[jira] [Updated] (HBASE-17281) FN should use datanode port from hdfs configuration
[
https://issues.apache.org/jira/browse/HBASE-17281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvel Thirumoolan updated HBASE-17281:
-
Attachment: HBASE-17281.master.005.patch
> FN should use datanode port from hdfs configuration
> ---
>
> Key: HBASE-17281
> URL: https://issues.apache.org/jira/browse/HBASE-17281
> Project: HBase
> Issue Type: Sub-task
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17281.master.001.patch,
> HBASE-17281.master.002.patch, HBASE-17281.master.003.patch,
> HBASE-17281.master.004.patch, HBASE-17281.master.005.patch
>
>
> Currently we use the ServerName port for providing favored node hints. We
> should use the DN port from hdfs-site.xml instead to avoid warning messages
> in region server logs. The warnings will be from this section of HDFS code,
> it moves across classes.
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java#L1758
> {code}
> private boolean[] getPinnings(DatanodeInfo[] nodes) {
> if (favoredNodes == null) {
> return null;
> } else {
> boolean[] pinnings = new boolean[nodes.length];
> HashSet favoredSet = new HashSet<>(Arrays.asList(favoredNodes));
> for (int i = 0; i < nodes.length; i++) {
> pinnings[i] = favoredSet.remove(nodes[i].getXferAddrWithHostname());
> LOG.debug("{} was chosen by name node (favored={}).",
> nodes[i].getXferAddrWithHostname(), pinnings[i]);
> }
> if (!favoredSet.isEmpty()) {
> // There is one or more favored nodes that were not allocated.
> LOG.warn("These favored nodes were specified but not chosen: "
> + favoredSet + " Specified favored nodes: "
> + Arrays.toString(favoredNodes));
> }
> return pinnings;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-15941) HBCK repair should not unsplit healthy splitted region
[ https://issues.apache.org/jira/browse/HBASE-15941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848067#comment-15848067 ] Stephen Yuan Jiang commented on HBASE-15941: [~esteban], please go ahead and take it. Thanks > HBCK repair should not unsplit healthy splitted region > -- > > Key: HBASE-15941 > URL: https://issues.apache.org/jira/browse/HBASE-15941 > Project: HBase > Issue Type: Improvement > Components: hbck >Affects Versions: 1.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > > Currently HBCK design in branch-1 has a flaw when repair option (the > -fixHdfsOverlaps option specifically) is specified: it would wrongly merge > split region (by looking at HDFS, it thinks that there exists overlapped > regions - parent region and daughter regions covers the same key range, of > course). See HBASE-15940 for details. > This JIRA tracks the improvement not-to-merge split region in HBCK repair. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17197) hfile does not work in 2.0
[
https://issues.apache.org/jira/browse/HBASE-17197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-17197:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.0.0
Release Note: The -f argument is no longer required specifying target
file; just pass the file as an argument.
Status: Resolved (was: Patch Available)
Pushed to master. Thanks for the fix [~balazs.meszaros]
> hfile does not work in 2.0
> --
>
> Key: HBASE-17197
> URL: https://issues.apache.org/jira/browse/HBASE-17197
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: Balazs Meszaros
> Fix For: 2.0.0
>
> Attachments: HBASE-17197-BM-01.patch
>
>
> I tried to use hfile in master branch, it does not print out kv pairs or meta
> as it is supposed to be.
> {code}
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/
> 53e9f9bc328f468b87831221de3a0c74 bdc6e1c4eea246a99e989e02d554cb03
> bf9275ac418d4d458904d81137e82683
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/bf9275ac418d4d458904d81137e82683
> -m
> 2016-11-29 12:25:22,019 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/bf9275ac418d4d458904d81137e82683
> -p
> 2016-11-29 12:25:27,333 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Scanned kv count -> 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17543) Create additional ReplicationEndpoint WALEntryFilters by configuration
[ https://issues.apache.org/jira/browse/HBASE-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848053#comment-15848053 ] Hudson commented on HBASE-17543: SUCCESS: Integrated in Jenkins build HBase-1.4 #607 (See [https://builds.apache.org/job/HBase-1.4/607/]) HBASE-17543 - Create additional ReplicationEndpoint WALEntryFilters by (tedyu: rev 3aac1b6884b43fbfd7a91d0f5cc765214e16d9a7) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/BaseReplicationEndpoint.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEndpoint.java > Create additional ReplicationEndpoint WALEntryFilters by configuration > -- > > Key: HBASE-17543 > URL: https://issues.apache.org/jira/browse/HBASE-17543 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17543-branch-1.patch, HBASE-17543.patch, > HBASE-17543.v2.patch, HBASE-17543.v3.patch > > > The existing BaseReplicationEndpoint creates a ChainWALEntryFilter containing > a NamespaceTableCfWALEntryFilter and a ScopeWALEntryFilter. Adding a custom > WALEntryFilter type to this chain requires creating an entirely new > ReplicationEndpoint subclass and creating a new peer on the running cluster, > which can be operationally complex to transition to without data loss in > cases such as master/master. > For WALEntryFilters without constructor parameters, it would be > straightforward to have a Configuration option to list additional > WALEntryFilter classes the operator wants to include in the filter chain in > the default endpoint, and then have the endpoint instantiate the filters via > reflection. Then filter logic could be added (or removed) with only a > hbase-site.xml change and a rolling restart. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17280) Add mechanism to control hbase cleaner behavior
[ https://issues.apache.org/jira/browse/HBASE-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848043#comment-15848043 ] Ajay Jadhav commented on HBASE-17280: - Yes, that was the original implementation wherein we either enable/ disable. But it is possible that the cleaner objects were not initialized/ instantiated properly in which case it might be possible that we cannot enable/ disable them. So, Ted suggested to change the API from IsCleanerChoreEnabled to getCleanerChoreState to just list the current state of cleaner chores. > Add mechanism to control hbase cleaner behavior > --- > > Key: HBASE-17280 > URL: https://issues.apache.org/jira/browse/HBASE-17280 > Project: HBase > Issue Type: Improvement > Components: Client, hbase, shell >Affects Versions: 2.0.0, 1.2.0 >Reporter: Ajay Jadhav >Priority: Minor > Fix For: 2.0.0, 1.2.0 > > Attachments: HBASE-17280.branch-1.2.patch, > HBASE-17280.branch-2.0.patch, HBASE-17280.master.003.patch, > HBASE-17280.master.004.patch, HBASE-17280.v1-branch-1.2.patch, > HBASE-17280.v2-branch-1.2.patch, HBASE-17280.v2-branch-2.patch > > > Cleaner is used to get rid of archived HFiles and old WALs in HBase. > In the case of heavy workload, cleaner can affect query performance by > creating a lot of connections to perform costly reads/ writes against > underlying filesystem. > This patch allows the user to control HBase cleaner behavior by providing > shell commands to enable/ disable and manually run it. > Our main intention with this patch was to avoid running the expensive cleaner > chore during peak times. During our experimentation, we saw a lot of HFiles > and WAL log related files getting created inside archive dir (didn't see > ZKlock related files). Since we were replacing hdfs with S3, these delete > calls will take forever to complete. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17280) Add mechanism to control hbase cleaner behavior
[ https://issues.apache.org/jira/browse/HBASE-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848041#comment-15848041 ] Anoop Sam John commented on HBASE-17280: bq.HFILE_CLEANER_ENABLED, LOG_CLEANER_ENABLED, BOTH_HFILE_LOG_CLEANER_ENABLED, BOTH_HFILE_LOG_CLEANER_DISABLED We have only one API to disable/enable both the cleaners.. So why is it possible that one of them is enabled while other disable? Do it really possible? > Add mechanism to control hbase cleaner behavior > --- > > Key: HBASE-17280 > URL: https://issues.apache.org/jira/browse/HBASE-17280 > Project: HBase > Issue Type: Improvement > Components: Client, hbase, shell >Affects Versions: 2.0.0, 1.2.0 >Reporter: Ajay Jadhav >Priority: Minor > Fix For: 2.0.0, 1.2.0 > > Attachments: HBASE-17280.branch-1.2.patch, > HBASE-17280.branch-2.0.patch, HBASE-17280.master.003.patch, > HBASE-17280.master.004.patch, HBASE-17280.v1-branch-1.2.patch, > HBASE-17280.v2-branch-1.2.patch, HBASE-17280.v2-branch-2.patch > > > Cleaner is used to get rid of archived HFiles and old WALs in HBase. > In the case of heavy workload, cleaner can affect query performance by > creating a lot of connections to perform costly reads/ writes against > underlying filesystem. > This patch allows the user to control HBase cleaner behavior by providing > shell commands to enable/ disable and manually run it. > Our main intention with this patch was to avoid running the expensive cleaner > chore during peak times. During our experimentation, we saw a lot of HFiles > and WAL log related files getting created inside archive dir (didn't see > ZKlock related files). Since we were replacing hdfs with S3, these delete > calls will take forever to complete. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847949#comment-15847949
]
Hadoop QA commented on HBASE-14123:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 6s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 23 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 22s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
14s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 53s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 59s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
4s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s
{color} | {color:red} The patch has 69 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 3s
{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 17s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 3m
46s {color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 55s
{color} | {color:red} hbase-server generated 4 new + 0 unchanged - 0 fixed = 4
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 50s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 88m
[jira] [Comment Edited] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly and beyond
[ https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847933#comment-15847933 ] Jingcheng Du edited comment on HBASE-16981 at 2/1/17 3:47 AM: -- Hi [~huaxiang], would you please take a look at the test failure? Thanks a lot! Would you mind uploading a new patch with format after fixing the failure? And I can use "git am" for it. Thanks a lot! was (Author: jingcheng.du): Hi [~huaxiang], would you please take a look at the test failure? Thanks a lot! > Expand Mob Compaction Partition policy from daily to weekly, monthly and > beyond > --- > > Key: HBASE-16981 > URL: https://issues.apache.org/jira/browse/HBASE-16981 > Project: HBase > Issue Type: New Feature > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-16981.master.001.patch, > HBASE-16981.master.002.patch, HBASE-16981.master.003.patch, > HBASE-16981.master.004.patch, HBASE-16981.master.005.patch, > HBASE-16981.master.006.patch, HBASE-16981.master.007.patch, > Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf > > > Today the mob region holds all mob files for all regions. With daily > partition mob compaction policy, after major mob compaction, there is still > one file per region daily. Given there is 365 days in one year, at least 365 > files per region. Since HDFS has limitation for number of files under one > folder, this is not going to scale if there are lots of regions. To reduce > mob file number, we want to introduce other partition policies such as > weekly, monthly to compact mob files within one week or month into one file. > This jira is create to track this effort. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly and beyond
[ https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847933#comment-15847933 ] Jingcheng Du commented on HBASE-16981: -- Hi [~huaxiang], would you please take a look at the test failure? Thanks a lot! > Expand Mob Compaction Partition policy from daily to weekly, monthly and > beyond > --- > > Key: HBASE-16981 > URL: https://issues.apache.org/jira/browse/HBASE-16981 > Project: HBase > Issue Type: New Feature > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-16981.master.001.patch, > HBASE-16981.master.002.patch, HBASE-16981.master.003.patch, > HBASE-16981.master.004.patch, HBASE-16981.master.005.patch, > HBASE-16981.master.006.patch, HBASE-16981.master.007.patch, > Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf > > > Today the mob region holds all mob files for all regions. With daily > partition mob compaction policy, after major mob compaction, there is still > one file per region daily. Given there is 365 days in one year, at least 365 > files per region. Since HDFS has limitation for number of files under one > folder, this is not going to scale if there are lots of regions. To reduce > mob file number, we want to introduce other partition policies such as > weekly, monthly to compact mob files within one week or month into one file. > This jira is create to track this effort. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-3462) Fix table.jsp in regards to splitting a region/table with an optional splitkey
[
https://issues.apache.org/jira/browse/HBASE-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847931#comment-15847931
]
Hudson commented on HBASE-3462:
---
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2422 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2422/])
HBASE-3462 Fix table.jsp in regards to splitting a region/table with an (stack:
rev ccd5b9f873d1d51041f17836dc17e96bec8793e5)
* (edit) hbase-server/src/main/resources/hbase-webapps/master/table.jsp
> Fix table.jsp in regards to splitting a region/table with an optional splitkey
> --
>
> Key: HBASE-3462
> URL: https://issues.apache.org/jira/browse/HBASE-3462
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 0.90.0
>Reporter: Lars George
>Assignee: Balazs Meszaros
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-3462-BM-01.patch, HBASE-3462-BM-01.patch
>
>
> After HBASE-3328 and HBASE-3437 went in there is also the table.jsp that
> needs updating to support the same features. Also, at the same time update
> the wording, for example
> {quote}
> This action will force a split of all eligible regions of the table, or, if a
> key is supplied, only the region containing the given key. An eligible region
> is one that does not contain any references to other regions. Split requests
> for noneligible regions will be ignored.
> {quote}
> I think it means it splits either all regions (that are splittable) or a
> specific one. It says though "the region containing the given key", that
> seems wrong in any event. Currently we do a split on the tablename when
> nothing was specified or else do an internal get(region), which is an exact
> match on the rows in .META.. In other words you need to match the region name
> exactly or else it fails. It reports it has accepted the request but logs
> internally
> {code}
> 2011-01-21 15:37:24,340 INFO org.apache.hadoop.hbase.client.HBaseAdmin: No
> server in .META. for csfsef; pair=null
> {code}
> Error reporting could be better but because of the async nature this is more
> difficult, yet it would be nice there is some concept of a Future to be able
> to poll the result if needed.
> Finally, when you go back to the previous page after submitting the split the
> entered values show up in the "compact" input fields, at least on my Chrome.
> The inputs in both forms are named the same so it seems to confuse it. This
> could be improved a lot by making the landing page reload the main one
> automatically or refresh on reload instead of submitting the request again.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16621) HBCK should have -fixHFileLinks
[
https://issues.apache.org/jira/browse/HBASE-16621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847929#comment-15847929
]
Hudson commented on HBASE-16621:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2422 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2422/])
HBASE-16621 HBCK should have -fixHFileLinks (Janos Gub) (tedyu: rev
34ffca1357a18750c9365890733dcff94a7198eb)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckOneRS.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckTwoRS.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
> HBCK should have -fixHFileLinks
> ---
>
> Key: HBASE-16621
> URL: https://issues.apache.org/jira/browse/HBASE-16621
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Janos Gub
> Labels: beginner, operability
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16621-2.patch, HBASE-16621-3.patch,
> HBASE-16621-4.patch, HBASE-16621-5.patch, HBASE-16621.patch, refactor-2.patch
>
>
> Similar to {{-fixReferenceFiles}}, HBCK should be able to sideline dangling
> HFile Links as well.
> We have seen a couple of cases, where due to HDFS-level fsck run which
> deleted files with missing blocks, the cluster is left with dangling HFIle
> Links, and regions cannot be opened because of these. Only a manual and
> error-prone finding and clearing of HFileLinks can save the tables regions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-15941) HBCK repair should not unsplit healthy splitted region
[ https://issues.apache.org/jira/browse/HBASE-15941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847849#comment-15847849 ] Esteban Gutierrez commented on HBASE-15941: --- Do you mind if I work on this [~syuanjiang]? I got a patch for this on the works. > HBCK repair should not unsplit healthy splitted region > -- > > Key: HBASE-15941 > URL: https://issues.apache.org/jira/browse/HBASE-15941 > Project: HBase > Issue Type: Improvement > Components: hbck >Affects Versions: 1.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > > Currently HBCK design in branch-1 has a flaw when repair option (the > -fixHdfsOverlaps option specifically) is specified: it would wrongly merge > split region (by looking at HDFS, it thinks that there exists overlapped > regions - parent region and daughter regions covers the same key range, of > course). See HBASE-15940 for details. > This JIRA tracks the improvement not-to-merge split region in HBCK repair. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17543) Create additional ReplicationEndpoint WALEntryFilters by configuration
[ https://issues.apache.org/jira/browse/HBASE-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17543: --- Resolution: Fixed Status: Resolved (was: Patch Available) Test failure was not related. Thanks for the patch, Geoffrey > Create additional ReplicationEndpoint WALEntryFilters by configuration > -- > > Key: HBASE-17543 > URL: https://issues.apache.org/jira/browse/HBASE-17543 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17543-branch-1.patch, HBASE-17543.patch, > HBASE-17543.v2.patch, HBASE-17543.v3.patch > > > The existing BaseReplicationEndpoint creates a ChainWALEntryFilter containing > a NamespaceTableCfWALEntryFilter and a ScopeWALEntryFilter. Adding a custom > WALEntryFilter type to this chain requires creating an entirely new > ReplicationEndpoint subclass and creating a new peer on the running cluster, > which can be operationally complex to transition to without data loss in > cases such as master/master. > For WALEntryFilters without constructor parameters, it would be > straightforward to have a Configuration option to list additional > WALEntryFilter classes the operator wants to include in the filter chain in > the default endpoint, and then have the endpoint instantiate the filters via > reflection. Then filter logic could be added (or removed) with only a > hbase-site.xml change and a rolling restart. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17543) Create additional ReplicationEndpoint WALEntryFilters by configuration
[
https://issues.apache.org/jira/browse/HBASE-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847829#comment-15847829
]
Hadoop QA commented on HBASE-17543:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 37s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 15s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
54s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 48s
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 22s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 57s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 29s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 72m 14s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.ipc.TestSimpleRpcScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:e01ee2f |
| JIRA Patch URL |
[jira] [Commented] (HBASE-17280) Add mechanism to control hbase cleaner behavior
[ https://issues.apache.org/jira/browse/HBASE-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847787#comment-15847787 ] Ajay Jadhav commented on HBASE-17280: - Thanks. Added them. > Add mechanism to control hbase cleaner behavior > --- > > Key: HBASE-17280 > URL: https://issues.apache.org/jira/browse/HBASE-17280 > Project: HBase > Issue Type: Improvement > Components: Client, hbase, shell >Affects Versions: 2.0.0, 1.2.0 >Reporter: Ajay Jadhav >Priority: Minor > Fix For: 2.0.0, 1.2.0 > > Attachments: HBASE-17280.branch-1.2.patch, > HBASE-17280.branch-2.0.patch, HBASE-17280.master.003.patch, > HBASE-17280.master.004.patch, HBASE-17280.v1-branch-1.2.patch, > HBASE-17280.v2-branch-1.2.patch, HBASE-17280.v2-branch-2.patch > > > Cleaner is used to get rid of archived HFiles and old WALs in HBase. > In the case of heavy workload, cleaner can affect query performance by > creating a lot of connections to perform costly reads/ writes against > underlying filesystem. > This patch allows the user to control HBase cleaner behavior by providing > shell commands to enable/ disable and manually run it. > Our main intention with this patch was to avoid running the expensive cleaner > chore during peak times. During our experimentation, we saw a lot of HFiles > and WAL log related files getting created inside archive dir (didn't see > ZKlock related files). Since we were replacing hdfs with S3, these delete > calls will take forever to complete. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17280) Add mechanism to control hbase cleaner behavior
[ https://issues.apache.org/jira/browse/HBASE-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Jadhav updated HBASE-17280: Release Note: The HBase cleaner chore process cleans up old WAL files and archived HFiles. Cleaner operation can affect query performance when running heavy workloads, so disable the cleaner during peak hours. The cleaner has the following HBase shell commands: - cleaner_chore_state: Queries the state of cleaner chore. Possible responses are as follows: HFILE_CLEANER_ENABLED, LOG_CLEANER_ENABLED, BOTH_HFILE_LOG_CLEANER_ENABLED, BOTH_HFILE_LOG_CLEANER_DISABLED. - cleaner_chore_run: Manually runs the cleaner to remove files. - cleaner_chore_switch: enables or disables the cleaner and returns the previous state of the cleaner. For example, cleaner-switch true enables the cleaner. Following APIs are added in Admin: - setCleanerChoreRunning(boolean on): Enable/Disable the cleaner chore - runCleanerChore(): Ask for cleaner chore to run - getCleanerChoreState(): Query the cleaner chore state > Add mechanism to control hbase cleaner behavior > --- > > Key: HBASE-17280 > URL: https://issues.apache.org/jira/browse/HBASE-17280 > Project: HBase > Issue Type: Improvement > Components: Client, hbase, shell >Affects Versions: 2.0.0, 1.2.0 >Reporter: Ajay Jadhav >Priority: Minor > Fix For: 2.0.0, 1.2.0 > > Attachments: HBASE-17280.branch-1.2.patch, > HBASE-17280.branch-2.0.patch, HBASE-17280.master.003.patch, > HBASE-17280.master.004.patch, HBASE-17280.v1-branch-1.2.patch, > HBASE-17280.v2-branch-1.2.patch, HBASE-17280.v2-branch-2.patch > > > Cleaner is used to get rid of archived HFiles and old WALs in HBase. > In the case of heavy workload, cleaner can affect query performance by > creating a lot of connections to perform costly reads/ writes against > underlying filesystem. > This patch allows the user to control HBase cleaner behavior by providing > shell commands to enable/ disable and manually run it. > Our main intention with this patch was to avoid running the expensive cleaner > chore during peak times. During our experimentation, we saw a lot of HFiles > and WAL log related files getting created inside archive dir (didn't see > ZKlock related files). Since we were replacing hdfs with S3, these delete > calls will take forever to complete. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-3462) Fix table.jsp in regards to splitting a region/table with an optional splitkey
[
https://issues.apache.org/jira/browse/HBASE-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-3462:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed to master branch. Thanks for the patch [~balazs.meszaros].
> Fix table.jsp in regards to splitting a region/table with an optional splitkey
> --
>
> Key: HBASE-3462
> URL: https://issues.apache.org/jira/browse/HBASE-3462
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 0.90.0
>Reporter: Lars George
>Assignee: Balazs Meszaros
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-3462-BM-01.patch, HBASE-3462-BM-01.patch
>
>
> After HBASE-3328 and HBASE-3437 went in there is also the table.jsp that
> needs updating to support the same features. Also, at the same time update
> the wording, for example
> {quote}
> This action will force a split of all eligible regions of the table, or, if a
> key is supplied, only the region containing the given key. An eligible region
> is one that does not contain any references to other regions. Split requests
> for noneligible regions will be ignored.
> {quote}
> I think it means it splits either all regions (that are splittable) or a
> specific one. It says though "the region containing the given key", that
> seems wrong in any event. Currently we do a split on the tablename when
> nothing was specified or else do an internal get(region), which is an exact
> match on the rows in .META.. In other words you need to match the region name
> exactly or else it fails. It reports it has accepted the request but logs
> internally
> {code}
> 2011-01-21 15:37:24,340 INFO org.apache.hadoop.hbase.client.HBaseAdmin: No
> server in .META. for csfsef; pair=null
> {code}
> Error reporting could be better but because of the async nature this is more
> difficult, yet it would be nice there is some concept of a Future to be able
> to poll the result if needed.
> Finally, when you go back to the previous page after submitting the split the
> entered values show up in the "compact" input fields, at least on my Chrome.
> The inputs in both forms are named the same so it seems to confuse it. This
> could be improved a lot by making the landing page reload the main one
> automatically or refresh on reload instead of submitting the request again.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-17437: -- Release Note: This patch adds support for specifying a WAL directory outside of the HBase root directory. Multiple configuration variables were added to accomplish this: hbase.wal.dir: used to configure where the root WAL directory is located. Could be on a different FileSystem than the root directory. WAL directory can not be set to a subdirectory of the root directory. The default value of this is the root directory if unset. hbase.rootdir.perms: Configures FileSystem permissions to set on the root directory. This is '700' by default. hbase.wal.dir.perms: Configures FileSystem permissions to set on the WAL directory FileSystem. This is '700' by default. > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Zach York > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, > HBASE-17437.master.001.patch, HBASE-17437.master.002.patch, > HBASE-17437.master.003.patch, HBASE-17437.master.004.patch, > HBASE-17437.master.005.patch, HBASE-17437.master.006.patch, > HBASE-17437.master.007.patch, HBASE-17437.master.008.patch, > HBASE-17437.master.009.patch, HBASE-17437.master.010.patch, > HBASE-17437.master.011.patch, HBASE-17437.master.012.patch, > hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-3462) Fix table.jsp in regards to splitting a region/table with an optional splitkey
[
https://issues.apache.org/jira/browse/HBASE-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847768#comment-15847768
]
Hadoop QA commented on HBASE-3462:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 39s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 44s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 104m 51s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 140m 38s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestZKAsyncRegistry |
| Timed out junit tests |
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.6 Server=1.12.6 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12850315/HBASE-3462-BM-01.patch
|
| JIRA Issue | HBASE-3462 |
| Optional Tests | asflicense javac javadoc unit |
| uname | Linux df2cb845dd0b 3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18
16:48:51 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 5ebaadf |
| Default Java | 1.8.0_121 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/5527/artifact/patchprocess/whitespace-tabs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5527/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/5527/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5527/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5527/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Fix table.jsp in regards to splitting a region/table with an optional splitkey
> --
>
> Key: HBASE-3462
> URL: https://issues.apache.org/jira/browse/HBASE-3462
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 0.90.0
>Reporter: Lars George
>Assignee: Balazs Meszaros
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-3462-BM-01.patch, HBASE-3462-BM-01.patch
>
>
> After HBASE-3328 and HBASE-3437 went in there is also the table.jsp that
> needs updating to support the same features. Also, at the same time update
> the wording, for example
> {quote}
> This action will force a split of all eligible regions of the table, or, if a
> key is supplied, only the region containing the given key. An eligible region
> is one that does not contain
[jira] [Commented] (HBASE-17197) hfile does not work in 2.0
[
https://issues.apache.org/jira/browse/HBASE-17197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847764#comment-15847764
]
Balazs Meszaros commented on HBASE-17197:
-
Yes, if args are empty then cmd.getArgList() will return an empty list, so the
lambda expression will append nothing to the list.
> hfile does not work in 2.0
> --
>
> Key: HBASE-17197
> URL: https://issues.apache.org/jira/browse/HBASE-17197
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: Balazs Meszaros
> Attachments: HBASE-17197-BM-01.patch
>
>
> I tried to use hfile in master branch, it does not print out kv pairs or meta
> as it is supposed to be.
> {code}
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/
> 53e9f9bc328f468b87831221de3a0c74 bdc6e1c4eea246a99e989e02d554cb03
> bf9275ac418d4d458904d81137e82683
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/bf9275ac418d4d458904d81137e82683
> -m
> 2016-11-29 12:25:22,019 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/bf9275ac418d4d458904d81137e82683
> -p
> 2016-11-29 12:25:27,333 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Scanned kv count -> 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17280) Add mechanism to control hbase cleaner behavior
[ https://issues.apache.org/jira/browse/HBASE-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847757#comment-15847757 ] Ted Yu commented on HBASE-17280: When you click on Edit button, you would see Release Note box where you can describe the new methods added to Admin.java and the new .rb scripts. > Add mechanism to control hbase cleaner behavior > --- > > Key: HBASE-17280 > URL: https://issues.apache.org/jira/browse/HBASE-17280 > Project: HBase > Issue Type: Improvement > Components: Client, hbase, shell >Affects Versions: 2.0.0, 1.2.0 >Reporter: Ajay Jadhav >Priority: Minor > Fix For: 2.0.0, 1.2.0 > > Attachments: HBASE-17280.branch-1.2.patch, > HBASE-17280.branch-2.0.patch, HBASE-17280.master.003.patch, > HBASE-17280.master.004.patch, HBASE-17280.v1-branch-1.2.patch, > HBASE-17280.v2-branch-1.2.patch, HBASE-17280.v2-branch-2.patch > > > Cleaner is used to get rid of archived HFiles and old WALs in HBase. > In the case of heavy workload, cleaner can affect query performance by > creating a lot of connections to perform costly reads/ writes against > underlying filesystem. > This patch allows the user to control HBase cleaner behavior by providing > shell commands to enable/ disable and manually run it. > Our main intention with this patch was to avoid running the expensive cleaner > chore during peak times. During our experimentation, we saw a lot of HFiles > and WAL log related files getting created inside archive dir (didn't see > ZKlock related files). Since we were replacing hdfs with S3, these delete > calls will take forever to complete. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17198) FN updates during region merge (follow up to Procedure v2 merge)
[ https://issues.apache.org/jira/browse/HBASE-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847730#comment-15847730 ] Hudson commented on HBASE-17198: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2421 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2421/]) HBASE-17198 Remove redundant FN updates to merged region (toffer: rev bd7c9581f20b864db41dfdf83b6240c8a5dcca9d) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java > FN updates during region merge (follow up to Procedure v2 merge) > > > Key: HBASE-17198 > URL: https://issues.apache.org/jira/browse/HBASE-17198 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-17198.master.001.patch > > > As mentioned in https://reviews.apache.org/r/53242/ (HBASE-16941), since the > procedure v2 merge changes are in development, there is a follow up > optimization/cleanup that can be done for favored nodes during merge. This > jira will be taken up once HBASE-16119 is complete. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17543) Create additional ReplicationEndpoint WALEntryFilters by configuration
[ https://issues.apache.org/jira/browse/HBASE-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847732#comment-15847732 ] Hudson commented on HBASE-17543: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2421 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2421/]) HBASE-17543 - Create additional ReplicationEndpoint WALEntryFilters by (tedyu: rev 5ebaadf1a6d8388f3c5633fb76ecfc8c0adc2da2) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/ReplicationManager.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEndpoint.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/BaseReplicationEndpoint.java > Create additional ReplicationEndpoint WALEntryFilters by configuration > -- > > Key: HBASE-17543 > URL: https://issues.apache.org/jira/browse/HBASE-17543 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17543-branch-1.patch, HBASE-17543.patch, > HBASE-17543.v2.patch, HBASE-17543.v3.patch > > > The existing BaseReplicationEndpoint creates a ChainWALEntryFilter containing > a NamespaceTableCfWALEntryFilter and a ScopeWALEntryFilter. Adding a custom > WALEntryFilter type to this chain requires creating an entirely new > ReplicationEndpoint subclass and creating a new peer on the running cluster, > which can be operationally complex to transition to without data loss in > cases such as master/master. > For WALEntryFilters without constructor parameters, it would be > straightforward to have a Configuration option to list additional > WALEntryFilter classes the operator wants to include in the filter chain in > the default endpoint, and then have the endpoint instantiate the filters via > reflection. Then filter logic could be added (or removed) with only a > hbase-site.xml change and a rolling restart. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17101) FavoredNodes should not apply to system tables
[ https://issues.apache.org/jira/browse/HBASE-17101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847733#comment-15847733 ] Hudson commented on HBASE-17101: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2421 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2421/]) HBASE-17101: FavoredNodes should not apply to system tables (toffer: rev 680289d67deb42922dd244daa11496a4c8a38f80) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/favored/FavoredNodeLoadBalancer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestTableFavoredNodes.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/favored/FavoredNodesManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java > FavoredNodes should not apply to system tables > -- > > Key: HBASE-17101 > URL: https://issues.apache.org/jira/browse/HBASE-17101 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-17101.master.001.patch, > HBASE-17101.master.002.patch, HBASE-17101.master.003.patch, > HBASE-17101.master.004.patch, HBASE_17101_rough_draft.patch > > > As described in the doc (see HBASE-15531), we would like to start with user > tables for favored nodes. This task ensures FN does not apply to system > tables. > System tables are in memory and won't benefit from favored nodes. Since we > also maintain FN information for user regions in meta, it helps to keep > implementation simpler by ignoring system tables for the first iterations. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847731#comment-15847731 ] Hudson commented on HBASE-17437: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2421 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2421/]) HBASE-17437 Support specifying a WAL directory outside of the root (enis: rev ae21797305188e82f7017ce5675e4fde950461aa) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALFactory.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractFSWALProvider.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterProcedureWalLease.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/WALLink.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALRecordReader.java * (edit) hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALPerformanceEvaluation.java * (edit) hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/ProcedureTestingUtility.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestFSHLog.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/AbstractTestFSWAL.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AsyncFSWALProvider.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterWalManager.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestWALProcedureStoreOnHDFS.java * (edit) hbase-common/src/main/resources/hbase-default.xml * (edit) hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALLoaderPerformanceEvaluation.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/DisabledWALProvider.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollAbort.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/FSHLogProvider.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/wal/IOTestProvider.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALRootDir.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFileSystemWithWALDir.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Zach York > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, > HBASE-17437.master.001.patch, HBASE-17437.master.002.patch, > HBASE-17437.master.003.patch, HBASE-17437.master.004.patch, > HBASE-17437.master.005.patch, HBASE-17437.master.006.patch, >
[jira] [Commented] (HBASE-17543) Create additional ReplicationEndpoint WALEntryFilters by configuration
[ https://issues.apache.org/jira/browse/HBASE-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847729#comment-15847729 ] Geoffrey Jacoby commented on HBASE-17543: - Uploaded backport to branch-1 > Create additional ReplicationEndpoint WALEntryFilters by configuration > -- > > Key: HBASE-17543 > URL: https://issues.apache.org/jira/browse/HBASE-17543 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17543-branch-1.patch, HBASE-17543.patch, > HBASE-17543.v2.patch, HBASE-17543.v3.patch > > > The existing BaseReplicationEndpoint creates a ChainWALEntryFilter containing > a NamespaceTableCfWALEntryFilter and a ScopeWALEntryFilter. Adding a custom > WALEntryFilter type to this chain requires creating an entirely new > ReplicationEndpoint subclass and creating a new peer on the running cluster, > which can be operationally complex to transition to without data loss in > cases such as master/master. > For WALEntryFilters without constructor parameters, it would be > straightforward to have a Configuration option to list additional > WALEntryFilter classes the operator wants to include in the filter chain in > the default endpoint, and then have the endpoint instantiate the filters via > reflection. Then filter logic could be added (or removed) with only a > hbase-site.xml change and a rolling restart. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17543) Create additional ReplicationEndpoint WALEntryFilters by configuration
[ https://issues.apache.org/jira/browse/HBASE-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Jacoby updated HBASE-17543: Status: Patch Available (was: Open) > Create additional ReplicationEndpoint WALEntryFilters by configuration > -- > > Key: HBASE-17543 > URL: https://issues.apache.org/jira/browse/HBASE-17543 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17543-branch-1.patch, HBASE-17543.patch, > HBASE-17543.v2.patch, HBASE-17543.v3.patch > > > The existing BaseReplicationEndpoint creates a ChainWALEntryFilter containing > a NamespaceTableCfWALEntryFilter and a ScopeWALEntryFilter. Adding a custom > WALEntryFilter type to this chain requires creating an entirely new > ReplicationEndpoint subclass and creating a new peer on the running cluster, > which can be operationally complex to transition to without data loss in > cases such as master/master. > For WALEntryFilters without constructor parameters, it would be > straightforward to have a Configuration option to list additional > WALEntryFilter classes the operator wants to include in the filter chain in > the default endpoint, and then have the endpoint instantiate the filters via > reflection. Then filter logic could be added (or removed) with only a > hbase-site.xml change and a rolling restart. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17543) Create additional ReplicationEndpoint WALEntryFilters by configuration
[ https://issues.apache.org/jira/browse/HBASE-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Jacoby updated HBASE-17543: Status: Open (was: Patch Available) > Create additional ReplicationEndpoint WALEntryFilters by configuration > -- > > Key: HBASE-17543 > URL: https://issues.apache.org/jira/browse/HBASE-17543 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17543-branch-1.patch, HBASE-17543.patch, > HBASE-17543.v2.patch, HBASE-17543.v3.patch > > > The existing BaseReplicationEndpoint creates a ChainWALEntryFilter containing > a NamespaceTableCfWALEntryFilter and a ScopeWALEntryFilter. Adding a custom > WALEntryFilter type to this chain requires creating an entirely new > ReplicationEndpoint subclass and creating a new peer on the running cluster, > which can be operationally complex to transition to without data loss in > cases such as master/master. > For WALEntryFilters without constructor parameters, it would be > straightforward to have a Configuration option to list additional > WALEntryFilter classes the operator wants to include in the filter chain in > the default endpoint, and then have the endpoint instantiate the filters via > reflection. Then filter logic could be added (or removed) with only a > hbase-site.xml change and a rolling restart. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17543) Create additional ReplicationEndpoint WALEntryFilters by configuration
[ https://issues.apache.org/jira/browse/HBASE-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Jacoby updated HBASE-17543: Attachment: HBASE-17543-branch-1.patch > Create additional ReplicationEndpoint WALEntryFilters by configuration > -- > > Key: HBASE-17543 > URL: https://issues.apache.org/jira/browse/HBASE-17543 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17543-branch-1.patch, HBASE-17543.patch, > HBASE-17543.v2.patch, HBASE-17543.v3.patch > > > The existing BaseReplicationEndpoint creates a ChainWALEntryFilter containing > a NamespaceTableCfWALEntryFilter and a ScopeWALEntryFilter. Adding a custom > WALEntryFilter type to this chain requires creating an entirely new > ReplicationEndpoint subclass and creating a new peer on the running cluster, > which can be operationally complex to transition to without data loss in > cases such as master/master. > For WALEntryFilters without constructor parameters, it would be > straightforward to have a Configuration option to list additional > WALEntryFilter classes the operator wants to include in the filter chain in > the default endpoint, and then have the endpoint instantiate the filters via > reflection. Then filter logic could be added (or removed) with only a > hbase-site.xml change and a rolling restart. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17280) Add mechanism to control hbase cleaner behavior
[ https://issues.apache.org/jira/browse/HBASE-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847711#comment-15847711 ] Ted Yu commented on HBASE-17280: Please fill in Release Note > Add mechanism to control hbase cleaner behavior > --- > > Key: HBASE-17280 > URL: https://issues.apache.org/jira/browse/HBASE-17280 > Project: HBase > Issue Type: Improvement > Components: Client, hbase, shell >Affects Versions: 2.0.0, 1.2.0 >Reporter: Ajay Jadhav >Priority: Minor > Fix For: 2.0.0, 1.2.0 > > Attachments: HBASE-17280.branch-1.2.patch, > HBASE-17280.branch-2.0.patch, HBASE-17280.master.003.patch, > HBASE-17280.master.004.patch, HBASE-17280.v1-branch-1.2.patch, > HBASE-17280.v2-branch-1.2.patch, HBASE-17280.v2-branch-2.patch > > > Cleaner is used to get rid of archived HFiles and old WALs in HBase. > In the case of heavy workload, cleaner can affect query performance by > creating a lot of connections to perform costly reads/ writes against > underlying filesystem. > This patch allows the user to control HBase cleaner behavior by providing > shell commands to enable/ disable and manually run it. > Our main intention with this patch was to avoid running the expensive cleaner > chore during peak times. During our experimentation, we saw a lot of HFiles > and WAL log related files getting created inside archive dir (didn't see > ZKlock related files). Since we were replacing hdfs with S3, these delete > calls will take forever to complete. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-14123: -- Attachment: 14123.master.v52.patch v52 > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Fix For: 2.0.0 > > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v20.txt, 14123-master.v21.txt, > 14123-master.v24.txt, 14123-master.v25.txt, 14123-master.v27.txt, > 14123-master.v28.txt, 14123-master.v29.full.txt, 14123-master.v2.txt, > 14123-master.v30.txt, 14123-master.v31.txt, 14123-master.v32.txt, > 14123-master.v33.txt, 14123-master.v34.txt, 14123-master.v35.txt, > 14123-master.v36.txt, 14123-master.v37.txt, 14123-master.v38.txt, > 14123.master.v39.patch, 14123-master.v3.txt, 14123.master.v40.patch, > 14123.master.v41.patch, 14123.master.v42.patch, 14123.master.v44.patch, > 14123.master.v45.patch, 14123.master.v46.patch, 14123.master.v48.patch, > 14123.master.v49.patch, 14123.master.v50.patch, 14123.master.v51.patch, > 14123.master.v52.patch, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v10.patch, HBASE-14123-v11.patch, HBASE-14123-v12.patch, > HBASE-14123-v13.patch, HBASE-14123-v15.patch, HBASE-14123-v16.patch, > HBASE-14123-v1.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Issue Comment Deleted] (HBASE-14167) hbase-spark integration tests do not respect -DskipIntegrationTests
[
https://issues.apache.org/jira/browse/HBASE-14167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liang updated HBASE-14167:
-
Comment: was deleted
(was: The reason why -DskipIntegrationTests not work is that hbase-spark has
two plugin for testing, one is Maven Surefire plugin for Java, and other is
Maven Scalatest plugin for scala. The -DskipIntegrationTests will only disable
Java test case, and would not do anything to Scala test cases. I can pick up
this one after I solved HBASE-17574. )
> hbase-spark integration tests do not respect -DskipIntegrationTests
> ---
>
> Key: HBASE-14167
> URL: https://issues.apache.org/jira/browse/HBASE-14167
> Project: HBase
> Issue Type: Improvement
> Components: pom, spark
>Affects Versions: 2.0.0
>Reporter: Andrew Purtell
>Priority: Minor
> Attachments: HBASE-14167.11.patch, HBASE-14167-master-v2.patch
>
>
> When running a build with {{mvn ... -DskipIntegrationTests}}, the hbase-spark
> module's integration tests do not respect the flag and run anyway. Fix.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17280) Add mechanism to control hbase cleaner behavior
[ https://issues.apache.org/jira/browse/HBASE-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847694#comment-15847694 ] Ajay Jadhav commented on HBASE-17280: - Thanks. Let me know if anything needs to be done from my side. > Add mechanism to control hbase cleaner behavior > --- > > Key: HBASE-17280 > URL: https://issues.apache.org/jira/browse/HBASE-17280 > Project: HBase > Issue Type: Improvement > Components: Client, hbase, shell >Affects Versions: 2.0.0, 1.2.0 >Reporter: Ajay Jadhav >Priority: Minor > Fix For: 2.0.0, 1.2.0 > > Attachments: HBASE-17280.branch-1.2.patch, > HBASE-17280.branch-2.0.patch, HBASE-17280.master.003.patch, > HBASE-17280.master.004.patch, HBASE-17280.v1-branch-1.2.patch, > HBASE-17280.v2-branch-1.2.patch, HBASE-17280.v2-branch-2.patch > > > Cleaner is used to get rid of archived HFiles and old WALs in HBase. > In the case of heavy workload, cleaner can affect query performance by > creating a lot of connections to perform costly reads/ writes against > underlying filesystem. > This patch allows the user to control HBase cleaner behavior by providing > shell commands to enable/ disable and manually run it. > Our main intention with this patch was to avoid running the expensive cleaner > chore during peak times. During our experimentation, we saw a lot of HFiles > and WAL log related files getting created inside archive dir (didn't see > ZKlock related files). Since we were replacing hdfs with S3, these delete > calls will take forever to complete. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14167) hbase-spark integration tests do not respect -DskipIntegrationTests
[
https://issues.apache.org/jira/browse/HBASE-14167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847690#comment-15847690
]
Yi Liang commented on HBASE-14167:
--
The reason why -DskipIntegrationTests not work is that hbase-spark has two
plugin for testing, one is Maven Surefire plugin for Java, and other is Maven
Scalatest plugin for scala. The -DskipIntegrationTests will only disable Java
test case, and would not do anything to Scala test cases. I can pick up this
one after I solved HBASE-17574.
> hbase-spark integration tests do not respect -DskipIntegrationTests
> ---
>
> Key: HBASE-14167
> URL: https://issues.apache.org/jira/browse/HBASE-14167
> Project: HBase
> Issue Type: Improvement
> Components: pom, spark
>Affects Versions: 2.0.0
>Reporter: Andrew Purtell
>Priority: Minor
> Attachments: HBASE-14167.11.patch, HBASE-14167-master-v2.patch
>
>
> When running a build with {{mvn ... -DskipIntegrationTests}}, the hbase-spark
> module's integration tests do not respect the flag and run anyway. Fix.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17280) Add mechanism to control hbase cleaner behavior
[ https://issues.apache.org/jira/browse/HBASE-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847688#comment-15847688 ] Ted Yu commented on HBASE-17280: Failure of TestHRegionWithInMemoryFlush indicates potential bug in in-memory compaction. Not related to your patch. > Add mechanism to control hbase cleaner behavior > --- > > Key: HBASE-17280 > URL: https://issues.apache.org/jira/browse/HBASE-17280 > Project: HBase > Issue Type: Improvement > Components: Client, hbase, shell >Affects Versions: 2.0.0, 1.2.0 >Reporter: Ajay Jadhav >Priority: Minor > Fix For: 2.0.0, 1.2.0 > > Attachments: HBASE-17280.branch-1.2.patch, > HBASE-17280.branch-2.0.patch, HBASE-17280.master.003.patch, > HBASE-17280.master.004.patch, HBASE-17280.v1-branch-1.2.patch, > HBASE-17280.v2-branch-1.2.patch, HBASE-17280.v2-branch-2.patch > > > Cleaner is used to get rid of archived HFiles and old WALs in HBase. > In the case of heavy workload, cleaner can affect query performance by > creating a lot of connections to perform costly reads/ writes against > underlying filesystem. > This patch allows the user to control HBase cleaner behavior by providing > shell commands to enable/ disable and manually run it. > Our main intention with this patch was to avoid running the expensive cleaner > chore during peak times. During our experimentation, we saw a lot of HFiles > and WAL log related files getting created inside archive dir (didn't see > ZKlock related files). Since we were replacing hdfs with S3, these delete > calls will take forever to complete. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17465) [C++] implement request retry mechanism over RPC
[
https://issues.apache.org/jira/browse/HBASE-17465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847685#comment-15847685
]
Xiaobing Zhou commented on HBASE-17465:
---
Posted v6 to add SERVICE template parameter for easy mock up test, e.g.
{code}
template
class SingleRequestCallerBuilder;
template
class AsyncSingleRequestRpcRetryingCaller;
{code}
> [C++] implement request retry mechanism over RPC
>
>
> Key: HBASE-17465
> URL: https://issues.apache.org/jira/browse/HBASE-17465
> Project: HBase
> Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HBASE-17465-HBASE-14850.000.patch,
> HBASE-17465-HBASE-14850.001.patch, HBASE-17465-HBASE-14850.002.patch,
> HBASE-17465-HBASE-14850.003.patch, HBASE-17465-HBASE-14850.004.patch,
> HBASE-17465-HBASE-14850.005.patch, HBASE-17465-HBASE-14850.006.patch
>
>
> HBASE-17051 implemented RPC layer. Requests retries will make system
> reliable. This JIRA proposes adding it, which corresponds to similar
> implementation in SingleRequestCallerBuilder (or BatchCallerBuilder,
> ScanSingleRegionCallerBuilder, SmallScanCallerBuilder, etc.) and
> AsyncSingleRequestRpcRetryingCaller. As a bonus, retry should be more
> generic, decoupled with HRegionLocation.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17465) [C++] implement request retry mechanism over RPC
[ https://issues.apache.org/jira/browse/HBASE-17465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HBASE-17465: -- Attachment: HBASE-17465-HBASE-14850.006.patch > [C++] implement request retry mechanism over RPC > > > Key: HBASE-17465 > URL: https://issues.apache.org/jira/browse/HBASE-17465 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-17465-HBASE-14850.000.patch, > HBASE-17465-HBASE-14850.001.patch, HBASE-17465-HBASE-14850.002.patch, > HBASE-17465-HBASE-14850.003.patch, HBASE-17465-HBASE-14850.004.patch, > HBASE-17465-HBASE-14850.005.patch, HBASE-17465-HBASE-14850.006.patch > > > HBASE-17051 implemented RPC layer. Requests retries will make system > reliable. This JIRA proposes adding it, which corresponds to similar > implementation in SingleRequestCallerBuilder (or BatchCallerBuilder, > ScanSingleRegionCallerBuilder, SmallScanCallerBuilder, etc.) and > AsyncSingleRequestRpcRetryingCaller. As a bonus, retry should be more > generic, decoupled with HRegionLocation. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16621) HBCK should have -fixHFileLinks
[
https://issues.apache.org/jira/browse/HBASE-16621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847678#comment-15847678
]
Hadoop QA commented on HBASE-16621:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 17s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 37s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 129m 9s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12850297/HBASE-16621-5.patch |
| JIRA Issue | HBASE-16621 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 16d4050d4278 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ae21797 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5526/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5526/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5526/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> HBCK should have -fixHFileLinks
> ---
>
> Key: HBASE-16621
> URL: https://issues.apache.org/jira/browse/HBASE-16621
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Janos Gub
> Labels: beginner, operability
> Fix For: 2.0.0,
[jira] [Commented] (HBASE-17574) Clean up how to run tests under hbase-spark module
[
https://issues.apache.org/jira/browse/HBASE-17574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847673#comment-15847673
]
Yi Liang commented on HBASE-17574:
--
Since Java and Scala has different maven plugin to run test case, java use
surefire and scala use scalatest, those two plugins are different at some
point, so what I can do is to provide a patch which have following usage
under hbase-spark dir or root dir
1. mvn test: run all the small java test, and all scala test(Still do not know
whether we can catogory scala test into small, medium, and large, if they can
be categorized, then this command can only apply to small test in scala)
2. mvn test -Dtest=TestJavaHBaseContext -DwildcardSuites=None(only run
specified java unit test)
3. mvn test -Dtest=None
-DwildcardSuites=org.apache.hadoop.hbase.spark.BulkLoadSuite (only run
specified scala unit test, scalatest plugin only support full class name)
Under root dir
1. mvn test -DskipSparkTests //skip all the hbase-spark tests
this jira is only for unit test, and if possible, I can open another jira to
try to add integration test case of hbase-spark into hbase-it module
> Clean up how to run tests under hbase-spark module
> ---
>
> Key: HBASE-17574
> URL: https://issues.apache.org/jira/browse/HBASE-17574
> Project: HBase
> Issue Type: Bug
> Components: spark
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 2.0.0
>
>
> In master brunch, the test of hbase-spark module needs clean-up.
> I think we need to let hbase-spark follow the rules that exist in the whole
> hbase project
> 1. In hbase-spark, all the scala test cases are regarded as integration test,
> i.e. we need to go to hbase-spark folder to use mvn verify to run the test
> case. I think these tests had better to be regard as unit test for the
> following reasons:
> (1) All the scala test are very small, most of them can be finished within
> 20s.
> (2) Integration test usually put into hbase-it module, not in its own module.
> (3) Hadoop QA could not run those scala test in hbase-spark, I guess Hadoop
> QA will only run mvn test under root dir, however hbase-spark need mvn verify.
> (4) From its pom.xml below, you can see that, both
> integration-test and test point to same
> test. From MVN reference,
> http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html#Built-in_Lifecycle_Bindings,
> we know that if a goal is bound to one or more build phases, that goal will
> be called in all those phases. it means that mvn test and mvn
> integration-test will do same thing, however true in
> test phase just disable the mvn test command. It is uncommon to have define
> like that.
> {code}
>
>
> test
> test
>
> test
>
>
> true
>
>
>
> integration-test
> integration-test
>
> test
>
>
> Integration-Test
>
> -Xmx1536m -XX:MaxPermSize=512m
> -XX:ReservedCodeCacheSize=512m
>
> false
>
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17574) Clean up how to run tests under hbase-spark module
[
https://issues.apache.org/jira/browse/HBASE-17574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liang updated HBASE-17574:
-
Description:
In master brunch, the test of hbase-spark module needs clean-up.
I think we need to let hbase-spark follow the rules that exist in the whole
hbase project
1. In hbase-spark, all the scala test cases are regarded as integration test,
i.e. we need to go to hbase-spark folder to use mvn verify to run the test
case. I think these tests had better to be regard as unit test for the
following reasons:
(1) All the scala test are very small, most of them can be finished within 20s.
(2) Integration test usually put into hbase-it module, not in its own module.
(3) Hadoop QA could not run those scala test in hbase-spark, I guess Hadoop QA
will only run mvn test under root dir, however hbase-spark need mvn verify.
(4) From its pom.xml below, you can see that, both
integration-test and test point to same
test. From MVN reference,
http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html#Built-in_Lifecycle_Bindings,
we know that if a goal is bound to one or more build phases, that goal will be
called in all those phases. it means that mvn test and mvn integration-test
will do same thing, however true in test phase just
disable the mvn test command. It is uncommon to have define like that.
{code}
test
test
test
true
integration-test
integration-test
test
Integration-Test
-Xmx1536m -XX:MaxPermSize=512m
-XX:ReservedCodeCacheSize=512m
false
{code}
was:
In master brunch, the test of hbase-spark module needs clean-up.
I think we need to let hbase-spark follow the rules that exist in the whole
hbase project
1. In hbase-spark, all the scala test cases are regarded as integration test,
i.e. we need to go to hbase-spark folder to use mvn verify to run the test
case. I think these tests had better to be regard as unit test for the
following reasons:
(1) All the scala test are very small, most of them can be finished within 20s.
(2) Integration test usually put into hbase-it module, not in its own module.
(3) Hadoop QA could not run those scala test in hbase-spark, I guess Hadoop QA
will only run mvn test under root dir, however hbase-spark need mvn verify.
(4) From its pom.xml below, you can see that, both
integration-test and test point to same
test. From MVN reference,
http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html#Built-in_Lifecycle_Bindings,
we know that if a goal is bound to one or more build phases, that goal will be
called in all those phases. it means that mvn test and mvn integration-test
will do same thing, however true in test phase just
disable the mvn test command. It is uncommon to have define like that.
{code}
test
test
test
true
integration-test
integration-test
test
Integration-Test
-Xmx1536m -XX:MaxPermSize=512m
-XX:ReservedCodeCacheSize=512m
false
{code}
> Clean up how to run tests under hbase-spark module
> ---
>
> Key: HBASE-17574
> URL: https://issues.apache.org/jira/browse/HBASE-17574
> Project: HBase
> Issue Type: Bug
> Components: spark
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 2.0.0
>
>
> In master brunch, the test of hbase-spark module needs clean-up.
> I think we need to let hbase-spark follow the rules that exist in the whole
> hbase project
> 1. In hbase-spark, all the scala test cases are regarded as integration test,
> i.e. we need to go to hbase-spark folder to use mvn
[jira] [Created] (HBASE-17574) Clean up how to run tests under hbase-spark module
Yi Liang created HBASE-17574:
Summary: Clean up how to run tests under hbase-spark module
Key: HBASE-17574
URL: https://issues.apache.org/jira/browse/HBASE-17574
Project: HBase
Issue Type: Bug
Components: spark
Affects Versions: 2.0.0
Reporter: Yi Liang
Assignee: Yi Liang
Fix For: 2.0.0
In master brunch, the test of hbase-spark module needs clean-up.
I think we need to let hbase-spark follow the rules that exist in the whole
hbase project
1. In hbase-spark, all the scala test cases are regarded as integration test,
i.e. we need to go to hbase-spark folder to use mvn verify to run the test
case. I think these tests had better to be regard as unit test for the
following reasons:
(1) All the scala test are very small, most of them can be finished within 20s.
(2) Integration test usually put into hbase-it module, not in its own module.
(3) Hadoop QA could not run those scala test in hbase-spark, I guess Hadoop QA
will only run mvn test under root dir, however hbase-spark need mvn verify.
(4) From its pom.xml below, you can see that, both
integration-test and test point to same
test. From MVN reference,
http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html#Built-in_Lifecycle_Bindings,
we know that if a goal is bound to one or more build phases, that goal will be
called in all those phases. it means that mvn test and mvn integration-test
will do same thing, however true in test phase just
disable the mvn test command. It is uncommon to have define like that.
{code}
test
test
test
true
integration-test
integration-test
test
Integration-Test
-Xmx1536m -XX:MaxPermSize=512m
-XX:ReservedCodeCacheSize=512m
false
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17569) HBase-Procedure module need to support mvn clean test -PskipProcedureTests to skip unit test
[
https://issues.apache.org/jira/browse/HBASE-17569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847656#comment-15847656
]
Yi Liang commented on HBASE-17569:
--
Correct it in addendum-v2, thanks [~stack]
> HBase-Procedure module need to support mvn clean test -PskipProcedureTests to
> skip unit test
>
>
> Key: HBASE-17569
> URL: https://issues.apache.org/jira/browse/HBASE-17569
> Project: HBase
> Issue Type: Improvement
> Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBase-17569-Addendum.patch,
> HBase-17569-Addendum-V2.patch, HBase-17569-V1.patch
>
>
> From Reference guide, we know that To skip the tests in the hbase-server
> module, you would run:
> {code}
> mvn clean test -PskipServerTests
> {code}
> We can also support this command in hbase-procedure module.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16621) HBCK should have -fixHFileLinks
[
https://issues.apache.org/jira/browse/HBASE-16621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847658#comment-15847658
]
Ted Yu commented on HBASE-16621:
Please attach patch for branch-1
> HBCK should have -fixHFileLinks
> ---
>
> Key: HBASE-16621
> URL: https://issues.apache.org/jira/browse/HBASE-16621
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Janos Gub
> Labels: beginner, operability
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16621-2.patch, HBASE-16621-3.patch,
> HBASE-16621-4.patch, HBASE-16621-5.patch, HBASE-16621.patch, refactor-2.patch
>
>
> Similar to {{-fixReferenceFiles}}, HBCK should be able to sideline dangling
> HFile Links as well.
> We have seen a couple of cases, where due to HDFS-level fsck run which
> deleted files with missing blocks, the cluster is left with dangling HFIle
> Links, and regions cannot be opened because of these. Only a manual and
> error-prone finding and clearing of HFileLinks can save the tables regions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17569) HBase-Procedure module need to support mvn clean test -PskipProcedureTests to skip unit test
[
https://issues.apache.org/jira/browse/HBASE-17569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liang updated HBASE-17569:
-
Attachment: HBase-17569-Addendum-V2.patch
> HBase-Procedure module need to support mvn clean test -PskipProcedureTests to
> skip unit test
>
>
> Key: HBASE-17569
> URL: https://issues.apache.org/jira/browse/HBASE-17569
> Project: HBase
> Issue Type: Improvement
> Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBase-17569-Addendum.patch,
> HBase-17569-Addendum-V2.patch, HBase-17569-V1.patch
>
>
> From Reference guide, we know that To skip the tests in the hbase-server
> module, you would run:
> {code}
> mvn clean test -PskipServerTests
> {code}
> We can also support this command in hbase-procedure module.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17197) hfile does not work in 2.0
[
https://issues.apache.org/jira/browse/HBASE-17197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847643#comment-15847643
]
stack commented on HBASE-17197:
---
If args are empty, does lambda do right thing?
> hfile does not work in 2.0
> --
>
> Key: HBASE-17197
> URL: https://issues.apache.org/jira/browse/HBASE-17197
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: Balazs Meszaros
> Attachments: HBASE-17197-BM-01.patch
>
>
> I tried to use hfile in master branch, it does not print out kv pairs or meta
> as it is supposed to be.
> {code}
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/
> 53e9f9bc328f468b87831221de3a0c74 bdc6e1c4eea246a99e989e02d554cb03
> bf9275ac418d4d458904d81137e82683
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/bf9275ac418d4d458904d81137e82683
> -m
> 2016-11-29 12:25:22,019 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/bf9275ac418d4d458904d81137e82683
> -p
> 2016-11-29 12:25:27,333 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Scanned kv count -> 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-17197) hfile does not work in 2.0
[
https://issues.apache.org/jira/browse/HBASE-17197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847635#comment-15847635
]
Hadoop QA commented on HBASE-17197:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 54s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 85m 9s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 124m 33s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12850295/HBASE-17197-BM-01.patch
|
| JIRA Issue | HBASE-17197 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 3d760bfd4c48 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ae21797 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5525/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5525/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> hfile does not work in 2.0
> --
>
> Key: HBASE-17197
> URL: https://issues.apache.org/jira/browse/HBASE-17197
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: Balazs Meszaros
>
[jira] (HBASE-17569) HBase-Procedure module need to support mvn clean test -PskipProcedureTests to skip unit test
[
https://issues.apache.org/jira/browse/HBASE-17569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847610#comment-15847610
]
stack commented on HBASE-17569:
---
Addendum is great except for the last change in hbase-protocol. It was wrong
before you changed it... should be hbase-protocol but it says hbase-rpc...
Thank you [~yiliang]
> HBase-Procedure module need to support mvn clean test -PskipProcedureTests to
> skip unit test
>
>
> Key: HBASE-17569
> URL: https://issues.apache.org/jira/browse/HBASE-17569
> Project: HBase
> Issue Type: Improvement
> Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBase-17569-Addendum.patch, HBase-17569-V1.patch
>
>
> From Reference guide, we know that To skip the tests in the hbase-server
> module, you would run:
> {code}
> mvn clean test -PskipServerTests
> {code}
> We can also support this command in hbase-procedure module.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-17280) Add mechanism to control hbase cleaner behavior
[ https://issues.apache.org/jira/browse/HBASE-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847612#comment-15847612 ] Ajay Jadhav commented on HBASE-17280: - I ran the org.apache.hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush test locally and it is running fine. In fact, the test is not even related to this patch. > Add mechanism to control hbase cleaner behavior > --- > > Key: HBASE-17280 > URL: https://issues.apache.org/jira/browse/HBASE-17280 > Project: HBase > Issue Type: Improvement > Components: Client, hbase, shell >Affects Versions: 2.0.0, 1.2.0 >Reporter: Ajay Jadhav >Priority: Minor > Fix For: 2.0.0, 1.2.0 > > Attachments: HBASE-17280.branch-1.2.patch, > HBASE-17280.branch-2.0.patch, HBASE-17280.master.003.patch, > HBASE-17280.master.004.patch, HBASE-17280.v1-branch-1.2.patch, > HBASE-17280.v2-branch-1.2.patch, HBASE-17280.v2-branch-2.patch > > > Cleaner is used to get rid of archived HFiles and old WALs in HBase. > In the case of heavy workload, cleaner can affect query performance by > creating a lot of connections to perform costly reads/ writes against > underlying filesystem. > This patch allows the user to control HBase cleaner behavior by providing > shell commands to enable/ disable and manually run it. > Our main intention with this patch was to avoid running the expensive cleaner > chore during peak times. During our experimentation, we saw a lot of HFiles > and WAL log related files getting created inside archive dir (didn't see > ZKlock related files). Since we were replacing hdfs with S3, these delete > calls will take forever to complete. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-16621) HBCK should have -fixHFileLinks
[
https://issues.apache.org/jira/browse/HBASE-16621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847591#comment-15847591
]
Hadoop QA commented on HBASE-16621:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 0s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 100m 45s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 150m 13s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12850293/HBASE-16621-4.patch |
| JIRA Issue | HBASE-16621 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 78f1573a9a91 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / bd7c958 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5524/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5524/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> HBCK should have -fixHFileLinks
> ---
>
> Key: HBASE-16621
> URL: https://issues.apache.org/jira/browse/HBASE-16621
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Janos Gub
> Labels: beginner, operability
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16621-2.patch, HBASE-16621-3.patch,
> HBASE-16621-4.patch,
[jira] (HBASE-17280) Add mechanism to control hbase cleaner behavior
[
https://issues.apache.org/jira/browse/HBASE-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847584#comment-15847584
]
Hadoop QA commented on HBASE-17280:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 0s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 0s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 30s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 14m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 0s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 14m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 6s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 19s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 154m 19s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 31s
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
53s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 246m 37s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA
[jira] (HBASE-17569) HBase-Procedure module need to support mvn clean test -PskipProcedureTests to skip unit test
[
https://issues.apache.org/jira/browse/HBASE-17569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liang updated HBASE-17569:
-
Attachment: HBase-17569-Addendum.patch
> HBase-Procedure module need to support mvn clean test -PskipProcedureTests to
> skip unit test
>
>
> Key: HBASE-17569
> URL: https://issues.apache.org/jira/browse/HBASE-17569
> Project: HBase
> Issue Type: Improvement
> Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBase-17569-Addendum.patch, HBase-17569-V1.patch
>
>
> From Reference guide, we know that To skip the tests in the hbase-server
> module, you would run:
> {code}
> mvn clean test -PskipServerTests
> {code}
> We can also support this command in hbase-procedure module.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-3462) Fix table.jsp in regards to splitting a region/table with an optional splitkey
[
https://issues.apache.org/jira/browse/HBASE-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-3462:
-
Attachment: HBASE-3462-BM-01.patch
Retry. Failures can't be related. The white space will be fixed by git on
commit.
> Fix table.jsp in regards to splitting a region/table with an optional splitkey
> --
>
> Key: HBASE-3462
> URL: https://issues.apache.org/jira/browse/HBASE-3462
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 0.90.0
>Reporter: Lars George
>Assignee: Balazs Meszaros
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-3462-BM-01.patch, HBASE-3462-BM-01.patch
>
>
> After HBASE-3328 and HBASE-3437 went in there is also the table.jsp that
> needs updating to support the same features. Also, at the same time update
> the wording, for example
> {quote}
> This action will force a split of all eligible regions of the table, or, if a
> key is supplied, only the region containing the given key. An eligible region
> is one that does not contain any references to other regions. Split requests
> for noneligible regions will be ignored.
> {quote}
> I think it means it splits either all regions (that are splittable) or a
> specific one. It says though "the region containing the given key", that
> seems wrong in any event. Currently we do a split on the tablename when
> nothing was specified or else do an internal get(region), which is an exact
> match on the rows in .META.. In other words you need to match the region name
> exactly or else it fails. It reports it has accepted the request but logs
> internally
> {code}
> 2011-01-21 15:37:24,340 INFO org.apache.hadoop.hbase.client.HBaseAdmin: No
> server in .META. for csfsef; pair=null
> {code}
> Error reporting could be better but because of the async nature this is more
> difficult, yet it would be nice there is some concept of a Future to be able
> to poll the result if needed.
> Finally, when you go back to the previous page after submitting the split the
> entered values show up in the "compact" input fields, at least on my Chrome.
> The inputs in both forms are named the same so it seems to confuse it. This
> could be improved a lot by making the landing page reload the main one
> automatically or refresh on reload instead of submitting the request again.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-16524) Procedure v2 - Compute WALs cleanup on wal modification and not on every sync
[ https://issues.apache.org/jira/browse/HBASE-16524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847555#comment-15847555 ] Appy commented on HBASE-16524: -- Not sure if it's an issue at this point, but definitely information that should go in code as comment. HBASE-17573 > Procedure v2 - Compute WALs cleanup on wal modification and not on every sync > - > > Key: HBASE-16524 > URL: https://issues.apache.org/jira/browse/HBASE-16524 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Appy >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: flame1.svg, HBASE-16524.master.001.patch, > HBASE-16524.master.002.patch, HBASE-16524-v2.patch, HBASE-16524-v3.patch, > HBASE-16524-v4.patch, HBASE-16524-v5.patch, HBASE-16524-v6.patch > > > Fix performance regression introduced by HBASE-16094. > Instead of scanning all the wals every time, we can rely on the > insert/update/delete events we have. > and since we want to delete the wals in order we can keep track of what is > "holding" that wal, and take a hit on scanning all the trackers only when we > remove the first log in the queue. > e.g. > WAL-1 [1, 2] > WAL-2 [1] -> "[2] is holding WAL-1" > WAL-3 [2] -> "WAL 1 can be removed, recompute what is holding WAL-2" -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-17573) Some refactor in proc wals code and adding comments on some corner cases.
[ https://issues.apache.org/jira/browse/HBASE-17573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy reassigned HBASE-17573: Assignee: Appy > Some refactor in proc wals code and adding comments on some corner cases. > - > > Key: HBASE-17573 > URL: https://issues.apache.org/jira/browse/HBASE-17573 > Project: HBase > Issue Type: Task >Reporter: Appy >Assignee: Appy > > Previous comment which outlines the work to be > done:https://issues.apache.org/jira/browse/HBASE-16524?focusedCommentId=15798436=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15798436 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-17573) Some refactor in proc wals code and adding comments on some corner cases.
Appy created HBASE-17573: Summary: Some refactor in proc wals code and adding comments on some corner cases. Key: HBASE-17573 URL: https://issues.apache.org/jira/browse/HBASE-17573 Project: HBase Issue Type: Task Reporter: Appy Previous comment which outlines the work to be done:https://issues.apache.org/jira/browse/HBASE-16524?focusedCommentId=15798436=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15798436 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-3462) Fix table.jsp in regards to splitting a region/table with an optional splitkey
[
https://issues.apache.org/jira/browse/HBASE-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847545#comment-15847545
]
Hadoop QA commented on HBASE-3462:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 49s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 59s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 6s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
2s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 139m 20s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestRowTooBig |
| | hadoop.hbase.regionserver.TestPerColumnFamilyFlush |
| Timed out junit tests |
org.apache.hadoop.hbase.regionserver.TestCorruptedRegionStoreFile |
| | org.apache.hadoop.hbase.mapred.TestTableInputFormat |
| | org.apache.hadoop.hbase.TestHBaseTestingUtility |
| | org.apache.hadoop.hbase.wal.TestWALSplitCompressed |
| | org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster |
| | org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter |
| | org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover |
| | org.apache.hadoop.hbase.wal.TestBoundedRegionGroupingStrategy |
| | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
| | org.apache.hadoop.hbase.wal.TestWALSplit |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.10.1 Server=1.10.1 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12850291/HBASE-3462-BM-01.patch
|
| JIRA Issue | HBASE-3462 |
| Optional Tests | asflicense javac javadoc unit |
| uname | Linux 15b580682602 3.19.0-25-generic #26~14.04.1-Ubuntu SMP Fri Jul
24 21:16:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 680289d |
| Default Java | 1.8.0_121 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/5522/artifact/patchprocess/whitespace-tabs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5522/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/5522/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5522/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5522/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Fix table.jsp in regards to splitting a region/table with an optional splitkey
> --
>
> Key: HBASE-3462
> URL: https://issues.apache.org/jira/browse/HBASE-3462
> Project: HBase
> Issue Type: Improvement
>
[jira] (HBASE-16524) Procedure v2 - Compute WALs cleanup on wal modification and not on every sync
[ https://issues.apache.org/jira/browse/HBASE-16524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847542#comment-15847542 ] stack commented on HBASE-16524: --- Want to file an issue to change above [~appy]? And thanks for the heads up. > Procedure v2 - Compute WALs cleanup on wal modification and not on every sync > - > > Key: HBASE-16524 > URL: https://issues.apache.org/jira/browse/HBASE-16524 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Appy >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: flame1.svg, HBASE-16524.master.001.patch, > HBASE-16524.master.002.patch, HBASE-16524-v2.patch, HBASE-16524-v3.patch, > HBASE-16524-v4.patch, HBASE-16524-v5.patch, HBASE-16524-v6.patch > > > Fix performance regression introduced by HBASE-16094. > Instead of scanning all the wals every time, we can rely on the > insert/update/delete events we have. > and since we want to delete the wals in order we can keep track of what is > "holding" that wal, and take a hit on scanning all the trackers only when we > remove the first log in the queue. > e.g. > WAL-1 [1, 2] > WAL-2 [1] -> "[2] is holding WAL-1" > WAL-3 [2] -> "WAL 1 can be removed, recompute what is holding WAL-2" -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-17543) Create additional ReplicationEndpoint WALEntryFilters by configuration
[
https://issues.apache.org/jira/browse/HBASE-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847539#comment-15847539
]
Hadoop QA commented on HBASE-17543:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 34s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 89m 13s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 132m 57s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12850286/HBASE-17543.v3.patch |
| JIRA Issue | HBASE-17543 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux ba6fadb409d3 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 680289d |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5523/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5523/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Create additional ReplicationEndpoint WALEntryFilters by configuration
> --
>
> Key: HBASE-17543
> URL: https://issues.apache.org/jira/browse/HBASE-17543
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
> Fix For: 2.0.0, 1.4.0
>
>
[jira] (HBASE-17543) Create additional ReplicationEndpoint WALEntryFilters by configuration
[ https://issues.apache.org/jira/browse/HBASE-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847505#comment-15847505 ] Ted Yu commented on HBASE-17543: Integrated to master branch. Please attach patch for branch-1 > Create additional ReplicationEndpoint WALEntryFilters by configuration > -- > > Key: HBASE-17543 > URL: https://issues.apache.org/jira/browse/HBASE-17543 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17543.patch, HBASE-17543.v2.patch, > HBASE-17543.v3.patch > > > The existing BaseReplicationEndpoint creates a ChainWALEntryFilter containing > a NamespaceTableCfWALEntryFilter and a ScopeWALEntryFilter. Adding a custom > WALEntryFilter type to this chain requires creating an entirely new > ReplicationEndpoint subclass and creating a new peer on the running cluster, > which can be operationally complex to transition to without data loss in > cases such as master/master. > For WALEntryFilters without constructor parameters, it would be > straightforward to have a Configuration option to list additional > WALEntryFilter classes the operator wants to include in the filter chain in > the default endpoint, and then have the endpoint instantiate the filters via > reflection. Then filter logic could be added (or removed) with only a > hbase-site.xml change and a rolling restart. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly and beyond
[
https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847486#comment-15847486
]
Hadoop QA commented on HBASE-16981:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 0s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 0s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 29s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 16s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 0s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 40s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 14s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 102m 41s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m 9s
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 161m 5s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mob.compactions.TestPartitionedMobCompactor
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12850276/HBASE-16981.master.007.patch
|
| JIRA Issue | HBASE-16981 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile rubocop ruby_lint |
| uname | Linux 214db9a27654 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] (HBASE-16621) HBCK should have -fixHFileLinks
[
https://issues.apache.org/jira/browse/HBASE-16621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Janos Gub updated HBASE-16621:
--
Attachment: HBASE-16621-5.patch
[~tedyu][~enis] Thank you for the reviews. Attaching a patch with these
modifications.
> HBCK should have -fixHFileLinks
> ---
>
> Key: HBASE-16621
> URL: https://issues.apache.org/jira/browse/HBASE-16621
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Janos Gub
> Labels: beginner, operability
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16621-2.patch, HBASE-16621-3.patch,
> HBASE-16621-4.patch, HBASE-16621-5.patch, HBASE-16621.patch, refactor-2.patch
>
>
> Similar to {{-fixReferenceFiles}}, HBCK should be able to sideline dangling
> HFile Links as well.
> We have seen a couple of cases, where due to HDFS-level fsck run which
> deleted files with missing blocks, the cluster is left with dangling HFIle
> Links, and regions cannot be opened because of these. Only a manual and
> error-prone finding and clearing of HFileLinks can save the tables regions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-17197) hfile does not work in 2.0
[
https://issues.apache.org/jira/browse/HBASE-17197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Balazs Meszaros updated HBASE-17197:
Assignee: Balazs Meszaros (was: huaxiang sun)
Status: Patch Available (was: Reopened)
> hfile does not work in 2.0
> --
>
> Key: HBASE-17197
> URL: https://issues.apache.org/jira/browse/HBASE-17197
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: Balazs Meszaros
> Attachments: HBASE-17197-BM-01.patch
>
>
> I tried to use hfile in master branch, it does not print out kv pairs or meta
> as it is supposed to be.
> {code}
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/
> 53e9f9bc328f468b87831221de3a0c74 bdc6e1c4eea246a99e989e02d554cb03
> bf9275ac418d4d458904d81137e82683
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/bf9275ac418d4d458904d81137e82683
> -m
> 2016-11-29 12:25:22,019 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/bf9275ac418d4d458904d81137e82683
> -p
> 2016-11-29 12:25:27,333 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Scanned kv count -> 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-17197) hfile does not work in 2.0
[
https://issues.apache.org/jira/browse/HBASE-17197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Balazs Meszaros updated HBASE-17197:
Attachment: HBASE-17197-BM-01.patch
> hfile does not work in 2.0
> --
>
> Key: HBASE-17197
> URL: https://issues.apache.org/jira/browse/HBASE-17197
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: HBASE-17197-BM-01.patch
>
>
> I tried to use hfile in master branch, it does not print out kv pairs or meta
> as it is supposed to be.
> {code}
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/
> 53e9f9bc328f468b87831221de3a0c74 bdc6e1c4eea246a99e989e02d554cb03
> bf9275ac418d4d458904d81137e82683
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/bf9275ac418d4d458904d81137e82683
> -m
> 2016-11-29 12:25:22,019 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/bf9275ac418d4d458904d81137e82683
> -p
> 2016-11-29 12:25:27,333 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Scanned kv count -> 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-3462) Fix table.jsp in regards to splitting a region/table with an optional splitkey
[
https://issues.apache.org/jira/browse/HBASE-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847425#comment-15847425
]
stack commented on HBASE-3462:
--
I tried it. Nice. I can use row key to dictate where the split or compaction
can be. I like it. Will commit after CI comes back.
> Fix table.jsp in regards to splitting a region/table with an optional splitkey
> --
>
> Key: HBASE-3462
> URL: https://issues.apache.org/jira/browse/HBASE-3462
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 0.90.0
>Reporter: Lars George
>Assignee: Balazs Meszaros
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-3462-BM-01.patch
>
>
> After HBASE-3328 and HBASE-3437 went in there is also the table.jsp that
> needs updating to support the same features. Also, at the same time update
> the wording, for example
> {quote}
> This action will force a split of all eligible regions of the table, or, if a
> key is supplied, only the region containing the given key. An eligible region
> is one that does not contain any references to other regions. Split requests
> for noneligible regions will be ignored.
> {quote}
> I think it means it splits either all regions (that are splittable) or a
> specific one. It says though "the region containing the given key", that
> seems wrong in any event. Currently we do a split on the tablename when
> nothing was specified or else do an internal get(region), which is an exact
> match on the rows in .META.. In other words you need to match the region name
> exactly or else it fails. It reports it has accepted the request but logs
> internally
> {code}
> 2011-01-21 15:37:24,340 INFO org.apache.hadoop.hbase.client.HBaseAdmin: No
> server in .META. for csfsef; pair=null
> {code}
> Error reporting could be better but because of the async nature this is more
> difficult, yet it would be nice there is some concept of a Future to be able
> to poll the result if needed.
> Finally, when you go back to the previous page after submitting the split the
> entered values show up in the "compact" input fields, at least on my Chrome.
> The inputs in both forms are named the same so it seems to confuse it. This
> could be improved a lot by making the landing page reload the main one
> automatically or refresh on reload instead of submitting the request again.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-16621) HBCK should have -fixHFileLinks
[
https://issues.apache.org/jira/browse/HBASE-16621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847423#comment-15847423
]
Enis Soztutar commented on HBASE-16621:
---
A nit is the method name {{offlineHLinkFileRepair}} should be
{{offlineHFileLinkRepair}}. Other than that looks pretty good.
> HBCK should have -fixHFileLinks
> ---
>
> Key: HBASE-16621
> URL: https://issues.apache.org/jira/browse/HBASE-16621
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Janos Gub
> Labels: beginner, operability
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16621-2.patch, HBASE-16621-3.patch,
> HBASE-16621-4.patch, HBASE-16621.patch, refactor-2.patch
>
>
> Similar to {{-fixReferenceFiles}}, HBCK should be able to sideline dangling
> HFile Links as well.
> We have seen a couple of cases, where due to HDFS-level fsck run which
> deleted files with missing blocks, the cluster is left with dangling HFIle
> Links, and regions cannot be opened because of these. Only a manual and
> error-prone finding and clearing of HFileLinks can save the tables regions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-17543) Create additional ReplicationEndpoint WALEntryFilters by configuration
[ https://issues.apache.org/jira/browse/HBASE-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847411#comment-15847411 ] Enis Soztutar commented on HBASE-17543: --- +1. Needs a release note (and maybe doc update after). > Create additional ReplicationEndpoint WALEntryFilters by configuration > -- > > Key: HBASE-17543 > URL: https://issues.apache.org/jira/browse/HBASE-17543 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17543.patch, HBASE-17543.v2.patch, > HBASE-17543.v3.patch > > > The existing BaseReplicationEndpoint creates a ChainWALEntryFilter containing > a NamespaceTableCfWALEntryFilter and a ScopeWALEntryFilter. Adding a custom > WALEntryFilter type to this chain requires creating an entirely new > ReplicationEndpoint subclass and creating a new peer on the running cluster, > which can be operationally complex to transition to without data loss in > cases such as master/master. > For WALEntryFilters without constructor parameters, it would be > straightforward to have a Configuration option to list additional > WALEntryFilter classes the operator wants to include in the filter chain in > the default endpoint, and then have the endpoint instantiate the filters via > reflection. Then filter logic could be added (or removed) with only a > hbase-site.xml change and a rolling restart. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847395#comment-15847395 ] Enis Soztutar commented on HBASE-17437: --- Pushed to master. Thanks [~yangyishan0901m] and [~zyork] for the patch! bq. Do you want it to be in the same JIRA or create a new JIRA for it? It can go here. We can keep this open for a couple of days if you can get the patch ready. Otherwise, we'll resolve this, and open another jira. > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Zach York > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, > HBASE-17437.master.001.patch, HBASE-17437.master.002.patch, > HBASE-17437.master.003.patch, HBASE-17437.master.004.patch, > HBASE-17437.master.005.patch, HBASE-17437.master.006.patch, > HBASE-17437.master.007.patch, HBASE-17437.master.008.patch, > HBASE-17437.master.009.patch, HBASE-17437.master.010.patch, > HBASE-17437.master.011.patch, HBASE-17437.master.012.patch, > hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-17522) RuntimeExceptions from MemoryMXBean should not take down server process
[ https://issues.apache.org/jira/browse/HBASE-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847394#comment-15847394 ] Hudson commented on HBASE-17522: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2420 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2420/]) HBASE-17522 Handle JVM throwing runtime exceptions when we ask for (busbey: rev 679182869876f834c7e0a62e06179fd928b83c3d) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheConfig.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * (edit) hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/ServerMetricsTmpl.jamon * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/util/MemorySizeUtil.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultMemStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HeapMemoryManager.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FlushType.java > RuntimeExceptions from MemoryMXBean should not take down server process > --- > > Key: HBASE-17522 > URL: https://issues.apache.org/jira/browse/HBASE-17522 > Project: HBase > Issue Type: Bug > Components: regionserver > Environment: java version "1.8.0_92" > Java(TM) SE Runtime Environment (build 1.8.0_92-b14) > Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode) > -XX:+UseG1GC -XX:+UseStringDeduplication -Xmx32768m >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 1.1.9 > > Attachments: HBASE-17522.0.patch, HBASE-17522.1.patch > > > RegionServer died after MemoryMXBean threw an IllegalArgumentException while > attempting to create a MemoryUsage object for the heap during construction of > the server load. > We shouldn't allow failure to get load information to take down the RS. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-17381) ReplicationSourceWorkerThread can die due to unhandled exceptions
[
https://issues.apache.org/jira/browse/HBASE-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847375#comment-15847375
]
Andrew Purtell commented on HBASE-17381:
I also favor aborting for unhandled exceptions.
Later, we can fix the handling.
> ReplicationSourceWorkerThread can die due to unhandled exceptions
> -
>
> Key: HBASE-17381
> URL: https://issues.apache.org/jira/browse/HBASE-17381
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Gary Helmling
>Assignee: huzheng
> Attachments: HBASE-17381.patch, HBASE-17381.v1.patch,
> HBASE-17381.v2.patch
>
>
> If a ReplicationSourceWorkerThread encounters an unexpected exception in the
> run() method (for example failure to allocate direct memory for the DFS
> client), the exception will be logged by the UncaughtExceptionHandler, but
> the thread will also die and the replication queue will back up indefinitely
> until the Regionserver is restarted.
> We should make sure the worker thread is resilient to all exceptions that it
> can actually handle. For those that it really can't, it seems better to
> abort the regionserver rather than just allow replication to stop with
> minimal signal.
> Here is a sample exception:
> {noformat}
> ERROR regionserver.ReplicationSource: Unexpected exception in
> ReplicationSourceWorkerThread,
> currentPath=hdfs://.../hbase/WALs/XXXwalfilenameXXX
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:693)
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
> at
> org.apache.hadoop.crypto.CryptoOutputStream.(CryptoOutputStream.java:96)
> at
> org.apache.hadoop.crypto.CryptoOutputStream.(CryptoOutputStream.java:113)
> at
> org.apache.hadoop.crypto.CryptoOutputStream.(CryptoOutputStream.java:108)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.createStreamPair(DataTransferSaslUtil.java:344)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:490)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getSaslStreams(SaslDataTransferClient.java:391)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:263)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:211)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.peerSend(SaslDataTransferClient.java:160)
> at
> org.apache.hadoop.hdfs.net.TcpPeerServer.peerFromSocketAndKey(TcpPeerServer.java:92)
> at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3444)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:778)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:695)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:356)
> at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:673)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)
> at java.io.DataInputStream.read(DataInputStream.java:100)
> at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:308)
> at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:276)
> at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:264)
> at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:423)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationWALReaderManager.openReader(ReplicationWALReaderManager.java:70)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceWorkerThread.openReader(ReplicationSource.java:830)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceWorkerThread.run(ReplicationSource.java:572)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847371#comment-15847371 ] stack commented on HBASE-17437: --- bq. What needs to go in ReleaseNotes? Just a write-up of new settings and how to use it? Yes sir. Target audience is the 'operator' or 'user' who is not interested in detail only how to make use of this nice new feature. So, yeah, how to configure it, how to turn it on, and a listing of anything that might 'surprise'. Can be short. Would be seed for documentation of the feature up in refguide. Thank you. > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Zach York > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, > HBASE-17437.master.001.patch, HBASE-17437.master.002.patch, > HBASE-17437.master.003.patch, HBASE-17437.master.004.patch, > HBASE-17437.master.005.patch, HBASE-17437.master.006.patch, > HBASE-17437.master.007.patch, HBASE-17437.master.008.patch, > HBASE-17437.master.009.patch, HBASE-17437.master.010.patch, > HBASE-17437.master.011.patch, HBASE-17437.master.012.patch, > hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847367#comment-15847367 ] Zach York commented on HBASE-17437: --- [~enis] Sure. Do you want it to be in the same JIRA or create a new JIRA for it? [~stack] What needs to go in ReleaseNotes? Just a write-up of new settings and how to use it? > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Zach York > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, > HBASE-17437.master.001.patch, HBASE-17437.master.002.patch, > HBASE-17437.master.003.patch, HBASE-17437.master.004.patch, > HBASE-17437.master.005.patch, HBASE-17437.master.006.patch, > HBASE-17437.master.007.patch, HBASE-17437.master.008.patch, > HBASE-17437.master.009.patch, HBASE-17437.master.010.patch, > HBASE-17437.master.011.patch, HBASE-17437.master.012.patch, > hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847363#comment-15847363 ] Enis Soztutar commented on HBASE-17437: --- No worries. Was waiting for you to +1. Let me commit. [~zyork] do you mind doing a backport patch to branch-1? > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Zach York > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, > HBASE-17437.master.001.patch, HBASE-17437.master.002.patch, > HBASE-17437.master.003.patch, HBASE-17437.master.004.patch, > HBASE-17437.master.005.patch, HBASE-17437.master.006.patch, > HBASE-17437.master.007.patch, HBASE-17437.master.008.patch, > HBASE-17437.master.009.patch, HBASE-17437.master.010.patch, > HBASE-17437.master.011.patch, HBASE-17437.master.012.patch, > hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-16621) HBCK should have -fixHFileLinks
[
https://issues.apache.org/jira/browse/HBASE-16621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Janos Gub updated HBASE-16621:
--
Attachment: HBASE-16621-4.patch
> HBCK should have -fixHFileLinks
> ---
>
> Key: HBASE-16621
> URL: https://issues.apache.org/jira/browse/HBASE-16621
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Janos Gub
> Labels: beginner, operability
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16621-2.patch, HBASE-16621-3.patch,
> HBASE-16621-4.patch, HBASE-16621.patch, refactor-2.patch
>
>
> Similar to {{-fixReferenceFiles}}, HBCK should be able to sideline dangling
> HFile Links as well.
> We have seen a couple of cases, where due to HDFS-level fsck run which
> deleted files with missing blocks, the cluster is left with dangling HFIle
> Links, and regions cannot be opened because of these. Only a manual and
> error-prone finding and clearing of HFileLinks can save the tables regions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-17198) FN updates during region merge (follow up to Procedure v2 merge)
[ https://issues.apache.org/jira/browse/HBASE-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-17198: Resolution: Fixed Status: Resolved (was: Patch Available) > FN updates during region merge (follow up to Procedure v2 merge) > > > Key: HBASE-17198 > URL: https://issues.apache.org/jira/browse/HBASE-17198 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-17198.master.001.patch > > > As mentioned in https://reviews.apache.org/r/53242/ (HBASE-16941), since the > procedure v2 merge changes are in development, there is a follow up > optimization/cleanup that can be done for favored nodes during merge. This > jira will be taken up once HBASE-16119 is complete. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-17281) FN should use datanode port from hdfs configuration
[
https://issues.apache.org/jira/browse/HBASE-17281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847351#comment-15847351
]
Francis Liu commented on HBASE-17281:
-
[~thiruvel] The patch needs a rebase prolly because I applied HBASE-17101
> FN should use datanode port from hdfs configuration
> ---
>
> Key: HBASE-17281
> URL: https://issues.apache.org/jira/browse/HBASE-17281
> Project: HBase
> Issue Type: Sub-task
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17281.master.001.patch,
> HBASE-17281.master.002.patch, HBASE-17281.master.003.patch,
> HBASE-17281.master.004.patch
>
>
> Currently we use the ServerName port for providing favored node hints. We
> should use the DN port from hdfs-site.xml instead to avoid warning messages
> in region server logs. The warnings will be from this section of HDFS code,
> it moves across classes.
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java#L1758
> {code}
> private boolean[] getPinnings(DatanodeInfo[] nodes) {
> if (favoredNodes == null) {
> return null;
> } else {
> boolean[] pinnings = new boolean[nodes.length];
> HashSet favoredSet = new HashSet<>(Arrays.asList(favoredNodes));
> for (int i = 0; i < nodes.length; i++) {
> pinnings[i] = favoredSet.remove(nodes[i].getXferAddrWithHostname());
> LOG.debug("{} was chosen by name node (favored={}).",
> nodes[i].getXferAddrWithHostname(), pinnings[i]);
> }
> if (!favoredSet.isEmpty()) {
> // There is one or more favored nodes that were not allocated.
> LOG.warn("These favored nodes were specified but not chosen: "
> + favoredSet + " Specified favored nodes: "
> + Arrays.toString(favoredNodes));
> }
> return pinnings;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-17572) HMaster: Caught throwable while processing event C_M_MERGE_REGION (UndeclaredThrowableException)
[
https://issues.apache.org/jira/browse/HBASE-17572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-17572:
---
Description:
Running ITBLL 1B rows against branch-1.3 compiled against Hadoop 2.7.3 with the
noKill monkey policy, I see both masters go down with
master.HMaster: Caught throwable while processing event C_M_MERGE_REGION
java.lang.reflect.UndeclaredThrowableException
In ServerManager#sendRegionsMerge we call ProtobufUtil#mergeRegions, which does
a doAs, and the code within that block invokes RSRpcServices#mergeRegions, but
is not resilient against RegionOpeningException ("region is opening")
An UndeclaredThrowableException is "thrown by a method invocation on a proxy
instance if its invocation handler's invoke method throws a checked exception
(a Throwable that is not assignable to RuntimeException or Error) that is not
assignable to any of the exception types declared in the throws clause of the
method that was invoked on the proxy instance and dispatched to the invocation
handler."
(http://docs.oracle.com/javase/7/docs/api/java/lang/reflect/UndeclaredThrowableException.html)
{noformat}
2017-01-31 07:21:17,495 FATAL [MASTER_TABLE_OPERATIONS-node-1:16000-0]
master.HMaster: Caught throwable while processing event C_M_MERGE_REGION
java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1737)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.mergeRegions(ProtobufUtil.java:1990)
at
org.apache.hadoop.hbase.master.ServerManager.sendRegionsMerge(ServerManager.java:925)
at
org.apache.hadoop.hbase.master.handler.DispatchMergingRegionHandler.process(DispatchMergingRegionHandler.java:153)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.google.protobuf.ServiceException:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.RegionOpeningException):
org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region
IntegrationTestBigLinkedList,|\xFFnk\x1C\x85<[\x1Ef\xFDE\xF9\xAA\xAC\x08,1485846598043.f56ad22121e872777468020c4452a7c7.
is opening on node-2.cluster,16020,1485822382322
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2964)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1139)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.mergeRegions(RSRpcServices.java:1497)
at
org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22749)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2355)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:244)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:340)
at
org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.mergeRegions(AdminProtos.java:23695)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil$1.run(ProtobufUtil.java:1993)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil$1.run(ProtobufUtil.java:1990)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1719)
... 7 more
Caused by:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.RegionOpeningException):
org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region
IntegrationTestBigLinkedList,|\xFFnk\x1C\x85<[\x1Ef\xFDE\xF9\xAA\xAC\x08,1485846598043.f56ad22121e872777468020c4452a7c7.
is opening on node-2.cluster,16020,1485822382322
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2964)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1139)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.mergeRegions(RSRpcServices.java:1497)
at
org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22749)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2355)
at
[jira] (HBASE-17281) FN should use datanode port from hdfs configuration
[
https://issues.apache.org/jira/browse/HBASE-17281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847346#comment-15847346
]
Francis Liu commented on HBASE-17281:
-
+1
> FN should use datanode port from hdfs configuration
> ---
>
> Key: HBASE-17281
> URL: https://issues.apache.org/jira/browse/HBASE-17281
> Project: HBase
> Issue Type: Sub-task
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17281.master.001.patch,
> HBASE-17281.master.002.patch, HBASE-17281.master.003.patch,
> HBASE-17281.master.004.patch
>
>
> Currently we use the ServerName port for providing favored node hints. We
> should use the DN port from hdfs-site.xml instead to avoid warning messages
> in region server logs. The warnings will be from this section of HDFS code,
> it moves across classes.
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java#L1758
> {code}
> private boolean[] getPinnings(DatanodeInfo[] nodes) {
> if (favoredNodes == null) {
> return null;
> } else {
> boolean[] pinnings = new boolean[nodes.length];
> HashSet favoredSet = new HashSet<>(Arrays.asList(favoredNodes));
> for (int i = 0; i < nodes.length; i++) {
> pinnings[i] = favoredSet.remove(nodes[i].getXferAddrWithHostname());
> LOG.debug("{} was chosen by name node (favored={}).",
> nodes[i].getXferAddrWithHostname(), pinnings[i]);
> }
> if (!favoredSet.isEmpty()) {
> // There is one or more favored nodes that were not allocated.
> LOG.warn("These favored nodes were specified but not chosen: "
> + favoredSet + " Specified favored nodes: "
> + Arrays.toString(favoredNodes));
> }
> return pinnings;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-3462) Fix table.jsp in regards to splitting a region/table with an optional splitkey
[
https://issues.apache.org/jira/browse/HBASE-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847336#comment-15847336
]
stack commented on HBASE-3462:
--
+1 Let me try it...
> Fix table.jsp in regards to splitting a region/table with an optional splitkey
> --
>
> Key: HBASE-3462
> URL: https://issues.apache.org/jira/browse/HBASE-3462
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 0.90.0
>Reporter: Lars George
>Assignee: Balazs Meszaros
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-3462-BM-01.patch
>
>
> After HBASE-3328 and HBASE-3437 went in there is also the table.jsp that
> needs updating to support the same features. Also, at the same time update
> the wording, for example
> {quote}
> This action will force a split of all eligible regions of the table, or, if a
> key is supplied, only the region containing the given key. An eligible region
> is one that does not contain any references to other regions. Split requests
> for noneligible regions will be ignored.
> {quote}
> I think it means it splits either all regions (that are splittable) or a
> specific one. It says though "the region containing the given key", that
> seems wrong in any event. Currently we do a split on the tablename when
> nothing was specified or else do an internal get(region), which is an exact
> match on the rows in .META.. In other words you need to match the region name
> exactly or else it fails. It reports it has accepted the request but logs
> internally
> {code}
> 2011-01-21 15:37:24,340 INFO org.apache.hadoop.hbase.client.HBaseAdmin: No
> server in .META. for csfsef; pair=null
> {code}
> Error reporting could be better but because of the async nature this is more
> difficult, yet it would be nice there is some concept of a Future to be able
> to poll the result if needed.
> Finally, when you go back to the previous page after submitting the split the
> entered values show up in the "compact" input fields, at least on my Chrome.
> The inputs in both forms are named the same so it seems to confuse it. This
> could be improved a lot by making the landing page reload the main one
> automatically or refresh on reload instead of submitting the request again.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-3462) Fix table.jsp in regards to splitting a region/table with an optional splitkey
[
https://issues.apache.org/jira/browse/HBASE-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-3462:
-
Fix Version/s: 2.0.0
Status: Patch Available (was: Open)
Submitting patch to get a CI run.
> Fix table.jsp in regards to splitting a region/table with an optional splitkey
> --
>
> Key: HBASE-3462
> URL: https://issues.apache.org/jira/browse/HBASE-3462
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 0.90.0
>Reporter: Lars George
>Assignee: Balazs Meszaros
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-3462-BM-01.patch
>
>
> After HBASE-3328 and HBASE-3437 went in there is also the table.jsp that
> needs updating to support the same features. Also, at the same time update
> the wording, for example
> {quote}
> This action will force a split of all eligible regions of the table, or, if a
> key is supplied, only the region containing the given key. An eligible region
> is one that does not contain any references to other regions. Split requests
> for noneligible regions will be ignored.
> {quote}
> I think it means it splits either all regions (that are splittable) or a
> specific one. It says though "the region containing the given key", that
> seems wrong in any event. Currently we do a split on the tablename when
> nothing was specified or else do an internal get(region), which is an exact
> match on the rows in .META.. In other words you need to match the region name
> exactly or else it fails. It reports it has accepted the request but logs
> internally
> {code}
> 2011-01-21 15:37:24,340 INFO org.apache.hadoop.hbase.client.HBaseAdmin: No
> server in .META. for csfsef; pair=null
> {code}
> Error reporting could be better but because of the async nature this is more
> difficult, yet it would be nice there is some concept of a Future to be able
> to poll the result if needed.
> Finally, when you go back to the previous page after submitting the split the
> entered values show up in the "compact" input fields, at least on my Chrome.
> The inputs in both forms are named the same so it seems to confuse it. This
> could be improved a lot by making the landing page reload the main one
> automatically or refresh on reload instead of submitting the request again.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-17572) HMaster: Caught throwable while processing event C_M_MERGE_REGION (UndeclaredThrowableException)
Andrew Purtell created HBASE-17572:
--
Summary: HMaster: Caught throwable while processing event
C_M_MERGE_REGION (UndeclaredThrowableException)
Key: HBASE-17572
URL: https://issues.apache.org/jira/browse/HBASE-17572
Project: HBase
Issue Type: Bug
Affects Versions: 1.3.0
Reporter: Andrew Purtell
Running ITBLL 1B rows against 1.3.0 compiled against Hadoop 2.7.3 with the
noKill monkey policy, I see both masters go down with
master.HMaster: Caught throwable while processing event C_M_MERGE_REGION
java.lang.reflect.UndeclaredThrowableException
In ServerManager#sendRegionsMerge we call ProtobufUtil#mergeRegions, which does
a doAs, and the code within that block invokes RSRpcServices#mergeRegions, but
is not resilient against RegionOpeningException ("region is opening")
An UndeclaredThrowableException is "thrown by a method invocation on a proxy
instance if its invocation handler's invoke method throws a checked exception
(a Throwable that is not assignable to RuntimeException or Error) that is not
assignable to any of the exception types declared in the throws clause of the
method that was invoked on the proxy instance and dispatched to the invocation
handler."
(http://docs.oracle.com/javase/7/docs/api/java/lang/reflect/UndeclaredThrowableException.html)
{noformat}
2017-01-31 07:21:17,495 FATAL [MASTER_TABLE_OPERATIONS-node-1:16000-0]
master.HMaster: Caught throwable while processing event C_M_MERGE_REGION
java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1737)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.mergeRegions(ProtobufUtil.java:1990)
at
org.apache.hadoop.hbase.master.ServerManager.sendRegionsMerge(ServerManager.java:925)
at
org.apache.hadoop.hbase.master.handler.DispatchMergingRegionHandler.process(DispatchMergingRegionHandler.java:153)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.google.protobuf.ServiceException:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.RegionOpeningException):
org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region
IntegrationTestBigLinkedList,|\xFFnk\x1C\x85<[\x1Ef\xFDE\xF9\xAA\xAC\x08,1485846598043.f56ad22121e872777468020c4452a7c7.
is opening on node-2.cluster,16020,1485822382322
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2964)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1139)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.mergeRegions(RSRpcServices.java:1497)
at
org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22749)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2355)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:244)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:340)
at
org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.mergeRegions(AdminProtos.java:23695)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil$1.run(ProtobufUtil.java:1993)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil$1.run(ProtobufUtil.java:1990)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1719)
... 7 more
Caused by:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.RegionOpeningException):
org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region
IntegrationTestBigLinkedList,|\xFFnk\x1C\x85<[\x1Ef\xFDE\xF9\xAA\xAC\x08,1485846598043.f56ad22121e872777468020c4452a7c7.
is opening on node-2.cluster,16020,1485822382322
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2964)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1139)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.mergeRegions(RSRpcServices.java:1497)
at
[jira] (HBASE-17543) Create additional ReplicationEndpoint WALEntryFilters by configuration
[ https://issues.apache.org/jira/browse/HBASE-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17543: --- Hadoop Flags: Reviewed Fix Version/s: 1.4.0 2.0.0 > Create additional ReplicationEndpoint WALEntryFilters by configuration > -- > > Key: HBASE-17543 > URL: https://issues.apache.org/jira/browse/HBASE-17543 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17543.patch, HBASE-17543.v2.patch, > HBASE-17543.v3.patch > > > The existing BaseReplicationEndpoint creates a ChainWALEntryFilter containing > a NamespaceTableCfWALEntryFilter and a ScopeWALEntryFilter. Adding a custom > WALEntryFilter type to this chain requires creating an entirely new > ReplicationEndpoint subclass and creating a new peer on the running cluster, > which can be operationally complex to transition to without data loss in > cases such as master/master. > For WALEntryFilters without constructor parameters, it would be > straightforward to have a Configuration option to list additional > WALEntryFilter classes the operator wants to include in the filter chain in > the default endpoint, and then have the endpoint instantiate the filters via > reflection. Then filter logic could be added (or removed) with only a > hbase-site.xml change and a rolling restart. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-3462) Fix table.jsp in regards to splitting a region/table with an optional splitkey
[
https://issues.apache.org/jira/browse/HBASE-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Balazs Meszaros updated HBASE-3462:
---
Attachment: HBASE-3462-BM-01.patch
Now compact/split requires row keys instead of region names (as UI suggests).
> Fix table.jsp in regards to splitting a region/table with an optional splitkey
> --
>
> Key: HBASE-3462
> URL: https://issues.apache.org/jira/browse/HBASE-3462
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 0.90.0
>Reporter: Lars George
>Assignee: Balazs Meszaros
> Labels: beginner
> Attachments: HBASE-3462-BM-01.patch
>
>
> After HBASE-3328 and HBASE-3437 went in there is also the table.jsp that
> needs updating to support the same features. Also, at the same time update
> the wording, for example
> {quote}
> This action will force a split of all eligible regions of the table, or, if a
> key is supplied, only the region containing the given key. An eligible region
> is one that does not contain any references to other regions. Split requests
> for noneligible regions will be ignored.
> {quote}
> I think it means it splits either all regions (that are splittable) or a
> specific one. It says though "the region containing the given key", that
> seems wrong in any event. Currently we do a split on the tablename when
> nothing was specified or else do an internal get(region), which is an exact
> match on the rows in .META.. In other words you need to match the region name
> exactly or else it fails. It reports it has accepted the request but logs
> internally
> {code}
> 2011-01-21 15:37:24,340 INFO org.apache.hadoop.hbase.client.HBaseAdmin: No
> server in .META. for csfsef; pair=null
> {code}
> Error reporting could be better but because of the async nature this is more
> difficult, yet it would be nice there is some concept of a Future to be able
> to poll the result if needed.
> Finally, when you go back to the previous page after submitting the split the
> entered values show up in the "compact" input fields, at least on my Chrome.
> The inputs in both forms are named the same so it seems to confuse it. This
> could be improved a lot by making the landing page reload the main one
> automatically or refresh on reload instead of submitting the request again.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-16621) HBCK should have -fixHFileLinks
[
https://issues.apache.org/jira/browse/HBASE-16621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847311#comment-15847311
]
Ted Yu commented on HBASE-16621:
Almost there.
{code}
+"-fixHFileLinkFiles");
{code}
Please change the name of new option (throughout the patch): -fixHFileLinkFiles
=> -fixHFileLinks
> HBCK should have -fixHFileLinks
> ---
>
> Key: HBASE-16621
> URL: https://issues.apache.org/jira/browse/HBASE-16621
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Janos Gub
> Labels: beginner, operability
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16621-2.patch, HBASE-16621-3.patch,
> HBASE-16621.patch, refactor-2.patch
>
>
> Similar to {{-fixReferenceFiles}}, HBCK should be able to sideline dangling
> HFile Links as well.
> We have seen a couple of cases, where due to HDFS-level fsck run which
> deleted files with missing blocks, the cluster is left with dangling HFIle
> Links, and regions cannot be opened because of these. Only a manual and
> error-prone finding and clearing of HFileLinks can save the tables regions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-17101) FavoredNodes should not apply to system tables
[ https://issues.apache.org/jira/browse/HBASE-17101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-17101: Resolution: Fixed Status: Resolved (was: Patch Available) > FavoredNodes should not apply to system tables > -- > > Key: HBASE-17101 > URL: https://issues.apache.org/jira/browse/HBASE-17101 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-17101.master.001.patch, > HBASE-17101.master.002.patch, HBASE-17101.master.003.patch, > HBASE-17101.master.004.patch, HBASE_17101_rough_draft.patch > > > As described in the doc (see HBASE-15531), we would like to start with user > tables for favored nodes. This task ensures FN does not apply to system > tables. > System tables are in memory and won't benefit from favored nodes. Since we > also maintain FN information for user regions in meta, it helps to keep > implementation simpler by ignoring system tables for the first iterations. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-17101) FavoredNodes should not apply to system tables
[ https://issues.apache.org/jira/browse/HBASE-17101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847310#comment-15847310 ] Francis Liu commented on HBASE-17101: - Committed to master. Thanks Thiruvel. > FavoredNodes should not apply to system tables > -- > > Key: HBASE-17101 > URL: https://issues.apache.org/jira/browse/HBASE-17101 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-17101.master.001.patch, > HBASE-17101.master.002.patch, HBASE-17101.master.003.patch, > HBASE-17101.master.004.patch, HBASE_17101_rough_draft.patch > > > As described in the doc (see HBASE-15531), we would like to start with user > tables for favored nodes. This task ensures FN does not apply to system > tables. > System tables are in memory and won't benefit from favored nodes. Since we > also maintain FN information for user regions in meta, it helps to keep > implementation simpler by ignoring system tables for the first iterations. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-17543) Create additional ReplicationEndpoint WALEntryFilters by configuration
[ https://issues.apache.org/jira/browse/HBASE-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Jacoby updated HBASE-17543: Status: Patch Available (was: Open) > Create additional ReplicationEndpoint WALEntryFilters by configuration > -- > > Key: HBASE-17543 > URL: https://issues.apache.org/jira/browse/HBASE-17543 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby > Attachments: HBASE-17543.patch, HBASE-17543.v2.patch, > HBASE-17543.v3.patch > > > The existing BaseReplicationEndpoint creates a ChainWALEntryFilter containing > a NamespaceTableCfWALEntryFilter and a ScopeWALEntryFilter. Adding a custom > WALEntryFilter type to this chain requires creating an entirely new > ReplicationEndpoint subclass and creating a new peer on the running cluster, > which can be operationally complex to transition to without data loss in > cases such as master/master. > For WALEntryFilters without constructor parameters, it would be > straightforward to have a Configuration option to list additional > WALEntryFilter classes the operator wants to include in the filter chain in > the default endpoint, and then have the endpoint instantiate the filters via > reflection. Then filter logic could be added (or removed) with only a > hbase-site.xml change and a rolling restart. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-17543) Create additional ReplicationEndpoint WALEntryFilters by configuration
[ https://issues.apache.org/jira/browse/HBASE-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Jacoby updated HBASE-17543: Status: Open (was: Patch Available) > Create additional ReplicationEndpoint WALEntryFilters by configuration > -- > > Key: HBASE-17543 > URL: https://issues.apache.org/jira/browse/HBASE-17543 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby > Attachments: HBASE-17543.patch, HBASE-17543.v2.patch, > HBASE-17543.v3.patch > > > The existing BaseReplicationEndpoint creates a ChainWALEntryFilter containing > a NamespaceTableCfWALEntryFilter and a ScopeWALEntryFilter. Adding a custom > WALEntryFilter type to this chain requires creating an entirely new > ReplicationEndpoint subclass and creating a new peer on the running cluster, > which can be operationally complex to transition to without data loss in > cases such as master/master. > For WALEntryFilters without constructor parameters, it would be > straightforward to have a Configuration option to list additional > WALEntryFilter classes the operator wants to include in the filter chain in > the default endpoint, and then have the endpoint instantiate the filters via > reflection. Then filter logic could be added (or removed) with only a > hbase-site.xml change and a rolling restart. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-17543) Create additional ReplicationEndpoint WALEntryFilters by configuration
[ https://issues.apache.org/jira/browse/HBASE-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Jacoby updated HBASE-17543: Attachment: HBASE-17543.v3.patch Fixed the import formatting in ReplicationManager requested by [~te...@apache.org]. I'll also backport to branch-1 and post soon > Create additional ReplicationEndpoint WALEntryFilters by configuration > -- > > Key: HBASE-17543 > URL: https://issues.apache.org/jira/browse/HBASE-17543 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby > Attachments: HBASE-17543.patch, HBASE-17543.v2.patch, > HBASE-17543.v3.patch > > > The existing BaseReplicationEndpoint creates a ChainWALEntryFilter containing > a NamespaceTableCfWALEntryFilter and a ScopeWALEntryFilter. Adding a custom > WALEntryFilter type to this chain requires creating an entirely new > ReplicationEndpoint subclass and creating a new peer on the running cluster, > which can be operationally complex to transition to without data loss in > cases such as master/master. > For WALEntryFilters without constructor parameters, it would be > straightforward to have a Configuration option to list additional > WALEntryFilter classes the operator wants to include in the filter chain in > the default endpoint, and then have the endpoint instantiate the filters via > reflection. Then filter logic could be added (or removed) with only a > hbase-site.xml change and a rolling restart. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-17101) FavoredNodes should not apply to system tables
[ https://issues.apache.org/jira/browse/HBASE-17101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847289#comment-15847289 ] Francis Liu commented on HBASE-17101: - +1 > FavoredNodes should not apply to system tables > -- > > Key: HBASE-17101 > URL: https://issues.apache.org/jira/browse/HBASE-17101 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-17101.master.001.patch, > HBASE-17101.master.002.patch, HBASE-17101.master.003.patch, > HBASE-17101.master.004.patch, HBASE_17101_rough_draft.patch > > > As described in the doc (see HBASE-15531), we would like to start with user > tables for favored nodes. This task ensures FN does not apply to system > tables. > System tables are in memory and won't benefit from favored nodes. Since we > also maintain FN information for user regions in meta, it helps to keep > implementation simpler by ignoring system tables for the first iterations. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-16621) HBCK should have -fixHFileLinks
[
https://issues.apache.org/jira/browse/HBASE-16621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847281#comment-15847281
]
Hadoop QA commented on HBASE-16621:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 28s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
37m 1s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 100m 34s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 153m 2s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12850176/HBASE-16621-3.patch |
| JIRA Issue | HBASE-16621 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 4054b15131fc 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6791828 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5518/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5518/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> HBCK should have -fixHFileLinks
> ---
>
> Key: HBASE-16621
> URL: https://issues.apache.org/jira/browse/HBASE-16621
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Janos Gub
> Labels: beginner, operability
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16621-2.patch, HBASE-16621-3.patch,
> HBASE-16621.patch, refactor-2.patch
>
[jira] (HBASE-17565) StochasticLoadBalancer may incorrectly skip balancing due to skewed multiplier sum
[
https://issues.apache.org/jira/browse/HBASE-17565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847278#comment-15847278
]
stack commented on HBASE-17565:
---
bq. I wasn't aware of Guanghao's fix over HBASE-17261 until you mentioned
HBASE-17261.
It happens. Perhaps do a search first next time.
bq. I left some comment at the end of HBASE-17261 w.r.t. his fix.
Thanks.
> StochasticLoadBalancer may incorrectly skip balancing due to skewed
> multiplier sum
> --
>
> Key: HBASE-17565
> URL: https://issues.apache.org/jira/browse/HBASE-17565
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 17565.v1.txt, 17565.v2.txt
>
>
> I was investigating why a 6 node cluster kept skipping balancing requests.
> Here were the region counts on the servers:
> 449, 448, 447, 449, 453, 0
> {code}
> 2017-01-26 22:04:47,145 INFO
> [RpcServer.deafult.FPBQ.Fifo.handler=1,queue=0,port=16000]
> balancer.StochasticLoadBalancer: Skipping load balancing because balanced
> cluster; total cost is 127.0171157050385, sum multiplier is 111087.0 min cost
> which need balance is 0.05
> {code}
> The big multiplier sum caught my eyes. Here was what additional debug logging
> showed:
> {code}
> 2017-01-27 23:25:31,749 DEBUG
> [RpcServer.deafult.FPBQ.Fifo.handler=9,queue=0,port=16000]
> balancer.StochasticLoadBalancer: class
> org.apache.hadoop.hbase.master.balancer.
> StochasticLoadBalancer$RegionReplicaHostCostFunction with multiplier 10.0
> 2017-01-27 23:25:31,749 DEBUG
> [RpcServer.deafult.FPBQ.Fifo.handler=9,queue=0,port=16000]
> balancer.StochasticLoadBalancer: class
> org.apache.hadoop.hbase.master.balancer.
> StochasticLoadBalancer$RegionReplicaRackCostFunction with multiplier 1.0
> {code}
> Note however, that no table in the cluster used read replica.
> I can think of two ways of fixing this situation:
> 1. If there is no read replica in the cluster, ignore the multipliers for the
> above two functions.
> 2. When cost() returned by the CostFunction is 0 (or very very close to 0.0),
> ignore the multiplier.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-17565) StochasticLoadBalancer may incorrectly skip balancing due to skewed multiplier sum
[
https://issues.apache.org/jira/browse/HBASE-17565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847273#comment-15847273
]
Ted Yu commented on HBASE-17565:
I wasn't aware of Guanghao's fix over HBASE-17261 until you mentioned
HBASE-17261.
I left some comment at the end of HBASE-17261 w.r.t. his fix.
> StochasticLoadBalancer may incorrectly skip balancing due to skewed
> multiplier sum
> --
>
> Key: HBASE-17565
> URL: https://issues.apache.org/jira/browse/HBASE-17565
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 17565.v1.txt, 17565.v2.txt
>
>
> I was investigating why a 6 node cluster kept skipping balancing requests.
> Here were the region counts on the servers:
> 449, 448, 447, 449, 453, 0
> {code}
> 2017-01-26 22:04:47,145 INFO
> [RpcServer.deafult.FPBQ.Fifo.handler=1,queue=0,port=16000]
> balancer.StochasticLoadBalancer: Skipping load balancing because balanced
> cluster; total cost is 127.0171157050385, sum multiplier is 111087.0 min cost
> which need balance is 0.05
> {code}
> The big multiplier sum caught my eyes. Here was what additional debug logging
> showed:
> {code}
> 2017-01-27 23:25:31,749 DEBUG
> [RpcServer.deafult.FPBQ.Fifo.handler=9,queue=0,port=16000]
> balancer.StochasticLoadBalancer: class
> org.apache.hadoop.hbase.master.balancer.
> StochasticLoadBalancer$RegionReplicaHostCostFunction with multiplier 10.0
> 2017-01-27 23:25:31,749 DEBUG
> [RpcServer.deafult.FPBQ.Fifo.handler=9,queue=0,port=16000]
> balancer.StochasticLoadBalancer: class
> org.apache.hadoop.hbase.master.balancer.
> StochasticLoadBalancer$RegionReplicaRackCostFunction with multiplier 1.0
> {code}
> Note however, that no table in the cluster used read replica.
> I can think of two ways of fixing this situation:
> 1. If there is no read replica in the cluster, ignore the multipliers for the
> above two functions.
> 2. When cost() returned by the CostFunction is 0 (or very very close to 0.0),
> ignore the multiplier.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-17261) Balancer makes no sense on tip of branch-1: says balanced when not
[
https://issues.apache.org/jira/browse/HBASE-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847271#comment-15847271
]
Ted Yu commented on HBASE-17261:
[~zghaobac]:
I wasn't aware of your patch until Stack mentioned this JIRA over HBASE-17565
where I have a patch
(https://issues.apache.org/jira/secure/attachment/12849799/17565.v2.txt) which
fixes the same problem.
Your patch doesn't really fix the problem:
In StochasticLoadBalancer#setConf():
{code}
minCostNeedBalance = conf.getFloat(MIN_COST_NEED_BALANCE_KEY,
minCostNeedBalance);
{code}
This means if user sets non-zero value for
"hbase.master.balancer.stochastic.minCostNeedBalance", the change is not
effective.
If default value is used:
{code}
protected boolean needsBalance(Cluster cluster) {
+if (minCostNeedBalance <= 0) {
+ return super.needsBalance(cluster);
{code}
The addition from HBASE-15529 would be skipped. I don't think that's what you
wanted.
What do you think ?
> Balancer makes no sense on tip of branch-1: says balanced when not
> --
>
> Key: HBASE-17261
> URL: https://issues.apache.org/jira/browse/HBASE-17261
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: Guanghao Zhang
> Attachments: HBASE-17261.patch
>
>
> Running ITBLL on tip of branch-1, I see this in log when I try to balance:
> {code}
> 2016-12-05 16:42:21,031 INFO
> [RpcServer.deafult.FPBQ.Fifo.handler=46,queue=1,port=16000]
> balancer.StochasticLoadBalancer: Skipping load balancing because balanced
> cluster; total cost is 525.2547686174673|
> , sum multiplier is 111087.0 min cost which need balance is 0.05
> {code}
> Its some old nonsense.
> Does this every time I balance. Can't even force a balance.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-17565) StochasticLoadBalancer may incorrectly skip balancing due to skewed multiplier sum
[
https://issues.apache.org/jira/browse/HBASE-17565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847258#comment-15847258
]
stack commented on HBASE-17565:
---
[~tedyu] Quit this forking of development/attention and duplication of effort.
HBase development is a collaborative affair. It is not a competition. If you
have problem or contrib to the patch up on HBASE-17261, please add comment
there. Your actions here only baffle the contributor as they ask themselves
what was wrong w/ their patch and why their effort cast aside. HBASE-17261 has
precedent, has a tie to the issue that introduced the bug, and it has a patch
that seems to address the issue. All that was missing was exercise of the
contributors posted patch. The decent thing would be resolve of this item as a
duplicate and help landing a fix, preferably by a new contributor.
> StochasticLoadBalancer may incorrectly skip balancing due to skewed
> multiplier sum
> --
>
> Key: HBASE-17565
> URL: https://issues.apache.org/jira/browse/HBASE-17565
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 17565.v1.txt, 17565.v2.txt
>
>
> I was investigating why a 6 node cluster kept skipping balancing requests.
> Here were the region counts on the servers:
> 449, 448, 447, 449, 453, 0
> {code}
> 2017-01-26 22:04:47,145 INFO
> [RpcServer.deafult.FPBQ.Fifo.handler=1,queue=0,port=16000]
> balancer.StochasticLoadBalancer: Skipping load balancing because balanced
> cluster; total cost is 127.0171157050385, sum multiplier is 111087.0 min cost
> which need balance is 0.05
> {code}
> The big multiplier sum caught my eyes. Here was what additional debug logging
> showed:
> {code}
> 2017-01-27 23:25:31,749 DEBUG
> [RpcServer.deafult.FPBQ.Fifo.handler=9,queue=0,port=16000]
> balancer.StochasticLoadBalancer: class
> org.apache.hadoop.hbase.master.balancer.
> StochasticLoadBalancer$RegionReplicaHostCostFunction with multiplier 10.0
> 2017-01-27 23:25:31,749 DEBUG
> [RpcServer.deafult.FPBQ.Fifo.handler=9,queue=0,port=16000]
> balancer.StochasticLoadBalancer: class
> org.apache.hadoop.hbase.master.balancer.
> StochasticLoadBalancer$RegionReplicaRackCostFunction with multiplier 1.0
> {code}
> Note however, that no table in the cluster used read replica.
> I can think of two ways of fixing this situation:
> 1. If there is no read replica in the cluster, ignore the multipliers for the
> above two functions.
> 2. When cost() returned by the CostFunction is 0 (or very very close to 0.0),
> ignore the multiplier.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-17508) Unify the implementation of small scan and regular scan for sync client
[
https://issues.apache.org/jira/browse/HBASE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847257#comment-15847257
]
Hadoop QA commented on HBASE-17508:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 14 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 43s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 8s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 4s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 127m 1s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestZKAsyncRegistry |
| | hadoop.hbase.client.TestMobRestoreSnapshotFromClient |
| | hadoop.hbase.mapreduce.TestSyncTable |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12850218/HBASE-17508-v2.patch |
| JIRA Issue | HBASE-17508 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 452820f73949 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6791828 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5519/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
[jira] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847234#comment-15847234 ] stack commented on HBASE-17437: --- Skimmed the patch up in RB. +1. [~enis] you want to commit? You did most review sir (I can do it for you if you busy). Needs a release note [~zyork]. > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Zach York > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, > HBASE-17437.master.001.patch, HBASE-17437.master.002.patch, > HBASE-17437.master.003.patch, HBASE-17437.master.004.patch, > HBASE-17437.master.005.patch, HBASE-17437.master.006.patch, > HBASE-17437.master.007.patch, HBASE-17437.master.008.patch, > HBASE-17437.master.009.patch, HBASE-17437.master.010.patch, > HBASE-17437.master.011.patch, HBASE-17437.master.012.patch, > hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly and beyond
[ https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847226#comment-15847226 ] huaxiang sun commented on HBASE-16981: -- HI [~jingcheng.du], I rebased the patch and attached is v7 patch, thanks! > Expand Mob Compaction Partition policy from daily to weekly, monthly and > beyond > --- > > Key: HBASE-16981 > URL: https://issues.apache.org/jira/browse/HBASE-16981 > Project: HBase > Issue Type: New Feature > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-16981.master.001.patch, > HBASE-16981.master.002.patch, HBASE-16981.master.003.patch, > HBASE-16981.master.004.patch, HBASE-16981.master.005.patch, > HBASE-16981.master.006.patch, HBASE-16981.master.007.patch, > Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf > > > Today the mob region holds all mob files for all regions. With daily > partition mob compaction policy, after major mob compaction, there is still > one file per region daily. Given there is 365 days in one year, at least 365 > files per region. Since HDFS has limitation for number of files under one > folder, this is not going to scale if there are lots of regions. To reduce > mob file number, we want to introduce other partition policies such as > weekly, monthly to compact mob files within one week or month into one file. > This jira is create to track this effort. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly and beyond
[ https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-16981: - Attachment: HBASE-16981.master.007.patch > Expand Mob Compaction Partition policy from daily to weekly, monthly and > beyond > --- > > Key: HBASE-16981 > URL: https://issues.apache.org/jira/browse/HBASE-16981 > Project: HBase > Issue Type: New Feature > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-16981.master.001.patch, > HBASE-16981.master.002.patch, HBASE-16981.master.003.patch, > HBASE-16981.master.004.patch, HBASE-16981.master.005.patch, > HBASE-16981.master.006.patch, HBASE-16981.master.007.patch, > Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf > > > Today the mob region holds all mob files for all regions. With daily > partition mob compaction policy, after major mob compaction, there is still > one file per region daily. Given there is 365 days in one year, at least 365 > files per region. Since HDFS has limitation for number of files under one > folder, this is not going to scale if there are lots of regions. To reduce > mob file number, we want to introduce other partition policies such as > weekly, monthly to compact mob files within one week or month into one file. > This jira is create to track this effort. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-17569) HBase-Procedure module need to support mvn clean test -PskipProcedureTests to skip unit test
[
https://issues.apache.org/jira/browse/HBASE-17569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847214#comment-15847214
]
Yi Liang commented on HBASE-17569:
--
Thanks [~stack] for reviewing, will provide a addendum to cover all other
modules that need this fix.
> HBase-Procedure module need to support mvn clean test -PskipProcedureTests to
> skip unit test
>
>
> Key: HBASE-17569
> URL: https://issues.apache.org/jira/browse/HBASE-17569
> Project: HBase
> Issue Type: Improvement
> Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBase-17569-V1.patch
>
>
> From Reference guide, we know that To skip the tests in the hbase-server
> module, you would run:
> {code}
> mvn clean test -PskipServerTests
> {code}
> We can also support this command in hbase-procedure module.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-17280) Add mechanism to control hbase cleaner behavior
[ https://issues.apache.org/jira/browse/HBASE-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Jadhav updated HBASE-17280: Status: Patch Available (was: Open) > Add mechanism to control hbase cleaner behavior > --- > > Key: HBASE-17280 > URL: https://issues.apache.org/jira/browse/HBASE-17280 > Project: HBase > Issue Type: Improvement > Components: Client, hbase, shell >Affects Versions: 1.2.0, 2.0.0 >Reporter: Ajay Jadhav >Priority: Minor > Fix For: 2.0.0, 1.2.0 > > Attachments: HBASE-17280.branch-1.2.patch, > HBASE-17280.branch-2.0.patch, HBASE-17280.master.003.patch, > HBASE-17280.master.004.patch, HBASE-17280.v1-branch-1.2.patch, > HBASE-17280.v2-branch-1.2.patch, HBASE-17280.v2-branch-2.patch > > > Cleaner is used to get rid of archived HFiles and old WALs in HBase. > In the case of heavy workload, cleaner can affect query performance by > creating a lot of connections to perform costly reads/ writes against > underlying filesystem. > This patch allows the user to control HBase cleaner behavior by providing > shell commands to enable/ disable and manually run it. > Our main intention with this patch was to avoid running the expensive cleaner > chore during peak times. During our experimentation, we saw a lot of HFiles > and WAL log related files getting created inside archive dir (didn't see > ZKlock related files). Since we were replacing hdfs with S3, these delete > calls will take forever to complete. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
