[jira] [Commented] (HBASE-16874) Potential NPE from ProcedureExecutor#stop()
[
https://issues.apache.org/jira/browse/HBASE-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586789#comment-15586789
]
Ted Yu commented on HBASE-16874:
Which test covers multi-executor thread case ?
> Potential NPE from ProcedureExecutor#stop()
> ---
>
> Key: HBASE-16874
> URL: https://issues.apache.org/jira/browse/HBASE-16874
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Reporter: Ted Yu
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: 16874.v1.txt, HBASE-16874-v0.patch
>
>
> When examining failed test :
> https://builds.apache.org/job/HBase-TRUNK_matrix/lastCompletedBuild/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.master.procedure/TestMasterFailoverWithProcedures/org_apache_hadoop_hbase_master_procedure_TestMasterFailoverWithProcedures/
> I noticed the following:
> {code}
> 2016-10-18 18:47:39,313 INFO [Time-limited test]
> procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state:
> TRUNCATE_TABLE_CLEAR_FS_LAYOUT
> Exception in thread "ProcedureExecutorWorker-1" java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.stop(ProcedureExecutor.java:533)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1197)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:959)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$700(ProcedureExecutor.java:73)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1405)
> {code}
> This seems to be the result of race between stop() and join() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16845) Run tests under hbase-spark module in test phase
[ https://issues.apache.org/jira/browse/HBASE-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587014#comment-15587014 ] Ted Yu commented on HBASE-16845: Is skipITs shorthand for skipIntegrationTests ? I didn't find skipITs in code base. Support for skipIntegrationTests is added in patch v2. > Run tests under hbase-spark module in test phase > > > Key: HBASE-16845 > URL: https://issues.apache.org/jira/browse/HBASE-16845 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16845.v1.txt, 16845.v2.txt > > > [~appy] reported over in the discussion thread titled 'Testing and CI' that > tests under hbase-spark module are not run by QA bot. > This issue is to run the unit tests in test phase so that we detect build > breakage early. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16827) Backup.proto refactoring
[ https://issues.apache.org/jira/browse/HBASE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587119#comment-15587119 ] Ted Yu commented on HBASE-16827: Have you run all backup / restore tests ? > Backup.proto refactoring > > > Key: HBASE-16827 > URL: https://issues.apache.org/jira/browse/HBASE-16827 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16827-v1.patch > > > Make server_name type ServerName (from HBase.proto) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16845) Run tests under hbase-spark module in test phase
[ https://issues.apache.org/jira/browse/HBASE-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587406#comment-15587406 ] Ted Yu commented on HBASE-16845: [~apurtell]: Can you give the full command involving -DskipITs ? > Run tests under hbase-spark module in test phase > > > Key: HBASE-16845 > URL: https://issues.apache.org/jira/browse/HBASE-16845 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16845.v1.txt, 16845.v2.txt > > > [~appy] reported over in the discussion thread titled 'Testing and CI' that > tests under hbase-spark module are not run by QA bot. > This issue is to run the unit tests in test phase so that we detect build > breakage early. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15589109#comment-15589109 ] Ted Yu commented on HBASE-14417: Discussed with Enis. The persistence of bulk loaded hfiles should be synchronous with the bulk load. Writing to hbase:backup table through postAppend() hook is tricky. The precedent of HBASE-13153 can be used for the persistence of reference to bulk loaded hfiles. > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417.v1.txt, 14417.v11.txt, 14417.v2.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14417: --- Attachment: 14417.v11.txt Patch v11 rebased on HBASE-7912 branch. > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417.v1.txt, 14417.v11.txt, 14417.v2.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16827) Backup.proto refactoring
[ https://issues.apache.org/jira/browse/HBASE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15589311#comment-15589311 ] Ted Yu commented on HBASE-16827: [~stack]: Is there other change you want to see for Backup.proto ? > Backup.proto refactoring > > > Key: HBASE-16827 > URL: https://issues.apache.org/jira/browse/HBASE-16827 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16827-v1.patch > > > Make server_name type ServerName (from HBase.proto) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16828) Make backup command-line tool interface consistent
[
https://issues.apache.org/jira/browse/HBASE-16828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15589588#comment-15589588
]
Ted Yu commented on HBASE-16828:
Vlad:
Mind rebasing the patch ?
> Make backup command-line tool interface consistent
> --
>
> Key: HBASE-16828
> URL: https://issues.apache.org/jira/browse/HBASE-16828
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-16828-v1.patch, HBASE-16828-v2.patch,
> HBASE-16828-v3.patch, HBASE-16828-v4.patch
>
>
> From Stacks review:
> {quote}
> This is interesting... if I pass a set and a table as command lets me do, it
> runs the backup twice. I was sort of expecting one backup dir with all
> content in it. No biggie. Here is the command I ran (the -set included the
> table so the tool behavior avoids backup overwrite).
> ./hbase/bin/hbase --config ~/conf_hbase backup create full
> hdfs://ve0524.halxg.cloudera.com:8020/backup clicks -set s
> {quote}
> {quote}
> The command I ran was $ ./hbase/bin/hbase --config ~/conf_hbase backup create
> full clicks I'm missing the backup location which probably explains the
> above. I'd think the tool would verify the command before going ahead and
> running a mapreduce job?
> {quote}
> {quote}
> stack@ve0524:~$ ./hbase/bin/hbase --config ~/conf_hbase backup create
> ... does the tables 'option' not take a hyphen but all others do? Most of the
> time options are delimited by space but not always
> {quote}
> {quote}
> When required args are not supplied, dumping help/usage rather than ERROR is
> usual and user friendly.
> {quote}
> {quote}
> What about consistency in errors, that sometimes commands are single letter,
> other times words, that descriptions in usage are not what is on the Option,
> that we do Options but don't use it to do Usage/help, and others.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16874) Potential NPE from ProcedureExecutor#stop()
[
https://issues.apache.org/jira/browse/HBASE-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15589633#comment-15589633
]
Ted Yu commented on HBASE-16874:
lgtm
> Potential NPE from ProcedureExecutor#stop()
> ---
>
> Key: HBASE-16874
> URL: https://issues.apache.org/jira/browse/HBASE-16874
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Reporter: Ted Yu
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: 16874.v1.txt, HBASE-16874-v0.patch, HBASE-16874-v1.patch
>
>
> When examining failed test :
> https://builds.apache.org/job/HBase-TRUNK_matrix/lastCompletedBuild/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.master.procedure/TestMasterFailoverWithProcedures/org_apache_hadoop_hbase_master_procedure_TestMasterFailoverWithProcedures/
> I noticed the following:
> {code}
> 2016-10-18 18:47:39,313 INFO [Time-limited test]
> procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state:
> TRUNCATE_TABLE_CLEAR_FS_LAYOUT
> Exception in thread "ProcedureExecutorWorker-1" java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.stop(ProcedureExecutor.java:533)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1197)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:959)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$700(ProcedureExecutor.java:73)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1405)
> {code}
> This seems to be the result of race between stop() and join() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16870) Add the metrics of replication sources which were transformed from other died rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15589643#comment-15589643
]
Ted Yu commented on HBASE-16870:
lgtm, pending QA.
> Add the metrics of replication sources which were transformed from other died
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16881) Address findbugs warnings
Ted Yu created HBASE-16881: -- Summary: Address findbugs warnings Key: HBASE-16881 URL: https://issues.apache.org/jira/browse/HBASE-16881 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: Ted Yu Multiple findbugs warnings were raised for mega patch v29. Largely this was due to fixes for findbugs warnings in mega patch v28 were dropped. This issue addresses the findbugs warnings for mega patch v29. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16881) Address findbugs warnings
[ https://issues.apache.org/jira/browse/HBASE-16881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16881: --- Attachment: 16881.v1.txt > Address findbugs warnings > - > > Key: HBASE-16881 > URL: https://issues.apache.org/jira/browse/HBASE-16881 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Labels: backup > Fix For: HBASE-7912 > > Attachments: 16881.v1.txt > > > Multiple findbugs warnings were raised for mega patch v29. > Largely this was due to fixes for findbugs warnings in mega patch v28 were > dropped. > This issue addresses the findbugs warnings for mega patch v29. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16881) Address findbugs warnings
[ https://issues.apache.org/jira/browse/HBASE-16881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16881: --- Attachment: (was: 16881.v1.txt) > Address findbugs warnings > - > > Key: HBASE-16881 > URL: https://issues.apache.org/jira/browse/HBASE-16881 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Labels: backup > Fix For: HBASE-7912 > > > Multiple findbugs warnings were raised for mega patch v29. > Largely this was due to fixes for findbugs warnings in mega patch v28 were > dropped. > This issue addresses the findbugs warnings for mega patch v29. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16881) Address findbugs warnings
[ https://issues.apache.org/jira/browse/HBASE-16881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16881: --- Attachment: 16881.v1.txt > Address findbugs warnings > - > > Key: HBASE-16881 > URL: https://issues.apache.org/jira/browse/HBASE-16881 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Labels: backup > Fix For: HBASE-7912 > > Attachments: 16881.v1.txt > > > Multiple findbugs warnings were raised for mega patch v29. > Largely this was due to fixes for findbugs warnings in mega patch v28 were > dropped. > This issue addresses the findbugs warnings for mega patch v29. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16881) Address findbugs warnings
[
https://issues.apache.org/jira/browse/HBASE-16881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15589780#comment-15589780
]
Ted Yu commented on HBASE-16881:
All backup tests passed.
{code}
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 28:56 min
[INFO] Finished at: 2016-10-19T20:32:47+00:00
{code}
> Address findbugs warnings
> -
>
> Key: HBASE-16881
> URL: https://issues.apache.org/jira/browse/HBASE-16881
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: backup
> Fix For: HBASE-7912
>
> Attachments: 16881.v1.txt
>
>
> Multiple findbugs warnings were raised for mega patch v29.
> Largely this was due to fixes for findbugs warnings in mega patch v28 were
> dropped.
> This issue addresses the findbugs warnings for mega patch v29.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-16881) Address findbugs warnings for backup mega patch
[ https://issues.apache.org/jira/browse/HBASE-16881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-16881. Resolution: Fixed Hadoop Flags: Reviewed Thanks, Vlad. > Address findbugs warnings for backup mega patch > --- > > Key: HBASE-16881 > URL: https://issues.apache.org/jira/browse/HBASE-16881 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Labels: backup > Fix For: HBASE-7912 > > Attachments: 16881.v1.txt > > > Multiple findbugs warnings were raised for mega patch v29. > Largely this was due to fixes for findbugs warnings in mega patch v28 were > dropped. > This issue addresses the findbugs warnings for mega patch v29. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16881) Address findbugs warnings for backup mega patch
[ https://issues.apache.org/jira/browse/HBASE-16881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16881: --- Summary: Address findbugs warnings for backup mega patch (was: Address findbugs warnings) > Address findbugs warnings for backup mega patch > --- > > Key: HBASE-16881 > URL: https://issues.apache.org/jira/browse/HBASE-16881 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Labels: backup > Fix For: HBASE-7912 > > Attachments: 16881.v1.txt > > > Multiple findbugs warnings were raised for mega patch v29. > Largely this was due to fixes for findbugs warnings in mega patch v28 were > dropped. > This issue addresses the findbugs warnings for mega patch v29. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16870:
---
Summary: Add the metrics of replication sources which were transformed from
other dead rs to ReplicationLoad (was: Add the metrics of replication sources
which were transformed from other died rs to ReplicationLoad)
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16870:
---
Hadoop Flags: Reviewed
Fix Version/s: 2.0.0
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16883) Fix javadoc warnings for mega patch
Ted Yu created HBASE-16883: -- Summary: Fix javadoc warnings for mega patch Key: HBASE-16883 URL: https://issues.apache.org/jira/browse/HBASE-16883 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: Ted Yu Priority: Minor There were 14 javadoc warnings for mega patch v29. This task fixes the warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16883) Fix javadoc warnings for mega patch
[ https://issues.apache.org/jira/browse/HBASE-16883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16883: --- Attachment: 16883.v1.txt > Fix javadoc warnings for mega patch > --- > > Key: HBASE-16883 > URL: https://issues.apache.org/jira/browse/HBASE-16883 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Labels: backup > Fix For: HBASE-7912 > > Attachments: 16883.v1.txt > > > There were 14 javadoc warnings for mega patch v29. > This task fixes the warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15590277#comment-15590277
]
Ted Yu commented on HBASE-16870:
Though the patch applies on branch-1, TestReplicationSmallTests keeps failing.
Guanghao
Can you check ?
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16885) Deprecate public classes which are removed in 2.0
Ted Yu created HBASE-16885: -- Summary: Deprecate public classes which are removed in 2.0 Key: HBASE-16885 URL: https://issues.apache.org/jira/browse/HBASE-16885 Project: HBase Issue Type: Bug Reporter: Ted Yu I tried to compile master branch of Phoenix against 2.0-SNAPSHOT and observed some compilation errors. /phoenix/phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/LocalIndexStoreFileScanner.java:[31,54] cannot find symbol symbol: class Reader location: class org.apache.hadoop.hbase.regionserver.StoreFile /phoenix/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/MultiHfileOutputFormat.java:[300,16] cannot find symbol symbol: class Writer location: class org.apache.hadoop.hbase.regionserver.StoreFile The above inner classes of StoreFile are marked public as of branch-1. We should deprecate them in 1.x release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16885) Deprecate public classes which are removed in 2.0
[ https://issues.apache.org/jira/browse/HBASE-16885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16885: --- Attachment: phoenix-compile-against-2.0.txt File containing compilation errors. > Deprecate public classes which are removed in 2.0 > - > > Key: HBASE-16885 > URL: https://issues.apache.org/jira/browse/HBASE-16885 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > Attachments: phoenix-compile-against-2.0.txt > > > I tried to compile master branch of Phoenix against 2.0-SNAPSHOT and observed > some compilation errors. > /phoenix/phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/LocalIndexStoreFileScanner.java:[31,54] > cannot find symbol > symbol: class Reader > location: class org.apache.hadoop.hbase.regionserver.StoreFile > /phoenix/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/MultiHfileOutputFormat.java:[300,16] > cannot find symbol > symbol: class Writer > location: class org.apache.hadoop.hbase.regionserver.StoreFile > The above inner classes of StoreFile are marked public as of branch-1. > We should deprecate them in 1.x release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16845) Run tests under hbase-spark module in test phase
[
https://issues.apache.org/jira/browse/HBASE-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15590640#comment-15590640
]
Ted Yu commented on HBASE-16845:
Looking at http://hbase.apache.org/book.html#integration.tests , I only see
{{-PskipIntegrationTests}} being described.
> Run tests under hbase-spark module in test phase
>
>
> Key: HBASE-16845
> URL: https://issues.apache.org/jira/browse/HBASE-16845
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16845.v1.txt, 16845.v2.txt
>
>
> [~appy] reported over in the discussion thread titled 'Testing and CI' that
> tests under hbase-spark module are not run by QA bot.
> This issue is to run the unit tests in test phase so that we detect build
> breakage early.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16885) Deprecate public classes which are removed in 2.0
[
https://issues.apache.org/jira/browse/HBASE-16885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15590716#comment-15590716
]
Ted Yu commented on HBASE-16885:
StoreFile is @InterfaceAudience.LimitedPrivate("Coprocessor")
> Deprecate public classes which are removed in 2.0
> -
>
> Key: HBASE-16885
> URL: https://issues.apache.org/jira/browse/HBASE-16885
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
> Attachments: phoenix-compile-against-2.0.txt
>
>
> I tried to compile master branch of Phoenix against 2.0-SNAPSHOT and observed
> some compilation errors.
> /phoenix/phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/LocalIndexStoreFileScanner.java:[31,54]
> cannot find symbol
> symbol: class Reader
> location: class org.apache.hadoop.hbase.regionserver.StoreFile
> /phoenix/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/MultiHfileOutputFormat.java:[300,16]
> cannot find symbol
> symbol: class Writer
> location: class org.apache.hadoop.hbase.regionserver.StoreFile
> The above inner classes of StoreFile are marked public as of branch-1.
> We should deprecate them in 1.x release.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16827) Backup.proto refactoring
[
https://issues.apache.org/jira/browse/HBASE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15590727#comment-15590727
]
Ted Yu commented on HBASE-16827:
{code}
message BackupManifest {
- required string version = 1;
- required string backup_id = 2;
- required BackupType type = 3;
+ //optional string version = 1;
+ //optional string backup_id = 2;
+ //optional BackupType type = 3;
{code}
Please drop commented out fields.
> Backup.proto refactoring
>
>
> Key: HBASE-16827
> URL: https://issues.apache.org/jira/browse/HBASE-16827
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-16827-v1.patch, HBASE-16827-v2.patch
>
>
> Make server_name type ServerName (from HBase.proto)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15590758#comment-15590758
]
Ted Yu commented on HBASE-16870:
Add license header to TestReplicationStatus
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16870-ADDENDUM.patch,
> HBASE-16870-branch-1.1.patch, HBASE-16870-branch-1.3.patch,
> HBASE-16870-branch-1.patch, HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15463) Region normalizer should check whether split/merge is enabled
[ https://issues.apache.org/jira/browse/HBASE-15463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15463: --- Attachment: HBASE-15463.v2.patch > Region normalizer should check whether split/merge is enabled > - > > Key: HBASE-15463 > URL: https://issues.apache.org/jira/browse/HBASE-15463 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Attachments: HBASE-15463.v1.patch, HBASE-15463.v2.patch > > > HBASE-15128 added switch for disabling split / merge. > When split / merge switch is turned off, region normalizer should not perform > split / merge operation, respectively. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15463) Region normalizer should check whether split/merge is enabled
[ https://issues.apache.org/jira/browse/HBASE-15463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15463: --- Attachment: HBASE-15463.v3.patch Patch v3 addresses checkstyle warning. > Region normalizer should check whether split/merge is enabled > - > > Key: HBASE-15463 > URL: https://issues.apache.org/jira/browse/HBASE-15463 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Attachments: HBASE-15463.v1.patch, HBASE-15463.v2.patch, > HBASE-15463.v3.patch > > > HBASE-15128 added switch for disabling split / merge. > When split / merge switch is turned off, region normalizer should not perform > split / merge operation, respectively. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15463) Region normalizer should check whether split/merge is enabled
[ https://issues.apache.org/jira/browse/HBASE-15463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15463: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.4.0 1.3.0 2.0.0 Status: Resolved (was: Patch Available) Thanks for the review, Heng. > Region normalizer should check whether split/merge is enabled > - > > Key: HBASE-15463 > URL: https://issues.apache.org/jira/browse/HBASE-15463 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15463.v1.patch, HBASE-15463.v2.patch, > HBASE-15463.v3.patch > > > HBASE-15128 added switch for disabling split / merge. > When split / merge switch is turned off, region normalizer should not perform > split / merge operation, respectively. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15456) CreateTableProcedure/ModifyTableProcedure needs to fail when there is no family in descriptor
[
https://issues.apache.org/jira/browse/HBASE-15456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15195726#comment-15195726
]
Ted Yu commented on HBASE-15456:
lgtm
{code}
+ throw new DoNotRetryIOException("Table " +
htableDescriptor.getNameAsString() +
+ "should have at least one column family.");
{code}
nit: add space before "should"
> CreateTableProcedure/ModifyTableProcedure needs to fail when there is no
> family in descriptor
> -
>
> Key: HBASE-15456
> URL: https://issues.apache.org/jira/browse/HBASE-15456
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Attachments: HBASE-15456-v001.patch, HBASE-15456-v002.patch,
> HBASE-15456-v002.patch
>
>
> If there is only one family in the table, DeleteColumnFamilyProcedure will
> fail.
> Currently, when hbase.table.sanity.checks is set to false, hbase master logs
> a warning and CreateTableProcedure/ModifyTableProcedure will succeed.
> This behavior is not consistent with DeleteColumnFamilyProcedure's.
> Another point, before HBASE-13145, PeriodicMemstoreFlusher will run into the
> following exception. lastStoreFlushTimeMap is populated for families, if
> there is no family in the table, there is no entry in lastStoreFlushTimeMap.
> {code}
> 16/02/01 11:14:26 ERROR regionserver.HRegionServer$PeriodicMemstoreFlusher:
> Caught exception
> java.util.NoSuchElementException
> at
> java.util.concurrent.ConcurrentHashMap$HashIterator.nextEntry(ConcurrentHashMap.java:1354)
>
> at
> java.util.concurrent.ConcurrentHashMap$ValueIterator.next(ConcurrentHashMap.java:1384)
>
> at java.util.Collections.min(Collections.java:628)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getEarliestFlushTimeForAllStores(HRegion.java:1572)
>
> at
> org.apache.hadoop.hbase.regionserver.HRegion.shouldFlush(HRegion.java:1904)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$PeriodicMemstoreFlusher.chore(HRegionServer.java:1509)
>
> at org.apache.hadoop.hbase.Chore.run(Chore.java:87)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15334) Add avro support for spark hbase connector
[ https://issues.apache.org/jira/browse/HBASE-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15196215#comment-15196215 ] Ted Yu commented on HBASE-15334: +1 > Add avro support for spark hbase connector > -- > > Key: HBASE-15334 > URL: https://issues.apache.org/jira/browse/HBASE-15334 > Project: HBase > Issue Type: Sub-task >Reporter: Zhan Zhang >Assignee: Zhan Zhang > Attachments: HBASE-15334-1.patch, HBASE-15334-2.patch, > HBASE-15334-3.patch, HBASE-15334-4.patch, HBASE-15334-5.patch > > > Avro is a popular format for hbase storage. User may want the support > natively in the connector. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15456) CreateTableProcedure/ModifyTableProcedure needs to fail when there is no family in table descriptor
[
https://issues.apache.org/jira/browse/HBASE-15456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-15456:
---
Summary: CreateTableProcedure/ModifyTableProcedure needs to fail when there
is no family in table descriptor (was:
CreateTableProcedure/ModifyTableProcedure needs to fail when there is no family
in descriptor)
> CreateTableProcedure/ModifyTableProcedure needs to fail when there is no
> family in table descriptor
> ---
>
> Key: HBASE-15456
> URL: https://issues.apache.org/jira/browse/HBASE-15456
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-15456-v001.patch, HBASE-15456-v002.patch,
> HBASE-15456-v002.patch, HBASE-15456-v003.patch
>
>
> If there is only one family in the table, DeleteColumnFamilyProcedure will
> fail.
> Currently, when hbase.table.sanity.checks is set to false, hbase master logs
> a warning and CreateTableProcedure/ModifyTableProcedure will succeed.
> This behavior is not consistent with DeleteColumnFamilyProcedure's.
> Another point, before HBASE-13145, PeriodicMemstoreFlusher will run into the
> following exception. lastStoreFlushTimeMap is populated for families, if
> there is no family in the table, there is no entry in lastStoreFlushTimeMap.
> {code}
> 16/02/01 11:14:26 ERROR regionserver.HRegionServer$PeriodicMemstoreFlusher:
> Caught exception
> java.util.NoSuchElementException
> at
> java.util.concurrent.ConcurrentHashMap$HashIterator.nextEntry(ConcurrentHashMap.java:1354)
>
> at
> java.util.concurrent.ConcurrentHashMap$ValueIterator.next(ConcurrentHashMap.java:1384)
>
> at java.util.Collections.min(Collections.java:628)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getEarliestFlushTimeForAllStores(HRegion.java:1572)
>
> at
> org.apache.hadoop.hbase.regionserver.HRegion.shouldFlush(HRegion.java:1904)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$PeriodicMemstoreFlusher.chore(HRegionServer.java:1509)
>
> at org.apache.hadoop.hbase.Chore.run(Chore.java:87)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15456) CreateTableProcedure/ModifyTableProcedure needs to fail when there is no family in table descriptor
[
https://issues.apache.org/jira/browse/HBASE-15456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-15456:
---
Hadoop Flags: Reviewed
Fix Version/s: 1.4.0
> CreateTableProcedure/ModifyTableProcedure needs to fail when there is no
> family in table descriptor
> ---
>
> Key: HBASE-15456
> URL: https://issues.apache.org/jira/browse/HBASE-15456
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15456-v001.patch, HBASE-15456-v002.patch,
> HBASE-15456-v002.patch, HBASE-15456-v003.patch
>
>
> If there is only one family in the table, DeleteColumnFamilyProcedure will
> fail.
> Currently, when hbase.table.sanity.checks is set to false, hbase master logs
> a warning and CreateTableProcedure/ModifyTableProcedure will succeed.
> This behavior is not consistent with DeleteColumnFamilyProcedure's.
> Another point, before HBASE-13145, PeriodicMemstoreFlusher will run into the
> following exception. lastStoreFlushTimeMap is populated for families, if
> there is no family in the table, there is no entry in lastStoreFlushTimeMap.
> {code}
> 16/02/01 11:14:26 ERROR regionserver.HRegionServer$PeriodicMemstoreFlusher:
> Caught exception
> java.util.NoSuchElementException
> at
> java.util.concurrent.ConcurrentHashMap$HashIterator.nextEntry(ConcurrentHashMap.java:1354)
>
> at
> java.util.concurrent.ConcurrentHashMap$ValueIterator.next(ConcurrentHashMap.java:1384)
>
> at java.util.Collections.min(Collections.java:628)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getEarliestFlushTimeForAllStores(HRegion.java:1572)
>
> at
> org.apache.hadoop.hbase.regionserver.HRegion.shouldFlush(HRegion.java:1904)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$PeriodicMemstoreFlusher.chore(HRegionServer.java:1509)
>
> at org.apache.hadoop.hbase.Chore.run(Chore.java:87)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15456) CreateTableProcedure/ModifyTableProcedure needs to fail when there is no family in table descriptor
[
https://issues.apache.org/jira/browse/HBASE-15456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-15456:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for the patch, huaxiang
Thanks for the reviews.
> CreateTableProcedure/ModifyTableProcedure needs to fail when there is no
> family in table descriptor
> ---
>
> Key: HBASE-15456
> URL: https://issues.apache.org/jira/browse/HBASE-15456
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15456-v001.patch, HBASE-15456-v002.patch,
> HBASE-15456-v002.patch, HBASE-15456-v003.patch
>
>
> If there is only one family in the table, DeleteColumnFamilyProcedure will
> fail.
> Currently, when hbase.table.sanity.checks is set to false, hbase master logs
> a warning and CreateTableProcedure/ModifyTableProcedure will succeed.
> This behavior is not consistent with DeleteColumnFamilyProcedure's.
> Another point, before HBASE-13145, PeriodicMemstoreFlusher will run into the
> following exception. lastStoreFlushTimeMap is populated for families, if
> there is no family in the table, there is no entry in lastStoreFlushTimeMap.
> {code}
> 16/02/01 11:14:26 ERROR regionserver.HRegionServer$PeriodicMemstoreFlusher:
> Caught exception
> java.util.NoSuchElementException
> at
> java.util.concurrent.ConcurrentHashMap$HashIterator.nextEntry(ConcurrentHashMap.java:1354)
>
> at
> java.util.concurrent.ConcurrentHashMap$ValueIterator.next(ConcurrentHashMap.java:1384)
>
> at java.util.Collections.min(Collections.java:628)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getEarliestFlushTimeForAllStores(HRegion.java:1572)
>
> at
> org.apache.hadoop.hbase.regionserver.HRegion.shouldFlush(HRegion.java:1904)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$PeriodicMemstoreFlusher.chore(HRegionServer.java:1509)
>
> at org.apache.hadoop.hbase.Chore.run(Chore.java:87)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12940) Expose listPeerConfigs and getPeerConfig to the HBase shell
[
https://issues.apache.org/jira/browse/HBASE-12940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15196605#comment-15196605
]
Ted Yu commented on HBASE-12940:
{code}
174 def list_peer_configs
175 @replication_admin.list_peer_configs
176 end
177
178 def get_peer_config(id)
179 @replication_admin.get_peer_config(id)
{code}
Should @replication_admin.listPeerConfigs and
@replication_admin.getPeerConfig() be called, respectively ?
> Expose listPeerConfigs and getPeerConfig to the HBase shell
> ---
>
> Key: HBASE-12940
> URL: https://issues.apache.org/jira/browse/HBASE-12940
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Kevin Risden
>Assignee: Geoffrey Jacoby
> Attachments: HBASE-12940.patch
>
>
> In HBASE-12867 found that listPeerConfigs and getPeerConfig from
> ReplicationAdmin are not exposed to the HBase shell. This makes looking at
> details for custom replication endpoints and testing of add_peer from
> HBASE-12867 impossible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15187) Integrate CSRF prevention filter to REST gateway
[ https://issues.apache.org/jira/browse/HBASE-15187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15196613#comment-15196613 ] Ted Yu commented on HBASE-15187: [~apurtell]: Can you take a look ? > Integrate CSRF prevention filter to REST gateway > > > Key: HBASE-15187 > URL: https://issues.apache.org/jira/browse/HBASE-15187 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: HBASE-15187-branch-1.v13.patch, HBASE-15187.v1.patch, > HBASE-15187.v10.patch, HBASE-15187.v11.patch, HBASE-15187.v12.patch, > HBASE-15187.v13.patch, HBASE-15187.v2.patch, HBASE-15187.v3.patch, > HBASE-15187.v4.patch, HBASE-15187.v5.patch, HBASE-15187.v6.patch, > HBASE-15187.v7.patch, HBASE-15187.v8.patch, HBASE-15187.v9.patch > > > HADOOP-12691 introduced a filter in Hadoop Common to help REST APIs guard > against cross-site request forgery attacks. > This issue tracks the integration of that filter into HBase REST gateway. > From REST section of refguide: > To delete a table, use a DELETE request with the /schema endpoint: > http://example.com:8000/schema > Suppose an attacker hosts a malicious web form on a domain under his control. > The form uses the DELETE action targeting a REST URL. Through social > engineering, the attacker tricks an authenticated user into accessing the > form and submitting it. > The browser sends the HTTP DELETE request to the REST gateway. > At REST gateway, the call is executed and user table is dropped -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-15456) CreateTableProcedure/ModifyTableProcedure needs to fail when there is no family in table descriptor
[
https://issues.apache.org/jira/browse/HBASE-15456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reopened HBASE-15456:
> CreateTableProcedure/ModifyTableProcedure needs to fail when there is no
> family in table descriptor
> ---
>
> Key: HBASE-15456
> URL: https://issues.apache.org/jira/browse/HBASE-15456
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15456-v001.patch, HBASE-15456-v002.patch,
> HBASE-15456-v002.patch, HBASE-15456-v003.patch
>
>
> If there is only one family in the table, DeleteColumnFamilyProcedure will
> fail.
> Currently, when hbase.table.sanity.checks is set to false, hbase master logs
> a warning and CreateTableProcedure/ModifyTableProcedure will succeed.
> This behavior is not consistent with DeleteColumnFamilyProcedure's.
> Another point, before HBASE-13145, PeriodicMemstoreFlusher will run into the
> following exception. lastStoreFlushTimeMap is populated for families, if
> there is no family in the table, there is no entry in lastStoreFlushTimeMap.
> {code}
> 16/02/01 11:14:26 ERROR regionserver.HRegionServer$PeriodicMemstoreFlusher:
> Caught exception
> java.util.NoSuchElementException
> at
> java.util.concurrent.ConcurrentHashMap$HashIterator.nextEntry(ConcurrentHashMap.java:1354)
>
> at
> java.util.concurrent.ConcurrentHashMap$ValueIterator.next(ConcurrentHashMap.java:1384)
>
> at java.util.Collections.min(Collections.java:628)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getEarliestFlushTimeForAllStores(HRegion.java:1572)
>
> at
> org.apache.hadoop.hbase.regionserver.HRegion.shouldFlush(HRegion.java:1904)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$PeriodicMemstoreFlusher.chore(HRegionServer.java:1509)
>
> at org.apache.hadoop.hbase.Chore.run(Chore.java:87)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15456) CreateTableProcedure/ModifyTableProcedure needs to fail when there is no family in table descriptor
[
https://issues.apache.org/jira/browse/HBASE-15456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15196741#comment-15196741
]
Ted Yu commented on HBASE-15456:
[~huaxiang]:
Can you attach patch for branch-1 ?
Thanks
> CreateTableProcedure/ModifyTableProcedure needs to fail when there is no
> family in table descriptor
> ---
>
> Key: HBASE-15456
> URL: https://issues.apache.org/jira/browse/HBASE-15456
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15456-v001.patch, HBASE-15456-v002.patch,
> HBASE-15456-v002.patch, HBASE-15456-v003.patch
>
>
> If there is only one family in the table, DeleteColumnFamilyProcedure will
> fail.
> Currently, when hbase.table.sanity.checks is set to false, hbase master logs
> a warning and CreateTableProcedure/ModifyTableProcedure will succeed.
> This behavior is not consistent with DeleteColumnFamilyProcedure's.
> Another point, before HBASE-13145, PeriodicMemstoreFlusher will run into the
> following exception. lastStoreFlushTimeMap is populated for families, if
> there is no family in the table, there is no entry in lastStoreFlushTimeMap.
> {code}
> 16/02/01 11:14:26 ERROR regionserver.HRegionServer$PeriodicMemstoreFlusher:
> Caught exception
> java.util.NoSuchElementException
> at
> java.util.concurrent.ConcurrentHashMap$HashIterator.nextEntry(ConcurrentHashMap.java:1354)
>
> at
> java.util.concurrent.ConcurrentHashMap$ValueIterator.next(ConcurrentHashMap.java:1384)
>
> at java.util.Collections.min(Collections.java:628)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getEarliestFlushTimeForAllStores(HRegion.java:1572)
>
> at
> org.apache.hadoop.hbase.regionserver.HRegion.shouldFlush(HRegion.java:1904)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$PeriodicMemstoreFlusher.chore(HRegionServer.java:1509)
>
> at org.apache.hadoop.hbase.Chore.run(Chore.java:87)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15474) Exception in HConnectionImplementation's constructor cause Zookeeper connnections leak
[
https://issues.apache.org/jira/browse/HBASE-15474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15202341#comment-15202341
]
Ted Yu commented on HBASE-15474:
You can name your patch HBASE-15474-branch-1.v2.patch
> Exception in HConnectionImplementation's constructor cause Zookeeper
> connnections leak
> ---
>
> Key: HBASE-15474
> URL: https://issues.apache.org/jira/browse/HBASE-15474
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Allan Yang
>Assignee: Allan Yang
> Labels: patch
> Fix For: 1.1.0
>
> Attachments: HBASE-15474v2.patch
>
>
> HConnectionImplementation creates a ZooKeeperKeepAliveConnection during
> construction, but if the constructor throw a exception, the zookeeper
> connection is not properly closed.
> {code}
> HConnectionImplementation(Configuration conf, boolean managed,
> ExecutorService pool, User user) throws IOException {
> this(conf);
> this.user = user;
> this.batchPool = pool;
> this.managed = managed;
> this.registry = setupRegistry();
> retrieveClusterId(); //here is the zookeeper connection created
> this.rpcClient = RpcClientFactory.createClient(this.conf,
> this.clusterId);// In our case, the exception happens here, so the zookeeper
> connection never closes
> this.rpcControllerFactory = RpcControllerFactory.instantiate(conf);
> ..
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15488) Add ACL for setting split merge switch
[ https://issues.apache.org/jira/browse/HBASE-15488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15488: --- Fix Version/s: (was: 1.3.0) 1.4.0 Status: Patch Available (was: Open) > Add ACL for setting split merge switch > -- > > Key: HBASE-15488 > URL: https://issues.apache.org/jira/browse/HBASE-15488 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15488.v1.patch > > > Currently there is no access control for the split merge switch setter in > MasterRpcServices. > This JIRA adds necessary coprocessor hook along with enforcing permission > check in AccessController through the new hook. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201469#comment-15201469
]
Ted Yu commented on HBASE-15406:
{code}
4283 ZKUtil.createSetData(zkw, splitOrMergeLock,
4284ProtobufUtil.prependPBMagic(builder.build().toByteArray()));
{code}
How about calling createEphemeralNodeAndWatch() in the above ?
Protection against changing the switch when hbck is running can be achieved by
checking presence of the lock znode (not its contents).
If hbck dies unexpectedly, the lock znode would disappear.
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15406.patch, HBASE-15406.v1.patch,
> HBASE-15406_v1.patch, test.patch, wip.patch
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15488) Add ACL for setting split merge switch
[ https://issues.apache.org/jira/browse/HBASE-15488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15488: --- Attachment: HBASE-15488.v1.patch > Add ACL for setting split merge switch > -- > > Key: HBASE-15488 > URL: https://issues.apache.org/jira/browse/HBASE-15488 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15488.v1.patch > > > Currently there is no access control for the split merge switch setter in > MasterRpcServices. > This JIRA adds necessary coprocessor hook along with enforcing permission > check in AccessController through the new hook. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15479) No more garbage or beware of autoboxing
[
https://issues.apache.org/jira/browse/HBASE-15479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15202107#comment-15202107
]
Ted Yu edited comment on HBASE-15479 at 3/18/16 9:39 PM:
-
Not running hbase-server has hidden cost:
{code}
Author: Mikhail Antonov
Date: Thu Mar 17 01:06:42 2016 -0700
HBASE-15390 Unnecessary MetaCache evictions cause elevated number of
requests to meta (ADDENDUM)
{code}
Currently Yetus only runs unit tests in modules which are touched by the patch.
This may lead to regression some cases where bug was detected when running
hbase-server module tests.
was (Author: yuzhih...@gmail.com):
Not running hbase-server has hidden cost:
{code}
Author: Mikhail Antonov
Date: Thu Mar 17 01:06:42 2016 -0700
HBASE-15390 Unnecessary MetaCache evictions cause elevated number of
requests to meta (ADDENDUM)
{code}
> No more garbage or beware of autoboxing
> ---
>
> Key: HBASE-15479
> URL: https://issues.apache.org/jira/browse/HBASE-15479
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-15479-v1.patch, HBASE-15479-v2.patch
>
>
> Quick journey with JMC in a profile mode revealed very interesting and
> unexpected heap polluter on a client side. Patch will shortly follow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15302) Reenable the other tests disabled by HBASE-14678
[
https://issues.apache.org/jira/browse/HBASE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15202540#comment-15202540
]
Ted Yu commented on HBASE-15302:
{code}
Fetching
https://builds.apache.org/job/HBase-1.4/31/jdk=latest1.7,label=yahoo-not-h2/console
Printing hanging tests
Hanging test : org.apache.hadoop.hbase.io.encoding.TestEncodedSeekers
Hanging test :
org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures
Hanging test : org.apache.hadoop.hbase.wal.TestWALSplit
Hanging test : org.apache.hadoop.hbase.io.encoding.TestChangingEncoding
...
Fetching
https://builds.apache.org/job/HBase-1.4/31/jdk=latest1.8,label=yahoo-not-h2/consoleFull
Building remotely on H9
(Mapreduce Falcon Hadoop Pig Zookeeper Tez Hdfs yahoo-not-h2) in workspace
/home/jenkins/jenkins-slave/workspace/HBase-1.4/jdk/latest1.8/label/yahoo-not-h2
Printing hanging tests
Hanging test :
org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures
Printing Failing tests
Failing test :
org.apache.hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController
Failing test : org.apache.hadoop.hbase.client.TestAsyncProcess
{code}
Reverted TestMasterFailoverWithProcedures.java, TestWALSplit.java and
TestWALSplitCompressed
> Reenable the other tests disabled by HBASE-14678
>
>
> Key: HBASE-15302
> URL: https://issues.apache.org/jira/browse/HBASE-15302
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.3.0, 1.2.1
>
> Attachments: HBASE-15302-branch-1-v1.patch, HBASE-15302-v1.txt,
> HBASE-15302-v1.txt
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15488) Add ACL for setting split merge switch
[ https://issues.apache.org/jira/browse/HBASE-15488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15488: --- Attachment: (was: HBASE-15488.v1.patch) > Add ACL for setting split merge switch > -- > > Key: HBASE-15488 > URL: https://issues.apache.org/jira/browse/HBASE-15488 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.4.0 > > > Currently there is no access control for the split merge switch setter in > MasterRpcServices. > This JIRA adds necessary coprocessor hook along with enforcing permission > check in AccessController through the new hook. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15488) Add ACL for setting split merge switch
Ted Yu created HBASE-15488: -- Summary: Add ACL for setting split merge switch Key: HBASE-15488 URL: https://issues.apache.org/jira/browse/HBASE-15488 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: Ted Yu Currently there is no access control for the split merge switch setter in MasterRpcServices. This JIRA adds necessary coprocessor hook along with enforcing permission check in AccessController through the new hook. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15433) SnapshotManager#restoreSnapshot not update table and region count quota correctly when encountering exception
[
https://issues.apache.org/jira/browse/HBASE-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201414#comment-15201414
]
Ted Yu edited comment on HBASE-15433 at 3/18/16 12:46 PM:
--
Jianwei:
I looked at the scenario (region counts of 8 and 10 in snapshots) above where
retrieving from region locator may produce inaccurate end result.
We can go with patch v4.
Since branch-1 is among the Fix versions, can you attach patch for branch-1 ?
branch-1 tests are more stable, hopefully we can get higher confidence.
Thanks
was (Author: yuzhih...@gmail.com):
Jianwei:
I looked at the scenario (region counts of 8 and 10 in snapshots) above where
retrieving from region locator may produce inaccurate end result.
We can go with path v4.
Since branch-1 is among the Fix versions, can you attach patch for branch-1 ?
branch-1 tests are more stable, hopefully we can get higher confidence.
Thanks
> SnapshotManager#restoreSnapshot not update table and region count quota
> correctly when encountering exception
> -
>
> Key: HBASE-15433
> URL: https://issues.apache.org/jira/browse/HBASE-15433
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 2.0.0
>Reporter: Jianwei Cui
>Assignee: Jianwei Cui
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.4.0, 1.1.5
>
> Attachments: HBASE-15433-trunk-v1.patch, HBASE-15433-trunk-v2.patch,
> HBASE-15433-trunk.patch, HBASE-15433-v3.patch, HBASE-15433-v4.patch
>
>
> In SnapshotManager#restoreSnapshot, the table and region quota will be
> checked and updated as:
> {code}
> try {
> // Table already exist. Check and update the region quota for this
> table namespace
> checkAndUpdateNamespaceRegionQuota(manifest, tableName);
> restoreSnapshot(snapshot, snapshotTableDesc);
> } catch (IOException e) {
>
> this.master.getMasterQuotaManager().removeTableFromNamespaceQuota(tableName);
> LOG.error("Exception occurred while restoring the snapshot " +
> snapshot.getName()
> + " as table " + tableName.getNameAsString(), e);
> throw e;
> }
> {code}
> The 'checkAndUpdateNamespaceRegionQuota' will fail if regions in the snapshot
> make the region count quota exceeded, then, the table will be removed in the
> 'catch' block. This will make the current table count and region count
> decrease, following table creation or region split will succeed even if the
> actual quota is exceeded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201485#comment-15201485
]
Ted Yu commented on HBASE-15406:
But shouldn't the admin rerun hbck if previous hbck run gets aborted ?
At the end of successful hbck run, switch state would be restored anyway.
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15406.patch, HBASE-15406.v1.patch,
> HBASE-15406_v1.patch, test.patch, wip.patch
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15482) Provide an option to skip calculating block locations for SnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200994#comment-15200994 ] Ted Yu commented on HBASE-15482: Welcome back, Liyin. Yeah. This makes sense. > Provide an option to skip calculating block locations for SnapshotInputFormat > - > > Key: HBASE-15482 > URL: https://issues.apache.org/jira/browse/HBASE-15482 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Liyin Tang >Priority: Minor > > When a MR job is reading from SnapshotInputFormat, it needs to calculate the > splits based on the block locations in order to get best locality. However, > this process may take a long time for large snapshots. > In some setup, the computing layer, Spark, Hive or Presto could run out side > of HBase cluster. In these scenarios, the block locality doesn't matter. > Therefore, it will be great to have an option to skip calculating the block > locations for every job. That will super useful for the Hive/Presto/Spark > connectors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12940) Expose listPeerConfigs and getPeerConfig to the HBase shell
[ https://issues.apache.org/jira/browse/HBASE-12940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198315#comment-15198315 ] Ted Yu commented on HBASE-12940: Can you let me know you verified the test result ? For me the following search for test case yielded nothing: grep 'simple.\*clusterKey' hbase-shell/target/surefire-reports/org.apache.hadoop.hbase.client.TestReplicationShell-output.txt > Expose listPeerConfigs and getPeerConfig to the HBase shell > --- > > Key: HBASE-12940 > URL: https://issues.apache.org/jira/browse/HBASE-12940 > Project: HBase > Issue Type: Bug > Components: shell >Reporter: Kevin Risden >Assignee: Geoffrey Jacoby > Attachments: HBASE-12940-v1.patch, HBASE-12940.patch > > > In HBASE-12867 found that listPeerConfigs and getPeerConfig from > ReplicationAdmin are not exposed to the HBase shell. This makes looking at > details for custom replication endpoints and testing of add_peer from > HBASE-12867 impossible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1511#comment-1511
]
Ted Yu commented on HBASE-14123:
For Backup.proto :
{code}
+ required uint32 bandwidth = 15;
{code}
I think convention is to use long for bandwidth.
e.g.
{code}
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java:
long bandwidth =
this.conf.getLong("replication.source.per.peer.node.bandwidth
{code}
{code}
- String targetRootDir) throws IOException {
+ String targetRootDir, int workers, int bandwidth) throws IOException {
{code}
Suggest using long for bandwidth.
{code}
+opt.addOption("n", true, "History length");
{code}
I don't see "h" or "n" taken in BackupDriver. Either one is more user friendly.
> HBase Backup/Restore Phase 2
>
>
> Key: HBASE-14123
> URL: https://issues.apache.org/jira/browse/HBASE-14123
> Project: HBase
> Issue Type: Umbrella
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-14123-v1.patch, HBASE-14123-v10.patch,
> HBASE-14123-v11.patch, HBASE-14123-v12.patch, HBASE-14123-v2.patch,
> HBASE-14123-v3.patch, HBASE-14123-v4.patch, HBASE-14123-v5.patch,
> HBASE-14123-v6.patch, HBASE-14123-v7.patch, HBASE-14123-v9.patch
>
>
> Phase 2 umbrella JIRA. See HBASE-7912 for design document and description.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HBASE-15456) CreateTableProcedure/ModifyTableProcedure needs to fail when there is no family in table descriptor
[
https://issues.apache.org/jira/browse/HBASE-15456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reopened HBASE-15456:
> CreateTableProcedure/ModifyTableProcedure needs to fail when there is no
> family in table descriptor
> ---
>
> Key: HBASE-15456
> URL: https://issues.apache.org/jira/browse/HBASE-15456
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15456-001_branch-1.patch,
> HBASE-15456-branch-1.patch, HBASE-15456-branch-1.patch,
> HBASE-15456-v001.patch, HBASE-15456-v002.patch, HBASE-15456-v002.patch,
> HBASE-15456-v003.patch
>
>
> If there is only one family in the table, DeleteColumnFamilyProcedure will
> fail.
> Currently, when hbase.table.sanity.checks is set to false, hbase master logs
> a warning and CreateTableProcedure/ModifyTableProcedure will succeed.
> This behavior is not consistent with DeleteColumnFamilyProcedure's.
> Another point, before HBASE-13145, PeriodicMemstoreFlusher will run into the
> following exception. lastStoreFlushTimeMap is populated for families, if
> there is no family in the table, there is no entry in lastStoreFlushTimeMap.
> {code}
> 16/02/01 11:14:26 ERROR regionserver.HRegionServer$PeriodicMemstoreFlusher:
> Caught exception
> java.util.NoSuchElementException
> at
> java.util.concurrent.ConcurrentHashMap$HashIterator.nextEntry(ConcurrentHashMap.java:1354)
>
> at
> java.util.concurrent.ConcurrentHashMap$ValueIterator.next(ConcurrentHashMap.java:1384)
>
> at java.util.Collections.min(Collections.java:628)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getEarliestFlushTimeForAllStores(HRegion.java:1572)
>
> at
> org.apache.hadoop.hbase.regionserver.HRegion.shouldFlush(HRegion.java:1904)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$PeriodicMemstoreFlusher.chore(HRegionServer.java:1509)
>
> at org.apache.hadoop.hbase.Chore.run(Chore.java:87)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15424) Add bulk load hfile-refs for replication in ZK after the event is appended in the WAL
[ https://issues.apache.org/jira/browse/HBASE-15424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15197586#comment-15197586 ] Ted Yu commented on HBASE-15424: lgtm Better get a good QA run. > Add bulk load hfile-refs for replication in ZK after the event is appended in > the WAL > - > > Key: HBASE-15424 > URL: https://issues.apache.org/jira/browse/HBASE-15424 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.3.0 >Reporter: Ashish Singhi >Assignee: Ashish Singhi >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15424.patch, HBASE-15424.v1.patch, > HBASE-15424.v1.patch, HBASE-15424.v2.patch > > > Currenlty hfile-refs znode used for tracking the bulk loaded data replication > is added first and then the bulk load event in appended in the WAL. So this > may lead to a issue where the znode is added in ZK but append to WAL is > failed(due to some probelm in DN), so this znode will be left in ZK as it is > and will not allow hfile to get deleted from archive directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15325) ResultScanner allowing partial result will miss the rest of the row if the region is moved between two rpc requests
[
https://issues.apache.org/jira/browse/HBASE-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200773#comment-15200773
]
Ted Yu commented on HBASE-15325:
{code}
Hunk #1 FAILED at 378.
1 out of 1 hunk FAILED -- saving rejects to file
hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java.rej
{code}
TestPartialResultsFromClientSide.java is not in branch-1
Phil:
Can you attach patch for branch-1 ?
Thanks
> ResultScanner allowing partial result will miss the rest of the row if the
> region is moved between two rpc requests
> ---
>
> Key: HBASE-15325
> URL: https://issues.apache.org/jira/browse/HBASE-15325
> Project: HBase
> Issue Type: Bug
> Components: dataloss, Scanners
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Phil Yang
>Assignee: Phil Yang
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.4.0
>
> Attachments: 15325-test.txt, HBASE-15325-v1.txt,
> HBASE-15325-v10.patch, HBASE-15325-v11.patch, HBASE-15325-v2.txt,
> HBASE-15325-v3.txt, HBASE-15325-v5.txt, HBASE-15325-v6.1.txt,
> HBASE-15325-v6.2.txt, HBASE-15325-v6.3.txt, HBASE-15325-v6.4.txt,
> HBASE-15325-v6.5.txt, HBASE-15325-v6.txt, HBASE-15325-v7.patch,
> HBASE-15325-v8.patch, HBASE-15325-v9.patch
>
>
> HBASE-11544 allow scan rpc return partial of a row to reduce memory usage for
> one rpc request. And client can setAllowPartial or setBatch to get several
> cells in a row instead of the whole row.
> However, the status of the scanner is saved on server and we need this to get
> the next part if there is a partial result before. If we move the region to
> another RS, client will get a NotServingRegionException and open a new
> scanner to the new RS which will be regarded as a new scan from the end of
> this row. So the rest cells of the row of last result will be missing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201553#comment-15201553
]
Ted Yu commented on HBASE-15406:
bq. Is there any difference whether they are the same user?
There is difference. split / merge switch normally should be on. The same admin
shouldn't leave an aborted hbck unattended and turn off split / merge switch.
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15406.patch, HBASE-15406.v1.patch,
> HBASE-15406_v1.patch, test.patch, wip.patch
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12940) Expose listPeerConfigs and getPeerConfig to the HBase shell
[ https://issues.apache.org/jira/browse/HBASE-12940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198603#comment-15198603 ] Ted Yu commented on HBASE-12940: Geoffrey: I am satisfied with your responses. > Expose listPeerConfigs and getPeerConfig to the HBase shell > --- > > Key: HBASE-12940 > URL: https://issues.apache.org/jira/browse/HBASE-12940 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Kevin Risden >Assignee: Geoffrey Jacoby > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-12940-v1.patch, HBASE-12940.patch > > > In HBASE-12867 found that listPeerConfigs and getPeerConfig from > ReplicationAdmin are not exposed to the HBase shell. This makes looking at > details for custom replication endpoints and testing of add_peer from > HBASE-12867 impossible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200824#comment-15200824
]
Ted Yu commented on HBASE-15406:
Heng:
I triggered new QA run.
Can you outline how you tested latest patch ?
Thanks
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15406.patch, HBASE-15406.v1.patch,
> HBASE-15406_v1.patch, wip.patch
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15411) Rewrite backup with Procedure V2
[ https://issues.apache.org/jira/browse/HBASE-15411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15411: --- Attachment: 15411-v15.txt Patch v15 adds workers and bandwidth parameters to FullTableBackupProcedure and IncrementalTableBackupProcedure ctors > Rewrite backup with Procedure V2 > > > Key: HBASE-15411 > URL: https://issues.apache.org/jira/browse/HBASE-15411 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15411-v1.txt, 15411-v11.txt, 15411-v12.txt, > 15411-v13.txt, 15411-v14.txt, 15411-v15.txt, 15411-v3.txt, 15411-v5.txt, > 15411-v6.txt, 15411-v7.txt, 15411-v9.txt, FullTableBackupProcedure.java > > > Currently full / incremental backup is driven by BackupHandler (see call() > method for flow). > This issue is to rewrite the flow using Procedure V2. > States (enum) for full / incremental backup would be introduced in > Backup.proto which correspond to the steps performed in BackupHandler#call(). > executeFromState() would pace the backup based on the current state. > serializeStateData() / deserializeStateData() would be used to persist state > into procedure WAL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12940) Expose listPeerConfigs and getPeerConfig to the HBase shell
[ https://issues.apache.org/jira/browse/HBASE-12940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12940: --- Hadoop Flags: Reviewed Fix Version/s: 1.4.0 2.0.0 Issue Type: Improvement (was: Bug) > Expose listPeerConfigs and getPeerConfig to the HBase shell > --- > > Key: HBASE-12940 > URL: https://issues.apache.org/jira/browse/HBASE-12940 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Kevin Risden >Assignee: Geoffrey Jacoby > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-12940-v1.patch, HBASE-12940.patch > > > In HBASE-12867 found that listPeerConfigs and getPeerConfig from > ReplicationAdmin are not exposed to the HBase shell. This makes looking at > details for custom replication endpoints and testing of add_peer from > HBASE-12867 impossible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201797#comment-15201797
]
Ted Yu commented on HBASE-14123:
{code}
+List list = new ArrayList();
+list.add(manifest.getBackupImage());
+List depList = manifest.getDependentListByTable(table);
+list.addAll(depList);
TreeSet restoreList =
-new TreeSet(manifest.getDependentListByTable(table));
+new TreeSet(list);
{code}
Consider constructing TreeSet and adding BackupImage's to it without using List.
{code}
+ public void cancelCopyJob(byte[] jobId) throws IOException {
...
+ if (job == null) {
+ LOG.error("No job found for " + id);
+ // should we throw exception
+ }
{code}
Should return type be boolean so that the caller knows whether the cancellation
is effective ?
{code}
+ } catch (InterruptedException e) {
+ throw new IOException(e);
{code}
throw InterruptedIOException instead
{code}
+ { logDirs, tableNames[i].getNameAsString(),
newTableNames[i].getNameAsString()/*,
+ "-Dwal.bulk.output=" + bulkOutputPath.toString() */};
{code}
Consider referring to WALPlayer#BULK_OUTPUT_CONF_KEY directly.
> HBase Backup/Restore Phase 2
>
>
> Key: HBASE-14123
> URL: https://issues.apache.org/jira/browse/HBASE-14123
> Project: HBase
> Issue Type: Umbrella
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-14123-v1.patch, HBASE-14123-v10.patch,
> HBASE-14123-v11.patch, HBASE-14123-v12.patch, HBASE-14123-v2.patch,
> HBASE-14123-v3.patch, HBASE-14123-v4.patch, HBASE-14123-v5.patch,
> HBASE-14123-v6.patch, HBASE-14123-v7.patch, HBASE-14123-v9.patch
>
>
> Phase 2 umbrella JIRA. See HBASE-7912 for design document and description.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12940) Expose listPeerConfigs and getPeerConfig to the HBase shell
[
https://issues.apache.org/jira/browse/HBASE-12940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198417#comment-15198417
]
Ted Yu commented on HBASE-12940:
I was looking for the following in test output:
{code}
+ formatter.row(["Cluster Key", cluster_key])
{code}
But didn't find any.
Do you know why ?
> Expose listPeerConfigs and getPeerConfig to the HBase shell
> ---
>
> Key: HBASE-12940
> URL: https://issues.apache.org/jira/browse/HBASE-12940
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Kevin Risden
>Assignee: Geoffrey Jacoby
> Attachments: HBASE-12940-v1.patch, HBASE-12940.patch
>
>
> In HBASE-12867 found that listPeerConfigs and getPeerConfig from
> ReplicationAdmin are not exposed to the HBase shell. This makes looking at
> details for custom replication endpoints and testing of add_peer from
> HBASE-12867 impossible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15334) Add avro support for spark hbase connector
[ https://issues.apache.org/jira/browse/HBASE-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15334: --- Hadoop Flags: Reviewed Fix Version/s: 2.0.0 > Add avro support for spark hbase connector > -- > > Key: HBASE-15334 > URL: https://issues.apache.org/jira/browse/HBASE-15334 > Project: HBase > Issue Type: Sub-task >Reporter: Zhan Zhang >Assignee: Zhan Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15334-1.patch, HBASE-15334-2.patch, > HBASE-15334-3.patch, HBASE-15334-4.patch, HBASE-15334-5.patch > > > Avro is a popular format for hbase storage. User may want the support > natively in the connector. > With the support, user can save serialized avro into hbase table, and then > query on top of it using spark sql. The conversion between avro and catalyst > datatype will be handled automatically. This is one way of support complex > data types. Otherwise, user has to define their own customized Serdes to > support complex data types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-12940) Expose listPeerConfigs and getPeerConfig to the HBase shell
[
https://issues.apache.org/jira/browse/HBASE-12940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198315#comment-15198315
]
Ted Yu edited comment on HBASE-12940 at 3/16/16 10:28 PM:
--
Can you let me know you verified the test result ?
For me the following search for test case yielded nothing:
grep 'simple.\*clusterKey'
hbase-shell/target/surefire-reports/org.apache.hadoop.hbase.client.TestReplicationShell-output.txt
I ran the test with the following change:
{code}
diff --git
a/hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestReplicationShell.java
b/hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestReplicationShell.ja
index 3f4af05..57bdab8 100644
---
a/hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestReplicationShell.java
+++
b/hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestReplicationShell.java
@@ -28,10 +28,11 @@ import org.junit.experimental.categories.Category;
@Category({ ClientTests.class, LargeTests.class })
public class TestReplicationShell extends AbstractTestShell {
- @Ignore ("Disabled because hangs on occasion.. about 10% of the time") @Test
+ //@Ignore ("Disabled because hangs on occasion.. about 10% of the time")
+ @Test
public void testRunShellTests() throws IOException {
System.setProperty("shell.test.include", "replication_admin_test.rb");
// Start all ruby tests
jruby.runScriptlet(PathType.ABSOLUTE, "src/test/ruby/tests_runner.rb");
}
-}
{code}
was (Author: yuzhih...@gmail.com):
Can you let me know you verified the test result ?
For me the following search for test case yielded nothing:
grep 'simple.\*clusterKey'
hbase-shell/target/surefire-reports/org.apache.hadoop.hbase.client.TestReplicationShell-output.txt
> Expose listPeerConfigs and getPeerConfig to the HBase shell
> ---
>
> Key: HBASE-12940
> URL: https://issues.apache.org/jira/browse/HBASE-12940
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Kevin Risden
>Assignee: Geoffrey Jacoby
> Attachments: HBASE-12940-v1.patch, HBASE-12940.patch
>
>
> In HBASE-12867 found that listPeerConfigs and getPeerConfig from
> ReplicationAdmin are not exposed to the HBase shell. This makes looking at
> details for custom replication endpoints and testing of add_peer from
> HBASE-12867 impossible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15155) Show All RPC handler tasks stops working after cluster is under heavy load for a while
[ https://issues.apache.org/jira/browse/HBASE-15155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200261#comment-15200261 ] Ted Yu commented on HBASE-15155: Tianying: I assume cluster testing has been done for this change ? > Show All RPC handler tasks stops working after cluster is under heavy load > for a while > -- > > Key: HBASE-15155 > URL: https://issues.apache.org/jira/browse/HBASE-15155 > Project: HBase > Issue Type: Bug > Components: monitoring >Affects Versions: 0.98.0, 1.0.0, 0.94.19 >Reporter: Tianying Chang >Assignee: Tianying Chang > Attachments: hbase-15155.patch > > > After we upgrade from 94.7 to 94.26 and 1.0, we found that "Show All RPC > handler status" link on RS webUI stops working after running in production > cluster with relatively high load for several days. > Turn out to be it is a bug introduced by > https://issues.apache.org/jira/browse/HBASE-10312 The BoundedFIFOBuffer cause > RPCHandler Status overriden/removed permanently when there is a spike of > non-RPC tasks status that is over the MAX_SIZE (1000). So as long as the RS > experienced "high" load once, the RPC status monitoring is gone forever, > until RS is restarted. > We added a unit test that can repro this. And the fix can pass the test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15325) ResultScanner allowing partial result will miss the rest of the row if the region is moved between two rpc requests
[ https://issues.apache.org/jira/browse/HBASE-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200065#comment-15200065 ] Ted Yu commented on HBASE-15325: Integrated to master branch. Once trunk Jenkins build comes back and is good, will integrate to other branches. Thanks, Phil. Thanks for the reviews, Ram and Anoop. > ResultScanner allowing partial result will miss the rest of the row if the > region is moved between two rpc requests > --- > > Key: HBASE-15325 > URL: https://issues.apache.org/jira/browse/HBASE-15325 > Project: HBase > Issue Type: Bug > Components: dataloss, Scanners >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.4.0 > > Attachments: 15325-test.txt, HBASE-15325-v1.txt, > HBASE-15325-v10.patch, HBASE-15325-v11.patch, HBASE-15325-v2.txt, > HBASE-15325-v3.txt, HBASE-15325-v5.txt, HBASE-15325-v6.1.txt, > HBASE-15325-v6.2.txt, HBASE-15325-v6.3.txt, HBASE-15325-v6.4.txt, > HBASE-15325-v6.5.txt, HBASE-15325-v6.txt, HBASE-15325-v7.patch, > HBASE-15325-v8.patch, HBASE-15325-v9.patch > > > HBASE-11544 allow scan rpc return partial of a row to reduce memory usage for > one rpc request. And client can setAllowPartial or setBatch to get several > cells in a row instead of the whole row. > However, the status of the scanner is saved on server and we need this to get > the next part if there is a partial result before. If we move the region to > another RS, client will get a NotServingRegionException and open a new > scanner to the new RS which will be regarded as a new scan from the end of > this row. So the rest cells of the row of last result will be missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15479) No more garbage or beware of autoboxing
[ https://issues.apache.org/jira/browse/HBASE-15479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200587#comment-15200587 ] Ted Yu commented on HBASE-15479: lgtm > No more garbage or beware of autoboxing > --- > > Key: HBASE-15479 > URL: https://issues.apache.org/jira/browse/HBASE-15479 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-15479-v1.patch > > > Quick journey with JMC in a profile mode revealed very interesting and > unexpected heap polluter on a client side. Patch will shortly follow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15411) Rewrite backup with Procedure V2
[ https://issues.apache.org/jira/browse/HBASE-15411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15411: --- Attachment: 15411-v13.txt Patch v13 changes to using shared lock on hbase:backup table. Also adds place holders for workers and bandwidth parameters for Admin API (implementation being part of phase 2). > Rewrite backup with Procedure V2 > > > Key: HBASE-15411 > URL: https://issues.apache.org/jira/browse/HBASE-15411 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15411-v1.txt, 15411-v11.txt, 15411-v12.txt, > 15411-v13.txt, 15411-v3.txt, 15411-v5.txt, 15411-v6.txt, 15411-v7.txt, > 15411-v9.txt, FullTableBackupProcedure.java > > > Currently full / incremental backup is driven by BackupHandler (see call() > method for flow). > This issue is to rewrite the flow using Procedure V2. > States (enum) for full / incremental backup would be introduced in > Backup.proto which correspond to the steps performed in BackupHandler#call(). > executeFromState() would pace the backup based on the current state. > serializeStateData() / deserializeStateData() would be used to persist state > into procedure WAL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15411) Rewrite backup with Procedure V2
[ https://issues.apache.org/jira/browse/HBASE-15411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15411: --- Attachment: 15411-v16.txt > Rewrite backup with Procedure V2 > > > Key: HBASE-15411 > URL: https://issues.apache.org/jira/browse/HBASE-15411 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15411-v1.txt, 15411-v11.txt, 15411-v12.txt, > 15411-v13.txt, 15411-v14.txt, 15411-v15.txt, 15411-v16.txt, 15411-v3.txt, > 15411-v5.txt, 15411-v6.txt, 15411-v7.txt, 15411-v9.txt, > FullTableBackupProcedure.java > > > Currently full / incremental backup is driven by BackupHandler (see call() > method for flow). > This issue is to rewrite the flow using Procedure V2. > States (enum) for full / incremental backup would be introduced in > Backup.proto which correspond to the steps performed in BackupHandler#call(). > executeFromState() would pace the backup based on the current state. > serializeStateData() / deserializeStateData() would be used to persist state > into procedure WAL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15488) Add ACL for setting split merge switch
[ https://issues.apache.org/jira/browse/HBASE-15488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15488: --- Attachment: HBASE-15488-branch-1.v1.patch > Add ACL for setting split merge switch > -- > > Key: HBASE-15488 > URL: https://issues.apache.org/jira/browse/HBASE-15488 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15488-branch-1.v1.patch, HBASE-15488.v1.patch > > > Currently there is no access control for the split merge switch setter in > MasterRpcServices. > This JIRA adds necessary coprocessor hook along with enforcing permission > check in AccessController through the new hook. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15479) No more garbage or beware of autoboxing
[
https://issues.apache.org/jira/browse/HBASE-15479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15202107#comment-15202107
]
Ted Yu commented on HBASE-15479:
Not running hbase-server has hidden cost:
{code}
Author: Mikhail Antonov
Date: Thu Mar 17 01:06:42 2016 -0700
HBASE-15390 Unnecessary MetaCache evictions cause elevated number of
requests to meta (ADDENDUM)
{code}
> No more garbage or beware of autoboxing
> ---
>
> Key: HBASE-15479
> URL: https://issues.apache.org/jira/browse/HBASE-15479
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-15479-v1.patch, HBASE-15479-v2.patch
>
>
> Quick journey with JMC in a profile mode revealed very interesting and
> unexpected heap polluter on a client side. Patch will shortly follow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15411) Rewrite backup with Procedure V2
[
https://issues.apache.org/jira/browse/HBASE-15411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200017#comment-15200017
]
Ted Yu commented on HBASE-15411:
I tried extracting common code between FullTableBackupProcedure and
IncrementalTableBackupProcedure into TableBackupProcedure:
{code}
public abstract class TableBackupProcedure
extends StateMachineProcedure
implements TableProcedureInterface {
{code}
But I realized the above wouldn't work because FullTableBackupProcedure and
IncrementalTableBackupProcedure use different enums for their states:
{code}
public class FullTableBackupProcedure
extends StateMachineProcedure
{code}
Since extending multiple base classes is not supported by Java, I don't see how
the common base class can be formed.
> Rewrite backup with Procedure V2
>
>
> Key: HBASE-15411
> URL: https://issues.apache.org/jira/browse/HBASE-15411
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15411-v1.txt, 15411-v11.txt, 15411-v12.txt,
> 15411-v13.txt, 15411-v14.txt, 15411-v3.txt, 15411-v5.txt, 15411-v6.txt,
> 15411-v7.txt, 15411-v9.txt, FullTableBackupProcedure.java
>
>
> Currently full / incremental backup is driven by BackupHandler (see call()
> method for flow).
> This issue is to rewrite the flow using Procedure V2.
> States (enum) for full / incremental backup would be introduced in
> Backup.proto which correspond to the steps performed in BackupHandler#call().
> executeFromState() would pace the backup based on the current state.
> serializeStateData() / deserializeStateData() would be used to persist state
> into procedure WAL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15456) CreateTableProcedure/ModifyTableProcedure needs to fail when there is no family in table descriptor
[
https://issues.apache.org/jira/browse/HBASE-15456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-15456:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> CreateTableProcedure/ModifyTableProcedure needs to fail when there is no
> family in table descriptor
> ---
>
> Key: HBASE-15456
> URL: https://issues.apache.org/jira/browse/HBASE-15456
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15456-001_branch-1.patch,
> HBASE-15456-branch-1.patch, HBASE-15456-branch-1.patch,
> HBASE-15456-v001.patch, HBASE-15456-v002.patch, HBASE-15456-v002.patch,
> HBASE-15456-v003.patch
>
>
> If there is only one family in the table, DeleteColumnFamilyProcedure will
> fail.
> Currently, when hbase.table.sanity.checks is set to false, hbase master logs
> a warning and CreateTableProcedure/ModifyTableProcedure will succeed.
> This behavior is not consistent with DeleteColumnFamilyProcedure's.
> Another point, before HBASE-13145, PeriodicMemstoreFlusher will run into the
> following exception. lastStoreFlushTimeMap is populated for families, if
> there is no family in the table, there is no entry in lastStoreFlushTimeMap.
> {code}
> 16/02/01 11:14:26 ERROR regionserver.HRegionServer$PeriodicMemstoreFlusher:
> Caught exception
> java.util.NoSuchElementException
> at
> java.util.concurrent.ConcurrentHashMap$HashIterator.nextEntry(ConcurrentHashMap.java:1354)
>
> at
> java.util.concurrent.ConcurrentHashMap$ValueIterator.next(ConcurrentHashMap.java:1384)
>
> at java.util.Collections.min(Collections.java:628)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getEarliestFlushTimeForAllStores(HRegion.java:1572)
>
> at
> org.apache.hadoop.hbase.regionserver.HRegion.shouldFlush(HRegion.java:1904)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$PeriodicMemstoreFlusher.chore(HRegionServer.java:1509)
>
> at org.apache.hadoop.hbase.Chore.run(Chore.java:87)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14030) HBase Backup/Restore Phase 1
[ https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14030: --- Attachment: HBASE-14030.v41.patch Patch v41 refactors RestoreClient API so that HBackupFileSystem (in hbase-server module) is not exposed to client. > HBase Backup/Restore Phase 1 > > > Key: HBASE-14030 > URL: https://issues.apache.org/jira/browse/HBASE-14030 > Project: HBase > Issue Type: Umbrella >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch, > HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch, > HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v15.patch, > HBASE-14030-v17.patch, HBASE-14030-v18.patch, HBASE-14030-v2.patch, > HBASE-14030-v20.patch, HBASE-14030-v21.patch, HBASE-14030-v22.patch, > HBASE-14030-v23.patch, HBASE-14030-v24.patch, HBASE-14030-v25.patch, > HBASE-14030-v26.patch, HBASE-14030-v27.patch, HBASE-14030-v28.patch, > HBASE-14030-v3.patch, HBASE-14030-v30.patch, HBASE-14030-v35.patch, > HBASE-14030-v37.patch, HBASE-14030-v4.patch, HBASE-14030-v5.patch, > HBASE-14030-v6.patch, HBASE-14030-v7.patch, HBASE-14030-v8.patch, > HBASE-14030.v38.patch, HBASE-14030.v39.patch, HBASE-14030.v40.patch, > HBASE-14030.v41.patch, hbase-14030_v36.patch > > > This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design > doc for the phase description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201501#comment-15201501
]
Ted Yu commented on HBASE-15406:
Can you identify the users in steps 1 and 3 above ?
Are they different users ?
I thought about the above scenario a week ago. Reference counting can be used
to solve the scenario but it is technically complicated (as of now).
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15406.patch, HBASE-15406.v1.patch,
> HBASE-15406_v1.patch, test.patch, wip.patch
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15302) Reenable the other tests disabled by HBASE-14678
[ https://issues.apache.org/jira/browse/HBASE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201701#comment-15201701 ] Ted Yu commented on HBASE-15302: Pushed to branch-1, first. > Reenable the other tests disabled by HBASE-14678 > > > Key: HBASE-15302 > URL: https://issues.apache.org/jira/browse/HBASE-15302 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15302-branch-1-v1.patch, HBASE-15302-v1.txt, > HBASE-15302-v1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15456) CreateTableProcedure/ModifyTableProcedure needs to fail when there is no family in table descriptor
[
https://issues.apache.org/jira/browse/HBASE-15456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198676#comment-15198676
]
Ted Yu commented on HBASE-15456:
I think the patch should be named HBASE-15456-branch-1.patch for QA bot to pick
up.
> CreateTableProcedure/ModifyTableProcedure needs to fail when there is no
> family in table descriptor
> ---
>
> Key: HBASE-15456
> URL: https://issues.apache.org/jira/browse/HBASE-15456
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15456-001_branch-1.patch, HBASE-15456-v001.patch,
> HBASE-15456-v002.patch, HBASE-15456-v002.patch, HBASE-15456-v003.patch
>
>
> If there is only one family in the table, DeleteColumnFamilyProcedure will
> fail.
> Currently, when hbase.table.sanity.checks is set to false, hbase master logs
> a warning and CreateTableProcedure/ModifyTableProcedure will succeed.
> This behavior is not consistent with DeleteColumnFamilyProcedure's.
> Another point, before HBASE-13145, PeriodicMemstoreFlusher will run into the
> following exception. lastStoreFlushTimeMap is populated for families, if
> there is no family in the table, there is no entry in lastStoreFlushTimeMap.
> {code}
> 16/02/01 11:14:26 ERROR regionserver.HRegionServer$PeriodicMemstoreFlusher:
> Caught exception
> java.util.NoSuchElementException
> at
> java.util.concurrent.ConcurrentHashMap$HashIterator.nextEntry(ConcurrentHashMap.java:1354)
>
> at
> java.util.concurrent.ConcurrentHashMap$ValueIterator.next(ConcurrentHashMap.java:1384)
>
> at java.util.Collections.min(Collections.java:628)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getEarliestFlushTimeForAllStores(HRegion.java:1572)
>
> at
> org.apache.hadoop.hbase.regionserver.HRegion.shouldFlush(HRegion.java:1904)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$PeriodicMemstoreFlusher.chore(HRegionServer.java:1509)
>
> at org.apache.hadoop.hbase.Chore.run(Chore.java:87)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15334) Add avro support for spark hbase connector
[ https://issues.apache.org/jira/browse/HBASE-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198300#comment-15198300 ] Ted Yu commented on HBASE-15334: [~jmhsieh] [~ted.m] : If you don't have further comment(s), I plan to integrate this tomorrow. > Add avro support for spark hbase connector > -- > > Key: HBASE-15334 > URL: https://issues.apache.org/jira/browse/HBASE-15334 > Project: HBase > Issue Type: Sub-task >Reporter: Zhan Zhang >Assignee: Zhan Zhang > Attachments: HBASE-15334-1.patch, HBASE-15334-2.patch, > HBASE-15334-3.patch, HBASE-15334-4.patch, HBASE-15334-5.patch > > > Avro is a popular format for hbase storage. User may want the support > natively in the connector. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15302) Reenable the other tests disabled by HBASE-14678
[ https://issues.apache.org/jira/browse/HBASE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201685#comment-15201685 ] Ted Yu commented on HBASE-15302: Based on the result from https://builds.apache.org/job/PreCommit-HBASE-Build/1070/artifact/patchprocess/patch-unit-hbase-server.txt , all tests passed. > Reenable the other tests disabled by HBASE-14678 > > > Key: HBASE-15302 > URL: https://issues.apache.org/jira/browse/HBASE-15302 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15302-branch-1-v1.patch, HBASE-15302-v1.txt, > HBASE-15302-v1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15433) SnapshotManager#restoreSnapshot not update table and region count quota correctly when encountering exception
[
https://issues.apache.org/jira/browse/HBASE-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201414#comment-15201414
]
Ted Yu commented on HBASE-15433:
Jianwei:
I looked at the scenario (region counts of 8 and 10 in snapshots) above where
retrieving from region locator may produce inaccurate end result.
We can go with path v4.
Since branch-1 is among the Fix versions, can you attach patch for branch-1 ?
branch-1 tests are more stable, hopefully we can get higher confidence.
Thanks
> SnapshotManager#restoreSnapshot not update table and region count quota
> correctly when encountering exception
> -
>
> Key: HBASE-15433
> URL: https://issues.apache.org/jira/browse/HBASE-15433
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 2.0.0
>Reporter: Jianwei Cui
>Assignee: Jianwei Cui
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.4.0, 1.1.5
>
> Attachments: HBASE-15433-trunk-v1.patch, HBASE-15433-trunk-v2.patch,
> HBASE-15433-trunk.patch, HBASE-15433-v3.patch, HBASE-15433-v4.patch
>
>
> In SnapshotManager#restoreSnapshot, the table and region quota will be
> checked and updated as:
> {code}
> try {
> // Table already exist. Check and update the region quota for this
> table namespace
> checkAndUpdateNamespaceRegionQuota(manifest, tableName);
> restoreSnapshot(snapshot, snapshotTableDesc);
> } catch (IOException e) {
>
> this.master.getMasterQuotaManager().removeTableFromNamespaceQuota(tableName);
> LOG.error("Exception occurred while restoring the snapshot " +
> snapshot.getName()
> + " as table " + tableName.getNameAsString(), e);
> throw e;
> }
> {code}
> The 'checkAndUpdateNamespaceRegionQuota' will fail if regions in the snapshot
> make the region count quota exceeded, then, the table will be removed in the
> 'catch' block. This will make the current table count and region count
> decrease, following table creation or region split will succeed even if the
> actual quota is exceeded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15456) CreateTableProcedure/ModifyTableProcedure needs to fail when there is no family in table descriptor
[
https://issues.apache.org/jira/browse/HBASE-15456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-15456:
---
Status: Patch Available (was: Reopened)
> CreateTableProcedure/ModifyTableProcedure needs to fail when there is no
> family in table descriptor
> ---
>
> Key: HBASE-15456
> URL: https://issues.apache.org/jira/browse/HBASE-15456
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15456-001_branch-1.patch, HBASE-15456-v001.patch,
> HBASE-15456-v002.patch, HBASE-15456-v002.patch, HBASE-15456-v003.patch
>
>
> If there is only one family in the table, DeleteColumnFamilyProcedure will
> fail.
> Currently, when hbase.table.sanity.checks is set to false, hbase master logs
> a warning and CreateTableProcedure/ModifyTableProcedure will succeed.
> This behavior is not consistent with DeleteColumnFamilyProcedure's.
> Another point, before HBASE-13145, PeriodicMemstoreFlusher will run into the
> following exception. lastStoreFlushTimeMap is populated for families, if
> there is no family in the table, there is no entry in lastStoreFlushTimeMap.
> {code}
> 16/02/01 11:14:26 ERROR regionserver.HRegionServer$PeriodicMemstoreFlusher:
> Caught exception
> java.util.NoSuchElementException
> at
> java.util.concurrent.ConcurrentHashMap$HashIterator.nextEntry(ConcurrentHashMap.java:1354)
>
> at
> java.util.concurrent.ConcurrentHashMap$ValueIterator.next(ConcurrentHashMap.java:1384)
>
> at java.util.Collections.min(Collections.java:628)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getEarliestFlushTimeForAllStores(HRegion.java:1572)
>
> at
> org.apache.hadoop.hbase.regionserver.HRegion.shouldFlush(HRegion.java:1904)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$PeriodicMemstoreFlusher.chore(HRegionServer.java:1509)
>
> at org.apache.hadoop.hbase.Chore.run(Chore.java:87)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15411) Rewrite backup with Procedure V2
[ https://issues.apache.org/jira/browse/HBASE-15411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15411: --- Status: Patch Available (was: Open) > Rewrite backup with Procedure V2 > > > Key: HBASE-15411 > URL: https://issues.apache.org/jira/browse/HBASE-15411 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15411-v1.txt, 15411-v11.txt, 15411-v12.txt, > 15411-v13.txt, 15411-v3.txt, 15411-v5.txt, 15411-v6.txt, 15411-v7.txt, > 15411-v9.txt, FullTableBackupProcedure.java > > > Currently full / incremental backup is driven by BackupHandler (see call() > method for flow). > This issue is to rewrite the flow using Procedure V2. > States (enum) for full / incremental backup would be introduced in > Backup.proto which correspond to the steps performed in BackupHandler#call(). > executeFromState() would pace the backup based on the current state. > serializeStateData() / deserializeStateData() would be used to persist state > into procedure WAL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15302) Reenable the other tests disabled by HBASE-14678
[ https://issues.apache.org/jira/browse/HBASE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201427#comment-15201427 ] Ted Yu commented on HBASE-15302: Please look at checkstyle warnings in WALSplitter.java which is modified by the patch. > Reenable the other tests disabled by HBASE-14678 > > > Key: HBASE-15302 > URL: https://issues.apache.org/jira/browse/HBASE-15302 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15302-branch-1-v1.patch, HBASE-15302-v1.txt, > HBASE-15302-v1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15199964#comment-15199964 ] Ted Yu commented on HBASE-14123: Is it possible to upload the patch onto review board ? > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-14123-v1.patch, HBASE-14123-v10.patch, > HBASE-14123-v11.patch, HBASE-14123-v12.patch, HBASE-14123-v2.patch, > HBASE-14123-v3.patch, HBASE-14123-v4.patch, HBASE-14123-v5.patch, > HBASE-14123-v6.patch, HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14030) HBase Backup/Restore Phase 1
[ https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14030: --- Attachment: HBASE-14030.v42.patch Patch v42 drops parameters backupRootPath and backupId from non-static HBackupFileSystem methods (Thanks to Vlad's review) > HBase Backup/Restore Phase 1 > > > Key: HBASE-14030 > URL: https://issues.apache.org/jira/browse/HBASE-14030 > Project: HBase > Issue Type: Umbrella >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch, > HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch, > HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v15.patch, > HBASE-14030-v17.patch, HBASE-14030-v18.patch, HBASE-14030-v2.patch, > HBASE-14030-v20.patch, HBASE-14030-v21.patch, HBASE-14030-v22.patch, > HBASE-14030-v23.patch, HBASE-14030-v24.patch, HBASE-14030-v25.patch, > HBASE-14030-v26.patch, HBASE-14030-v27.patch, HBASE-14030-v28.patch, > HBASE-14030-v3.patch, HBASE-14030-v30.patch, HBASE-14030-v35.patch, > HBASE-14030-v37.patch, HBASE-14030-v4.patch, HBASE-14030-v5.patch, > HBASE-14030-v6.patch, HBASE-14030-v7.patch, HBASE-14030-v8.patch, > HBASE-14030.v38.patch, HBASE-14030.v39.patch, HBASE-14030.v40.patch, > HBASE-14030.v41.patch, HBASE-14030.v42.patch, hbase-14030_v36.patch > > > This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design > doc for the phase description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12940) Expose listPeerConfigs and getPeerConfig to the HBase shell
[ https://issues.apache.org/jira/browse/HBASE-12940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12940: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Expose listPeerConfigs and getPeerConfig to the HBase shell > --- > > Key: HBASE-12940 > URL: https://issues.apache.org/jira/browse/HBASE-12940 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Kevin Risden >Assignee: Geoffrey Jacoby > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-12940-v1.patch, HBASE-12940.patch > > > In HBASE-12867 found that listPeerConfigs and getPeerConfig from > ReplicationAdmin are not exposed to the HBase shell. This makes looking at > details for custom replication endpoints and testing of add_peer from > HBASE-12867 impossible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15334) Add avro support for spark hbase connector
[ https://issues.apache.org/jira/browse/HBASE-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15334: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the patch, Zhan. > Add avro support for spark hbase connector > -- > > Key: HBASE-15334 > URL: https://issues.apache.org/jira/browse/HBASE-15334 > Project: HBase > Issue Type: Sub-task >Reporter: Zhan Zhang >Assignee: Zhan Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15334-1.patch, HBASE-15334-2.patch, > HBASE-15334-3.patch, HBASE-15334-4.patch, HBASE-15334-5.patch > > > Avro is a popular format for hbase storage. User may want the support > natively in the connector. > With the support, user can save serialized avro into hbase table, and then > query on top of it using spark sql. The conversion between avro and catalyst > datatype will be handled automatically. This is one way of support complex > data types. Otherwise, user has to define their own customized Serdes to > support complex data types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15479) No more garbage or beware of autoboxing
[ https://issues.apache.org/jira/browse/HBASE-15479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15479: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.1.5 1.4.0 0.98.19 1.2.1 1.3.0 Status: Resolved (was: Patch Available) Thanks for the patch, Vlad. > No more garbage or beware of autoboxing > --- > > Key: HBASE-15479 > URL: https://issues.apache.org/jira/browse/HBASE-15479 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0, 1.3.0, 1.2.1, 0.98.19, 1.4.0, 1.1.5 > > Attachments: HBASE-15479-v1.patch, HBASE-15479-v2.patch > > > Quick journey with JMC in a profile mode revealed very interesting and > unexpected heap polluter on a client side. Patch will shortly follow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15490) Two CompactionThroughputControllerFactory co-exist in branch-1
[
https://issues.apache.org/jira/browse/HBASE-15490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203095#comment-15203095
]
Ted Yu commented on HBASE-15490:
lgtm
> Two CompactionThroughputControllerFactory co-exist in branch-1
> --
>
> Key: HBASE-15490
> URL: https://issues.apache.org/jira/browse/HBASE-15490
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-15490.branch-1.patch, HBASE-15490.patch
>
>
> Currently there're two {{CompactionThroughputControllerFactory}} in our
> branch-1 code base (one in {{o.a.h.h.regionserver.compactions}} package, the
> other in {{o.a.h.h.regionserver.throttle}}) and both are in use.
> This is a regression of HBASE-14969 and only exists in branch-1. We should
> remove the one in {{o.a.h.h.regionserver.compactions}}, and change the
> default compaction throughput controller back to
> {{NoLimitThroughputController}} to keep compatible with previous branch-1
> version
> Thanks [~ghelmling] for pointing out the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15474) Exception in HConnectionImplementation's constructor cause Zookeeper connnections leak
[
https://issues.apache.org/jira/browse/HBASE-15474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-15474:
---
Resolution: Duplicate
Fix Version/s: (was: 1.1.0)
Status: Resolved (was: Patch Available)
Dup of HBASE-14485
> Exception in HConnectionImplementation's constructor cause Zookeeper
> connnections leak
> ---
>
> Key: HBASE-15474
> URL: https://issues.apache.org/jira/browse/HBASE-15474
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Allan Yang
>Assignee: Allan Yang
> Labels: patch
> Attachments: HBASE-15474-branch-1.v3.patch, HBASE-15474v2.patch
>
>
> HConnectionImplementation creates a ZooKeeperKeepAliveConnection during
> construction, but if the constructor throw a exception, the zookeeper
> connection is not properly closed.
> {code}
> HConnectionImplementation(Configuration conf, boolean managed,
> ExecutorService pool, User user) throws IOException {
> this(conf);
> this.user = user;
> this.batchPool = pool;
> this.managed = managed;
> this.registry = setupRegistry();
> retrieveClusterId(); //here is the zookeeper connection created
> this.rpcClient = RpcClientFactory.createClient(this.conf,
> this.clusterId);// In our case, the exception happens here, so the zookeeper
> connection never closes
> this.rpcControllerFactory = RpcControllerFactory.instantiate(conf);
> ..
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201887#comment-15201887
]
Ted Yu commented on HBASE-14123:
{code}
+conf.setBoolean("hbase.loadincremental.validate.hfile", false);
{code}
Describe in release notes that hfile validation is turned off.
{code}
+ master.createTable(backupHTD, null, HConstants.NO_NONCE,
HConstants.NO_NONCE);
+ LOG.info("Created "+ BackupSystemTable.getTableName()+" table");
+} catch(TableExistsException e) {
+ e.printStackTrace();
{code}
Better check the existence of table before issuing table creation request.
{code}
-Job job = Job.getInstance(conf, conf.get(JOB_NAME_CONF_KEY, NAME + "_" +
inputDir));
+Job job = Job.getInstance(conf, conf.get(JOB_NAME_CONF_KEY, NAME + "_" +
System.currentTimeMillis()));
{code}
Keep the input dir in job name.
Please use EnvironmentEdgeManager.
> HBase Backup/Restore Phase 2
>
>
> Key: HBASE-14123
> URL: https://issues.apache.org/jira/browse/HBASE-14123
> Project: HBase
> Issue Type: Umbrella
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-14123-v1.patch, HBASE-14123-v10.patch,
> HBASE-14123-v11.patch, HBASE-14123-v12.patch, HBASE-14123-v2.patch,
> HBASE-14123-v3.patch, HBASE-14123-v4.patch, HBASE-14123-v5.patch,
> HBASE-14123-v6.patch, HBASE-14123-v7.patch, HBASE-14123-v9.patch
>
>
> Phase 2 umbrella JIRA. See HBASE-7912 for design document and description.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15389) Write out multiple files when compaction
[ https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198668#comment-15198668 ] Ted Yu commented on HBASE-15389: +1, assuming tests pass. > Write out multiple files when compaction > > > Key: HBASE-15389 > URL: https://issues.apache.org/jira/browse/HBASE-15389 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 0.98.19 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.3.0, 0.98.19 > > Attachments: HBASE-15389-0.98.patch, HBASE-15389-uc.patch, > HBASE-15389-v1.patch, HBASE-15389-v10.patch, HBASE-15389-v11.patch, > HBASE-15389-v2.patch, HBASE-15389-v3.patch, HBASE-15389-v4.patch, > HBASE-15389-v5.patch, HBASE-15389-v6.patch, HBASE-15389-v7.patch, > HBASE-15389-v8.patch, HBASE-15389-v9.patch, HBASE-15389.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15488) Add ACL for setting split merge switch
[ https://issues.apache.org/jira/browse/HBASE-15488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15488: --- Attachment: HBASE-15488.v1.patch > Add ACL for setting split merge switch > -- > > Key: HBASE-15488 > URL: https://issues.apache.org/jira/browse/HBASE-15488 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15488.v1.patch > > > Currently there is no access control for the split merge switch setter in > MasterRpcServices. > This JIRA adds necessary coprocessor hook along with enforcing permission > check in AccessController through the new hook. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15456) CreateTableProcedure/ModifyTableProcedure needs to fail when there is no family in table descriptor
[
https://issues.apache.org/jira/browse/HBASE-15456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201412#comment-15201412
]
Ted Yu commented on HBASE-15456:
Integrated to branch-1 after running TestThriftServer locally.
If the build is green, will integrate to other branches.
> CreateTableProcedure/ModifyTableProcedure needs to fail when there is no
> family in table descriptor
> ---
>
> Key: HBASE-15456
> URL: https://issues.apache.org/jira/browse/HBASE-15456
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15456-001_branch-1.patch,
> HBASE-15456-branch-1.patch, HBASE-15456-branch-1.patch,
> HBASE-15456-branch-1_v002.patch, HBASE-15456-v001.patch,
> HBASE-15456-v002.patch, HBASE-15456-v002.patch, HBASE-15456-v003.patch,
> HBASE-15456-v004.patch
>
>
> If there is only one family in the table, DeleteColumnFamilyProcedure will
> fail.
> Currently, when hbase.table.sanity.checks is set to false, hbase master logs
> a warning and CreateTableProcedure/ModifyTableProcedure will succeed.
> This behavior is not consistent with DeleteColumnFamilyProcedure's.
> Another point, before HBASE-13145, PeriodicMemstoreFlusher will run into the
> following exception. lastStoreFlushTimeMap is populated for families, if
> there is no family in the table, there is no entry in lastStoreFlushTimeMap.
> {code}
> 16/02/01 11:14:26 ERROR regionserver.HRegionServer$PeriodicMemstoreFlusher:
> Caught exception
> java.util.NoSuchElementException
> at
> java.util.concurrent.ConcurrentHashMap$HashIterator.nextEntry(ConcurrentHashMap.java:1354)
>
> at
> java.util.concurrent.ConcurrentHashMap$ValueIterator.next(ConcurrentHashMap.java:1384)
>
> at java.util.Collections.min(Collections.java:628)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getEarliestFlushTimeForAllStores(HRegion.java:1572)
>
> at
> org.apache.hadoop.hbase.regionserver.HRegion.shouldFlush(HRegion.java:1904)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$PeriodicMemstoreFlusher.chore(HRegionServer.java:1509)
>
> at org.apache.hadoop.hbase.Chore.run(Chore.java:87)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201402#comment-15201402
]
Ted Yu commented on HBASE-15406:
bq. it will cause conflicts when two hbck run at the same time.
There is lock mechanism in place to prevent two hbck instances from running at
the same time. You don't need to add extra code covering this.
Since the key can be ignored, it is not that useful. You can remove the code
related to key.
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15406.patch, HBASE-15406.v1.patch,
> HBASE-15406_v1.patch, test.patch, wip.patch
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15411) Rewrite backup with Procedure V2
[ https://issues.apache.org/jira/browse/HBASE-15411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15411: --- Attachment: 15411-v14.txt Patch v14 simplifies restore related tests by dropping HBackupFileSystem since HBackupFileSystem is no longer needed in client API. > Rewrite backup with Procedure V2 > > > Key: HBASE-15411 > URL: https://issues.apache.org/jira/browse/HBASE-15411 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15411-v1.txt, 15411-v11.txt, 15411-v12.txt, > 15411-v13.txt, 15411-v14.txt, 15411-v3.txt, 15411-v5.txt, 15411-v6.txt, > 15411-v7.txt, 15411-v9.txt, FullTableBackupProcedure.java > > > Currently full / incremental backup is driven by BackupHandler (see call() > method for flow). > This issue is to rewrite the flow using Procedure V2. > States (enum) for full / incremental backup would be introduced in > Backup.proto which correspond to the steps performed in BackupHandler#call(). > executeFromState() would pace the backup based on the current state. > serializeStateData() / deserializeStateData() would be used to persist state > into procedure WAL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15456) CreateTableProcedure/ModifyTableProcedure needs to fail when there is no family in table descriptor
[
https://issues.apache.org/jira/browse/HBASE-15456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200706#comment-15200706
]
Ted Yu commented on HBASE-15456:
[~huaxiang]:
Looks like there were some thrift tests which should have been modified - even
for master branch.
Can you run tests in modules other than hbase-server so that we get this right ?
Thanks
> CreateTableProcedure/ModifyTableProcedure needs to fail when there is no
> family in table descriptor
> ---
>
> Key: HBASE-15456
> URL: https://issues.apache.org/jira/browse/HBASE-15456
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15456-001_branch-1.patch,
> HBASE-15456-branch-1.patch, HBASE-15456-branch-1.patch,
> HBASE-15456-v001.patch, HBASE-15456-v002.patch, HBASE-15456-v002.patch,
> HBASE-15456-v003.patch
>
>
> If there is only one family in the table, DeleteColumnFamilyProcedure will
> fail.
> Currently, when hbase.table.sanity.checks is set to false, hbase master logs
> a warning and CreateTableProcedure/ModifyTableProcedure will succeed.
> This behavior is not consistent with DeleteColumnFamilyProcedure's.
> Another point, before HBASE-13145, PeriodicMemstoreFlusher will run into the
> following exception. lastStoreFlushTimeMap is populated for families, if
> there is no family in the table, there is no entry in lastStoreFlushTimeMap.
> {code}
> 16/02/01 11:14:26 ERROR regionserver.HRegionServer$PeriodicMemstoreFlusher:
> Caught exception
> java.util.NoSuchElementException
> at
> java.util.concurrent.ConcurrentHashMap$HashIterator.nextEntry(ConcurrentHashMap.java:1354)
>
> at
> java.util.concurrent.ConcurrentHashMap$ValueIterator.next(ConcurrentHashMap.java:1384)
>
> at java.util.Collections.min(Collections.java:628)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getEarliestFlushTimeForAllStores(HRegion.java:1572)
>
> at
> org.apache.hadoop.hbase.regionserver.HRegion.shouldFlush(HRegion.java:1904)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$PeriodicMemstoreFlusher.chore(HRegionServer.java:1509)
>
> at org.apache.hadoop.hbase.Chore.run(Chore.java:87)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-15456) CreateTableProcedure/ModifyTableProcedure needs to fail when there is no family in table descriptor
[
https://issues.apache.org/jira/browse/HBASE-15456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HBASE-15456.
Resolution: Fixed
> CreateTableProcedure/ModifyTableProcedure needs to fail when there is no
> family in table descriptor
> ---
>
> Key: HBASE-15456
> URL: https://issues.apache.org/jira/browse/HBASE-15456
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15456-001_branch-1.patch,
> HBASE-15456-branch-1.patch, HBASE-15456-branch-1.patch,
> HBASE-15456-branch-1_v002.patch, HBASE-15456-v001.patch,
> HBASE-15456-v002.patch, HBASE-15456-v002.patch, HBASE-15456-v003.patch,
> HBASE-15456-v004.patch
>
>
> If there is only one family in the table, DeleteColumnFamilyProcedure will
> fail.
> Currently, when hbase.table.sanity.checks is set to false, hbase master logs
> a warning and CreateTableProcedure/ModifyTableProcedure will succeed.
> This behavior is not consistent with DeleteColumnFamilyProcedure's.
> Another point, before HBASE-13145, PeriodicMemstoreFlusher will run into the
> following exception. lastStoreFlushTimeMap is populated for families, if
> there is no family in the table, there is no entry in lastStoreFlushTimeMap.
> {code}
> 16/02/01 11:14:26 ERROR regionserver.HRegionServer$PeriodicMemstoreFlusher:
> Caught exception
> java.util.NoSuchElementException
> at
> java.util.concurrent.ConcurrentHashMap$HashIterator.nextEntry(ConcurrentHashMap.java:1354)
>
> at
> java.util.concurrent.ConcurrentHashMap$ValueIterator.next(ConcurrentHashMap.java:1384)
>
> at java.util.Collections.min(Collections.java:628)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getEarliestFlushTimeForAllStores(HRegion.java:1572)
>
> at
> org.apache.hadoop.hbase.regionserver.HRegion.shouldFlush(HRegion.java:1904)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$PeriodicMemstoreFlusher.chore(HRegionServer.java:1509)
>
> at org.apache.hadoop.hbase.Chore.run(Chore.java:87)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12940) Expose listPeerConfigs and getPeerConfig to the HBase shell
[
https://issues.apache.org/jira/browse/HBASE-12940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198391#comment-15198391
]
Ted Yu commented on HBASE-12940:
I was searching for individual test case names to see if they were run.
Have you noticed the following in test output ?
{code}
2016-03-16 15:22:40,098 INFO
[RpcServer.reader=0,bindAddress=10.22.16.220,port=61976]
ipc.RpcServer$Connection(1740): Connection from 10.22.16.220 port: 62069 with
version info: version: "2.0.0-SNAPSHOT" url:
"git://TYus-MacBook-Pro.local/Users/tyu/trunk" revision:
"3bf0945a1149e518a49d14d4cc930383a4f311da" user: "tyu" date: "Wed Mar 16
15:21:57 PDT 2016" src_checksum: "fed529d9fec612f1276ebb401ea46a06"
version_major: 2 version_minor: 0
2016-03-16 15:22:40,107 ERROR [main-EventThread]
regionserver.ReplicationSourceManager(596): Error while adding a new peer
org.apache.hadoop.hbase.replication.ReplicationException: Error adding peer
with id=1
at
org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.createAndAddPeer(ReplicationPeersZKImpl.java:425)
at
org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.peerAdded(ReplicationPeersZKImpl.java:397)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.peerListChanged(ReplicationSourceManager.java:591)
at
org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl$PeersWatcher.nodeChildrenChanged(ReplicationTrackerZKImpl.java:187)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:628)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
Caused by: org.apache.hadoop.hbase.replication.ReplicationException: Error
starting the peer state tracker for peerId=1
at
org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.createPeer(ReplicationPeersZKImpl.java:493)
at
org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.createAndAddPeer(ReplicationPeersZKImpl.java:423)
... 6 more
Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /hbase/replication/peers/1/peer-state
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:575)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:554)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createNodeIfNotExistsAndWatch(ZKUtil.java:1009)
at
org.apache.hadoop.hbase.replication.ReplicationPeerZKImpl.ensurePeerEnabled(ReplicationPeerZKImpl.java:238)
at
org.apache.hadoop.hbase.replication.ReplicationPeerZKImpl.startStateTracker(ReplicationPeerZKImpl.java:96)
{code}
> Expose listPeerConfigs and getPeerConfig to the HBase shell
> ---
>
> Key: HBASE-12940
> URL: https://issues.apache.org/jira/browse/HBASE-12940
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Kevin Risden
>Assignee: Geoffrey Jacoby
> Attachments: HBASE-12940-v1.patch, HBASE-12940.patch
>
>
> In HBASE-12867 found that listPeerConfigs and getPeerConfig from
> ReplicationAdmin are not exposed to the HBase shell. This makes looking at
> details for custom replication endpoints and testing of add_peer from
> HBASE-12867 impossible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
