[jira] [Created] (HBASE-16564) ITBLL run failed with hdfs 2.7.2 on branch 0.98
Heng Chen created HBASE-16564:
-
Summary: ITBLL run failed with hdfs 2.7.2 on branch 0.98
Key: HBASE-16564
URL: https://issues.apache.org/jira/browse/HBASE-16564
Project: HBase
Issue Type: Bug
Reporter: Heng Chen
Priority: Minor
0.98 compiled with hdfs 2.2.0, so it has some compatibility issues with hdfs
2.7.2 (it seems 2.5.0+ has the same issue), some counter has been removed.
IMO we should catch the exception so our ITBLL could go on.
{code}
16/09/06 15:39:33 INFO hbase.HBaseCluster: Added new HBaseAdmin
16/09/06 15:39:33 INFO hbase.HBaseCluster: Restoring cluster - done
16/09/06 15:39:33 INFO hbase.HBaseCommonTestingUtility: Stopping mini mapreduce
cluster...
16/09/06 15:39:33 INFO Configuration.deprecation: mapred.job.tracker is
deprecated. Instead, use mapreduce.jobtracker.address
16/09/06 15:39:33 INFO hbase.HBaseCommonTestingUtility: Mini mapreduce cluster
stopped
16/09/06 15:39:33 ERROR util.AbstractHBaseTool: Error running command-line tool
java.lang.IllegalArgumentException: No enum constant
org.apache.hadoop.mapreduce.JobCounter.MB_MILLIS_MAPS
at java.lang.Enum.valueOf(Enum.java:238)
at
org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.valueOf(FrameworkCounterGroup.java:148)
at
org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.findCounter(FrameworkCounterGroup.java:182)
at
org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:154)
at
org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:240)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:370)
at
org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1289)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator.jobCompletion(IntegrationTestBigLinkedList.java:543)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator.runRandomInputGenerator(IntegrationTestBigLinkedList.java:505)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator.run(IntegrationTestBigLinkedList.java:553)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Loop.runGenerator(IntegrationTestBigLinkedList.java:842)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Loop.run(IntegrationTestBigLinkedList.java:892)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList.runTestFromCommandLine(IntegrationTestBigLinkedList.java:1237)
at
org.apache.hadoop.hbase.IntegrationTestBase.doWork(IntegrationTestBase.java:115)
at
org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList.main(IntegrationTestBigLinkedList.java:1272)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [VOTE] The 1st HBase 0.98.22 release candidate (RC0) is available
+1
- Unpacked source and binary tarballs: layout looks good
- Started up a 3-node cluster (Hadoop 2.7.2, Oracle JDK 8u20, 2 master, 3
rs) from binary tarballs.
- Verified that the web UI works and shell works
- build from source and run test case (JDK 8u20), passed. (There is some
failed test case about thrift server, but could pass when rerun manually,
list the failed test case below)
TestThriftServer.beforeClass:97 » IO Shutting down
TestThriftServerCmdLine.setUpBeforeClass:119 » IO Shutting down
TestThriftHBaseServiceHandler.beforeClass:135 » IO Shutting down
TestThriftHBaseServiceHandlerWithLabels.beforeClass:135 » IO Shutting
down
- Run LTT with 1M rows (100 writers, 30 readers (100%), 10 updaters
(20%)) all keys verified, no warns, no errors, no failed, latencies lgtm
- Run ITBLL with 2M rows (slowDeterministic), passed.
- Run ITBLL with 2.5M rows (serverKilling), passed.
Some notes: because 0.98 compiled with hadoop 2.2.0, so when i run ITBLL
on hadoop 2.7.2, it failed due to compatibiltiy issue, see HBASE-16564, so
i replace hadoop-2.2.0 jar with hadoop 2.5.1, and pass the ITBLL. Still
give +1 because it is MapReduce issue not HBase
2016-09-05 13:41 GMT+08:00 Dima Spivak :
> Ugh, sorry guys, I'm dumb. I was running 1 mapper per RS before, but
> switched to a d2.4xlarge instance today and, after noticing cores sitting
> idly, decided to try setting the number of mappers and reducers to the
> number of cores to speed testing up (RAM is still grossly underutilized
> with less than 16 GB/122 GB in use at any one time). This definitely made
> runs go faster (generation took less than 3 hours, verification took about
> 1 hour), but I just realized that the number of nodes I picked (6250)
> isn't a multiple of 25,000,000 and so the list won't wrap properly. I'll
> rerun and confirm, but I'm guessing this is a false alarm.
>
> Sorry again. :(
>
> -Dima
>
> On Sun, Sep 4, 2016 at 9:56 PM, Andrew Purtell
> wrote:
>
> > I will also try your incantation (and JRE version) on this RC and 0.98.21
> > next week to answer those same questions.
> >
> > Looks like you are using a multiple of RSes (16) as numMappers? Is that
> > 4x? On what kind of instance type? I am (also, I think) using a 5 node
> > "cluster" with 4 RS nodes but numMappers 4 and numNodes 25000. Since
> > with clusterdock everything is contending for one instance's resources I
> > didn't want to overdo and so have started at 1 mapper per RS. Since you
> > appear to be using a higher value, I'm curious if you've found that you
> > will get stable results with that, if more mappers in this configuration
> > does a better job finding problems in your experience, and what instance
> > type are you using? I've been using a d2.4xlarge.
> >
> > > On Sep 4, 2016, at 9:04 PM, Andrew Purtell
> > wrote:
> > >
> > > I've been running 1B tests with slowDeterministic. 0.98.21 and this
> > 0.98.22 RC. I get 1B referenced, all ok.
> > >
> > > Did you run serverKilling with 0.98.21? And did it pass? Or does
> 0.98.21
> > pass for you now? If so then we have a regression. If not then it's
> > something to look at for 0.98.23 I'd say.
> > >
> > >> On Sep 4, 2016, at 8:44 PM, Dima Spivak
> wrote:
> > >>
> > >> Anyone else running ITBLL seeing issues? I just ran a 5-node
> clusterdock
> > >> cluster with JDK 7u79 of this RC and tried out ITBLL with 1 billion
> rows
> > >> and the serverKilling monkey (`hbase
> > >> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList -m
> > serverKilling
> > >> loop 1 16 6250 ${RANDOM} 16`). This failed for me because of
> > >> unreferenced list nodes:
> > >>
> > >> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$
> Verify$Counts
> > >> REFERENCED=732006926
> > >> UNREFERENCED=12003580
> > >>
> > >> Perhaps this is similar to what Mikhail saw a while back with later
> > >> releases?
> > >>
> > >> -Dima
> > >>
> > >>> On Sat, Sep 3, 2016 at 8:34 AM, Andrew Purtell
> > wrote:
> > >>>
> > >>> The 1st HBase 0.98.2
> > >>> 2 release candidate (RC0) is available for download at
> > >>> https://dist.apache.org/repos/dist/dev/hbase/hbase-0.98.22RC0 and
> > Maven
> > >>> artifacts are also available in the temporary repository
> > >>> https://repository.apache.org/content/repositories/
> orgapachehbase-1151
> > .
> > >>>
> > >>> The detailed source and binary compatibility report for this release
> > with
> > >>> respect to the previous is available for your review at
> > >>> https://dist.apache.org/repos/dist/dev/hbase/hbase-0.98.
> > >>> 22RC0/0.98.21_0.98.22RC0_compat_report.html
> > >>> . There are no reported compatibility issues.
> > >>>
> > >>> The
> > >>> 25
> > >>> issues resolved in this release can be found at
> > https://s.apache.org/C7SV
> > >>> .
> > >>>
> > >>> I have made the following assessments of this candidate:
> > >>> - Release audit check
> > >>> : pass
> > >>>
> > >>> -
> > >>> Unit test suite: pass 10/10 (7u79)
> > >>>
> > >>> - Loaded 1M
[jira] [Resolved] (HBASE-16564) ITBLL run failed with hadoop 2.7.2 on branch 0.98
[
https://issues.apache.org/jira/browse/HBASE-16564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen resolved HBASE-16564.
---
Resolution: Invalid
As [~Apache9] said, the best solution is upgrade hadoop client, so close this
issue as invalid.
> ITBLL run failed with hadoop 2.7.2 on branch 0.98
> -
>
> Key: HBASE-16564
> URL: https://issues.apache.org/jira/browse/HBASE-16564
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Priority: Minor
>
> 0.98 compiled with hadoop 2.2.0, so it has some compatibility issues with
> hadoop 2.7.2 (it seems 2.5.0+ has the same issue), some counter has been
> removed.
> IMO we should catch the exception so our ITBLL could go on.
> {code}
> 16/09/06 15:39:33 INFO hbase.HBaseCluster: Added new HBaseAdmin
> 16/09/06 15:39:33 INFO hbase.HBaseCluster: Restoring cluster - done
> 16/09/06 15:39:33 INFO hbase.HBaseCommonTestingUtility: Stopping mini
> mapreduce cluster...
> 16/09/06 15:39:33 INFO Configuration.deprecation: mapred.job.tracker is
> deprecated. Instead, use mapreduce.jobtracker.address
> 16/09/06 15:39:33 INFO hbase.HBaseCommonTestingUtility: Mini mapreduce
> cluster stopped
> 16/09/06 15:39:33 ERROR util.AbstractHBaseTool: Error running command-line
> tool
> java.lang.IllegalArgumentException: No enum constant
> org.apache.hadoop.mapreduce.JobCounter.MB_MILLIS_MAPS
> at java.lang.Enum.valueOf(Enum.java:238)
> at
> org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.valueOf(FrameworkCounterGroup.java:148)
> at
> org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.findCounter(FrameworkCounterGroup.java:182)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:154)
> at
> org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:240)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:370)
> at
> org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
> at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
> at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
> at
> org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
> at
> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1289)
> at
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator.jobCompletion(IntegrationTestBigLinkedList.java:543)
> at
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator.runRandomInputGenerator(IntegrationTestBigLinkedList.java:505)
> at
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator.run(IntegrationTestBigLinkedList.java:553)
> at
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Loop.runGenerator(IntegrationTestBigLinkedList.java:842)
> at
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Loop.run(IntegrationTestBigLinkedList.java:892)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList.runTestFromCommandLine(IntegrationTestBigLinkedList.java:1237)
> at
> org.apache.hadoop.hbase.IntegrationTestBase.doWork(IntegrationTestBase.java:115)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList.main(IntegrationTestBigLinkedList.java:1272)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-16545) Add backup test where data is ingested during backup procedure
[ https://issues.apache.org/jira/browse/HBASE-16545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-16545. Resolution: Fixed Assignee: Ted Yu Hadoop Flags: Reviewed Thanks for the review, Vlad. > Add backup test where data is ingested during backup procedure > -- > > Key: HBASE-16545 > URL: https://issues.apache.org/jira/browse/HBASE-16545 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu > Labels: backup > Attachments: 16545.v1.txt, 16545.v2.txt > > > Currently the backup / restore tests do the following: > * ingest data > * perform full backup > * ingest more data > Data ingestion in step 3 above is after the completion of backup. > This issue is to add concurrent data ingestion in the presence of on-going > backup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Successful: HBase Generate Website
Build status: Successful
If successful, the website and docs have been generated. To update the live
site, follow the instructions below. If failed, skip to the bottom of this
email.
Use the following commands to download the patch and apply it to a clean branch
based on origin/asf-site. If you prefer to keep the hbase-site repo around
permanently, you can skip the clone step.
git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git
cd hbase-site
wget -O-
https://builds.apache.org/job/hbase_generate_website/334/artifact/website.patch.zip
| funzip > b6ba13c37715422710a142f6f82ba4817129c3d6.patch
git fetch
git checkout -b asf-site-b6ba13c37715422710a142f6f82ba4817129c3d6
origin/asf-site
git am --whitespace=fix b6ba13c37715422710a142f6f82ba4817129c3d6.patch
At this point, you can preview the changes by opening index.html or any of the
other HTML pages in your local
asf-site-b6ba13c37715422710a142f6f82ba4817129c3d6 branch.
There are lots of spurious changes, such as timestamps and CSS styles in
tables, so a generic git diff is not very useful. To see a list of files that
have been added, deleted, renamed, changed type, or are otherwise interesting,
use the following command:
git diff --name-status --diff-filter=ADCRTXUB origin/asf-site
To see only files that had 100 or more lines changed:
git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'
When you are satisfied, publish your changes to origin/asf-site using these
commands:
git commit --allow-empty -m "Empty commit" # to work around a current ASF
INFRA bug
git push origin asf-site-b6ba13c37715422710a142f6f82ba4817129c3d6:asf-site
git checkout asf-site
git branch -D asf-site-b6ba13c37715422710a142f6f82ba4817129c3d6
Changes take a couple of minutes to be propagated. You can verify whether they
have been propagated by looking at the Last Published date at the bottom of
http://hbase.apache.org/. It should match the date in the index.html on the
asf-site branch in Git.
As a courtesy- reply-all to this email to let other committers know you pushed
the site.
If failed, see https://builds.apache.org/job/hbase_generate_website/334/console
[jira] [Resolved] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-15449. Resolution: Fixed Hadoop Flags: Reviewed Thanks for the review, Vlad. > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v10.txt, 15449.v11.txt, > 15449.v12.txt, 15449.v2.txt, 15449.v4.txt, 15449.v5.txt, 15449.v7.txt, > 15449.v8.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] First release candidate for hbase-1.2.3 (RC0) is available
Thank you Heng for the retry (and Dima for the help). Letting the vote go another day or so. There is another vote to come in. Thanks, St.Ack On Mon, Sep 5, 2016 at 7:44 AM, Heng Chen wrote: > Thanks Dima! I rerun test with you suggestion, and it passed! (hbase > --config ~/hadoop/hbase/f04_conf/ > org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList loop 1 3 250 > /tmp/itbll 3 10 25 -m serverKilling) > > So +1 for me > > 2016-09-05 21:33 GMT+08:00 Dima Spivak : > > > Hey Heng, > > > > You need to ensure that the number of nodes in ITBLL is a multiple of > > width*wrap (defaults to 25M, but can be configured by adding two more > args > > to the test invocation). See: the 0.98.22 RC0 thread, where I forgot this > > :). > > > > On Monday, September 5, 2016, Heng Chen > wrote: > > > > > I will test it with 1.2.2 again. Not sure about it now. > > > > > > 2016-09-05 20:26 GMT+08:00 Stack >: > > > > > > > Thanks Heng. Do you know if 1.2.2 or earlier versions of 1.2 failed > in > > > > similar way or is this new phenomenon? > > > > Thanks, > > > > St.Ack > > > > > > > > On Mon, Sep 5, 2016 at 1:26 AM, Heng Chen > > > > > > > wrote: > > > > > > > > > - Unpacked source and binary tarballs: layout looks good > > > > > - Started up a 3-node cluster (Hadoop 2.7.2, Oracle JDK 8u20, 2 > > > master, 2 > > > > > rs) from binary tarballs. > > > > > - Verified that the web UI works and shell works > > > > > - build from source and run test case (JDK 8u20), passed. > > > > > - Run LTT with 1M rows (100 writers, 30 readers (100%), 10 > updaters > > > > > (20%)) all keys verified, no warns, no errors, no failed, > > latencies > > > > > lgtm > > > > > - Run ITBLL with 2M rows (slowDeterministic), passed. > > > > > > > > > > Run ITBLL with 2M rows (serverKilling) has some issues. I run two > > > times, > > > > > all failed ( hbase --config ~/hadoop/hbase/f04_conf/ > > > > > org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList loop 1 1 > > > > > 200 /tmp/it_16224_v0 1 -m serverKilling) > > > > > > > > > > The first time, after kill components in cluster serval times (1 > > > active > > > > > master, 1 backup master, 1 active RS, 1 dead RS), some regions fall > > in > > > > > Failed_open state on active RS, so balance could not run, and > ITBLL > > > > hang. > > > > > Then i start the dead RS manually, and stop the active RS, > ITBLL > > > > could > > > > > go on. But after some time, ITBLL hang again due to backup master > > > could > > > > > not be startup, so i do manually again, and ITBLL go on , and at > > > least > > > > > verified failed. > > > > > > > > > > > > > > > The second time, there is no abnormal issues during ITBLL, but > > > verified > > > > > failed. > > > > > > > > > > Upload the logs > > > > > > > > > > > > > > > > > > > > 2016-09-03 0:33 GMT+08:00 Misty Stanley-Jones > > >: > > > > > > > > > >> +1 based on OSX 10.11.6 > > > > >> > > > > >> Steps taken: > > > > >> > > > > >> Binary tar.gz: > > > > >> - Download the tarball > > > > >> - Test the MD5sum, it matched > > > > >> - Extract the tarball > > > > >> - Start HBase in standalone mode > > > > >> - Start the CLI > > > > >> - Access the master and regionServer web UIs > > > > >> - Stop HBase > > > > >> > > > > >> Source tar.gz: > > > > >> - Download the tarball > > > > >> - Test the MD5sum, it matched > > > > >> - Extract the tarball > > > > >> - Build using Maven 3.39 and JDK 1.8.0_102 on OSX and 'mvn clean > > > install > > > > >> --fail-at-end' and let the full test suite run > > > > >> -- A few failed tests but each passed when I ran it separately. > Not > > > > >> surprising since I was running all this on a Macbook. > > > > >> - Start HBase in standalone mode > > > > >> - Start the CLI > > > > >> - Access the master and regionServer web UIs > > > > >> - Stop HBase > > > > >> > > > > >> > > > > >> On Tue, Aug 30, 2016, at 11:14 AM, Stack wrote: > > > > >> > The first release candidate for HBase 1.2.3 (hbase-1.2.3RC0) is > > > > >> > available for download at: > > > > >> > > > > > >> > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.2.3RC0/ > > > > >> > > > > > >> > Maven artifacts are also available in a staging repository at: > > > > >> > > > > > >> > https://repository.apache.org/content/repositories/orgapache > > > > >> hbase-1149/ > > > > >> > > > > > >> > Artifacts are signed with my key (30CD0996) published up in our > > KEYS > > > > >> > file at https://www-us.apache.org/dist/hbase/KEYS. > > > > >> > > > > > >> > The RC is tagged 1.2.3RC0 (I'll sign the tag the next time > > > through...) > > > > >> > > > > > >> > The detailed source and binary compatibility report vs 1.2.2 has > > > been > > > > >> > published for your review, at: > > > > >> > > > > > >> > http://people.apache.org/~stack/1.2.2_1.2.3RC0_compat_ > > report.html > > > > >> > > > > > >> > HBase 1.2.3 is the third patch release in the HBase 1.2 line, > > > > continuing > > > > >> > on > > > > >> > the theme of bringing a
info:sn and info:server in hbase:meta table
I scanned the hbase:meta table in a fresh-build master branch and get the
following results. Seems to me that info:sn and info:server are some kind
of duplicate (well, info:sn has the timestamp of RS, we could store this in
info:server too if we want). Searching the code in master branch, I saw
some comment on "info:server", but could not find any information on
"info:sn". Is this "info:sn" legacy information? If so, for a new 2.0
cluster, should we not create the value in meta?
hbase(main):002:0> scan 'hbase:meta'
ROW
COLUMN+CELL
hbase:meta
column=table:state, timestamp=1473185666041, value=\x08\x00
hbase:namespace
column=table:state, timestamp=1473185667442,
value=\x08\x00
hbase:namespace,,1473185666375.4c7fc12f243854e0af6c9e30ce7d6d90.
column=info:regioninfo, timestamp=1473185667273,
value={ENCODED => 4c7fc12f243854e0af6c9e30ce7d6d90, NAME =>
'hbase:namespace,,1473185666375.4c7fc12f243854e0af6c9e30ce7d6d90.',
STARTKEY => '', ENDKEY => ''}
hbase:namespace,,1473185666375.4c7fc12f243854e0af6c9e30ce7d6d90.
column=info:seqnumDuringOpen, timestamp=1473185667603,
value=\x00\x00\x00\x00\x00\x00\x00\x02
hbase:namespace,,1473185666375.4c7fc12f243854e0af6c9e30ce7d6d90.
column=info:server, timestamp=1473185667603, value=
10.22.2.122:58498
hbase:namespace,,1473185666375.4c7fc12f243854e0af6c9e30ce7d6d90.
column=info:serverstartcode, timestamp=1473185667603,
value=1473185651426
hbase:namespace,,1473185666375.4c7fc12f243854e0af6c9e30ce7d6d90.
column=info:sn, timestamp=1473185667421,
value=10.22.2.122,58498,1473185651426
hbase:namespace,,1473185666375.4c7fc12f243854e0af6c9e30ce7d6d90.
column=info:state, timestamp=1473185667610, value=OPEN
3 row(s)
Re: [VOTE] The 1st HBase 0.98.22 release candidate (RC0) is available
Thanks for the +1, Heng.
> TestThriftServer.beforeClass:97 » IO Shutting down
Looks like the minicluster failed to launch. Port binding problem, perhaps?
It passes when rerun manually because probably no other test is executing
concurrently. By default our build runs unit tests with some parallelism.
FWIW this can be disabled with '-Dsurefire.firstPartForkCount=1
-Dsurefire.secondPartForkCount=1'.
Also, I use '-Dsurefire.rerunFailingTestsCount=2' to help distinguish
between failures and flakes.
On Tue, Sep 6, 2016 at 1:57 AM, Heng Chen wrote:
> +1
>
> - Unpacked source and binary tarballs: layout looks good
>
> - Started up a 3-node cluster (Hadoop 2.7.2, Oracle JDK 8u20, 2 master, 3
> rs) from binary tarballs.
>
> - Verified that the web UI works and shell works
>
> - build from source and run test case (JDK 8u20), passed. (There is some
> failed test case about thrift server, but could pass when rerun manually,
> list the failed test case below)
>
> TestThriftServer.beforeClass:97 » IO Shutting down
>
> TestThriftServerCmdLine.setUpBeforeClass:119 » IO Shutting down
>
> TestThriftHBaseServiceHandler.beforeClass:135 » IO Shutting down
>
> TestThriftHBaseServiceHandlerWithLabels.beforeClass:135 » IO Shutting
> down
>
> - Run LTT with 1M rows (100 writers, 30 readers (100%), 10 updaters
> (20%)) all keys verified, no warns, no errors, no failed, latencies lgtm
>
> - Run ITBLL with 2M rows (slowDeterministic), passed.
>
> - Run ITBLL with 2.5M rows (serverKilling), passed.
>
> Some notes: because 0.98 compiled with hadoop 2.2.0, so when i run ITBLL
> on hadoop 2.7.2, it failed due to compatibiltiy issue, see HBASE-16564, so
> i replace hadoop-2.2.0 jar with hadoop 2.5.1, and pass the ITBLL. Still
> give +1 because it is MapReduce issue not HBase
>
>
>
>
> 2016-09-05 13:41 GMT+08:00 Dima Spivak :
>
> > Ugh, sorry guys, I'm dumb. I was running 1 mapper per RS before, but
> > switched to a d2.4xlarge instance today and, after noticing cores sitting
> > idly, decided to try setting the number of mappers and reducers to the
> > number of cores to speed testing up (RAM is still grossly underutilized
> > with less than 16 GB/122 GB in use at any one time). This definitely made
> > runs go faster (generation took less than 3 hours, verification took
> about
> > 1 hour), but I just realized that the number of nodes I picked (6250)
> > isn't a multiple of 25,000,000 and so the list won't wrap properly. I'll
> > rerun and confirm, but I'm guessing this is a false alarm.
> >
> > Sorry again. :(
> >
> > -Dima
> >
> > On Sun, Sep 4, 2016 at 9:56 PM, Andrew Purtell >
> > wrote:
> >
> > > I will also try your incantation (and JRE version) on this RC and
> 0.98.21
> > > next week to answer those same questions.
> > >
> > > Looks like you are using a multiple of RSes (16) as numMappers? Is that
> > > 4x? On what kind of instance type? I am (also, I think) using a 5 node
> > > "cluster" with 4 RS nodes but numMappers 4 and numNodes 25000.
> Since
> > > with clusterdock everything is contending for one instance's resources
> I
> > > didn't want to overdo and so have started at 1 mapper per RS. Since you
> > > appear to be using a higher value, I'm curious if you've found that you
> > > will get stable results with that, if more mappers in this
> configuration
> > > does a better job finding problems in your experience, and what
> instance
> > > type are you using? I've been using a d2.4xlarge.
> > >
> > > > On Sep 4, 2016, at 9:04 PM, Andrew Purtell >
> > > wrote:
> > > >
> > > > I've been running 1B tests with slowDeterministic. 0.98.21 and this
> > > 0.98.22 RC. I get 1B referenced, all ok.
> > > >
> > > > Did you run serverKilling with 0.98.21? And did it pass? Or does
> > 0.98.21
> > > pass for you now? If so then we have a regression. If not then it's
> > > something to look at for 0.98.23 I'd say.
> > > >
> > > >> On Sep 4, 2016, at 8:44 PM, Dima Spivak
> > wrote:
> > > >>
> > > >> Anyone else running ITBLL seeing issues? I just ran a 5-node
> > clusterdock
> > > >> cluster with JDK 7u79 of this RC and tried out ITBLL with 1 billion
> > rows
> > > >> and the serverKilling monkey (`hbase
> > > >> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList -m
> > > serverKilling
> > > >> loop 1 16 6250 ${RANDOM} 16`). This failed for me because of
> > > >> unreferenced list nodes:
> > > >>
> > > >> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$
> > Verify$Counts
> > > >> REFERENCED=732006926
> > > >> UNREFERENCED=12003580
> > > >>
> > > >> Perhaps this is similar to what Mikhail saw a while back with later
> > > >> releases?
> > > >>
> > > >> -Dima
> > > >>
> > > >>> On Sat, Sep 3, 2016 at 8:34 AM, Andrew Purtell <
> apurt...@apache.org>
> > > wrote:
> > > >>>
> > > >>> The 1st HBase 0.98.2
> > > >>> 2 release candidate (RC0) is available for download at
> > > >>> https://dist.apache.org/repos/dist/dev/hbase/hbase-0.98.22RC0 and
> > > Maven
> > > >>> artifacts are
[jira] [Created] (HBASE-16565) Add metrics for backup / restore
Ted Yu created HBASE-16565: -- Summary: Add metrics for backup / restore Key: HBASE-16565 URL: https://issues.apache.org/jira/browse/HBASE-16565 Project: HBase Issue Type: Improvement Reporter: Ted Yu Exposing metrics for backup / restore would give admin insight on the overall operations. The metrics should include (but are not limited to): * number of backups performed (full / incremental) * number of restore's performed (full / incremental) * number of aborted backups * number of aborted restore's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16566) Add nonce support to TableBackupProcedure
Ted Yu created HBASE-16566: -- Summary: Add nonce support to TableBackupProcedure Key: HBASE-16566 URL: https://issues.apache.org/jira/browse/HBASE-16566 Project: HBase Issue Type: Improvement Reporter: Ted Yu We should pass in nonce to avoid duplicate table backup RPC (the same RPC sent to server multiple times). The duplicate table backup RPC may happen due to master failover. If there is no nonce, same procedure may be executed more than once. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Successful: HBase Generate Website
Pushed.
On Wed, Sep 7, 2016, at 12:55 AM, Apache Jenkins Server wrote:
> Build status: Successful
>
> If successful, the website and docs have been generated. To update the
> live site, follow the instructions below. If failed, skip to the bottom
> of this email.
>
> Use the following commands to download the patch and apply it to a clean
> branch based on origin/asf-site. If you prefer to keep the hbase-site
> repo around permanently, you can skip the clone step.
>
> git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git
>
> cd hbase-site
> wget -O-
>
> https://builds.apache.org/job/hbase_generate_website/334/artifact/website.patch.zip
> | funzip > b6ba13c37715422710a142f6f82ba4817129c3d6.patch
> git fetch
> git checkout -b asf-site-b6ba13c37715422710a142f6f82ba4817129c3d6
> origin/asf-site
> git am --whitespace=fix b6ba13c37715422710a142f6f82ba4817129c3d6.patch
>
> At this point, you can preview the changes by opening index.html or any
> of the other HTML pages in your local
> asf-site-b6ba13c37715422710a142f6f82ba4817129c3d6 branch.
>
> There are lots of spurious changes, such as timestamps and CSS styles in
> tables, so a generic git diff is not very useful. To see a list of files
> that have been added, deleted, renamed, changed type, or are otherwise
> interesting, use the following command:
>
> git diff --name-status --diff-filter=ADCRTXUB origin/asf-site
>
> To see only files that had 100 or more lines changed:
>
> git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'
>
> When you are satisfied, publish your changes to origin/asf-site using
> these commands:
>
> git commit --allow-empty -m "Empty commit" # to work around a current
> ASF INFRA bug
> git push origin
> asf-site-b6ba13c37715422710a142f6f82ba4817129c3d6:asf-site
> git checkout asf-site
> git branch -D asf-site-b6ba13c37715422710a142f6f82ba4817129c3d6
>
> Changes take a couple of minutes to be propagated. You can verify whether
> they have been propagated by looking at the Last Published date at the
> bottom of http://hbase.apache.org/. It should match the date in the
> index.html on the asf-site branch in Git.
>
> As a courtesy- reply-all to this email to let other committers know you
> pushed the site.
>
>
>
> If failed, see
> https://builds.apache.org/job/hbase_generate_website/334/console
Re: info:sn and info:server in hbase:meta table
>From javadoc:
/**
* The serverName column qualifier. Its the server where the region is
* transitioning on, while column server is the server where the region is
* opened on. They are the same when the region is in state OPEN.
*/
public static final String SERVERNAME_QUALIFIER_STR = "sn";
We could rename the column to be more explanatory, but we have already
released the code that serializes in this column, so may not be worth the
trouble.
On Tue, Sep 6, 2016 at 11:31 AM, Stephen Jiang
wrote:
> I scanned the hbase:meta table in a fresh-build master branch and get the
> following results. Seems to me that info:sn and info:server are some kind
> of duplicate (well, info:sn has the timestamp of RS, we could store this in
> info:server too if we want). Searching the code in master branch, I saw
> some comment on "info:server", but could not find any information on
> "info:sn". Is this "info:sn" legacy information? If so, for a new 2.0
> cluster, should we not create the value in meta?
>
> hbase(main):002:0> scan 'hbase:meta'
> ROW
> COLUMN+CELL
> hbase:meta
> column=table:state, timestamp=1473185666041,
> value=\x08\x00
> hbase:namespace
> column=table:state, timestamp=1473185667442,
> value=\x08\x00
> hbase:namespace,,1473185666375.4c7fc12f243854e0af6c9e30ce7d6d90.
> column=info:regioninfo, timestamp=1473185667273,
> value={ENCODED => 4c7fc12f243854e0af6c9e30ce7d6d90, NAME =>
> 'hbase:namespace,,1473185666375.4c7fc12f243854e0af6c9e30ce7d6d90.',
> STARTKEY => '', ENDKEY => ''}
> hbase:namespace,,1473185666375.4c7fc12f243854e0af6c9e30ce7d6d90.
> column=info:seqnumDuringOpen, timestamp=1473185667603,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> hbase:namespace,,1473185666375.4c7fc12f243854e0af6c9e30ce7d6d90.
> column=info:server, timestamp=1473185667603, value=
> 10.22.2.122:58498
> hbase:namespace,,1473185666375.4c7fc12f243854e0af6c9e30ce7d6d90.
> column=info:serverstartcode, timestamp=1473185667603,
> value=1473185651426
> hbase:namespace,,1473185666375.4c7fc12f243854e0af6c9e30ce7d6d90.
> column=info:sn, timestamp=1473185667421,
> value=10.22.2.122,58498,1473185651426
> hbase:namespace,,1473185666375.4c7fc12f243854e0af6c9e30ce7d6d90.
> column=info:state, timestamp=1473185667610, value=OPEN
> 3 row(s)
>
[jira] [Created] (HBASE-16567) Upgrade to protobuf3
stack created HBASE-16567: - Summary: Upgrade to protobuf3 Key: HBASE-16567 URL: https://issues.apache.org/jira/browse/HBASE-16567 Project: HBase Issue Type: Task Components: Protobufs Affects Versions: 2.0.0 Reporter: stack Assignee: stack Priority: Critical Move master branch on to protobuf3. See https://github.com/google/protobuf/releases We'd do it because pb3 saves some on byte copies can work with offheap buffers -- needed for the off-heap write path project -- thought read-time is still a TODO. HBASE-15638 has us first shading protobufs before upgrading. Let us list here issues just going to pb3 without shading if only for completeness sake; i.e. do we have to shade? * pb3 is by default wire compatible with pb2. * protoc3 run against our .protos works fine except pb3 breaks our HBaseZeroCopyLiteralByteString hack. * Starting up a cluster that is all pb3'd seems to work fine. * A pb2 branch-1 can read and write against the pb3 master cluster. What will break if we just upgrade to pb3? * We should be able to write HDFS messages on our AsyncWAL using pb3; the pb2 HDFS should be able to read them (not tested). Or maybe not. See policy here: https://github.com/google/protobuf/issues/1852 * Core Coprocessor Endpoints such as AccessControl seem to just work (their protos will have been protoc3'd). I did simple test with a server from master branch up on pb3 and then going against it with a branch-1 client on pb2. I was able to add grants. * For non-core CPEPs where the protos are pb2 still, it might just work. To test. It would not be the end-of-the-world if they did not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16568) Remove Cygwin-oriented instructions (for installing HBase in Windows OS) from official reference materials
Daniel Vimont created HBASE-16568: - Summary: Remove Cygwin-oriented instructions (for installing HBase in Windows OS) from official reference materials Key: HBASE-16568 URL: https://issues.apache.org/jira/browse/HBASE-16568 Project: HBase Issue Type: Improvement Components: documentation Affects Versions: 2.0.0 Reporter: Daniel Vimont Assignee: Daniel Vimont Priority: Minor Cygwin-oriented instructions in the official reference materials (for installing HBase in a Windows environment) seem to be out of date and incorrect; a number of unresolved/unresolvable requests for help have been posted to d...@hbase.org and u...@hbase.org mailing lists. Discussions on d...@apache.org and HBase Slack channel resulted in (1) no volunteers to update/maintain the Cygwin-oriented instructions, and (2) several "+" votes and no "-" votes on the suggestion of removing Cygwin-oriented instructions from the official reference materials. FUTURE POSSIBLE FOLLOW-UP: For the sake of setting up a development/testing/sandbox environment in Windows, it might be helpful to recommend the installation of a virtual machine environment (e.g. VirtualBox) in a Windows OS, followed by the installation of an appropriate flavor of Linux (e.g., Ubuntu) in the VM. After this, all the standard HBase installation/config/usage instructions can be followed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
Andrew: Do you think you would have some time this week ? Thanks On Thu, Sep 1, 2016 at 8:47 AM, Andrew Purtell wrote: > Busy at work, aiming for next week. > > > On Sep 1, 2016, at 8:44 AM, Ted Yu wrote: > > > > Andrew: > > HBASE-16255 has been resolved. > > > > Kindly provide your feedback. > > > > Thanks > > > > On Sat, Aug 20, 2016 at 11:06 AM, Andrew Purtell < > andrew.purt...@gmail.com> > > wrote: > > > >> I plan to spin up a test cluster with clusterdock and try running the IT > >> under a number of different scenarios. I understand snapshots have to > >> function so baseline would be the calm monkey. > >> > >> Unless you have some other automated way for me to run the new > >> functionality repeatedly, the IT is it. > >> > On Aug 20, 2016, at 10:59 AM, Vladimir Rodionov < > vladrodio...@gmail.com> > >>> wrote: > >>> > >>> Not sure what do you mean, Andrew by "trying out the branch via the > IT", > >>> but we do not recommend running this with monkey enabled. > >>> It has not been tested in a such scenario yet and frankly speaking it > is > >>> not supposed to work (snapshots will fail anyway and we depends on > >>> snapshots) > >>> > >>> -Vladimir > >>> > >>> On Sat, Aug 20, 2016 at 10:29 AM, Andrew Purtell < > >> andrew.purt...@gmail.com> > >>> wrote: > >>> > Let's commit the IT to the branch, if you think the v5 patch is ready > >> for > commit Ted. > > I will be able to spend some time next week trying out the branch via > >> the > IT, and poking around with the new tools. After that I feel like I'll > be > informed enough to vote on a branch merge vote. > > > On Aug 19, 2016, at 12:38 PM, Ted Yu wrote: > > > > IT test is provided on HBASE-16255. > > > > Any other comment ? > > > > Thanks > > > >> On Tue, Aug 2, 2016 at 9:09 PM, Dima Spivak > wrote: > >> > >> Any chance for an IT test being added to the branch first? I'd love > to > put > >> it through the paces with clusterdock to make sure it behaves well > >> with > >> fault injection and the like. > >> > >> -Dima > >> > >>> On Tuesday, August 2, 2016, Ted Yu wrote: > >>> > >>> Any more comments from the community on whether the merge can be > >> conducted > >>> ? > >>> > >>> Thanks > >>> > >>> On Mon, Aug 1, 2016 at 12:03 PM, Vladimir Rodionov < > >> vladrodio...@gmail.com > >>> > > >>> wrote: > >>> > Carter Shanklin posted a blog article about the feature: > Some use cases and examples of a command line interface usage. > >>> https://hortonworks.com/blog/coming-hdp-2-5-incremental- > >> backup-restore-apache-hbase-apache-phoenix/ > > -Vlad > > On Wed, Jul 20, 2016 at 1:25 PM, Vladimir Rodionov < > >>> vladrodio...@gmail.com > wrote: > > > Ok, got it. > > > > -Vlad > > > > On Wed, Jul 20, 2016 at 12:15 PM, Enis Söztutar >>> > wrote: > > > >> We keep the WALs which can accumulate a lot if the use case is > to > >> only > do > >> backups infrequently. This will definitely cause issues since > HDFS > >>> space > >> will get filled up. That is why we may need an option for having > >> incremental backups not used, and WAL references being deleted. > >> > >> Enis > >> > >> On Tue, Jul 19, 2016 at 6:33 PM, Vladimir Rodionov < > >> vladrodio...@gmail.com > > >> wrote: > >> > >>> Why anyone will ever need disabling incremental backups? If you > >> do > >>> not > >> need > >>> it - just run only full backups. > >>> > >>> -Vlad > >>> > >>> On Tue, Jul 19, 2016 at 6:21 PM, Enis Söztutar < > e...@apache.org > >>> > > wrote: > >>> > Thanks Matteo for chiming in. > > On Tue, Jul 19, 2016 at 5:02 PM, Matteo Bertozzi < > >>> theo.berto...@gmail.com > > wrote: > > > I did some review in the early beginning, but then lost track > >> of > the > > changes. > > but I'd like to give a quick review to the full code once > >> people > >> here > >>> are > > ok with getting this feature in master (2.0). > > (let say we put a deadline for reviews, like 1 week for > >>> reviewing > >> the > full > > stuff after everyone agrees to get this in. just to avoid > >>> holding > >> this > for > > too long, but still enough time to have people that are > >>> interested > >> to > look > > at it. with did the same thing for MOB with a mega patch > > https://reviews.apache.org/r/36391/) > >
Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
So, the answer to Sean's original question is "as robust as snapshots presently are"? (independence of backup/restore failure tolerance from snapshot failure tolerance) Is this just a question WRT context of the change, or is it means for a veto from you, Sean? Just trying to make sure I'm following along adequately. Vladimir Rodionov wrote: Snapshot robustness is better now with introduction of region splits/merges on/off feature. Region splits during snapshots was the major problem. -Vlad On Fri, Sep 2, 2016 at 8:12 AM, Vladimir Rodionov wrote: Are they independent enough that we can get backup/restore tolerant to failures prior to merge to master? Prior to backport to branch-1? As we stated already, snapshots are not part of the feature, snapshots has been merged into the master long time ago and as far as I understood - without requiring them to be 100% robust and fault tolerant and they are widely used in many production systems nevertheless. https://issues.apache.org/jira/browse/HBASE-14415 relies on Snapshots v2 but we can reconsider it, there are some thoughts how to make backups snapshotless. Backups are fault tolerant to some extent - in case of failure (and failures can happen) we clean everything up and do not leave system table in inconsistent state. Would it be enough, Sean Busbey? -Vlad On Fri, Sep 2, 2016 at 7:38 AM, Ted Yu wrote: We're continuing to make backup / restore more robust. Work in progress (both are close to being integrated): HBASE-15565 Rewrite restore with Procedure V2 HBASE-15449 Support physical table layout change Since snapshot is dependency in the full backup, backup / restore wouldn't be more robust than snapshot is. On Fri, Sep 2, 2016 at 7:03 AM, Sean Busbey wrote: right, they're separate features but when asked about "robust backup/restore" (which is what I care about for this feature getting merged) things were pawned off on snapshots. Are they independent enough that we can get backup/restore tolerant to failures prior to merge to master? Prior to backport to branch-1? On Thu, Sep 1, 2016 at 1:11 PM, Andrew Purtell wrote: I agree these are separate features FWIW On Thu, Sep 1, 2016 at 11:09 AM, Vladimir Rodionov< vladrodio...@gmail.com> wrote: Do we have JIRA issue(s) covering making snapshots robust in the face of monkeys? I would like to mention that "robust snapshots" and "table backup/restore" are totally separate features, but we have separate JIRA for fault tolerance (HBASE-14413). -Vlad On Thu, Sep 1, 2016 at 9:28 AM, Ted Yu wrote: Sean: Please see HBASE-14413 for the last question. FYI On Thu, Sep 1, 2016 at 9:24 AM, Sean Busbey wrote: On Sat, Aug 20, 2016 at 12:59 PM, Vladimir Rodionov wrote: Not sure what do you mean, Andrew by "trying out the branch via the IT", but we do not recommend running this with monkey enabled. It has not been tested in a such scenario yet and frankly speaking it is not supposed to work (snapshots will fail anyway and we depends on snapshots) Also won't have time to test out the branch this week, but if we're not going to handle failures do we have tools or guidance on recovering in the case of things falling over? Do we have JIRA issue(s) covering making snapshots robust in the face of monkeys? -- busbey -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) -- busbey
Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
Fine I'll cast a vote as -0. If I find time to test that could easily change to +1. Perhaps my vote won't be needed. I don't wish to block you. > On Sep 6, 2016, at 6:55 PM, Ted Yu wrote: > > Andrew: > Do you think you would have some time this week ? > > Thanks > > On Thu, Sep 1, 2016 at 8:47 AM, Andrew Purtell > wrote: > >> Busy at work, aiming for next week. >> >>> On Sep 1, 2016, at 8:44 AM, Ted Yu wrote: >>> >>> Andrew: >>> HBASE-16255 has been resolved. >>> >>> Kindly provide your feedback. >>> >>> Thanks >>> >>> On Sat, Aug 20, 2016 at 11:06 AM, Andrew Purtell < >> andrew.purt...@gmail.com> >>> wrote: >>> I plan to spin up a test cluster with clusterdock and try running the IT under a number of different scenarios. I understand snapshots have to function so baseline would be the calm monkey. Unless you have some other automated way for me to run the new functionality repeatedly, the IT is it. >> On Aug 20, 2016, at 10:59 AM, Vladimir Rodionov < >> vladrodio...@gmail.com> > wrote: > > Not sure what do you mean, Andrew by "trying out the branch via the >> IT", > but we do not recommend running this with monkey enabled. > It has not been tested in a such scenario yet and frankly speaking it >> is > not supposed to work (snapshots will fail anyway and we depends on > snapshots) > > -Vladimir > > On Sat, Aug 20, 2016 at 10:29 AM, Andrew Purtell < andrew.purt...@gmail.com> > wrote: > >> Let's commit the IT to the branch, if you think the v5 patch is ready for >> commit Ted. >> >> I will be able to spend some time next week trying out the branch via the >> IT, and poking around with the new tools. After that I feel like I'll >> be >> informed enough to vote on a branch merge vote. >> >>> On Aug 19, 2016, at 12:38 PM, Ted Yu wrote: >>> >>> IT test is provided on HBASE-16255. >>> >>> Any other comment ? >>> >>> Thanks >>> On Tue, Aug 2, 2016 at 9:09 PM, Dima Spivak >> wrote: Any chance for an IT test being added to the branch first? I'd love >> to >> put it through the paces with clusterdock to make sure it behaves well with fault injection and the like. -Dima > On Tuesday, August 2, 2016, Ted Yu wrote: > > Any more comments from the community on whether the merge can be conducted > ? > > Thanks > > On Mon, Aug 1, 2016 at 12:03 PM, Vladimir Rodionov < vladrodio...@gmail.com > > > wrote: > >> Carter Shanklin posted a blog article about the feature: >> Some use cases and examples of a command line interface usage. > https://hortonworks.com/blog/coming-hdp-2-5-incremental- backup-restore-apache-hbase-apache-phoenix/ >> >> -Vlad >> >> On Wed, Jul 20, 2016 at 1:25 PM, Vladimir Rodionov < > vladrodio...@gmail.com >> wrote: >> >>> Ok, got it. >>> >>> -Vlad >>> >>> On Wed, Jul 20, 2016 at 12:15 PM, Enis Söztutar > wrote: >>> We keep the WALs which can accumulate a lot if the use case is >> to only >> do backups infrequently. This will definitely cause issues since >> HDFS > space will get filled up. That is why we may need an option for having incremental backups not used, and WAL references being deleted. Enis On Tue, Jul 19, 2016 at 6:33 PM, Vladimir Rodionov < vladrodio...@gmail.com > wrote: > Why anyone will ever need disabling incremental backups? If you do > not need > it - just run only full backups. > > -Vlad > > On Tue, Jul 19, 2016 at 6:21 PM, Enis Söztutar < >> e...@apache.org > > >> wrote: > >> Thanks Matteo for chiming in. >> >> On Tue, Jul 19, 2016 at 5:02 PM, Matteo Bertozzi < > theo.berto...@gmail.com > >> wrote: >> >>> I did some review in the early beginning, but then lost track of >> the >>> changes. >>> but I'd like to give a quick review to the full code once people here > are >>> ok with getting this feature in master (2.0). >>> (let say we put a deadline for reviews, like 1 week for > reviewing the >> full >>> stuff after everyone agrees to get this in. just to avoid > holding this >> for >
[ANNOUNCE] Duo Zhang (张铎) joins the Apache HBase PMC
On behalf of the Apache HBase PMC I am pleased to announce that 张铎 has accepted our invitation to become a PMC member on the Apache HBase project. Duo has healthy notions on where the project should be headed and over the last year and more has been working furiously to take us there. Please join me in welcoming Duo to the HBase PMC! One of us! St.Ack
Re: [ANNOUNCE] Duo Zhang (张铎) joins the Apache HBase PMC
Congratulations, Duo. > On Sep 6, 2016, at 9:26 PM, Stack wrote: > > On behalf of the Apache HBase PMC I am pleased to announce that 张铎 > has accepted our invitation to become a PMC member on the Apache > HBase project. Duo has healthy notions on where the project should be > headed and over the last year and more has been working furiously to take > us there. > > Please join me in welcoming Duo to the HBase PMC! > > One of us! > St.Ack
Re: [ANNOUNCE] Duo Zhang (张铎) joins the Apache HBase PMC
Congratulations! > On Sep 6, 2016, at 9:26 PM, Stack wrote: > > On behalf of the Apache HBase PMC I am pleased to announce that 张铎 > has accepted our invitation to become a PMC member on the Apache > HBase project. Duo has healthy notions on where the project should be > headed and over the last year and more has been working furiously to take > us there. > > Please join me in welcoming Duo to the HBase PMC! > > One of us! > St.Ack
Re: [ANNOUNCE] Duo Zhang (张铎) joins the Apache HBase PMC
Yay Duo! On Tuesday, September 6, 2016, Stack wrote: > On behalf of the Apache HBase PMC I am pleased to announce that 张铎 > has accepted our invitation to become a PMC member on the Apache > HBase project. Duo has healthy notions on where the project should be > headed and over the last year and more has been working furiously to take > us there. > > Please join me in welcoming Duo to the HBase PMC! > > One of us! > St.Ack > -- -Dima
Re: [ANNOUNCE] Duo Zhang (张铎) joins the Apache HBase PMC
Congratulations~ -Shaohui Liu On 09/07/2016 12:26 PM, Stack wrote: On behalf of the Apache HBase PMC I am pleased to announce that 张铎 has accepted our invitation to become a PMC member on the Apache HBase project. Duo has healthy notions on where the project should be headed and over the last year and more has been working furiously to take us there. Please join me in welcoming Duo to the HBase PMC! One of us! St.Ack
Re: [ANNOUNCE] Duo Zhang (张铎) joins the Apache HBase PMC
Congrats Duo..!!! On Wed, Sep 7, 2016 at 10:16 AM, liushaohui wrote: > Congratulations~ > > -Shaohui Liu > > > On 09/07/2016 12:26 PM, Stack wrote: > >> On behalf of the Apache HBase PMC I am pleased to announce that 张铎 >> has accepted our invitation to become a PMC member on the Apache >> HBase project. Duo has healthy notions on where the project should be >> headed and over the last year and more has been working furiously to take >> us there. >> >> Please join me in welcoming Duo to the HBase PMC! >> >> One of us! >> St.Ack >> >> >
Re: [ANNOUNCE] Duo Zhang (张铎) joins the Apache HBase PMC
Congratulations! 2016-09-07 12:56 GMT+08:00 ramkrishna vasudevan < ramkrishna.s.vasude...@gmail.com>: > Congrats Duo..!!! > > On Wed, Sep 7, 2016 at 10:16 AM, liushaohui wrote: > > > Congratulations~ > > > > -Shaohui Liu > > > > > > On 09/07/2016 12:26 PM, Stack wrote: > > > >> On behalf of the Apache HBase PMC I am pleased to announce that 张铎 > >> has accepted our invitation to become a PMC member on the Apache > >> HBase project. Duo has healthy notions on where the project should be > >> headed and over the last year and more has been working furiously to > take > >> us there. > >> > >> Please join me in welcoming Duo to the HBase PMC! > >> > >> One of us! > >> St.Ack > >> > >> > > >
Re: [ANNOUNCE] Duo Zhang (张铎) joins the Apache HBase PMC
Congratulations, 张铎！ Excellent work. Jerry On Sep 6, 2016 9:57 PM, "Guanghao Zhang" wrote: > Congratulations! > > 2016-09-07 12:56 GMT+08:00 ramkrishna vasudevan < > ramkrishna.s.vasude...@gmail.com>: > > > Congrats Duo..!!! > > > > On Wed, Sep 7, 2016 at 10:16 AM, liushaohui > wrote: > > > > > Congratulations~ > > > > > > -Shaohui Liu > > > > > > > > > On 09/07/2016 12:26 PM, Stack wrote: > > > > > >> On behalf of the Apache HBase PMC I am pleased to announce that 张铎 > > >> has accepted our invitation to become a PMC member on the Apache > > >> HBase project. Duo has healthy notions on where the project should be > > >> headed and over the last year and more has been working furiously to > > take > > >> us there. > > >> > > >> Please join me in welcoming Duo to the HBase PMC! > > >> > > >> One of us! > > >> St.Ack > > >> > > >> > > > > > >
RE: [ANNOUNCE] Duo Zhang (张铎) joins the Apache HBase PMC
Congratulations! -Original Message- From: saint@gmail.com [mailto:saint@gmail.com] On Behalf Of Stack Sent: 07 September 2016 09:56 To: HBase Dev List; Hbase-User Subject: [ANNOUNCE] Duo Zhang (张铎) joins the Apache HBase PMC On behalf of the Apache HBase PMC I am pleased to announce that 张铎 has accepted our invitation to become a PMC member on the Apache HBase project. Duo has healthy notions on where the project should be headed and over the last year and more has been working furiously to take us there. Please join me in welcoming Duo to the HBase PMC! One of us! St.Ack
答复: [ANNOUNCE] Duo Zhang (张铎) joins the Apache HBase PMC
Congratulations, Duo. 发件人: Ted Yu 发送时间: 2016年9月7日 12:31 收件人: u...@hbase.apache.org 抄送: HBase Dev List 主题: Re: [ANNOUNCE] Duo Zhang (张铎) joins the Apache HBase PMC Congratulations, Duo. > On Sep 6, 2016, at 9:26 PM, Stack wrote: > > On behalf of the Apache HBase PMC I am pleased to announce that 张铎 > has accepted our invitation to become a PMC member on the Apache > HBase project. Duo has healthy notions on where the project should be > headed and over the last year and more has been working furiously to take > us there. > > Please join me in welcoming Duo to the HBase PMC! > > One of us! > St.Ack
Re: 答复: [ANNOUNCE] Duo Zhang (张铎) joins the Apache HBase PMC
Thanks all. Will do my best. 2016-09-07 13:58 GMT+08:00 Honghua Feng 冯宏华 : > Congratulations, Duo. > > 发件人: Ted Yu > 发送时间: 2016年9月7日 12:31 > 收件人: u...@hbase.apache.org > 抄送: HBase Dev List > 主题: Re: [ANNOUNCE] Duo Zhang (张铎) joins the Apache HBase PMC > > Congratulations, Duo. > > > On Sep 6, 2016, at 9:26 PM, Stack wrote: > > > > On behalf of the Apache HBase PMC I am pleased to announce that 张铎 > > has accepted our invitation to become a PMC member on the Apache > > HBase project. Duo has healthy notions on where the project should be > > headed and over the last year and more has been working furiously to take > > us there. > > > > Please join me in welcoming Duo to the HBase PMC! > > > > One of us! > > St.Ack >
Re: 答复: [ANNOUNCE] Duo Zhang (张铎) joins the Apache HBase PMC
Congratulations! -Mikhail On Tue, Sep 6, 2016 at 10:58 PM, Honghua Feng 冯宏华 wrote: > Congratulations, Duo. > > 发件人: Ted Yu > 发送时间: 2016年9月7日 12:31 > 收件人: u...@hbase.apache.org > 抄送: HBase Dev List > 主题: Re: [ANNOUNCE] Duo Zhang (张铎) joins the Apache HBase PMC > > Congratulations, Duo. > > > On Sep 6, 2016, at 9:26 PM, Stack wrote: > > > > On behalf of the Apache HBase PMC I am pleased to announce that 张铎 > > has accepted our invitation to become a PMC member on the Apache > > HBase project. Duo has healthy notions on where the project should be > > headed and over the last year and more has been working furiously to take > > us there. > > > > Please join me in welcoming Duo to the HBase PMC! > > > > One of us! > > St.Ack > -- Thanks, Michael Antonov
