Hadoop 3.x profile working for hbase 2.0 [Re: HBASE 2.0]
I'd like to get the -Dhadoop.profile=3.0 at least to compiling and passing licensing working for the first hbase alpha (or whatever we end up calling it) I'll propose these items: 1) peg to one of the recent hadoop alphas (hadoop 3.0.0-alpha1 is the most recent). currently we are against 3.0.0-SNAPSHOT. 2) add precompile checks against a hadoop 3.x (HBASE-16733) 3) get 'mvn test install -Dskiptests' to succeed without licensing issues (HBASE-16712) 4) Have a job setup in jenkins so that we can gain insight and burn down unit tests failures against hadoop3. These items have a good chance of landing in the next week or two. Other related issues that are nice to have but wouldn't block an hbase alpha include: 1) having no always failing unit tests against hadoop3 (HBASE-6581) Thoughts? Jon. On Mon, Oct 3, 2016 at 3:27 PM, Stephen Jiang wrote: > Hello, All, > > It is time to discuss about the schedule of HBase 2.0 release. HBase 2.0 > release is a big major release. When we release 1.0, we had 0.99 as dev > preview/beta release. We should do something similar for the 2.0 release. > > Matteo and I talked about this. We think about that we need some > Alpha/Beta milestones before the RC and final Release-to-Web 2.0 release. > > I don't know whether there is any discussion on this community about the > Alpha/Beta release criteria. My proposal is that once we cut the branch-2, > we should only have new features that are absolutely needed for major > release (festures cannot be added in minor release) and those features > should be "almost ready". For Alpha releases, we can still accept these > kind of features; for Beta release, only bug fixes and performance > improvement on existing features (should we also accept existing feature > improvement in Beta? Maybe Beta 1, Not in Beta 2 - that is my take). > > This is a big release and requires a lot of work from Release Manager. I > asked Matteo whether I could help to be some kind of backup / hot-standby / > assistant RM. I think he is very happy to have someone to share some RM > duties. Thus, I will help make this 2.0 release as smooth as possible. > > Here is a tentative plan: > - For now, we are thinking of creating branch-2 middle of this month and > have 2.0-Alpha1 release at the end of this month or begin of Nov. The > definition of Alpha1 is that we could deploy to a cluster and it can do > some simple CRUD and table DDLs; and not crash (of course, UT passing). > > - Then we will have 2.0-Alpha2 in 4-6 weeks after Apha1. It would hold > higher bar. We will run some IT tests to make sure that it would > functional. > > - At that time, we will lock down and not allow any new features, every 4-6 > weeks, we will have a Beta release (my realistic guess is that we would hit > the US Christmas holiday at that time, so first Beta would take longer than > 6 weeks). For Beta release, we would fix bugs and do performance tuning. > Planning to have 2 Betas. (Question: in the past, do we need vote to have > a Beta release?) > > - Once the code are in the stable stage, then we will have RCs and vote for > the final release. > > Please let us know whether this is a reasonable approach that will make the > release successful. > > Currently, we are aware of the following on-going new features for 2.0: new > Assignment Manager, backup/restore, off-heap, protobuff 3, Hybrid Logical > Clock, and maybe AsyncRegion / C++ client). If you have a feature that > wants to be part of 2.0 release, please send discussion to dev community > and we can make a call on accepting/rejecting. > > Thanks > Stephen > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh
[jira] [Resolved] (HBASE-16372) References to previous cell in read path should be avoided
[ https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan resolved HBASE-16372. Resolution: Fixed > References to previous cell in read path should be avoided > -- > > Key: HBASE-16372 > URL: https://issues.apache.org/jira/browse/HBASE-16372 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16372_1.patch, HBASE-16372_10.patch, > HBASE-16372_2.patch, HBASE-16372_3.patch, HBASE-16372_5_withfactory.patch, > HBASE-16372_6_withfactory.patch, HBASE-16372_7_withfactory.patch, > HBASE-16372_9.patch, HBASE-16372_testcase.patch, HBASE-16372_testcase_1.patch > > > Came as part of review discussion in HBASE-15554. If there are references > kept to previous cells in the read path, with the Ref count based eviction > mechanism in trunk, then chances are there to evict a block backing the > previous cell but the read path still does some operations on that garbage > collected previous cell leading to incorrect results. > Areas to target > -> Storescanner > -> Bloom filters (particularly in compaction path) > Thanks to [~anoop.hbase] to point out this in bloomfilter path. But we found > it could be in other areas also. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16782) Create compaction specific writer that are of type ShipperListener
ramkrishna.s.vasudevan created HBASE-16782: -- Summary: Create compaction specific writer that are of type ShipperListener Key: HBASE-16782 URL: https://issues.apache.org/jira/browse/HBASE-16782 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Minor Fix For: 2.0.0 In HBASE-16372, there was a discussion around having Compaction specific writers so that only those writers are of type ShipperListener. Though some of the patches attached there was able to do it, still changin StoreFileWriter was affecting MOB and MR as the builder inside it was private. This improvement JIRA is to ensure all we create compaction specific writers. For now only the compaciton flow calls ShipperListener#beforeShipped() so should be fine as of now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16783) Use ByteBufferPool for the header and message during Rpc response
ramkrishna.s.vasudevan created HBASE-16783: -- Summary: Use ByteBufferPool for the header and message during Rpc response Key: HBASE-16783 URL: https://issues.apache.org/jira/browse/HBASE-16783 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Minor With ByteBufferPool in place we could avoid the byte[] creation in RpcServer#createHeaderAndMessageBytes and try using the Buffer from the pool rather than creating byte[] every time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: HBASE 2.0
On Tue, Oct 4, 2016 at 12:00 PM, Andrew Purtell wrote: >> Currently, we are aware of the following on-going new features for 2.0: > new Assignment Manager, backup/restore, off-heap, protobuff 3, Hybrid > Logical Clock, and maybe AsyncRegion / C++ client). > > For what it is worth, depending on the state of them, I would really like > to see the new AM, incremental backup/restore (with changes as discussed by > the community), PB3, and HLC. Let's see if it is possible to get them all > in and produce a high quality stable release. Implicitly: A simple 1.x to > 2.x upgrade path with bulletproof rollback. I'd like to get our use of HDFS cleaned up in time for 2.0; so far it looks like it'll probably impact some of our operational tooling. I don't think it ought to be a blocker, so I'm aiming for the something later than the first round of early-releases (alpha/beta/whatever we're calling it). -- busbey
Successful: HBase Generate Website
Build status: Successful
If successful, the website and docs have been generated. To update the live
site, follow the instructions below. If failed, skip to the bottom of this
email.
Use the following commands to download the patch and apply it to a clean branch
based on origin/asf-site. If you prefer to keep the hbase-site repo around
permanently, you can skip the clone step.
git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git
cd hbase-site
wget -O-
https://builds.apache.org/job/hbase_generate_website/368/artifact/website.patch.zip
| funzip > 7a38509782ba571cf6b4a5c02ed0ce693ae88ad7.patch
git fetch
git checkout -b asf-site-7a38509782ba571cf6b4a5c02ed0ce693ae88ad7
origin/asf-site
git am --whitespace=fix 7a38509782ba571cf6b4a5c02ed0ce693ae88ad7.patch
At this point, you can preview the changes by opening index.html or any of the
other HTML pages in your local
asf-site-7a38509782ba571cf6b4a5c02ed0ce693ae88ad7 branch.
There are lots of spurious changes, such as timestamps and CSS styles in
tables, so a generic git diff is not very useful. To see a list of files that
have been added, deleted, renamed, changed type, or are otherwise interesting,
use the following command:
git diff --name-status --diff-filter=ADCRTXUB origin/asf-site
To see only files that had 100 or more lines changed:
git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'
When you are satisfied, publish your changes to origin/asf-site using these
commands:
git commit --allow-empty -m "Empty commit" # to work around a current ASF
INFRA bug
git push origin asf-site-7a38509782ba571cf6b4a5c02ed0ce693ae88ad7:asf-site
git checkout asf-site
git branch -D asf-site-7a38509782ba571cf6b4a5c02ed0ce693ae88ad7
Changes take a couple of minutes to be propagated. You can verify whether they
have been propagated by looking at the Last Published date at the bottom of
http://hbase.apache.org/. It should match the date in the index.html on the
asf-site branch in Git.
As a courtesy- reply-all to this email to let other committers know you pushed
the site.
If failed, see https://builds.apache.org/job/hbase_generate_website/368/console
Issues while running HBase native client build (Ref: HBASE-14850)
Hi Team, I am facing segfaults while running the simple-client present in hbase-native-client. As per the build steps, when I executed the binary from the docker environemnt, I get the foll error:- 2016-10-06 10:08:53,536:93(0x7f6cc256a900):ZOO_ERROR@getaddrs@613: getaddrinfo: No such file or directory So, I tried installing all the dependencies on my local environment and tried running the simple-client on it, instead of docker environment. This time I got a different segfault as below. io/async/AsyncSocket.cpp:875: virtual void folly::AsyncSocket::closeNow(): Assertion `eventBase_ == nullptr || eventBase_->isInEventBaseThread()' failed. Can you please help me with some pointers. -- Warm Regards, Sudeep Sunthankar
Re: Hadoop 3.x profile working for hbase 2.0 [Re: HBASE 2.0]
Jon: Once the goals you outlined below are achieved, would user be able to use build artifacts compiled against hadoop 2.7.1 on a cluster deployed with hadoop 3.0.0-alpha1 ? Cheers On Thu, Oct 6, 2016 at 12:07 AM, Jonathan Hsieh wrote: > I'd like to get the -Dhadoop.profile=3.0 at least to compiling and passing > licensing working for the first hbase alpha (or whatever we end up calling > it) > > I'll propose these items: > 1) peg to one of the recent hadoop alphas (hadoop 3.0.0-alpha1 is the most > recent). currently we are against 3.0.0-SNAPSHOT. > 2) add precompile checks against a hadoop 3.x (HBASE-16733) > 3) get 'mvn test install -Dskiptests' to succeed without licensing issues > (HBASE-16712) > 4) Have a job setup in jenkins so that we can gain insight and burn down > unit tests failures against hadoop3. > > These items have a good chance of landing in the next week or two. > > Other related issues that are nice to have but wouldn't block an hbase > alpha include: > 1) having no always failing unit tests against hadoop3 (HBASE-6581) > > Thoughts? > Jon. > > On Mon, Oct 3, 2016 at 3:27 PM, Stephen Jiang > wrote: > > > Hello, All, > > > > It is time to discuss about the schedule of HBase 2.0 release. HBase 2.0 > > release is a big major release. When we release 1.0, we had 0.99 as dev > > preview/beta release. We should do something similar for the 2.0 > release. > > > > Matteo and I talked about this. We think about that we need some > > Alpha/Beta milestones before the RC and final Release-to-Web 2.0 release. > > > > I don't know whether there is any discussion on this community about the > > Alpha/Beta release criteria. My proposal is that once we cut the > branch-2, > > we should only have new features that are absolutely needed for major > > release (festures cannot be added in minor release) and those features > > should be "almost ready". For Alpha releases, we can still accept these > > kind of features; for Beta release, only bug fixes and performance > > improvement on existing features (should we also accept existing feature > > improvement in Beta? Maybe Beta 1, Not in Beta 2 - that is my take). > > > > This is a big release and requires a lot of work from Release Manager. I > > asked Matteo whether I could help to be some kind of backup / > hot-standby / > > assistant RM. I think he is very happy to have someone to share some RM > > duties. Thus, I will help make this 2.0 release as smooth as possible. > > > > Here is a tentative plan: > > - For now, we are thinking of creating branch-2 middle of this month and > > have 2.0-Alpha1 release at the end of this month or begin of Nov. The > > definition of Alpha1 is that we could deploy to a cluster and it can do > > some simple CRUD and table DDLs; and not crash (of course, UT passing). > > > > - Then we will have 2.0-Alpha2 in 4-6 weeks after Apha1. It would hold > > higher bar. We will run some IT tests to make sure that it would > > functional. > > > > - At that time, we will lock down and not allow any new features, every > 4-6 > > weeks, we will have a Beta release (my realistic guess is that we would > hit > > the US Christmas holiday at that time, so first Beta would take longer > than > > 6 weeks). For Beta release, we would fix bugs and do performance tuning. > > Planning to have 2 Betas. (Question: in the past, do we need vote to > have > > a Beta release?) > > > > - Once the code are in the stable stage, then we will have RCs and vote > for > > the final release. > > > > Please let us know whether this is a reasonable approach that will make > the > > release successful. > > > > Currently, we are aware of the following on-going new features for 2.0: > new > > Assignment Manager, backup/restore, off-heap, protobuff 3, Hybrid Logical > > Clock, and maybe AsyncRegion / C++ client). If you have a feature that > > wants to be part of 2.0 release, please send discussion to dev community > > and we can make a call on accepting/rejecting. > > > > Thanks > > Stephen > > > > > > -- > // Jonathan Hsieh (shay) > // HBase Tech Lead, Software Engineer, Cloudera > // j...@cloudera.com // @jmhsieh >
[jira] [Created] (HBASE-16784) Make use of ExtendedCell#write(OutputStream os) for the default HFileWriter#append()
ramkrishna.s.vasudevan created HBASE-16784: -- Summary: Make use of ExtendedCell#write(OutputStream os) for the default HFileWriter#append() Key: HBASE-16784 URL: https://issues.apache.org/jira/browse/HBASE-16784 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Initially this I was thinking we need to add an interface to represent the fact that the key is contiguous. But since Extendedcell is added encapsulating all the internal interfaces and adds a write(OutputStream , boolean) and tries to exploit the fact that the Cell is in KV serialized format. Hence we can make use of it in HFileWriter#append() code in case of No encoding case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Hadoop 3.x profile working for hbase 2.0 [Re: HBASE 2.0]
Yes. The goal is to produce one set of hbase binaries that will work against multiple hadoops such as the 2.7 and 3.0.0-alpha1 versions, but preferentially tested against and likely including binaries from a stable hadoop version. Up until recently, compiling against the hadoop 3 profile fails because of compilation issues and licensing issues, Another issue, HBASE-16711 has already landed which fixed compilation against hadoop2 and hadoop3. What remains is on the short proposed list makes sure licensing enforcers are satisfied correctly and getting build infrastructure precommit checks in place so we don't inadvertently introduce new problems. Jon. On Thu, Oct 6, 2016 at 9:02 AM, Ted Yu wrote: > Jon: > Once the goals you outlined below are achieved, would user be able to use > build artifacts compiled against hadoop 2.7.1 on a cluster deployed with > hadoop > 3.0.0-alpha1 ? > > Cheers > > On Thu, Oct 6, 2016 at 12:07 AM, Jonathan Hsieh wrote: > > > I'd like to get the -Dhadoop.profile=3.0 at least to compiling and > passing > > licensing working for the first hbase alpha (or whatever we end up > calling > > it) > > > > I'll propose these items: > > 1) peg to one of the recent hadoop alphas (hadoop 3.0.0-alpha1 is the > most > > recent). currently we are against 3.0.0-SNAPSHOT. > > 2) add precompile checks against a hadoop 3.x (HBASE-16733) > > 3) get 'mvn test install -Dskiptests' to succeed without licensing issues > > (HBASE-16712) > > 4) Have a job setup in jenkins so that we can gain insight and burn down > > unit tests failures against hadoop3. > > > > These items have a good chance of landing in the next week or two. > > > > Other related issues that are nice to have but wouldn't block an hbase > > alpha include: > > 1) having no always failing unit tests against hadoop3 (HBASE-6581) > > > > Thoughts? > > Jon. > > > > On Mon, Oct 3, 2016 at 3:27 PM, Stephen Jiang > > wrote: > > > > > Hello, All, > > > > > > It is time to discuss about the schedule of HBase 2.0 release. HBase > 2.0 > > > release is a big major release. When we release 1.0, we had 0.99 as > dev > > > preview/beta release. We should do something similar for the 2.0 > > release. > > > > > > Matteo and I talked about this. We think about that we need some > > > Alpha/Beta milestones before the RC and final Release-to-Web 2.0 > release. > > > > > > I don't know whether there is any discussion on this community about > the > > > Alpha/Beta release criteria. My proposal is that once we cut the > > branch-2, > > > we should only have new features that are absolutely needed for major > > > release (festures cannot be added in minor release) and those features > > > should be "almost ready". For Alpha releases, we can still accept > these > > > kind of features; for Beta release, only bug fixes and performance > > > improvement on existing features (should we also accept existing > feature > > > improvement in Beta? Maybe Beta 1, Not in Beta 2 - that is my take). > > > > > > This is a big release and requires a lot of work from Release > Manager. I > > > asked Matteo whether I could help to be some kind of backup / > > hot-standby / > > > assistant RM. I think he is very happy to have someone to share some > RM > > > duties. Thus, I will help make this 2.0 release as smooth as possible. > > > > > > Here is a tentative plan: > > > - For now, we are thinking of creating branch-2 middle of this month > and > > > have 2.0-Alpha1 release at the end of this month or begin of Nov. The > > > definition of Alpha1 is that we could deploy to a cluster and it can do > > > some simple CRUD and table DDLs; and not crash (of course, UT passing). > > > > > > - Then we will have 2.0-Alpha2 in 4-6 weeks after Apha1. It would hold > > > higher bar. We will run some IT tests to make sure that it would > > > functional. > > > > > > - At that time, we will lock down and not allow any new features, every > > 4-6 > > > weeks, we will have a Beta release (my realistic guess is that we would > > hit > > > the US Christmas holiday at that time, so first Beta would take longer > > than > > > 6 weeks). For Beta release, we would fix bugs and do performance > tuning. > > > Planning to have 2 Betas. (Question: in the past, do we need vote to > > have > > > a Beta release?) > > > > > > - Once the code are in the stable stage, then we will have RCs and vote > > for > > > the final release. > > > > > > Please let us know whether this is a reasonable approach that will make > > the > > > release successful. > > > > > > Currently, we are aware of the following on-going new features for 2.0: > > new > > > Assignment Manager, backup/restore, off-heap, protobuff 3, Hybrid > Logical > > > Clock, and maybe AsyncRegion / C++ client). If you have a feature that > > > wants to be part of 2.0 release, please send discussion to dev > community > > > and we can make a call on accepting/rejecting. > > > > > > Thanks > > > Stephen > > > > > > > > > > > -- > > // Jo
Re: Hadoop 3.x profile working for hbase 2.0 [Re: HBASE 2.0]
bq. one set of hbase binaries that will work against multiple hadoops I would be interested to know what tests are / will be performed against 3.0.0-alpha1 (using artifacts built against 2.7.1). Thanks On Thu, Oct 6, 2016 at 10:41 AM, Jonathan Hsieh wrote: > Yes. > > The goal is to produce one set of hbase binaries that will work against > multiple hadoops such as the 2.7 and 3.0.0-alpha1 versions, but > preferentially tested against and likely including binaries from a stable > hadoop version. > > Up until recently, compiling against the hadoop 3 profile fails because of > compilation issues and licensing issues, Another issue, HBASE-16711 has > already landed which fixed compilation against hadoop2 and hadoop3. What > remains is on the short proposed list makes sure licensing enforcers are > satisfied correctly and getting build infrastructure precommit checks in > place so we don't inadvertently introduce new problems. > > Jon. > > On Thu, Oct 6, 2016 at 9:02 AM, Ted Yu wrote: > > > Jon: > > Once the goals you outlined below are achieved, would user be able to use > > build artifacts compiled against hadoop 2.7.1 on a cluster deployed with > > hadoop > > 3.0.0-alpha1 ? > > > > Cheers > > > > On Thu, Oct 6, 2016 at 12:07 AM, Jonathan Hsieh > wrote: > > > > > I'd like to get the -Dhadoop.profile=3.0 at least to compiling and > > passing > > > licensing working for the first hbase alpha (or whatever we end up > > calling > > > it) > > > > > > I'll propose these items: > > > 1) peg to one of the recent hadoop alphas (hadoop 3.0.0-alpha1 is the > > most > > > recent). currently we are against 3.0.0-SNAPSHOT. > > > 2) add precompile checks against a hadoop 3.x (HBASE-16733) > > > 3) get 'mvn test install -Dskiptests' to succeed without licensing > issues > > > (HBASE-16712) > > > 4) Have a job setup in jenkins so that we can gain insight and burn > down > > > unit tests failures against hadoop3. > > > > > > These items have a good chance of landing in the next week or two. > > > > > > Other related issues that are nice to have but wouldn't block an hbase > > > alpha include: > > > 1) having no always failing unit tests against hadoop3 (HBASE-6581) > > > > > > Thoughts? > > > Jon. > > > > > > On Mon, Oct 3, 2016 at 3:27 PM, Stephen Jiang > > > > wrote: > > > > > > > Hello, All, > > > > > > > > It is time to discuss about the schedule of HBase 2.0 release. HBase > > 2.0 > > > > release is a big major release. When we release 1.0, we had 0.99 as > > dev > > > > preview/beta release. We should do something similar for the 2.0 > > > release. > > > > > > > > Matteo and I talked about this. We think about that we need some > > > > Alpha/Beta milestones before the RC and final Release-to-Web 2.0 > > release. > > > > > > > > I don't know whether there is any discussion on this community about > > the > > > > Alpha/Beta release criteria. My proposal is that once we cut the > > > branch-2, > > > > we should only have new features that are absolutely needed for major > > > > release (festures cannot be added in minor release) and those > features > > > > should be "almost ready". For Alpha releases, we can still accept > > these > > > > kind of features; for Beta release, only bug fixes and performance > > > > improvement on existing features (should we also accept existing > > feature > > > > improvement in Beta? Maybe Beta 1, Not in Beta 2 - that is my take). > > > > > > > > This is a big release and requires a lot of work from Release > > Manager. I > > > > asked Matteo whether I could help to be some kind of backup / > > > hot-standby / > > > > assistant RM. I think he is very happy to have someone to share some > > RM > > > > duties. Thus, I will help make this 2.0 release as smooth as > possible. > > > > > > > > Here is a tentative plan: > > > > - For now, we are thinking of creating branch-2 middle of this month > > and > > > > have 2.0-Alpha1 release at the end of this month or begin of Nov. > The > > > > definition of Alpha1 is that we could deploy to a cluster and it can > do > > > > some simple CRUD and table DDLs; and not crash (of course, UT > passing). > > > > > > > > - Then we will have 2.0-Alpha2 in 4-6 weeks after Apha1. It would > hold > > > > higher bar. We will run some IT tests to make sure that it would > > > > functional. > > > > > > > > - At that time, we will lock down and not allow any new features, > every > > > 4-6 > > > > weeks, we will have a Beta release (my realistic guess is that we > would > > > hit > > > > the US Christmas holiday at that time, so first Beta would take > longer > > > than > > > > 6 weeks). For Beta release, we would fix bugs and do performance > > tuning. > > > > Planning to have 2 Betas. (Question: in the past, do we need vote to > > > have > > > > a Beta release?) > > > > > > > > - Once the code are in the stable stage, then we will have RCs and > vote > > > for > > > > the final release. > > > > > > > > Please let us know whether this is a r
Re: Hadoop 3.x profile working for hbase 2.0 [Re: HBASE 2.0]
Again my goal is to have the hadoop 3 profile that already exist provide some signal and not be broken as it is currently. There are are other options but this one seems reasonable since hadoop3 releases are starting to show up. The milestone I propose is it compiles properly, and it passes our licensing enforcers. A further milestone for another release is passing unit tests. This is essentially no different for what we do for all the different hadoop version (2.6.x 2.7.x etc we support hbase 1.x on.), or the different builds with the different jdks. Jon. On Thu, Oct 6, 2016 at 10:47 AM, Ted Yu wrote: > bq. one set of hbase binaries that will work against multiple hadoops > > I would be interested to know what tests are / will be performed > against 3.0.0-alpha1 > (using artifacts built against 2.7.1). > > Thanks > > > On Thu, Oct 6, 2016 at 10:41 AM, Jonathan Hsieh wrote: > > > Yes. > > > > The goal is to produce one set of hbase binaries that will work against > > multiple hadoops such as the 2.7 and 3.0.0-alpha1 versions, but > > preferentially tested against and likely including binaries from a stable > > hadoop version. > > > > Up until recently, compiling against the hadoop 3 profile fails because > of > > compilation issues and licensing issues, Another issue, HBASE-16711 has > > already landed which fixed compilation against hadoop2 and hadoop3. What > > remains is on the short proposed list makes sure licensing enforcers are > > satisfied correctly and getting build infrastructure precommit checks in > > place so we don't inadvertently introduce new problems. > > > > Jon. > > > > On Thu, Oct 6, 2016 at 9:02 AM, Ted Yu wrote: > > > > > Jon: > > > Once the goals you outlined below are achieved, would user be able to > use > > > build artifacts compiled against hadoop 2.7.1 on a cluster deployed > with > > > hadoop > > > 3.0.0-alpha1 ? > > > > > > Cheers > > > > > > On Thu, Oct 6, 2016 at 12:07 AM, Jonathan Hsieh > > wrote: > > > > > > > I'd like to get the -Dhadoop.profile=3.0 at least to compiling and > > > passing > > > > licensing working for the first hbase alpha (or whatever we end up > > > calling > > > > it) > > > > > > > > I'll propose these items: > > > > 1) peg to one of the recent hadoop alphas (hadoop 3.0.0-alpha1 is the > > > most > > > > recent). currently we are against 3.0.0-SNAPSHOT. > > > > 2) add precompile checks against a hadoop 3.x (HBASE-16733) > > > > 3) get 'mvn test install -Dskiptests' to succeed without licensing > > issues > > > > (HBASE-16712) > > > > 4) Have a job setup in jenkins so that we can gain insight and burn > > down > > > > unit tests failures against hadoop3. > > > > > > > > These items have a good chance of landing in the next week or two. > > > > > > > > Other related issues that are nice to have but wouldn't block an > hbase > > > > alpha include: > > > > 1) having no always failing unit tests against hadoop3 (HBASE-6581) > > > > > > > > Thoughts? > > > > Jon. > > > > > > > > On Mon, Oct 3, 2016 at 3:27 PM, Stephen Jiang < > syuanjiang...@gmail.com > > > > > > > wrote: > > > > > > > > > Hello, All, > > > > > > > > > > It is time to discuss about the schedule of HBase 2.0 release. > HBase > > > 2.0 > > > > > release is a big major release. When we release 1.0, we had 0.99 > as > > > dev > > > > > preview/beta release. We should do something similar for the 2.0 > > > > release. > > > > > > > > > > Matteo and I talked about this. We think about that we need some > > > > > Alpha/Beta milestones before the RC and final Release-to-Web 2.0 > > > release. > > > > > > > > > > I don't know whether there is any discussion on this community > about > > > the > > > > > Alpha/Beta release criteria. My proposal is that once we cut the > > > > branch-2, > > > > > we should only have new features that are absolutely needed for > major > > > > > release (festures cannot be added in minor release) and those > > features > > > > > should be "almost ready". For Alpha releases, we can still accept > > > these > > > > > kind of features; for Beta release, only bug fixes and performance > > > > > improvement on existing features (should we also accept existing > > > feature > > > > > improvement in Beta? Maybe Beta 1, Not in Beta 2 - that is my > take). > > > > > > > > > > This is a big release and requires a lot of work from Release > > > Manager. I > > > > > asked Matteo whether I could help to be some kind of backup / > > > > hot-standby / > > > > > assistant RM. I think he is very happy to have someone to share > some > > > RM > > > > > duties. Thus, I will help make this 2.0 release as smooth as > > possible. > > > > > > > > > > Here is a tentative plan: > > > > > - For now, we are thinking of creating branch-2 middle of this > month > > > and > > > > > have 2.0-Alpha1 release at the end of this month or begin of Nov. > > The > > > > > definition of Alpha1 is that we could deploy to a cluster and it > can > > do > > > > > some simple CRUD and table D
Re: Hadoop 3.x profile working for hbase 2.0 [Re: HBASE 2.0]
I assume we will have matrix testing of HBase versions against designated upstream build targets for Hadoop 2.x and Hadoop 3.x? If not, if not too much trouble would be a good idea. We're ad hoc about checking if a commit to 0.98 breaks the Hadoop 1 build. Every few release candidates I have to commit addendums to fix as part of the RC process. For what it's worth. On Thu, Oct 6, 2016 at 11:58 AM, Jonathan Hsieh wrote: > Again my goal is to have the hadoop 3 profile that already exist provide > some signal and not be broken as it is currently. There are are other > options but this one seems reasonable since hadoop3 releases are starting > to show up. > > The milestone I propose is it compiles properly, and it passes our > licensing enforcers. A further milestone for another release is passing > unit tests. > > This is essentially no different for what we do for all the different > hadoop version (2.6.x 2.7.x etc we support hbase 1.x on.), or the different > builds with the different jdks. > > Jon. > > On Thu, Oct 6, 2016 at 10:47 AM, Ted Yu wrote: > > > bq. one set of hbase binaries that will work against multiple hadoops > > > > I would be interested to know what tests are / will be performed > > against 3.0.0-alpha1 > > (using artifacts built against 2.7.1). > > > > Thanks > > > > > > On Thu, Oct 6, 2016 at 10:41 AM, Jonathan Hsieh > wrote: > > > > > Yes. > > > > > > The goal is to produce one set of hbase binaries that will work against > > > multiple hadoops such as the 2.7 and 3.0.0-alpha1 versions, but > > > preferentially tested against and likely including binaries from a > stable > > > hadoop version. > > > > > > Up until recently, compiling against the hadoop 3 profile fails because > > of > > > compilation issues and licensing issues, Another issue, HBASE-16711 > has > > > already landed which fixed compilation against hadoop2 and hadoop3. > What > > > remains is on the short proposed list makes sure licensing enforcers > are > > > satisfied correctly and getting build infrastructure precommit checks > in > > > place so we don't inadvertently introduce new problems. > > > > > > Jon. > > > > > > On Thu, Oct 6, 2016 at 9:02 AM, Ted Yu wrote: > > > > > > > Jon: > > > > Once the goals you outlined below are achieved, would user be able to > > use > > > > build artifacts compiled against hadoop 2.7.1 on a cluster deployed > > with > > > > hadoop > > > > 3.0.0-alpha1 ? > > > > > > > > Cheers > > > > > > > > On Thu, Oct 6, 2016 at 12:07 AM, Jonathan Hsieh > > > wrote: > > > > > > > > > I'd like to get the -Dhadoop.profile=3.0 at least to compiling and > > > > passing > > > > > licensing working for the first hbase alpha (or whatever we end up > > > > calling > > > > > it) > > > > > > > > > > I'll propose these items: > > > > > 1) peg to one of the recent hadoop alphas (hadoop 3.0.0-alpha1 is > the > > > > most > > > > > recent). currently we are against 3.0.0-SNAPSHOT. > > > > > 2) add precompile checks against a hadoop 3.x (HBASE-16733) > > > > > 3) get 'mvn test install -Dskiptests' to succeed without licensing > > > issues > > > > > (HBASE-16712) > > > > > 4) Have a job setup in jenkins so that we can gain insight and burn > > > down > > > > > unit tests failures against hadoop3. > > > > > > > > > > These items have a good chance of landing in the next week or two. > > > > > > > > > > Other related issues that are nice to have but wouldn't block an > > hbase > > > > > alpha include: > > > > > 1) having no always failing unit tests against hadoop3 (HBASE-6581) > > > > > > > > > > Thoughts? > > > > > Jon. > > > > > > > > > > On Mon, Oct 3, 2016 at 3:27 PM, Stephen Jiang < > > syuanjiang...@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > Hello, All, > > > > > > > > > > > > It is time to discuss about the schedule of HBase 2.0 release. > > HBase > > > > 2.0 > > > > > > release is a big major release. When we release 1.0, we had 0.99 > > as > > > > dev > > > > > > preview/beta release. We should do something similar for the 2.0 > > > > > release. > > > > > > > > > > > > Matteo and I talked about this. We think about that we need > some > > > > > > Alpha/Beta milestones before the RC and final Release-to-Web 2.0 > > > > release. > > > > > > > > > > > > I don't know whether there is any discussion on this community > > about > > > > the > > > > > > Alpha/Beta release criteria. My proposal is that once we cut the > > > > > branch-2, > > > > > > we should only have new features that are absolutely needed for > > major > > > > > > release (festures cannot be added in minor release) and those > > > features > > > > > > should be "almost ready". For Alpha releases, we can still > accept > > > > these > > > > > > kind of features; for Beta release, only bug fixes and > performance > > > > > > improvement on existing features (should we also accept existing > > > > feature > > > > > > improvement in Beta? Maybe Beta 1, Not in Beta 2 - that is my > > take). > > > > > > > > > > > > This
[jira] [Created] (HBASE-16785) We are not running all tests
stack created HBASE-16785: - Summary: We are not running all tests Key: HBASE-16785 URL: https://issues.apache.org/jira/browse/HBASE-16785 Project: HBase Issue Type: Bug Components: build, test Reporter: stack Assignee: stack Noticed by [~mbertozzi] We have some modules where we tried to 'skip' the running of the second part of tests -- medium and larges. That might have made sense once when the module was originally added when there may have been just a few small tests to run but as time goes by and the module accumulates more tests in a few cases we've added mediums and larges but we've not removed the 'skip' config. Matteo noticed this happened in hbase-procedure. In hbase-client, there is at least a medium test that is being skipped. Let me try purging this trick everywhere. It doesn't seem to save us anything going by build time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16786) Procedure V2 - Move ZK-lock's uses to Procedure framework locks (LockProcedure)
Appy created HBASE-16786: Summary: Procedure V2 - Move ZK-lock's uses to Procedure framework locks (LockProcedure) Key: HBASE-16786 URL: https://issues.apache.org/jira/browse/HBASE-16786 Project: HBase Issue Type: Bug Reporter: Appy Assignee: Appy -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16787) update MasterStorage / RegionStorage to have a exists-in-storage check and archive methods
Sean Busbey created HBASE-16787: --- Summary: update MasterStorage / RegionStorage to have a exists-in-storage check and archive methods Key: HBASE-16787 URL: https://issues.apache.org/jira/browse/HBASE-16787 Project: HBase Issue Type: Sub-task Components: Filesystem Integration Reporter: Sean Busbey Assignee: Umesh Agashe Fix For: hbase-14439 remove filesystem calls from CatalogueJanitor and implement needed APIs in Master/Region Storage to facilitate that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16788) Race in compacted file deletion between HStore close() and closeAndArchiveCompactedFiles()
Gary Helmling created HBASE-16788:
-
Summary: Race in compacted file deletion between HStore close()
and closeAndArchiveCompactedFiles()
Key: HBASE-16788
URL: https://issues.apache.org/jira/browse/HBASE-16788
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 1.3.0
Reporter: Gary Helmling
Assignee: Gary Helmling
Priority: Blocker
HBASE-13082 changed the way that compacted files are archived from being done
inline on compaction completion to an async cleanup by the
CompactedHFilesDischarger chore. It looks like the changes to HStore to
support this introduced a race condition in the compacted HFile archiving.
In the following sequence, we can wind up with two separate threads trying to
archive the same HFiles, causing a regionserver abort:
# compaction completes normally and the compacted files are added to
{{compactedfiles}} in HStore's DefaultStoreFileManager
# *threadA*: CompactedHFilesDischargeHandler runs in a RS executor service,
calling closeAndArchiveCompactedFiles()
## obtains HStore readlock
## gets a copy of compactedfiles
## releases readlock
# *threadB*: calls HStore.close() as part of region close
## obtains HStore writelock
## calls DefaultStoreFileManager.clearCompactedfiles(), getting a copy of same
compactedfiles
# *threadA*: calls HStore.removeCompactedfiles(compactedfiles)
## archives files in {compactedfiles} in HRegionFileSystem.removeStoreFiles()
## call HStore.clearCompactedFiles()
## waits on write lock
# *threadB*: continues with close()
## calls removeCompactedfiles(compactedfiles)
## calls HRegionFIleSystem.removeStoreFiles() ->
HFileArchiver.archiveStoreFiles()
## receives FileNotFoundException because the files have already been archived
by threadA
## throws IOException
# RS aborts
I think the combination of fetching the compactedfiles list and removing the
files needs to be covered by locking. Options I see are:
* Modify HStore.closeAndArchiveCompactedFiles(): use writelock instead of
readlock and move the call to removeCompactedfiles() inside the lock. This
means the read operations will be blocked while the files are being archived,
which is bad.
* Synchronize closeAndArchiveCompactedFiles() and modify close() to call it
instead of calling removeCompactedfiles() directly
* Add a separate lock for compacted files removal and use in
closeAndArchiveCompactedFiles() and close()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-14138) HBase Backup/Restore Phase 3: Security
[ https://issues.apache.org/jira/browse/HBASE-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-14138. Resolution: Later Resolving as Later since implementation is moved to client. > HBase Backup/Restore Phase 3: Security > -- > > Key: HBASE-14138 > URL: https://issues.apache.org/jira/browse/HBASE-14138 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu > Labels: backup > Fix For: 2.0.0 > > > Security is not supported. Only authorized user (GLOBAL ADMIN) must be > allowed to perform backup/restore. See: HBASE-7367 for good discussion on > snapshot security model. > * Backup between two secured (Kerberos) clusters (Cross-realm authentication > is required to use distcp/export snapshot between two secured cluster?) > * Backup between secured (Kerberos) and secured non-Kerberos cluster (AWS) > * Backup between secured and unsecured cluster > * Restore between two Kerberos clusters > * Restore from non-Kerberos (AWS) to Kerberos > * Restore from unsecured to secured (Kerberos) > * Users must be able to run backup/restore for table if they have admin > privileges for a table (?) > Some relevant JIRAs > https://issues.apache.org/jira/browse/HADOOP-8828 > https://issues.apache.org/jira/browse/HDFS-6776 > Links: > http://henning.kropponline.de/2015/10/04/distcp-between-kerberized-and-none-kerberized-cluster/ > http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_Sys_Admin_Guides/content/distcp_and_security_settings.html > https://discuss.zendesk.com/hc/en-us/articles/203176207-Setting-up-a-kerberos-cross-realm-trust-for-distcp > http://www.cloudera.com/documentation/enterprise/5-5-x/topics/cdh_admin_distcp_secure_insecure.html > https://www.cloudera.com/documentation/enterprise/5-4-x/topics/cdh_admin_distcp_data_cluster_migrate.html > https://www.cloudera.com/documentation/enterprise/5-7-x/topics/cdh_admin_distcp_data_cluster_migrate.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16789) Remove directory layout/ filesystem references from CompactionTool
Umesh Agashe created HBASE-16789: Summary: Remove directory layout/ filesystem references from CompactionTool Key: HBASE-16789 URL: https://issues.apache.org/jira/browse/HBASE-16789 Project: HBase Issue Type: Sub-task Components: Filesystem Integration Reporter: Umesh Agashe Assignee: Umesh Agashe Remove directory layout/ filesystem references from CompactionTool and use APIs provided by MasterStorage/ RegionStorage instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14613) Remove MemStoreChunkPool?
[ https://issues.apache.org/jira/browse/HBASE-14613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-14613. --- Resolution: Won't Fix > Remove MemStoreChunkPool? > - > > Key: HBASE-14613 > URL: https://issues.apache.org/jira/browse/HBASE-14613 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 14613-0.98.txt, gc.png, writes.png > > > I just stumbled across MemStoreChunkPool. The idea behind is to reuse chunks > of allocations rather than letting the GC handle this. > Now, it's off by default, and it seems to me to be of dubious value. I'd > recommend just removing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
