Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
On Thu, Jul 18, 2013 at 3:54 PM, Ted Yu wrote: > bq. Francis' namespaces is in need of review > > I plan to review this feature. > > The latest update to https://github.com/francisliu/hbase_namespace lagged > the last comment on HBASE-8015 > I wonder if there are more significant changes to be made. > He moved his work to the github mentioned in the issue. Go there for his latest work. St.Ack
Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
bq. Francis' namespaces is in need of review I plan to review this feature. The latest update to https://github.com/francisliu/hbase_namespace lagged the last comment on HBASE-8015 I wonder if there are more significant changes to be made. Cheers On Wed, Jul 17, 2013 at 12:12 PM, Stack wrote: > STATUS > > +Build > > A few of us have been working on broke junit tests. Progress is slow but > steady. Currently I am looking into zombie tests -- "all" tests complete > but the build still fails. Andrew Purtell has set me up w/ a clone box > from his ec2 rig which is helping figure the zombie phenomenon. I also > just added our hadoopqa zombie detector up on apache trunk build to see if > it yields clues at nkeywals' suggestion. I also need to fix > https://issues.apache.org/jira/browse/HBASE-8764 so the ec2 integration > test build passes (ec2 build runs integration tests which apache build does > not) > > +Reviews > > Francis' namespaces is in need of review as are other 'Patch Available' > criticals that we'd like committed such as Sergey's compaction changes. If > any volunteers are game, see the 'patch available' issues here [1]. > Contrib appreciated. > > +Testing > I know of at least one rig that is going up to do long-term hbase-it tests. > Any other testing going on that folks would like to talk up? > > +Migration > > Testing and finish documentation on moving from 0.94 pedigree to 0.96. > Still TODO. > > +Packaging > > Making it so we can publish an hbase 0.96 built against hadoop1 and > hadoop2. Still TODO. > > What else? > > I am still heading for end of July for a complete '0.96'. > > Thanks, > St.Ack > > 1. > > https://issues.apache.org/jira/browse/HBASE/fixforversion/12320040#selectedTab=com.atlassian.jira.plugin.system.project%3Aversion-issues-panel > > > > > > On Tue, Jul 9, 2013 at 1:14 PM, Stack wrote: > > > I am shooting for end of July for 0.96 being 'complete'. I would like > to > > make a 0.96 release in August. We have some criticals outstanding but I > > think we could ship even if these are not fixed in time (excepting > > migration polish and of course remaining build fixes). See [1.] for the > > current list of issues. Please re-prioritize issues as you see fit (or > > better, move issues out of 0.95.2 if you do not think they will be done > in > > time). > > > > What to do with namespaces -- the last 0.96 'feature' -- given the above > > timeline? Currently it is a massive patch out on a branch. It is still > > not done, in want of review, and the author is going on holidays for a > few > > weeks soon. My thinking as of now, going by the rate of change over the > > last few weeks and estimating what is yet to be done, is that namespaces > > will not make it. I am willing to be convinced otherwise but that is how > > it looks to me currently. > > > > I am going to start just disabling flakey unit tests in 0.95 from here on > > out. When folks get the itch, they can fix at leisure first on trunk and > > then over in 0.95. > > > > What else? > > > > Thanks, > > St.Ack > > > > 1. > > > https://issues.apache.org/jira/browse/HBASE/fixforversion/12320040#selectedTab=com.atlassian.jira.plugin.system.project%3Aversion-issues-panel > > > > > > > > On Mon, Jun 24, 2013 at 8:45 PM, Stack wrote: > > > >> (Changed the subject) > >> > >> On Mon, Jun 24, 2013 at 4:04 PM, Nick Dimiduk > wrote: > >> > >>> I want to see initial data type APIs ship out with 0.95.2. A patch for > >>> ordered byte serialization is up (HBASE-8201) and is nearing > >>> steady-state. > >>> However, sershe is the only person who's left feedback. I just posted > an > >>> early patch for the data type API itself (HBASE-8693). It should get > some > >>> eyes from all manor of interested parties, but I'll settle for folk > from > >>> Phoenix for now. > >>> > >>> > >> It would be cool if Phoenix and Kiji fellows and any one else interested > >> would weigh in and take a look see. > >> > >> This does not strike me as something we should hold up the release for > >> though. It looks like something that could go in at any time? > >> > >> > >> > >>> Should these tasks be escalated to criticals in order to grab > attention? > >>> > >>> > >> I don't think that works going by past experience (and I don't think > this > >> a blocker on 0.96) > >> > >> > >> > >>> Additional comments inline. > >>> > >>> > >> ... > >> > >> > >> > >>> Namespaces is the long pole and progress seems slow. Do we hold up the > >>> > release for them? How can we hurry this effort along? Swat team > >>> descends > >>> > on Y!? > >>> > > >>> > >>> It would be a shame to not get a decision on this in for 0.96. > >>> > >>> > >> Agree. We need to get 0.96 out though. It has been too long. > >> > >> > >> > >>> + Is anyone testing? Integration tests fail on ec2 build from time to > >>> time > >>> > [2]. Our Elliott dug in on one of the failures a few days back and > >>> found > >>> > legit issue w/ no retry on admin tasks (I heart hba
Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
> These are both criticals. Maybe they should be blocker altogether since > can't ship w/o them. > Ok, did that. > > > https://issues.apache.org/jira/browse/HBASE-6929 (same for 0.94 and > 0.96) > > > > > This is a 0.94 issue. Could fall out from hbase-8224 work. > Agreed, as long as HBASE-8224 will enable us to publish 0.96-hadoop2 to mvn, and at the time of release we do that, we are golden. 0.94-hadoop2 is not a blocker. > > > > > For migration, there are some steps that the a user has to perform before > > going to 0.96 (recompile with 0.96, select hadoop1 or hadoop2, some minor > > class renamings, etc), we should document these steps in the book. (I > don't > > remember we already have an open jira for this.) > > > > > HBASE-8348 Polish the migration to 0.96 > > > > For testing, IT tests are rapidly improving. We will also start running > > those (or a subset of those) against 0.96. > > > > > Sweet. > > St.Ack >
Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
On Wed, Jul 17, 2013 at 3:21 PM, Andrew Purtell wrote: > On Wed, Jul 17, 2013 at 2:58 PM, Stack wrote: > > > I love the idea of bigtop packaging. Should we do new issue for that? > Or > just use 8224. > > Ok, did both. See https://issues.apache.org/jira/browse/BIGTOP-1029, and > linked to 8224, with comment there. > > Excellent. > > http://hbase.apache.org/book.html#upgrade0.96 > > Let's see about scripting that for repeatable testing. > > Yeah. I can work on that do the old school tarball of 0.94 data, undo it in a test, and run 0.96 over it. > > it unlikely that a *design* at this stage will make the cut given > completed patches may not make it for want of review, etc., etc > > Patch, not design, didn't mean to imply otherwise. > > Ok. Patch has better chance than a design of making it in. Good on you A, St.Ack
Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
On Wed, Jul 17, 2013 at 4:21 PM, Enis Söztutar wrote: > Thanks Stack for pushing us. > > For packaging, I think we should fix: > https://issues.apache.org/jira/browse/HBASE-8224 > https://issues.apache.org/jira/browse/HBASE-8488 These are both criticals. Maybe they should be blocker altogether since can't ship w/o them. > > https://issues.apache.org/jira/browse/HBASE-6929 (same for 0.94 and 0.96) > > This is a 0.94 issue. Could fall out from hbase-8224 work. > For migration, there are some steps that the a user has to perform before > going to 0.96 (recompile with 0.96, select hadoop1 or hadoop2, some minor > class renamings, etc), we should document these steps in the book. (I don't > remember we already have an open jira for this.) > > HBASE-8348 Polish the migration to 0.96 > For testing, IT tests are rapidly improving. We will also start running > those (or a subset of those) against 0.96. > Sweet. St.Ack
Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Thanks Stack for pushing us. For packaging, I think we should fix: https://issues.apache.org/jira/browse/HBASE-8224 https://issues.apache.org/jira/browse/HBASE-8488 https://issues.apache.org/jira/browse/HBASE-6929 (same for 0.94 and 0.96) For migration, there are some steps that the a user has to perform before going to 0.96 (recompile with 0.96, select hadoop1 or hadoop2, some minor class renamings, etc), we should document these steps in the book. (I don't remember we already have an open jira for this.) For testing, IT tests are rapidly improving. We will also start running those (or a subset of those) against 0.96. Enis On Wed, Jul 17, 2013 at 3:21 PM, Andrew Purtell wrote: > On Wed, Jul 17, 2013 at 2:58 PM, Stack wrote: > > > I love the idea of bigtop packaging. Should we do new issue for that? > Or > just use 8224. > > Ok, did both. See https://issues.apache.org/jira/browse/BIGTOP-1029, and > linked to 8224, with comment there. > > > http://hbase.apache.org/book.html#upgrade0.96 > > Let's see about scripting that for repeatable testing. > > > it unlikely that a *design* at this stage will make the cut given > completed patches may not make it for want of review, etc., etc > > Patch, not design, didn't mean to imply otherwise. > > > -- > Best regards, > >- Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White) >
Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
On Wed, Jul 17, 2013 at 2:58 PM, Stack wrote: > I love the idea of bigtop packaging. Should we do new issue for that? Or just use 8224. Ok, did both. See https://issues.apache.org/jira/browse/BIGTOP-1029, and linked to 8224, with comment there. > http://hbase.apache.org/book.html#upgrade0.96 Let's see about scripting that for repeatable testing. > it unlikely that a *design* at this stage will make the cut given completed patches may not make it for want of review, etc., etc Patch, not design, didn't mean to imply otherwise. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
On Wed, Jul 17, 2013 at 1:45 PM, Andrew Purtell wrote: > > +Packaging > > I would like to help you on this. Can do packaging and testing using the > Bigtop framework. I see HBASE-8187, but it's resolved. Elsewhere we > can coordinate? Could open a Bigtop jira. Whatever you prefer. > > I was most concerned about https://issues.apache.org/jira/browse/HBASE-8224-- publishing to mvn. That is what has me stumped at the moment. I love the idea of bigtop packaging. Should we do new issue for that? Or just use 8224. It has a few watchers -- downstreamers -- so it might be better to hang there at least at first. > > +Migration > > As part of the above. Test by starting with an 0.94 install on a small > cluster, load up some data, shut down and upgrade packages, try to > (re)launch... > > Yes, following the (as-yet-incomplete) instruction here: http://hbase.apache.org/book.html#upgrade0.96 > > What else? > > I beg your patience as RM. Some guys here have been working diligently for > the past couple of months on something that is in our estimation 0.96able. > Won't go into it here because they have been passing around a proposal for > dev@ since the weekend, to make sure it says what they want. Will be out > in > the next day or so for the consideration of all. > > Ain't no need to beg. Lets have a look at it before making a call. I'll spare the rehearsal that it is very late in the game, etc., etc., and that it unlikely that a *design* at this stage will make the cut given completed patches may not make it for want of review, etc., etc But lets see. Thanks Andrew, St.Ack
Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
> +Packaging I would like to help you on this. Can do packaging and testing using the Bigtop framework. I see HBASE-8187, but it's resolved. Elsewhere we can coordinate? Could open a Bigtop jira. Whatever you prefer. > +Migration As part of the above. Test by starting with an 0.94 install on a small cluster, load up some data, shut down and upgrade packages, try to (re)launch... > What else? I beg your patience as RM. Some guys here have been working diligently for the past couple of months on something that is in our estimation 0.96able. Won't go into it here because they have been passing around a proposal for dev@ since the weekend, to make sure it says what they want. Will be out in the next day or so for the consideration of all. On Wed, Jul 17, 2013 at 12:12 PM, Stack wrote: > STATUS > > +Build > > A few of us have been working on broke junit tests. Progress is slow but > steady. Currently I am looking into zombie tests -- "all" tests complete > but the build still fails. Andrew Purtell has set me up w/ a clone box > from his ec2 rig which is helping figure the zombie phenomenon. I also > just added our hadoopqa zombie detector up on apache trunk build to see if > it yields clues at nkeywals' suggestion. I also need to fix > https://issues.apache.org/jira/browse/HBASE-8764 so the ec2 integration > test build passes (ec2 build runs integration tests which apache build does > not) > > +Reviews > > Francis' namespaces is in need of review as are other 'Patch Available' > criticals that we'd like committed such as Sergey's compaction changes. If > any volunteers are game, see the 'patch available' issues here [1]. > Contrib appreciated. > > +Testing > I know of at least one rig that is going up to do long-term hbase-it tests. > Any other testing going on that folks would like to talk up? > > +Migration > > Testing and finish documentation on moving from 0.94 pedigree to 0.96. > Still TODO. > > +Packaging > > Making it so we can publish an hbase 0.96 built against hadoop1 and > hadoop2. Still TODO. > > What else? > > I am still heading for end of July for a complete '0.96'. > > Thanks, > St.Ack > > 1. > > https://issues.apache.org/jira/browse/HBASE/fixforversion/12320040#selectedTab=com.atlassian.jira.plugin.system.project%3Aversion-issues-panel > > > > > > On Tue, Jul 9, 2013 at 1:14 PM, Stack wrote: > > > I am shooting for end of July for 0.96 being 'complete'. I would like > to > > make a 0.96 release in August. We have some criticals outstanding but I > > think we could ship even if these are not fixed in time (excepting > > migration polish and of course remaining build fixes). See [1.] for the > > current list of issues. Please re-prioritize issues as you see fit (or > > better, move issues out of 0.95.2 if you do not think they will be done > in > > time). > > > > What to do with namespaces -- the last 0.96 'feature' -- given the above > > timeline? Currently it is a massive patch out on a branch. It is still > > not done, in want of review, and the author is going on holidays for a > few > > weeks soon. My thinking as of now, going by the rate of change over the > > last few weeks and estimating what is yet to be done, is that namespaces > > will not make it. I am willing to be convinced otherwise but that is how > > it looks to me currently. > > > > I am going to start just disabling flakey unit tests in 0.95 from here on > > out. When folks get the itch, they can fix at leisure first on trunk and > > then over in 0.95. > > > > What else? > > > > Thanks, > > St.Ack > > > > 1. > > > https://issues.apache.org/jira/browse/HBASE/fixforversion/12320040#selectedTab=com.atlassian.jira.plugin.system.project%3Aversion-issues-panel > > > > > > > > On Mon, Jun 24, 2013 at 8:45 PM, Stack wrote: > > > >> (Changed the subject) > >> > >> On Mon, Jun 24, 2013 at 4:04 PM, Nick Dimiduk > wrote: > >> > >>> I want to see initial data type APIs ship out with 0.95.2. A patch for > >>> ordered byte serialization is up (HBASE-8201) and is nearing > >>> steady-state. > >>> However, sershe is the only person who's left feedback. I just posted > an > >>> early patch for the data type API itself (HBASE-8693). It should get > some > >>> eyes from all manor of interested parties, but I'll settle for folk > from > >>> Phoenix for now. > >>> > >>> > >> It would be cool if Phoenix and Kiji fellows and any one else interested > >> would weigh in and take a look see. > >> > >> This does not strike me as something we should hold up the release for > >> though. It looks like something that could go in at any time? > >> > >> > >> > >>> Should these tasks be escalated to criticals in order to grab > attention? > >>> > >>> > >> I don't think that works going by past experience (and I don't think > this > >> a blocker on 0.96) > >> > >> > >> > >>> Additional comments inline. > >>> > >>> > >> ... > >> > >> > >> > >>> Namespaces is the long pole and progress seems slow. Do we hold up the > >>> > release
Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
STATUS +Build A few of us have been working on broke junit tests. Progress is slow but steady. Currently I am looking into zombie tests -- "all" tests complete but the build still fails. Andrew Purtell has set me up w/ a clone box from his ec2 rig which is helping figure the zombie phenomenon. I also just added our hadoopqa zombie detector up on apache trunk build to see if it yields clues at nkeywals' suggestion. I also need to fix https://issues.apache.org/jira/browse/HBASE-8764 so the ec2 integration test build passes (ec2 build runs integration tests which apache build does not) +Reviews Francis' namespaces is in need of review as are other 'Patch Available' criticals that we'd like committed such as Sergey's compaction changes. If any volunteers are game, see the 'patch available' issues here [1]. Contrib appreciated. +Testing I know of at least one rig that is going up to do long-term hbase-it tests. Any other testing going on that folks would like to talk up? +Migration Testing and finish documentation on moving from 0.94 pedigree to 0.96. Still TODO. +Packaging Making it so we can publish an hbase 0.96 built against hadoop1 and hadoop2. Still TODO. What else? I am still heading for end of July for a complete '0.96'. Thanks, St.Ack 1. https://issues.apache.org/jira/browse/HBASE/fixforversion/12320040#selectedTab=com.atlassian.jira.plugin.system.project%3Aversion-issues-panel On Tue, Jul 9, 2013 at 1:14 PM, Stack wrote: > I am shooting for end of July for 0.96 being 'complete'. I would like to > make a 0.96 release in August. We have some criticals outstanding but I > think we could ship even if these are not fixed in time (excepting > migration polish and of course remaining build fixes). See [1.] for the > current list of issues. Please re-prioritize issues as you see fit (or > better, move issues out of 0.95.2 if you do not think they will be done in > time). > > What to do with namespaces -- the last 0.96 'feature' -- given the above > timeline? Currently it is a massive patch out on a branch. It is still > not done, in want of review, and the author is going on holidays for a few > weeks soon. My thinking as of now, going by the rate of change over the > last few weeks and estimating what is yet to be done, is that namespaces > will not make it. I am willing to be convinced otherwise but that is how > it looks to me currently. > > I am going to start just disabling flakey unit tests in 0.95 from here on > out. When folks get the itch, they can fix at leisure first on trunk and > then over in 0.95. > > What else? > > Thanks, > St.Ack > > 1. > https://issues.apache.org/jira/browse/HBASE/fixforversion/12320040#selectedTab=com.atlassian.jira.plugin.system.project%3Aversion-issues-panel > > > > On Mon, Jun 24, 2013 at 8:45 PM, Stack wrote: > >> (Changed the subject) >> >> On Mon, Jun 24, 2013 at 4:04 PM, Nick Dimiduk wrote: >> >>> I want to see initial data type APIs ship out with 0.95.2. A patch for >>> ordered byte serialization is up (HBASE-8201) and is nearing >>> steady-state. >>> However, sershe is the only person who's left feedback. I just posted an >>> early patch for the data type API itself (HBASE-8693). It should get some >>> eyes from all manor of interested parties, but I'll settle for folk from >>> Phoenix for now. >>> >>> >> It would be cool if Phoenix and Kiji fellows and any one else interested >> would weigh in and take a look see. >> >> This does not strike me as something we should hold up the release for >> though. It looks like something that could go in at any time? >> >> >> >>> Should these tasks be escalated to criticals in order to grab attention? >>> >>> >> I don't think that works going by past experience (and I don't think this >> a blocker on 0.96) >> >> >> >>> Additional comments inline. >>> >>> >> ... >> >> >> >>> Namespaces is the long pole and progress seems slow. Do we hold up the >>> > release for them? How can we hurry this effort along? Swat team >>> descends >>> > on Y!? >>> > >>> >>> It would be a shame to not get a decision on this in for 0.96. >>> >>> >> Agree. We need to get 0.96 out though. It has been too long. >> >> >> >>> + Is anyone testing? Integration tests fail on ec2 build from time to >>> time >>> > [2]. Our Elliott dug in on one of the failures a few days back and >>> found >>> > legit issue w/ no retry on admin tasks (I heart hbase-it tests). Our >>> unit >>> > test story is better [3] but there are still the odd failures. >>> > >>> >>> With the creation of the new list, noticing these issues is going to push >>> further back-burner. Nannying this stuff should retain focus. I'll >>> volunteer to track on these issues as I see them. >>> >>> >> Thank you Nick. >> >> >> >>> Notice though that some of the more recent failures are caused by lack of >>> disk space on the Jenkins build host. >>> >>> >> Oh. Missed that. Let me dig in. >> >> >> St.Ack >> > >
Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
I am shooting for end of July for 0.96 being 'complete'. I would like to make a 0.96 release in August. We have some criticals outstanding but I think we could ship even if these are not fixed in time (excepting migration polish and of course remaining build fixes). See [1.] for the current list of issues. Please re-prioritize issues as you see fit (or better, move issues out of 0.95.2 if you do not think they will be done in time). What to do with namespaces -- the last 0.96 'feature' -- given the above timeline? Currently it is a massive patch out on a branch. It is still not done, in want of review, and the author is going on holidays for a few weeks soon. My thinking as of now, going by the rate of change over the last few weeks and estimating what is yet to be done, is that namespaces will not make it. I am willing to be convinced otherwise but that is how it looks to me currently. I am going to start just disabling flakey unit tests in 0.95 from here on out. When folks get the itch, they can fix at leisure first on trunk and then over in 0.95. What else? Thanks, St.Ack 1. https://issues.apache.org/jira/browse/HBASE/fixforversion/12320040#selectedTab=com.atlassian.jira.plugin.system.project%3Aversion-issues-panel On Mon, Jun 24, 2013 at 8:45 PM, Stack wrote: > (Changed the subject) > > On Mon, Jun 24, 2013 at 4:04 PM, Nick Dimiduk wrote: > >> I want to see initial data type APIs ship out with 0.95.2. A patch for >> ordered byte serialization is up (HBASE-8201) and is nearing steady-state. >> However, sershe is the only person who's left feedback. I just posted an >> early patch for the data type API itself (HBASE-8693). It should get some >> eyes from all manor of interested parties, but I'll settle for folk from >> Phoenix for now. >> >> > It would be cool if Phoenix and Kiji fellows and any one else interested > would weigh in and take a look see. > > This does not strike me as something we should hold up the release for > though. It looks like something that could go in at any time? > > > >> Should these tasks be escalated to criticals in order to grab attention? >> >> > I don't think that works going by past experience (and I don't think this > a blocker on 0.96) > > > >> Additional comments inline. >> >> > ... > > > >> Namespaces is the long pole and progress seems slow. Do we hold up the >> > release for them? How can we hurry this effort along? Swat team >> descends >> > on Y!? >> > >> >> It would be a shame to not get a decision on this in for 0.96. >> >> > Agree. We need to get 0.96 out though. It has been too long. > > > >> + Is anyone testing? Integration tests fail on ec2 build from time to >> time >> > [2]. Our Elliott dug in on one of the failures a few days back and >> found >> > legit issue w/ no retry on admin tasks (I heart hbase-it tests). Our >> unit >> > test story is better [3] but there are still the odd failures. >> > >> >> With the creation of the new list, noticing these issues is going to push >> further back-burner. Nannying this stuff should retain focus. I'll >> volunteer to track on these issues as I see them. >> >> > Thank you Nick. > > > >> Notice though that some of the more recent failures are caused by lack of >> disk space on the Jenkins build host. >> >> > Oh. Missed that. Let me dig in. > > > St.Ack >
Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
On Mon, Jul 1, 2013 at 3:56 PM, Andrew Purtell wrote: > On Mon, Jul 1, 2013 at 3:34 PM, Stack wrote: > > > That patch won't go in to hbase. > > > What if we and Hadoop can come up with a common RPC connection setup, like > PB-based SASL negotiation and a common version header? Then there are > options post-singularity. > > Hopefully, the HADOOP-9421 is a step in that direction. I am going to spend some time on HADOOP-9421. > > -- > Best regards, > >- Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White) >
Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
On Mon, Jul 1, 2013 at 3:34 PM, Stack wrote: > That patch won't go in to hbase. What if we and Hadoop can come up with a common RPC connection setup, like PB-based SASL negotiation and a common version header? Then there are options post-singularity. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
On Mon, Jul 1, 2013 at 3:34 PM, Stack wrote: > On Mon, Jul 1, 2013 at 2:54 PM, Devaraj Das wrote: > > > > so we might look at using Hadoop RPC directly again? > > > > Totally agree with that thought, Andrew. I must admit that I don't know > > about all the context on why the HBase RPC forked in the first place :-) > > > > > Bug fixes and speed optimizations > > > > > Your suggestion on using the Hadoop RPC work here makes sense. I am not > too > > sure whether that is practical in the short term though (we would need to > > put in Tracing framework calls in Hadoop RPC and so on). IMO this will > > further delay the 0.96 shipping as well. However doing the equivalent > patch > > for HADOOP-9421 in HBase will probably be easier and doable for 0.96... > > > > > That patch won't go in to hbase. Can you sketch the back and forth finally > decided upon in an issue? If you do this, I could help on how you might > get it in. > > I meant to say that using Hadoop RPC with changes needed for HBase to work makes sense (and, of course, not the patch directly from HADOOP-9421 :-) ) But I'll write up the proposal implemented in HADOOP-9421 in a HBASE jira and we can go from there. > St.Ack >
Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
On Mon, Jul 1, 2013 at 2:54 PM, Devaraj Das wrote: > However doing the equivalent patch > for HADOOP-9421 in HBase will probably be easier and doable for 0.96... > If someone shows up with a patch equivalent to HADOOP-9421 for HBase real soon now let's look at it. Maybe that is good enough and we will be able to finesse common authentication on top of two divergent RPC stacks. Let me throw out there a third way: Hadoop and HBase gets together right now in a "protobuf RPC wire format working group" and solves at least the wire side of the problem before one or the other ships something that has to be set going forward. On the HBase side, that's happening real soon now, perhaps as early as the end of the month. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
On Mon, Jul 1, 2013 at 2:54 PM, Devaraj Das wrote: > > so we might look at using Hadoop RPC directly again? > > Totally agree with that thought, Andrew. I must admit that I don't know > about all the context on why the HBase RPC forked in the first place :-) > > Bug fixes and speed optimizations > Your suggestion on using the Hadoop RPC work here makes sense. I am not too > sure whether that is practical in the short term though (we would need to > put in Tracing framework calls in Hadoop RPC and so on). IMO this will > further delay the 0.96 shipping as well. However doing the equivalent patch > for HADOOP-9421 in HBase will probably be easier and doable for 0.96... > > That patch won't go in to hbase. Can you sketch the back and forth finally decided upon in an issue? If you do this, I could help on how you might get it in. St.Ack
Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
On Mon, Jul 1, 2013 at 1:40 PM, Devaraj Das wrote: > BTW I just wanted to bring to everyone's attention the work done over in > HADOOP-9421. Sanjay Radia has been telling me that a corresponding work > should be done in HBase as well. That will help keep the compatibility > story straight when the work to do with pluggable authentication mechanisms > happening elsewhere in the ecosystem takes shape.. > > Any thoughts? > There is no description of what was finally decided on? Looks like client initiates and then server and client can negotiate? Can't we up the rpc version when we want to add support for negotiated protocols? If so, this does not have to hold up 0.96? St.Ack
Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
> so we might look at using Hadoop RPC directly again? Totally agree with that thought, Andrew. I must admit that I don't know about all the context on why the HBase RPC forked in the first place :-) Your suggestion on using the Hadoop RPC work here makes sense. I am not too sure whether that is practical in the short term though (we would need to put in Tracing framework calls in Hadoop RPC and so on). IMO this will further delay the 0.96 shipping as well. However doing the equivalent patch for HADOOP-9421 in HBase will probably be easier and doable for 0.96... On Mon, Jul 1, 2013 at 1:54 PM, Andrew Purtell wrote: > On Mon, Jul 1, 2013 at 1:40 PM, Devaraj Das wrote: > > > HADOOP-9421 > > > How about turning that around? (smile) Any thoughts on removing the > indirection in Hadoop RPC like Stack did with HBase PB RPC so we might look > at using Hadoop RPC directly again? Seems if the goal is to support > pluggable authentication mechanisms being developed in core then we should > be taking that up through a core artifact, not duplicating the work over > here? > > > -- > Best regards, > >- Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White) >
Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
On Mon, Jul 1, 2013 at 1:40 PM, Devaraj Das wrote: > HADOOP-9421 How about turning that around? (smile) Any thoughts on removing the indirection in Hadoop RPC like Stack did with HBase PB RPC so we might look at using Hadoop RPC directly again? Seems if the goal is to support pluggable authentication mechanisms being developed in core then we should be taking that up through a core artifact, not duplicating the work over here? -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
BTW I just wanted to bring to everyone's attention the work done over in HADOOP-9421. Sanjay Radia has been telling me that a corresponding work should be done in HBase as well. That will help keep the compatibility story straight when the work to do with pluggable authentication mechanisms happening elsewhere in the ecosystem takes shape.. Any thoughts? Thanks Devaraj On Mon, Jun 24, 2013 at 8:45 PM, Stack wrote: > (Changed the subject) > > On Mon, Jun 24, 2013 at 4:04 PM, Nick Dimiduk wrote: > > > I want to see initial data type APIs ship out with 0.95.2. A patch for > > ordered byte serialization is up (HBASE-8201) and is nearing > steady-state. > > However, sershe is the only person who's left feedback. I just posted an > > early patch for the data type API itself (HBASE-8693). It should get some > > eyes from all manor of interested parties, but I'll settle for folk from > > Phoenix for now. > > > > > It would be cool if Phoenix and Kiji fellows and any one else interested > would weigh in and take a look see. > > This does not strike me as something we should hold up the release for > though. It looks like something that could go in at any time? > > > > > Should these tasks be escalated to criticals in order to grab attention? > > > > > I don't think that works going by past experience (and I don't think this a > blocker on 0.96) > > > > > Additional comments inline. > > > > > ... > > > > > Namespaces is the long pole and progress seems slow. Do we hold up the > > > release for them? How can we hurry this effort along? Swat team > > descends > > > on Y!? > > > > > > > It would be a shame to not get a decision on this in for 0.96. > > > > > Agree. We need to get 0.96 out though. It has been too long. > > > > > + Is anyone testing? Integration tests fail on ec2 build from time to > time > > > [2]. Our Elliott dug in on one of the failures a few days back and > found > > > legit issue w/ no retry on admin tasks (I heart hbase-it tests). Our > > unit > > > test story is better [3] but there are still the odd failures. > > > > > > > With the creation of the new list, noticing these issues is going to push > > further back-burner. Nannying this stuff should retain focus. I'll > > volunteer to track on these issues as I see them. > > > > > Thank you Nick. > > > > > Notice though that some of the more recent failures are caused by lack of > > disk space on the Jenkins build host. > > > > > Oh. Missed that. Let me dig in. > > > St.Ack >
[DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
(Changed the subject) On Mon, Jun 24, 2013 at 4:04 PM, Nick Dimiduk wrote: > I want to see initial data type APIs ship out with 0.95.2. A patch for > ordered byte serialization is up (HBASE-8201) and is nearing steady-state. > However, sershe is the only person who's left feedback. I just posted an > early patch for the data type API itself (HBASE-8693). It should get some > eyes from all manor of interested parties, but I'll settle for folk from > Phoenix for now. > > It would be cool if Phoenix and Kiji fellows and any one else interested would weigh in and take a look see. This does not strike me as something we should hold up the release for though. It looks like something that could go in at any time? > Should these tasks be escalated to criticals in order to grab attention? > > I don't think that works going by past experience (and I don't think this a blocker on 0.96) > Additional comments inline. > > ... > Namespaces is the long pole and progress seems slow. Do we hold up the > > release for them? How can we hurry this effort along? Swat team > descends > > on Y!? > > > > It would be a shame to not get a decision on this in for 0.96. > > Agree. We need to get 0.96 out though. It has been too long. > + Is anyone testing? Integration tests fail on ec2 build from time to time > > [2]. Our Elliott dug in on one of the failures a few days back and found > > legit issue w/ no retry on admin tasks (I heart hbase-it tests). Our > unit > > test story is better [3] but there are still the odd failures. > > > > With the creation of the new list, noticing these issues is going to push > further back-burner. Nannying this stuff should retain focus. I'll > volunteer to track on these issues as I see them. > > Thank you Nick. > Notice though that some of the more recent failures are caused by lack of > disk space on the Jenkins build host. > > Oh. Missed that. Let me dig in. St.Ack