When adding back the pre(Flush/Compact/Store)ScannerOpen methods in HBASE-19033, I found that there maybe a hole in our CP hools. It is the in memory compaction.
As Anoop suggested in HBASE-19001, we still need to give user the ability to extend the max versions and TTL config, so I plan to add back the hooks above to let CP users can change the max versions and TTL of a ScanInfo object. But I'm not sure whether in memory compaction will also discard expired cells, if so then we are in trouble... Thanks. 2017-10-25 6:03 GMT+08:00 Stack <[email protected]>: > Chatting with my coworker Mr. Mike Drob, we were batting back and forth > what remains to be done. Surfacing our thoughts here so you all clued > in....Or if you think otherwise, please speak up. > > We have ~13 issues to land: > https://issues.apache.org/jira/projects/HBASE/versions/12341594 About two > are meta-issues that are about process which leaves 11. > > Duo and Zheng Hu are to merge the FilterList fixes improvements > (HBASE-19057, HBASE-18410 et al.). These are blocker because some changed > API/semantic that we need to get out earlier rather than later. > > Once the above is merged, HBASE-13346, change of Filter method names to > mention Cell instead of KeyValue can land. > > HBASE-199048 needs a review (Anoop will probably do it), removing > IA.Private objects as params to MasterObserver... Hopefully this goes in > soon. > > Duo is hard at work on trackers for flush and compaction for CPs > (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)? > > I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after Duo is > done w/ his work above. > > I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after all the > purges allowing CPs do direct calls against Regions in same Host. > > Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil. > > Another day or two? > > St.Ack > > > > On Mon, Oct 23, 2017 at 2:52 PM, Stack <[email protected]> wrote: > > > > > On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <[email protected]> wrote: > > > >> +1 > >> > >> I was trying to work on helping out on the outstanding alpha-4 stuff > last > >> week -- will be continuing to try to do the same this week. > >> > >> If you need any help, Stack, or if others need reviews where I haven't > >> noticed on my own: feel free to @mention me. > >> > >> > > Thanks for the offer Josh. All items seem assigned and are being actively > > worked on. If you get a moment, reviews by you (or anyone else) helps > move > > the process along. > > > > We need to merge in HBASE-18410 branch to pick up Filter improvements. > > Then HBASE-13346 can go in. > > > > You are already helping out on HBASE-18906, thanks. Looks like that will > > be addressed by other alpha-4s about to land. > > > > St.Ack > > TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594 > > > > > > > > > > > > > > > > > > > >> On 10/23/17 12:53 PM, Stack wrote: > >> > >>> (Reviving this thread) > >>> > >>> Lets push out alpha-4 this week. Alpha-4 is the release that has the > >>> refactor of the Coprocessor API shutting down access to internals > marked > >>> InterfaceAudience.Private. > >>> > >>> The outstanding list is here: > >>> https://issues.apache.org/jira/projects/HBASE/versions/12341594 > >>> > >>> Please push in anything marked alpha-4 that belongs to you. > >>> > >>> If issue, talk out loud on this thread. If you need a review to land an > >>> item, shout on the issue and here; we'll help you out. > >>> > >>> As is, items are coming along nicely I'd say. We need to merge the > filter > >>> branch -- HBASE-18410 -- so APIs are finished for hbase2. > >>> > >>> Post alpha-4, we'll have to hunt down our downstreamers and help them > >>> test > >>> on top of alpha-4 so rolling into beta-1, we have confidence our > >>> downstreamers know what to expect (or we discover what we missed BEFORE > >>> we > >>> beta-1). > >>> > >>> Thanks for time, > >>> S > >>> > >>> > >>> > >>> > >>> > >>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <[email protected]> wrote: > >>> > >>> I'll put up an alpha3 RC Monday, probably Monday night. That should be > >>>> time, if we all sprint, for the public-facing API fixes to be done. > >>>> > >>>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3 > but > >>>> it is plain that more time is needed (in spite of valiant effort so > far > >>>> by > >>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose > >>>> theme is > >>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the > following > >>>> week. > >>>> > >>>> We should then be ready for beta (beta == no new features, no API > >>>> changes, > >>>> just fixes). > >>>> > >>>> Thanks, > >>>> St.Ack > >>>> > >>>> > >>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <[email protected]> wrote: > >>>> > >>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it. > >>>>> > >>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a > >>>>> release out in the next week or so. > >>>>> > >>>>> I did a weeding of 2.0.0 issues over the last day. If folks are > >>>>> interested in helping out, below are the items I think we need done > for > >>>>> alpha3 (below are at least 'Critical' status, are API possibly > altering > >>>>> items, and are absent those JIRAs that are making active progress, > >>>>> i.e. the > >>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs > >>>>> doing is > >>>>> what Andrew did comparing 1.3. and 1.4 APIs > >>>>> > >>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and > >>>>> branch-2 > >>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do it > >>>>> between branch-1 and branch-2. > >>>>> > >>>>> * HBASE-10462 Recategorize some of the client facing Public / Private > >>>>> interfaces > >>>>> This one is almost done. It could do with a finish, attention to the > >>>>> items in last comment, and then our codebase could do with another > >>>>> sweep > >>>>> after the spirit of this issue since a bunch has gone in since the > pass > >>>>> that was the basis of this issue. > >>>>> > >>>>> * HBASE-10504 Define Replication Interface > >>>>> I was going to take a crack at this as part of the revamp forced by > >>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service' > >>>>> but if > >>>>> anyone else is interested, be my guest. > >>>>> > >>>>> * HBASE-14996 Some more API cleanup for 2.0 > >>>>> Has a bunch of subtasks, some of which are being worked on. Needs > >>>>> finishing. > >>>>> > >>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and > >>>>> cleanup > >>>>> Needs a pass. Small issue I think. Could also look at new AsyncClient > >>>>> and > >>>>> make sure symmetry. > >>>>> > >>>>> * HBASE-15607 Remove PB references from Admin for 2.0 > >>>>> Predicated on result of an ongoing DISCUSSION thread but needs to be > >>>>> done. > >>>>> > >>>>> Rolling upgrade will have implications for our API. Would be good to > >>>>> try > >>>>> it and figure what needs fixup (as said above, according to trial by > >>>>> Sean, > >>>>> we might not be too bad here): > >>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0 > >>>>> cluster > >>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and > >>>>> 1.x > >>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2. > >>>>> > >>>>> * HBASE-17442 Move most of the replication related classes to > >>>>> hbase-server package > >>>>> The above would be good to do generally but it may make for ripples > in > >>>>> API so would be good to do now. > >>>>> > >>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo > >>>>> Balazs is working on this. The idea is that we avoid adding two new > >>>>> types > >>>>> to our API, two types that are nought but curtailed, read-only views > on > >>>>> internals. Input if you have time appreciated. > >>>>> > >>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a > hbase2 > >>>>> cluster; verify > >>>>> Esteban is looking at this one > >>>>> > >>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core > >>>>> * HBASE-17143 Scan improvement > >>>>> > >>>>> Our Coprocessor Interface needs a tough edit. It exposes > >>>>> implementations > >>>>> marked audience Private and returns implementations rather than > >>>>> Interfaces. > >>>>> In a few locations, we allow returning an alternate implementation > >>>>> altogether which is probably something we don't want a CP doing. To > >>>>> that > >>>>> end, the following issues started by Duo and Anoop need to be taken > to > >>>>> the > >>>>> finish line; ideally they'd have an owner: > >>>>> > >>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The > >>>>> umbrella issue. > >>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose > >>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and > >>>>> RegionServerObserver > >>>>> > >>>>> > >>>>> Nice-to-haves: > >>>>> > >>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove > unused > >>>>> TimeRange constructors > >>>>> > >>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references > >>>>> existing in the code > >>>>> This is the end of an old long-running project moving up on to Cell > >>>>> Interface. We think it is done but for a few little items (deprecate > KV > >>>>> methods in MR and provide Cell versions instead...) > >>>>> > >>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs > >>>>> fixing > >>>>> > >>>>> * HBASE-13346 Clean up Filter package for post 1.0 > >>>>> > >>>>> * HBASE-14255 Simplify Cell creation post 1.0 > >>>>> * HBASE-14997 > >>>>> Move compareOp and Comparators out of filter to client package > >>>>> > >>>>> * HBASE-13740 Stop using Hadoop private interfaces > >>>>> > >>>>> What about: > >>>>> > >>>>> * HBASE-18601 Remove Htrace 3.2 > >>>>> As has been noted, the HTrace API is our 'trace' API. > >>>>> > >>>>> If interested in any of the above and you need a legup, just ask in > the > >>>>> issue and I'll be by.... > >>>>> > >>>>> Thanks, > >>>>> St.Ack > >>>>> > >>>>> > >>>>> > >>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <[email protected]> wrote: > >>>>> > >>>>> Heads-up: > >>>>>> > >>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme > is > >>>>>> updated dependencies, reliance on relocated popular libs (guava, > >>>>>> netty, > >>>>>> protobuf), purge of checked-in generated src, and > >>>>>> master-carries-no-regions > >>>>>> by default. > >>>>>> > >>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme > will > >>>>>> be > >>>>>> settling the APIs and compatibility (At first blush, we are not > >>>>>> looking too > >>>>>> bad; our Sean ran some tests over weekend that have hbase-1 client > >>>>>> running > >>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp > >>>>>> should be > >>>>>> done by alpha3 (i.e. returning Interfaces rather than > >>>>>> Implementations, and > >>>>>> our shutdown of CPs accessing classes in hbase marked > >>>>>> InterfaceAudience). > >>>>>> We'll also have purged thirdparty classes from our API; e.g. guava > >>>>>> 0.12 > >>>>>> Service showing through in our replication API and protobufs in > Admin > >>>>>> Interface. On alpha3, we will have to do a bunch of outreach to make > >>>>>> sure > >>>>>> our downstreamers are up on what is coming down the pipe. > >>>>>> > >>>>>> Beta1 in mid-September? > >>>>>> > >>>>>> I encourage you to check out the items marked for hbase2: > >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188 > Edit > >>>>>> as > >>>>>> you see appropriate. Punt if you know the JIRA will not get any > >>>>>> attention > >>>>>> in next month or so. > >>>>>> > >>>>>> A bunch of issues marked blocker are unassigned. I'll leave them as > is > >>>>>> another while but I'll boot them soon. > >>>>>> > >>>>>> While I have your attention: > >>>>>> > >>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase > thrift > >>>>>> to 0.10.0 will break existing clients. The change is easy enough if > >>>>>> folks > >>>>>> need to upgrade their hbase thrift. See HBASE-18591. > >>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first > >>>>>> (0.98?). > >>>>>> > >>>>>> St.Ack > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <[email protected]> wrote: > >>>>>> > >>>>>> > >>>>>>> > >>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <[email protected]> > >>>>>>> wrote: > >>>>>>> > >>>>>>> > >>>>>>>> > >>>>>>>> On 7/31/17 9:00 AM, Stack wrote: > >>>>>>>> > >>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<[email protected]> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> ... > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the > >>>>>>>>>> feature a > >>>>>>>>>> bit more "real", but am obviously a little nervous (I have no > >>>>>>>>>> reason > >>>>>>>>>> to be > >>>>>>>>>> nervous though). I am pretty happy with the feature in terms of > >>>>>>>>>> how > >>>>>>>>>> much it > >>>>>>>>>> is covered via testing. > >>>>>>>>>> > >>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh. > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> Last think on space quota says that need doc too. See 'Space > >>>>>>>>> Quota' in > >>>>>>>>> here: > >>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i > >>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5 > >>>>>>>>> Does this little section need an update Josh? > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> S > >>>>>>>>> > >>>>>>>>> > >>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some > docs > >>>>>>>> already included for 17748. Happy to put that on my plate if > >>>>>>>> you're good > >>>>>>>> with it. I can reasonably assume that no one is against it :) > >>>>>>>> > >>>>>>>> I think I had knocked out docs for the "phase 1" stuff before we > >>>>>>>> merged it in from the original feature branch. I'll double check > >>>>>>>> and update > >>>>>>>> the gdoc. Perhaps this was just a timing thing. > >>>>>>>> > >>>>>>>> > >>>>>>> Thanks Josh, > >>>>>>> S > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > > >
