> I suspect this feature will be considered risky but I'd be willing to try it > out and give it some production-like load (and with chaos). By the time we > would be iterating our platform on 2.0 we'd have the type of production loads > fielded that could overpressure meta on a single server.
Thanks Andy. Am working on a 1.x (for internal use) and will try to put both a 1.x (for reference) and 2.x version. On Tuesday, November 22, 2016 10:05 AM, Andrew Purtell <andrew.purt...@gmail.com> wrote: Thanks. Are your experiences with FNv2 in production documented? Perhaps on the JIRA? That would go a long way. > On Nov 22, 2016, at 12:44 PM, Thiruvel Thirumoolan <thiru...@yahoo-inc.com> > wrote: > > Hi Andrew, > > Thanks for your comments. > > FN v2 happened due to the same reasons you mention. We believe v2 is complete > based on our experience using and operating it in production since early > 2016. We have a design doc @ umbrella jira HBASE-15531 and have responded to > all questions we had so far. More eyes will always help. > > -Thiruvel > > > On Tuesday, November 22, 2016, 7:13:08 AM PST, Andrew Purtell > <andrew.purt...@gmail.com> wrote: > Favored nodes version 1 did not see much adoption and was hard to use or > unusable, depending on who you ask, so please expect a focus on usability and > feature completeness when/if v2 is reviewed for possible inclusion in 2.0. > While I am not suggesting this is the case, if the v2 candidate shares the > feature incompleteness or usability problems of its predecessor it should not > go in. > > > > On Nov 21, 2016, at 6:01 PM, Thiruvel Thirumoolan > > <thiru...@yahoo-inc.com.INVALID> wrote: > > > > We would like to add the favored node enhancements as part of > > https://issues.apache.org/jira/browse/HBASE-15531 to 2.0. Most of the code > > should be new, there will be very less code changes to Master/AM etc. > > > > -Thiruvel > > > > On Monday, November 14, 2016, 10:06:49 PM PST, Anoop John > > <anoop.hb...@gmail.com> wrote:>- Anyone (Ram, Anoop?) wants to post a > > high-level writeup on the current > > up-to-date state of offheaping? > > > > Off heaping the read path (HBASE-11425) this is completed some time > > back. As per my knowledge at least 2 users back ported this work to > > 1.x based version and even running in production. Am leaving it them > > for giving details.. > > > > Write path off heaping HBASE-15179 is the umbrella issue. As u can > > see most of the sub tasks are done by now.. Mainly 2 more sub tasks > > are yet to get committed. Both are some what bigger sized also.. > > Patch is already available for them. We hope it can be completed by > > Nov end. > > > > Some more off heaping work, (in compaction path etc) might be taken up > > after write path work. Alibaba guys might be joining with that. Had > > some discuss with them.. We will come up with jira and doc for that > > before actual work begin. But as Enis said ya all that can go in 2.x > > (x>0) :-) > > > > Just giving some high level status update. Thanks .. > > > > -Anoop- > > > > > >> On Tue, Nov 15, 2016 at 7:31 AM, Nick Dimiduk <ndimi...@gmail.com> wrote: > >> Enis, by your criteria it seems the log4j2 upgrade ticket should be > >> considered for 2.0 as well: HBASE-12341. > >> > >>> On Monday, November 14, 2016, Enis Söztutar <enis....@gmail.com> wrote: > >>> > >>> Another way to look at whether a feature is a "blocker" or not for 2.0 is > >>> a > >>> simple test: > >>> - Can this feature be committed for 2.1, 2.2, etc or not assuming it does > >>> not make into 2.0. > >>> > >>> It is a simple test, but affectively validates whether the feature is > >>> client-visible and have backwards compatibility implications. If it is ok > >>> to bring the feature in for 2.1 or later, it means that the release should > >>> never block on it. Otherwise, we should at least have the API / Client > >>> visible parts of it in 2.0 if not the full thing. > >>> > >>> Looking at the features under discussion through this lens reveals easier > >>> decision points I think. A couple of examples: > >>> - Java async client / C++ Client: It is independent work, can come in 2.1 > >>> or 2.2, etc. Whenever it is completed. > >>> - Offheaping : transparent ti clients, so can come in incrementally in > >>> 2.0, 2.1, 2.2, etc. > >>> > >>> On the other hand, 1.x releases have been going on for >1.5 years, and > >>> will > >>> likely go on for another year at least. So the choices (dependency, APIs, > >>> etc) that we make now for 2.0 will likely live for >2 years. In that > >>> respect I would rather be a bit more aggressive in terms of dropping > >>> support for older stuff and updating wherever we can. For example require > >>> Hadoop-2.6+, Java-8, etc. > >>> > >>> Enis > >>> > >>> On Mon, Nov 14, 2016 at 12:15 PM, Mikhail Antonov <olorinb...@gmail.com > >>> <javascript:;>> > >>> wrote: > >>> > >>>> Great to see progress here. > >>>> > >>>> As I'm reading through the list, here're some high-level questions I > >>> have: > >>>> > >>>> - Regarding the work on new AssignmentManager - any notes on > >>>> perf/stability testing? Are you guys running tips of master branch > >>> through > >>>> ITBLL setup? > >>>> - Anyone (Ram, Anoop?) wants to post a high-level writeup on the > >>> current > >>>> up-to-date state of offheaping? > >>>> - Logical clock - at this point is it more like a nice to have feature > >>>> than "need to be done before 2.0"? What are the features blocked or > >>>> affected by lack thereof? > >>>> > >>>> -Mikhail > >>>> > >>>> On Sun, Nov 13, 2016 at 3:57 PM, Stack <st...@duboce.net <javascript:;>> > >>> wrote: > >>>> > >>>>> Thanks for the writeup Stephen. > >>>>> > >>>>> See below. > >>>>> > >>>>> On Fri, Nov 11, 2016 at 5:15 PM, Stephen Jiang < > >>> syuanjiang...@gmail.com <javascript:;>> > >>>>> wrote: > >>>>> > >>>>>> Hello, fellow HBASE developers, > >>>>>> > >>>>>> We are making progress towards HBASE 2.0 releases. I am using the > >>>>>> following queries to search for on-going HBASE 2.0 feature work items > >>>>>> (project = HBase AND (fixVersion = 2.0.0 OR affectedVersion = 2.0.0) > >>>> AND > >>>>>> resolution is EMPTY AND (issuetype != Bug AND issuetype != Test AND > >>>>>> issuetype != Sub-task) ORDER BY issuetype DESC), at this time, we > >>> have > >>>>>> 247! That is a lot. At some time in near future, we definitely need > >>>> to > >>>>>> trim down the list; otherwise, 2.0 will never be released. > >>>>>> > >>>>>> > >>>>> Agree. Suggest we all do our own review but at a certain stage, it is > >>> up > >>>> to > >>>>> the RMs to make a call on what shape of 2.0 will be. > >>>>> > >>>>> > >>>>> > >>>>>> For now, Matteo and I are tracking some big projects that are > >>> on-going: > >>>>>> > >>>>>> (1). HBASE-14350 the stable Assignment Manager (using Procedure V2) > >>>>>> - This is a blocker to have branch-2 cut. In the past few weeks, we > >>>> made > >>>>>> good progress and majority of implementation is done. The patches > >>> are > >>>>>> under review and testing. Matto is drafting a document for review. > >>>>>> > >>>>>> (2). HBASE-14414 Backup/Restore Phase 3 > >>>>>> - Currently it is blocked by HBASE-14123. The giant HBASE-14123 > >>>> patches > >>>>>> was discussed and reviewed from the community (see > >>>>>> http://www.mail-archive.com/dev@hbase.apache.org/msg41090.html for > >>> the > >>>>>> long > >>>>>> discussion); and all feedback were taken care of in the latest patch. > >>>>>> Currently I marked HBASE-14123 as 2.0 blocker, as without it, the > >>>> further > >>>>>> develpment of backup/restore is blocked - the backup/restore is a key > >>>>>> enterprise feature for 2.0 release. I think HBASE-14123 is ready for > >>>>>> another round of vote. > >>>>>> > >>>>>> > >>>>> IMO, not a blocker since it ancillary function but something we should > >>>> get > >>>>> in. > >>>>> > >>>>> > >>>>> > >>>>>> (3). HBASE-15179 Offheap > >>>>>> - This is another important feature that I think it is MUST for 2.0. > >>>>> Since > >>>>>> stack works closely with Intel developers, he has some insight on > >>> this > >>>>>> project: "Intel are betting on this. Alibaba are using the offheap > >>> read > >>>>>> path and interested in write path too. This work is still very much > >>> up > >>>> in > >>>>>> the air and being worked out as we go (especially the Y! Israel > >>>> inmemory > >>>>>> compaction component). It is a little shakey dependent on mslab > >>>> pooling, > >>>>>> blockcache being on by default, async wal being default, and then > >>>>> dependent > >>>>>> on lots of perf and ITBLL testing." > >>>>>> > >>>>>> > >>>>> The above is from a private, hurried email that does not do the current > >>>>> state justice. > >>>>> > >>>>> Ram and Anoop are the authorities on offheaping and have been keeping > >>> up > >>>>> their state in an associated doc. The lads might be persuaded do a > >>>> summary > >>>>> of current state here. > >>>>> > >>>>> Ditto for inmemory compaction. Our Anastasia and Eshcar are working > >>> hard > >>>> at > >>>>> the moment doing laste pieces and perf testing. Maybe we can a status > >>>>> dumped here in dev. > >>>>> > >>>>> > >>>>> > >>>>>> (4). HBASE-16952 Protobuf3 > >>>>>> - Good news, stack got the majority of work done already. This is a > >>>> MUST > >>>>>> for 2.0 release. Now we only have a small sub-task HBASE-16967 > >>>>> (findbugs) > >>>>>> left. > >>>>>> > >>>>>> > >>>>> To be clear, pb3 is under the offheaping umbrella. Let's add async WAL > >>> to > >>>>> this list, also needed by offheaping. > >>>>> > >>>>> Will be back after taking a look at current 2.0 list. > >>>>> > >>>>> Agree that hbase2 needs to support hadoop3. > >>>>> > >>>>> St.Ack > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>> (5). HBASE-14070 Logical Clock > >>>>>> - This needs to be done before 2.0 release. At this time, seems not > >>>>> making > >>>>>> much progress. > >>>>>> > >>>>>> (6). HBASE-16833 JAVA Async Client > >>>>>> - A lot of progress was made by Duo and his associates. We should be > >>>> in > >>>>> a > >>>>>> good shape in JAVA client when 2.0 release. > >>>>>> > >>>>>> (7). HBASE-14850 C++ Async Client > >>>>>> - Another project that is making good progress. I know some customer > >>>>> wants > >>>>>> this. This is long overdue project. Hopefully we will make those > >>>>> customer > >>>>>> happy in 2.0 release with a good performed C++ client. > >>>>>> > >>>>>> Please let me know other on-going projects that needs to be in HBASE > >>>> 2.0 > >>>>>> release (stack mentioned "logical file system tier", but I am not > >>> sure > >>>>>> whether it would make enough progress to have a realistic chance > >>> making > >>>>>> 2.0). As I said at the beginning of this email, at some point soon, > >>> we > >>>>>> will be more picky and trim down the unfinished features in 2.0. > >>>>>> > >>>>>> Happy Weekend! > >>>>>> Stephen > >>>>>> > >>>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> Thanks, > >>>> Michael Antonov > >>>>