Hi Andrew, Thanks for your comments. FN v2 happened due to the same reasons you mention. We believe v2 is complete based on our experience using and operating it in production since early 2016. We have a design doc @ umbrella jira HBASE-15531 and have responded to all questions we had so far. More eyes will always help. -Thiruvel
On Tuesday, November 22, 2016, 7:13:08 AM PST, Andrew Purtell <andrew.purt...@gmail.com> wrote:Favored nodes version 1 did not see much adoption and was hard to use or unusable, depending on who you ask, so please expect a focus on usability and feature completeness when/if v2 is reviewed for possible inclusion in 2.0. While I am not suggesting this is the case, if the v2 candidate shares the feature incompleteness or usability problems of its predecessor it should not go in. > On Nov 21, 2016, at 6:01 PM, Thiruvel Thirumoolan > <thiru...@yahoo-inc.com.INVALID> wrote: > > We would like to add the favored node enhancements as part of > https://issues.apache.org/jira/browse/HBASE-15531 to 2.0. Most of the code > should be new, there will be very less code changes to Master/AM etc. > > -Thiruvel > > On Monday, November 14, 2016, 10:06:49 PM PST, Anoop John > <anoop.hb...@gmail.com> wrote:>- Anyone (Ram, Anoop?) wants to post a > high-level writeup on the current > up-to-date state of offheaping? > > Off heaping the read path (HBASE-11425) this is completed some time > back. As per my knowledge at least 2 users back ported this work to > 1.x based version and even running in production. Am leaving it them > for giving details.. > > Write path off heaping HBASE-15179 is the umbrella issue. As u can > see most of the sub tasks are done by now.. Mainly 2 more sub tasks > are yet to get committed. Both are some what bigger sized also.. > Patch is already available for them. We hope it can be completed by > Nov end. > > Some more off heaping work, (in compaction path etc) might be taken up > after write path work. Alibaba guys might be joining with that. Had > some discuss with them.. We will come up with jira and doc for that > before actual work begin. But as Enis said ya all that can go in 2.x > (x>0) :-) > > Just giving some high level status update. Thanks .. > > -Anoop- > > >> On Tue, Nov 15, 2016 at 7:31 AM, Nick Dimiduk <ndimi...@gmail.com> wrote: >> Enis, by your criteria it seems the log4j2 upgrade ticket should be >> considered for 2.0 as well: HBASE-12341. >> >>> On Monday, November 14, 2016, Enis Söztutar <enis....@gmail.com> wrote: >>> >>> Another way to look at whether a feature is a "blocker" or not for 2.0 is a >>> simple test: >>> - Can this feature be committed for 2.1, 2.2, etc or not assuming it does >>> not make into 2.0. >>> >>> It is a simple test, but affectively validates whether the feature is >>> client-visible and have backwards compatibility implications. If it is ok >>> to bring the feature in for 2.1 or later, it means that the release should >>> never block on it. Otherwise, we should at least have the API / Client >>> visible parts of it in 2.0 if not the full thing. >>> >>> Looking at the features under discussion through this lens reveals easier >>> decision points I think. A couple of examples: >>> - Java async client / C++ Client: It is independent work, can come in 2.1 >>> or 2.2, etc. Whenever it is completed. >>> - Offheaping : transparent ti clients, so can come in incrementally in >>> 2.0, 2.1, 2.2, etc. >>> >>> On the other hand, 1.x releases have been going on for >1.5 years, and will >>> likely go on for another year at least. So the choices (dependency, APIs, >>> etc) that we make now for 2.0 will likely live for >2 years. In that >>> respect I would rather be a bit more aggressive in terms of dropping >>> support for older stuff and updating wherever we can. For example require >>> Hadoop-2.6+, Java-8, etc. >>> >>> Enis >>> >>> On Mon, Nov 14, 2016 at 12:15 PM, Mikhail Antonov <olorinb...@gmail.com >>> <javascript:;>> >>> wrote: >>> >>>> Great to see progress here. >>>> >>>> As I'm reading through the list, here're some high-level questions I >>> have: >>>> >>>> - Regarding the work on new AssignmentManager - any notes on >>>> perf/stability testing? Are you guys running tips of master branch >>> through >>>> ITBLL setup? >>>> - Anyone (Ram, Anoop?) wants to post a high-level writeup on the >>> current >>>> up-to-date state of offheaping? >>>> - Logical clock - at this point is it more like a nice to have feature >>>> than "need to be done before 2.0"? What are the features blocked or >>>> affected by lack thereof? >>>> >>>> -Mikhail >>>> >>>> On Sun, Nov 13, 2016 at 3:57 PM, Stack <st...@duboce.net <javascript:;>> >>> wrote: >>>> >>>>> Thanks for the writeup Stephen. >>>>> >>>>> See below. >>>>> >>>>> On Fri, Nov 11, 2016 at 5:15 PM, Stephen Jiang < >>> syuanjiang...@gmail.com <javascript:;>> >>>>> wrote: >>>>> >>>>>> Hello, fellow HBASE developers, >>>>>> >>>>>> We are making progress towards HBASE 2.0 releases. I am using the >>>>>> following queries to search for on-going HBASE 2.0 feature work items >>>>>> (project = HBase AND (fixVersion = 2.0.0 OR affectedVersion = 2.0.0) >>>> AND >>>>>> resolution is EMPTY AND (issuetype != Bug AND issuetype != Test AND >>>>>> issuetype != Sub-task) ORDER BY issuetype DESC), at this time, we >>> have >>>>>> 247! That is a lot. At some time in near future, we definitely need >>>> to >>>>>> trim down the list; otherwise, 2.0 will never be released. >>>>>> >>>>>> >>>>> Agree. Suggest we all do our own review but at a certain stage, it is >>> up >>>> to >>>>> the RMs to make a call on what shape of 2.0 will be. >>>>> >>>>> >>>>> >>>>>> For now, Matteo and I are tracking some big projects that are >>> on-going: >>>>>> >>>>>> (1). HBASE-14350 the stable Assignment Manager (using Procedure V2) >>>>>> - This is a blocker to have branch-2 cut. In the past few weeks, we >>>> made >>>>>> good progress and majority of implementation is done. The patches >>> are >>>>>> under review and testing. Matto is drafting a document for review. >>>>>> >>>>>> (2). HBASE-14414 Backup/Restore Phase 3 >>>>>> - Currently it is blocked by HBASE-14123. The giant HBASE-14123 >>>> patches >>>>>> was discussed and reviewed from the community (see >>>>>> http://www.mail-archive.com/dev@hbase.apache.org/msg41090.html for >>> the >>>>>> long >>>>>> discussion); and all feedback were taken care of in the latest patch. >>>>>> Currently I marked HBASE-14123 as 2.0 blocker, as without it, the >>>> further >>>>>> develpment of backup/restore is blocked - the backup/restore is a key >>>>>> enterprise feature for 2.0 release. I think HBASE-14123 is ready for >>>>>> another round of vote. >>>>>> >>>>>> >>>>> IMO, not a blocker since it ancillary function but something we should >>>> get >>>>> in. >>>>> >>>>> >>>>> >>>>>> (3). HBASE-15179 Offheap >>>>>> - This is another important feature that I think it is MUST for 2.0. >>>>> Since >>>>>> stack works closely with Intel developers, he has some insight on >>> this >>>>>> project: "Intel are betting on this. Alibaba are using the offheap >>> read >>>>>> path and interested in write path too. This work is still very much >>> up >>>> in >>>>>> the air and being worked out as we go (especially the Y! Israel >>>> inmemory >>>>>> compaction component). It is a little shakey dependent on mslab >>>> pooling, >>>>>> blockcache being on by default, async wal being default, and then >>>>> dependent >>>>>> on lots of perf and ITBLL testing." >>>>>> >>>>>> >>>>> The above is from a private, hurried email that does not do the current >>>>> state justice. >>>>> >>>>> Ram and Anoop are the authorities on offheaping and have been keeping >>> up >>>>> their state in an associated doc. The lads might be persuaded do a >>>> summary >>>>> of current state here. >>>>> >>>>> Ditto for inmemory compaction. Our Anastasia and Eshcar are working >>> hard >>>> at >>>>> the moment doing laste pieces and perf testing. Maybe we can a status >>>>> dumped here in dev. >>>>> >>>>> >>>>> >>>>>> (4). HBASE-16952 Protobuf3 >>>>>> - Good news, stack got the majority of work done already. This is a >>>> MUST >>>>>> for 2.0 release. Now we only have a small sub-task HBASE-16967 >>>>> (findbugs) >>>>>> left. >>>>>> >>>>>> >>>>> To be clear, pb3 is under the offheaping umbrella. Let's add async WAL >>> to >>>>> this list, also needed by offheaping. >>>>> >>>>> Will be back after taking a look at current 2.0 list. >>>>> >>>>> Agree that hbase2 needs to support hadoop3. >>>>> >>>>> St.Ack >>>>> >>>>> >>>>> >>>>> >>>>>> (5). HBASE-14070 Logical Clock >>>>>> - This needs to be done before 2.0 release. At this time, seems not >>>>> making >>>>>> much progress. >>>>>> >>>>>> (6). HBASE-16833 JAVA Async Client >>>>>> - A lot of progress was made by Duo and his associates. We should be >>>> in >>>>> a >>>>>> good shape in JAVA client when 2.0 release. >>>>>> >>>>>> (7). HBASE-14850 C++ Async Client >>>>>> - Another project that is making good progress. I know some customer >>>>> wants >>>>>> this. This is long overdue project. Hopefully we will make those >>>>> customer >>>>>> happy in 2.0 release with a good performed C++ client. >>>>>> >>>>>> Please let me know other on-going projects that needs to be in HBASE >>>> 2.0 >>>>>> release (stack mentioned "logical file system tier", but I am not >>> sure >>>>>> whether it would make enough progress to have a realistic chance >>> making >>>>>> 2.0). As I said at the beginning of this email, at some point soon, >>> we >>>>>> will be more picky and trim down the unfinished features in 2.0. >>>>>> >>>>>> Happy Weekend! >>>>>> Stephen >>>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Thanks, >>>> Michael Antonov >>>>
