On Mon, May 9, 2016 at 11:59 PM, Gary Helmling <[email protected]> wrote: ...
> To me, it seems much safer to actively try to push this upstream into HDFS > right now, and still pointing to its optional, non-default use in HBase as > a compelling story. I don't understand why making it the default in 2.0 is > necessary for this. Do you really think it will make that big a difference > for upstreaming? Once it's actually in Hadoop and maintained, it seems > like a no-brainer to make it the default. > > The suggestion is that we make this new client the default now in master branch so we have plenty of time to find any issues with the implementation. We'd also enable it as the default because the improvement is dramatic (performance, less moving parts, comprehensible, etc.) and we think this async, lightweight, WAL-purposed client the way to go moving forward. We'd also spare our users having to make a choice; if optional, if they trip over the feature at all, they'll be wary enabling such a fundamental afraid that it experimental. Going this route we are taking on risk as you call out Gary but the suggestion is that the benefits far outweigh the downsides (In mitigation, I don't think we've ever run against HDFS free of reflection code though, true, we are into a new level of violation with this client). We are not arguing that it needs to be default to help get the async client upstreamed into HDFS. Maybe it would help but going by the issue cited by Duo below, it seems like there are a bunch of other concerns and dimensions to be considered first; it may take a while for an async DFS client to land (if at all). We should push on the upstreaming yes to close down the risk you note, but let us not predicate our use of async WAL on it first being committed to HDFS. Thanks, St.Ack > On Mon, May 9, 2016 at 5:09 PM Stack <[email protected]> wrote: > > > Any other suggestions/objections here? If not, will make the cut over in > > next day or so. > > Thanks, > > St.Ack > > > > On Thu, May 5, 2016 at 10:02 PM, Stack <[email protected]> wrote: > > > > > On Thu, May 5, 2016 at 7:39 PM, Yu Li <[email protected]> wrote: > > > > > >> Almost miss the party... > > >> > > >> bq. Do you think it worth to backport this feature to branch-1 and > > release > > >> it in the next 1.x release? This may introduce a compatibility issue > as > > >> said > > >> in HBASE-14949 that we need HBASE-14949 to make sure that the rolling > > >> upgrade > > >> does not lose data... > > >> From current perf data I think the effort is worthwhile, we already > > >> started > > >> some work here and will run it on production after some carefully > > testing > > >> (and of course, if the perf number confirmed, but I'm optimistic > somehow > > >> :-P). Regarding HBASE-14949, I guess a two-step rolling upgrade will > > make > > >> it work, right? (And I guess this will also be a question when we > > upgrade > > >> from 1.x to 2.0 later?) > > >> > > >> > > > Or a clean shutdown and restart? Or a fresh install? I'd think backport > > > would be fine if you have to enable it and it has warnings and is clear > > on > > > circumstances under which there could be dataloss. > > > > > > St.Ack > > > > > > > > > > > >> btw, I'm +1 about making asyncfswal as default in 2.0 :-) > > >> > > >> Best Regards, > > >> Yu > > >> > > >> On 6 May 2016 at 09:49, Ted Yu <[email protected]> wrote: > > >> > > >> > Thanks for your effort, Duo. > > >> > > > >> > I am in favor of turning AsyncWAL as default in master branch. > > >> > > > >> > Cheers > > >> > > > >> > On Thu, May 5, 2016 at 6:03 PM, 张铎 <[email protected]> wrote: > > >> > > > >> > > Some progress. > > >> > > > > >> > > I have filed HBASE-15743 for the transparent encryption support, > > >> > > and HBASE-15754 for the AES encryption UT. Now both of them are > > >> resolved. > > >> > > Let's resume the discussion here. > > >> > > > > >> > > Thanks. > > >> > > > > >> > > 2016-05-03 10:09 GMT+08:00 张铎 <[email protected]>: > > >> > > > > >> > > > Fine, will add the testcase. > > >> > > > > > >> > > > And for the RPC, we only implement a new client side DTP here > and > > >> still > > >> > > > use the original RPC. > > >> > > > > > >> > > > Thanks. > > >> > > > > > >> > > > 2016-05-03 3:20 GMT+08:00 Gary Helmling <[email protected]>: > > >> > > > > > >> > > >> On Fri, Apr 29, 2016 at 6:24 PM 张铎 <[email protected]> > > wrote: > > >> > > >> > > >> > > >> > Yes, it does. There is testcase that enumerates all the > > possible > > >> > > >> protection > > >> > > >> > level(authentication, integrity and privacy) and encryption > > >> > > >> algorithm(none, > > >> > > >> > 3des, rc4). > > >> > > >> > > > >> > > >> > > > >> > > >> > > > >> > > >> > > >> > > > > >> > > > >> > > > https://github.com/apache/hbase/blob/master/hbase-server/src/test/java/org/apache/hadoop/hbase/io/asyncfs/TestSaslFanOutOneBlockAsyncDFSOutput.java > > >> > > >> > > > >> > > >> > I have also tested it in a secure > cluster(hbase-2.0.0-SNAPSHOT > > >> and > > >> > > >> > hadoop-2.4.0). > > >> > > >> > > > >> > > >> > > >> > > >> Thanks. Can you add in support for testing with AES > > >> > > >> (dfs.encrypt.data.transfer.cipher.suites=AES/CTR/NoPadding)? > > This > > >> is > > >> > > only > > >> > > >> available in Hadoop 2.6.0+, but I think is far more likely to > be > > >> used > > >> > in > > >> > > >> production than 3des or rc4. > > >> > > > > > >> > > > > > >> > > >> Also, have you been following HADOOP-10768? That is changing > > >> Hadoop > > >> > RPC > > >> > > >> encryption negotiation to support more performant AES wrapping, > > >> > similar > > >> > > to > > >> > > >> what is now supported in the data transfer pipeline. > > >> > > >> > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > > > > >
