RE: Apache Hadoop 2.8.3 Release Plan

ly fashion, let's try to address that... This is interesting. Do you aware any means to do that? Thanks! Regards, Kai -Original Message- From: Andrew Wang [mailto:andrew.w...@cloudera.com] Sent: Tuesday, November 21, 2017 2:22 PM To: Zheng, Kai <kai.zh...@intel.com> Cc: Junping

RE: Apache Hadoop 2.8.3 Release Plan

Hi Junping, Thank you for making 2.8.2 happen and now planning the 2.8.3 release. I have an ask, is it convenient to include the back port work for OSS connector module? We have some Hadoop users that wish to have it by default for convenience, though in the past they used it by back porting

RE: Backporting OSS module to branch 2.x

being voted and goes fine. My idea is to put the work in branch-2.9 first and expect some new release after the 2.9.0 one. Sure let me raise it on the 2.9 release thread when it's the right time. Thanks Chris again for the education and the thoughts. Regards, Kai -Original Message- From: Z

Backporting OSS module to branch 2.x

There was some discussion about backporting OSS module to branch 2.x and per Chris's suggestion we should do it in the dev list. -Original Message- From: Chris Douglas [mailto:cdoug...@apache.org] Sent: Thursday, November 16, 2017 1:20 AM To: Zheng, Kai <kai.zh...@intel.

RE: [VOTE] Merge yarn-native-services branch into trunk

Cool to have this feature! Thanks Jian and all. Regards, Kai -Original Message- From: Vinod Kumar Vavilapalli [mailto:vino...@apache.org] Sent: Tuesday, November 07, 2017 7:20 AM To: Jian He Cc: yarn-...@hadoop.apache.org; common-dev@hadoop.apache.org; Hdfs-dev

RE: [DISCUSS] A final minor release off branch-2?

Thanks Vinod. >> Of the top of my head, one of the biggest areas is application >> compatibility. When folks move from 2.x to 3.x, are their apps binary >> compatible? Source compatible? Or need changes? I thought these are good concerns from overall perspective. On the other hand, I've

RE: [VOTE] HADOOP-12756 - Aliyun OSS Support branch merge

comments in HADOOP-12756 and Genmao Yu for addressing them, doing the complete "hadoop fs" tests. Regards, Kai -Original Message----- From: Zheng, Kai [mailto:kai.zh...@intel.com] Sent: Wednesday, September 28, 2016 10:35 AM To: common-dev@hadoop.apache.org Subject: [VOTE] HADOOP-12756 -

RE: [VOTE] HADOOP-12756 - Aliyun OSS Support branch merge

Could I extend this a bit longer considering the PRC holiday (during Oct 1 and Oct 7)? If sounds good I'd like to have another week (next Wednesday) for this. Please advise if you'd like to think otherwise. Thanks. Regards, Kai -Original Message- From: Zheng, Kai [mailto:kai.zh

RE: desc error on official site http://hadoop.apache.org/

Thank you for the catch. This should go to the common-dev mailing list. Would you fire an issue to fix this? Regards, Kai -Original Message- From: 444...@qq.com [mailto:444...@qq.com] Sent: Wednesday, September 28, 2016 9:10 AM To: general Subject: desc

[VOTE] HADOOP-12756 - Aliyun OSS Support branch merge

Hi all, I would like to propose a merge vote for HADOOP-12756 branch to trunk. This branch develops support for Aliyun OSS (another cloud storage) in Hadoop. The voting starts now and will run for 7 days till Oct 5, 2016 07:00 PM PDT. Aliyun OSS is widely used among China's cloud users, and

RE: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0

Thanks Sammi. My non-binding +1 to make the release candidate. Regards, Kai -Original Message- From: Chen, Sammi Sent: Friday, September 02, 2016 4:59 PM To: Zheng, Kai <kai.zh...@intel.com>; Andrew Wang <andrew.w...@cloudera.com>; Arun Suresh <asur...@apache.org&

RE: [DISCUSS] The order of classpath isolation work and updating/shading dependencies on trunk

For the leveldb thing, wouldn't we have an alternative option in Java for the platforms where leveldb isn't supported yet due to whatever reasons. IMO, native library would be best to be used for optimization and production for performance. For development and pure Java platform, by default

RE: Setting JIRA fix versions for 3.0.0 releases

My humble feeling is almost the same regarding the urgent need of a 3.0 alpha release. Considering EC, shell-script rewriting and etc. are significant changes and there are interested users that want to evaluate EC storage method, an alpha 3.0 release will definitely help a lot allowing users

RE: Not being able to add HDFS contributors

Thanks Andrew for the work around!! It works great ... Regards, Kai -Original Message- From: Andrew Wang [mailto:andrew.w...@cloudera.com] Sent: Wednesday, July 20, 2016 8:10 AM To: Chris Douglas <chris.doug...@gmail.com> Cc: Zheng, Kai <kai.zh...@intel.com>

Not being able to add HDFS contributors

Hi, I tried many times in the week at different time but just found it's not possible to add more HDFS contributors. I can add some Hadoop ones, though. It becomes an issue because without adding someone and assigning issues to him first, he won't be able to work on it and upload patches ...

Clean up target/fix versions

Hi, I noticed it's pretty hard to opt a version (say 3.0-alpha1) in the fix/target version box in the JIRA system since the list is pretty long and not well sorted. Could we clean it up or resort/re-list them in order for the most possibly used ones to be displayed first. Regards, Kai

RE: Different JIRA permissions for HADOOP and HDFS

Yeah, this would be great, so some guys like me won't need to trouble you asking the question again and again :). Thanks a lot. Regards, Kai -Original Message- From: Akira AJISAKA [mailto:ajisa...@oss.nttdata.co.jp] Sent: Monday, June 20, 2016 3:17 PM To: Zheng, Kai <kai

RE: Different JIRA permissions for HADOOP and HDFS

To: Zheng, Kai <kai.zh...@intel.com>; common-dev@hadoop.apache.org Subject: Re: Different JIRA permissions for HADOOP and HDFS Yes, the role allows committers to add/remove all the roles. Now about 400 accounts have contributors roles in Hadoop common, and about 1000 contributors in h

RE: A top container module like hadoop-cloud for cloud integration modules

...@hortonworks.com] Sent: Wednesday, June 15, 2016 6:16 PM To: Zheng, Kai <kai.zh...@intel.com> Cc: common-dev@hadoop.apache.org Subject: Re: A top container module like hadoop-cloud for cloud integration modules > On 13 Jun 2016, at 14:02, Zheng, Kai <kai.zh...@intel.com>

RE: Different JIRA permissions for HADOOP and HDFS

, June 18, 2016 12:56 PM To: Zheng, Kai <kai.zh...@intel.com>; common-dev@hadoop.apache.org Subject: Re: Different JIRA permissions for HADOOP and HDFS I'm doing the following steps to reduce the number of contributors: 1. Find committers who have only contributor role 2. Add them into com

RE: Different JIRA permissions for HADOOP and HDFS

> > >> It looks like, I too dont have permissions in projects except >>>>HDFS. >>>>> > >> >>>>> > >> Please grant me also to the group. >>>>> > >> >>>>> > >> Thanks

A top container module like hadoop-cloud for cloud integration modules

Hi, Noticed it's an obvious trend Hadoop is supporting more and more cloud platforms, I suggest we have a top container module to hold such integration modules, like the ones for aws, openstack, azure and upcoming one aliyun. The rational is simple besides the trend: 1. Existing modules

RE: About fix versions

se. Regards, Kai -Original Message- From: Allen Wittenauer [mailto:allenwittena...@yahoo.com] Sent: Sunday, May 29, 2016 12:17 PM To: Zheng, Kai <kai.zh...@intel.com> Cc: common-dev@hadoop.apache.org Subject: Re: About fix versions > On May 28, 2016, at 5:13 PM, Zheng, Kai

About fix versions

Hi, This may be a stupid question but I want to make sure. What fix versions would we fill with when a committer just wants to commit a patch to trunk or branch-2 branch? I remembered it's a release manager's role to decide which jira/patch to include when working on a release. Would anyone

RE: Release numbering for 3.x leading up to GA

Thanks for thinking about this, Andrew and Zhe. I updated the patch today for HADOOP-13010 (rather large) and would be great if we could get it in the week. Regards, Kai -Original Message- From: Zhe Zhang [mailto:zhe.zhang.resea...@gmail.com] Sent: Friday, May 20, 2016 2:38 PM To:

RE: Different JIRA permissions for HADOOP and HDFS

It works for me now, thanks Andrew! Regards, Kai -Original Message- From: Andrew Wang [mailto:andrew.w...@cloudera.com] Sent: Monday, May 16, 2016 12:14 AM To: Zheng, Kai <kai.zh...@intel.com> Cc: common-dev@hadoop.apache.org Subject: Re: Different JIRA permissions for HADOOP an

RE: Different JIRA permissions for HADOOP and HDFS

state? Thanks! -Original Message- From: Zheng, Kai [mailto:kai.zh...@intel.com] Sent: Sunday, May 15, 2016 3:20 PM To: Allen Wittenauer <allenwittena...@yahoo.com.INVALID> Cc: common-dev@hadoop.apache.org Subject: RE: Different JIRA permissions for HADOOP and HDFS Thanks

RE: Different JIRA permissions for HADOOP and HDFS

] Sent: Saturday, May 14, 2016 9:38 AM To: Zheng, Kai <kai.zh...@intel.com> Cc: common-dev@hadoop.apache.org Subject: Re: Different JIRA permissions for HADOOP and HDFS > On May 14, 2016, at 7:07 AM, Zheng, Kai <kai.zh...@intel.com> wrote: > > Hi, > > Noticed this diff

RE: Different JIRA permissions for HADOOP and HDFS

reason? https://issues.apache.org/jira/browse/HADOOP-12782 https://issues.apache.org/jira/browse/HDFS-10285 Regards, Kai From: Ted Yu [mailto:yuzhih...@gmail.com] Sent: Saturday, May 14, 2016 7:28 AM To: Zheng, Kai <kai.zh...@intel.com> Cc: common-dev@hadoop.apache.org Subject: Re: Differen

Different JIRA permissions for HADOOP and HDFS

Hi, Noticed this difference but not sure if it's intended. YARN is similar with HDFS. It's not convenient. Any clarifying? Thanks. -kai [cid:image001.png@01D1ADAF.16160940] [cid:image002.png@01D1ADAF.16160940]

RE: Release numbering for 3.x leading up to GA

Ok, got it. Thanks for the explanation. Regards, Kai From: Andrew Wang [mailto:andrew.w...@cloudera.com] Sent: Tuesday, May 03, 2016 5:41 AM To: Zheng, Kai <kai.zh...@intel.com> Cc: common-dev@hadoop.apache.org Subject: Re: Release numbering for 3.x leading up to GA >> but I'm go

RE: Release numbering for 3.x leading up to GA

Thanks for driving this, Andrew. Sounds great. >> but I'm going to spend time on our first RC this week. Sorry what does this mean? Did you mean the first RC version or 3.0.0-alpha1 will be cut out this week? Anyway will try to get some tasks done sooner. Regards, Kai -Original

RE: hadoop 2.7.2 build failure with error "plugin descriptor"

Looks like you missed some maven plugin. May be helpful to run `mvn install` before the building. -Original Message- From: ? ? [mailto:yu20...@hotmail.com] Sent: Tuesday, March 29, 2016 3:39 PM To: common-dev@hadoop.apache.org Subject: hadoop 2.7.2 build failure with error "plugin

RE: [VOTE] Accept Chimera as new Apache Commons Component

Nice proposal. The high performance crypto offering is nice to be broadly accessible. Thanks. Non-binding +1. Regards, Kai -Original Message- From: Chris Nauroth [mailto:cnaur...@hortonworks.com] Sent: Wednesday, March 23, 2016 5:47 AM To: common-dev@hadoop.apache.org; Commons

RE: Style checking related to getters

'll review. > >Best, >Andrew > >On Sun, Feb 28, 2016 at 11:28 PM, Zheng, Kai <kai.zh...@intel.com> wrote: > >> Hi, >> >> I'm wondering if we could get rid of the style checking in getters >> like the following (from HDFS-9733). It's annoying because i

RE: Style checking related to getters

and post patch to disable that checkstyle rule? You can look at HADOOP-12713 for an example. Ping me and I'll review. Best, Andrew On Sun, Feb 28, 2016 at 11:28 PM, Zheng, Kai <kai.zh...@intel.com> wrote: > Hi, > > I'm wondering if we could get rid of the style checking in

Style checking related to getters

Hi, I'm wondering if we could get rid of the style checking in getters like the following (from HDFS-9733). It's annoying because it's a common Java practice and widely used in the project. void setBlockLocations(LocatedBlocks blockLocations) {:42: 'blockLocations' hides a field. void

RE: Introduce Apache Kerby to Hadoop

RPC calls or similar still. Throughput drops 3-4x, or > worse. > > > On Feb 22, 2016, at 4:56 PM, Zheng, Kai <kai.zh...@intel.com> wrote: > > > > Thanks for the confirm and further inputs, Steve. > > > >>> the latter would dramatically reduce the

RE: Introduce Apache Kerby to Hadoop

worse. > On Feb 22, 2016, at 4:56 PM, Zheng, Kai <kai.zh...@intel.com> wrote: > > Thanks for the confirm and further inputs, Steve. > >>> the latter would dramatically reduce the cost of wire-encrypting IPC. > Yes to optimize Hadoop IPC/RPC encryption is another oppo

RE: Introduce Apache Kerby to Hadoop

and ultimately get better APIs for > doing Kerberos and SASL —the latter would dramatically reduce the cost > of wire-encrypting IPC. > > For now, I'd like to see basic steps -upgrading minkdc to krypto, see > how it works. > > Long term, I'd like Hadoop 3 to be Kerby-ized &

Introduce Apache Kerby to Hadoop

Hi folks, I'd like to mention Apache Kerby [1] here to the community and propose to introduce the project to Hadoop, a sub project of Apache Directory project. Apache Kerby is a Kerberos centric project and aims to provide a first Java Kerberos library that contains both client and server

RE: Looking to a Hadoop 3 release

Thanks Andrew for driving this. Wonder if it's a good chance for HADOOP-12579 (Deprecate and remove WriteableRPCEngine) to be in. Note it's not an incompatible change, but feel better to be done in the major release. Regards, Kai -Original Message- From: Andrew Wang

Some questions about DistCp

Hi, I recently did some investigation about DistCp and have some questions. I thought before diving into JIRA things it would be good to discuss them first here. I read the doc at the following link and regard it as the latest revision that corresponds with the trunk codebase.

RE: Question about subtype exceptions in the thrown list in addition to a more general one

sult, any exception that needs to cross a process boundary over RPC generally ends up needing to subclass IOException. I don't think this is something that can be changed easily. --Chris Nauroth On 12/28/15, 7:46 PM, "Zheng, Kai" <kai.zh...@intel.com> wrote: > >Hi, >

Question about subtype exceptions in the thrown list in addition to a more general one

Hi, Would it be good to add to throw a subtype exception in addition to a more general exception already there in the thrown list? Is it some coding style that's required to follow or developers can do it as they like? It's often seen that only the general exception is in the thrown list,

RE: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk

Non-binding +1 According to our extensive performance tests, striping + ISA-L coder based erasure coding not only can save storage, but also can increase the throughput of a client or a cluster. It will be a great addition to HDFS and its users. Based on the latest branch codes, we also

RE: Jira down :(

It's OK now. Regards, Kai -Original Message- From: Vinayakumar B [mailto:vinayakum...@apache.org] Sent: Friday, June 05, 2015 3:12 PM To: common-dev@hadoop.apache.org Subject: Re: Jira down :( Thanks Tsuyoshi. Regards, Vinay On Fri, Jun 5, 2015 at 12:38 PM, Tsuyoshi Ozawa

RE: IMPORTANT: testing patches for branches

-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org Subject: Re: IMPORTANT: testing patches for branches On Apr 22, 2015, at 11:34 PM, Zheng, Kai kai.zh...@intel.com wrote: Hi Allen, This sounds great. Naming a patch foo-HDFS-7285.00.patch should get tested on the HDFS

RE: IMPORTANT: testing patches for branches

Hi Allen, This sounds great. Naming a patch foo-HDFS-7285.00.patch should get tested on the HDFS-7285 branch. Does it happen locally in developer's machine when running test-patch.sh, or also mean something in Hadoop Jenkins building when a JIRA becoming patch available? Thanks. Regards,

RE: [RFE] Support MIT Kerberos localauth plugin API

Hello Leo/Liou And the plugin interface can be as simple as this function (error handling ignored here) ... I thought it's good to have the pluggable allowing to customize the method how to perform the mapping. You could open a JIRA for this. If you'd like to work on it and need help,

RE: 2.7 status

Kumar Vavilapalli [mailto:vino...@hortonworks.com] Sent: Tuesday, March 03, 2015 11:31 AM To: Zheng, Kai Cc: mapreduce-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; Hadoop Common; yarn-...@hadoop.apache.org Subject: Re: 2.7 status Kai, please ping the reviewers that were already looking

RE: Looking to a Hadoop 3 release

Might I have some comments for this, just providing my thought. Thanks. If we start now, it might make it out by 2016. If we start now, downstreamers can start aligning themselves to land versions that suit at about the same time. Not only for down streamers to align with the long term

RE: Looking to a Hadoop 3 release

JDK8 support is in the consideration, looks like many issues were reported and resolved already. https://issues.apache.org/jira/browse/HADOOP-11090 -Original Message- From: Andrew Wang [mailto:andrew.w...@cloudera.com] Sent: Tuesday, March 03, 2015 7:20 AM To:

RE: 2.7 status

Is it interested to get the following issues in the release ? Thanks ! HADOOP-10670 HADOOP-10671 Regards, Kai -Original Message- From: Yongjun Zhang [mailto:yzh...@cloudera.com] Sent: Monday, March 02, 2015 4:46 AM To: hdfs-...@hadoop.apache.org Cc: Vinod Kumar Vavilapalli; Hadoop

RE: Looking to a Hadoop 3 release

Sorry for the bad. I thought it was sending to my colleagues. By the way, for the JDK8 support, we (Intel) would like to investigate further and help, thanks. Regards, Kai -Original Message- From: Zheng, Kai Sent: Tuesday, March 03, 2015 8:49 AM To: common-dev@hadoop.apache.org

RE: Anyone know how to mock a secured hdfs for unit test?

with something that more closely approximates a secured deployment. Chris Nauroth Hortonworks http://hortonworks.com/ On Thu, Jun 26, 2014 at 7:27 AM, Zheng, Kai kai.zh...@intel.com wrote: Hi David, Quite some time ago I opened HADOOP-9952 and planned to create secured MiniClusters by making use

RE: [DISCUSS] Security Efforts and Branching

] Sent: Wednesday, September 18, 2013 6:27 AM To: Zheng, Kai; Chen, Haifeng; common-dev@hadoop.apache.org Subject: Re: [DISCUSS] Security Efforts and Branching All - I apologize for not following up sooner. I have been heads down on some other matters that required my attention. It seems

RE: [DISCUSS] Security Efforts and Branching

. Regards, Kai From: larry mccay [mailto:larry.mc...@gmail.com] Sent: Wednesday, September 18, 2013 6:27 AM To: Zheng, Kai; Chen, Haifeng; common-dev@hadoop.apache.org Subject: Re: [DISCUSS] Security Efforts and Branching All - I apologize for not following up sooner. I have been heads down on some

RE: [DISCUSS] Hadoop SSO/Token Server Components

Got it Suresh. So I guess HADOOP-9797 (and the family) for the UGI change would be a fit to this rule right. The refactoring is improving and cleaning UGI, also preparing for TokenAuth feature. According to this rule the changes would be in trunk first. Thanks for your guidance. Regards, Kai

RE: Need help for building haddop source on windows 7

Perhaps you could try the 'branch-trunk-win' branch, and look for some building instructions in it. Not sure this works or not, though. -Original Message- From: Ranjan Dutta [mailto:rdbmsdata.ran...@gmail.com] Sent: Thursday, September 05, 2013 3:43 PM To: common-dev@hadoop.apache.org

RE: [DISCUSS] Hadoop SSO/Token Server Components

. If this is what others feel the next topic should be. If we can identify a jira home for it, we can do it there - otherwise we can create another DISCUSS thread for it. thanks, --larry On Jul 3, 2013, at 2:39 PM, Zheng, Kai kai.zh...@intel.com wrote: Hi

RE: [DISCUSS] Hadoop SSO/Token Server Components

a jira home for it, we can do it there - otherwise we can create another DISCUSS thread for it. thanks, --larry On Jul 3, 2013, at 2:39 PM, Zheng, Kai kai.zh...@intel.com wrote: Hi Larry, Thanks for the update. Good to see that with this update we are now

RE: [DISCUSS] Hadoop SSO/Token Server Components

to contribute code to these JIRAs. Regards, Kai -Original Message- From: Larry McCay [mailto:lmc...@hortonworks.com] Sent: Thursday, July 04, 2013 4:10 AM To: Zheng, Kai Cc: common-dev@hadoop.apache.org Subject: Re: [DISCUSS] Hadoop SSO/Token Server Components Hi Kai - I think that I need

RE: [DISCUSS] Hadoop SSO/Token Server Components

Hi Larry, Thanks for the update. Good to see that with this update we are now aligned on most points. I have also updated our TokenAuth design in HADOOP-9392. The new revision incorporates feedback and suggestions in related discussion with the community, particularly from Microsoft and

RE: Fostering a Hadoop security dev community

In my view it should be for the whole ecosystem. One inspiration of this is to ease the collaboration and discussion for the work on going about token based authentication and SSO, which absolutely targets the ecosystem, although the coming up libraries and facilities might reside in hadoop

RE: [jira] [Updated] (HADOOP-9477) posixGroups support for LDAP groups mapping service

Hi, Can anyone help take a look at this why the patch submitting won't trigger the checking of HADOOP-QA? Thanks. Regards, Kai -Original Message- From: Dapeng Sun (JIRA) [mailto:j...@apache.org] Sent: Sunday, May 05, 2013 11:00 PM To: Zheng, Kai Subject: [jira] [Updated] (HADOOP-9477

RE: [jira] [Updated] (HADOOP-9477) posixGroups support for LDAP groups mapping service

-9477) posixGroups support for LDAP groups mapping service Hi Zheng You can resubmit the patch again. Sometimes the Jenkins may not work well. On Mon, 2013-05-06 at 07:36 +, Zheng, Kai wrote: Hi, Can anyone help take a look at this why the patch submitting won't trigger the checking

About FileBasedGroupMapping provider and Virtual Groups

Hi everyone, Before I open a JIRA, I'd like to know how you like it, a file based group mapping provider. The idea is as follows. 1. Have a new user group mapping provider such as FileBasedGroupMapping, which consumes a mapping file like below: $HADOOP_CONF/groupsMapping.txt: group1:user1,user2

Questions and possible improvements for LdapGroupsMapping

Hi All, Regarding LdapGroupsMapping, I have following questions: 1. Is it possible to use ShellBasedUnixGroupsMapping for Hadoop service principals/users, and LdapGroupsMapping for end user accounts? In our environment, normal end users (along with their groups info) for Hadoop cluster

RE: Questions and possible improvements for LdapGroupsMapping

happen to be geared towards Active Directory. Thanks, Natty -Original Message- From: Zheng, Kai [mailto:kai.zh...@intel.com] Sent: Friday, October 19, 2012 8:32 AM To: common-dev@hadoop.apache.org Subject: Questions and possible improvements for LdapGroupsMapping Hi All, Regarding

RE: Questions and possible improvements for LdapGroupsMapping

JIRA is opened for this: https://issues.apache.org/jira/browse/HADOOP-8943 -Original Message- From: Zheng, Kai [mailto:kai.zh...@intel.com] Sent: Friday, October 19, 2012 10:17 AM To: common-dev@hadoop.apache.org; na...@cloudera.com Subject: RE: Questions and possible improvements