[ANNOUNCE] New Hadoop PMC member - Mukund Thakur

[Vinayakumar B]

On behalf of the Apache Hadoop PMC, I am pleased to announce that Mukund
Thakur (mthakur) has accepted the PMC's invitation to become a PMC member on
the project. We appreciate all of Mukund’s generous contributions thus far
and look forward to his continued involvement.

Congratulations and welcome, Mukund !

-Vinayakumar B
-- 
-Vinay

Re: [DISCUSS] Race condition in ProtobufRpcEngine2

[Vinayakumar B]

Call.toString() was used only while sending the response, in the response
processor thread. At this point, header would be already read and
processed. So no chance of race conditions.

This race condition was not introduced by the HADOOP-17046
, instead existing code
was never intended to be used for logging purposes by multiple threads.
Though the locking mechanism proposed in HADOOP-18143
 solves the problems,
it adds overhead in both memory as well as performance.

This is a critical path for RPC processing, and adding a
locking/synchronization in this place would affect the overall performance
(may not be huge).

I would advise to remove logging and locking both altogether and leave the
code as is in Reader thread.

If logging is required, then add it in the response processor thread or the
handler threads.

-Vinay


On Mon, Feb 28, 2022 at 10:02 PM Ayush Saxena  wrote:

> Hey Andras,
> I had a quick look at HADOOP-18143, the methods in question in
> ProtobufRpcEngine2 are identical to the ones in ProtobufRpcEngine. So, I am
> not very sure how the  race condition doesn't happen in  ProtobufRpcEngine.
> I have to debug and spend some more time, considering that I have
> reverted HADOOP-18082 for now to unblock YARN. Though the issue would still
> be there as you said, but will give us some time to analyse.
>
> Thanks
> -Ayush
>
> On Mon, 28 Feb 2022 at 21:26, Gyori Andras 
> wrote:
>
>> Hey everyone!
>>
>> We have started seeing test failures in YARN PRs for a while. We have
>> identified the problematic commit, which is HADOOP-18082
>> , however, this
>> change
>> just revealed the race condition lying in ProtobufRpcEngine2 introduced in
>> HADOOP-17046 . We
>> have
>> also fixed the underlying issue via a locking mechanism, presented in
>> HADOOP-18143 , but
>> since it is out of our area of expertise, we can neither verify nor
>> guarantee that it will not cause some subtle issues in the RPC system.
>> As we think it is a core part of Hadoop, we would use feedback from
>> someone
>> who is proficient in this part.
>>
>> Regards:
>> Andras
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

[Vinayakumar B]

+1 (Binding)

1. Built from Tag.
2. Successful Native Build on Ubuntu 20.04
3. Verified Checksums
4. Deployed the docker cluster with 3 nodes
5. Ran sample MR Jobs

-Vinay


On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena  wrote:

> +1,
> Built from Source.
> Successful Native Build on Ubuntu 20.04
> Verified Checksums
> Ran basic hdfs shell commands.
> Ran simple MR jobs.
> Browsed NN,DN,RM and NM UI.
>
> Thanx Wei-Chiu for driving the release.
>
> -Ayush
>
>
> > On 12-Jun-2021, at 1:45 AM, epa...@apache.org wrote:
> >
> > +1 (binding)
> > Eric
> >
> >
> > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
> weic...@apache.org> wrote:
> >
> > Hi community,
> >
> > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> blocker
> > issues have been resolved [1] again.
> >
> > There are 2 additional issues resolved for RC3:
> > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > HADOOP-16878
> > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> source
> > and destination are the same
> >
> > There are 4 issues resolved for RC2:
> > * HADOOP-17666. Update LICENSE for 3.3.1
> > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> >
> > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> > fixes compared to hadoop-thirdparty 1.1.0:
> > * HADOOP-17707. Remove jaeger document from site index.
> > * HADOOP-17730. Add back error_prone
> >
> > *RC tag is release-3.3.1-RC3
> > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> >
> > *The RC3 artifacts are at*:
> > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> >
> > *The maven artifacts are hosted here:*
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> >
> > *My public key is available here:*
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> >
> > Things I've verified:
> > * all blocker issues targeting 3.3.1 have been resolved.
> > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > * LICENSE and NOTICE files checked
> > * RELEASENOTES and CHANGELOG
> > * rat check passed.
> > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> > Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> and
> > dependency divergence. Issues are being identified but so far nothing
> > blocker for Hadoop itself.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > My +1 to start,
> >
> > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > [2]
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Re: [DISCUSS] Hadoop 3.3.1 release

[Vinayakumar B]

Hi David,

>> Still hoping for help here:

>> https://issues.apache.org/jira/browse/HDFS-15790

I will raise a PR for the said solution soon (in a day or two).

-Vinay

On Thu, 11 Mar 2021 at 5:39 AM, David  wrote:

> Hello,
>
> Still hoping for help here:
>
> https://issues.apache.org/jira/browse/HDFS-15790
>
> Looks like it has been worked on, not sure how to best move it forward.
>
> On Wed, Mar 10, 2021, 12:21 PM Steve Loughran  >
> wrote:
>
> > I'm going to argue its too late to do IPv6 support close to a release, as
> > it's best if its on developer machines for some time to let all quirks
> > surface. It's not so much IPv6 itself, but do we cause any regressions on
> > IPv4?
> >
> > But: it can/should go into trunk and stabilize there
> >
> > On Thu, 4 Mar 2021 at 03:52, Muralikrishna Dmmkr <
> > muralikrishna.dm...@gmail.com> wrote:
> >
> > > Hi Brahma,
> > >
> > > I have missed out mentioning about the IPV6 feature in the last mail,
> > > Support for IPV6 has been in development since 2015 and We have done a
> > good
> > > amount of testing at our organisation, the feature is stable and used
> by
> > > our customers extensively in the last one year. I think it is a good
> time
> > > to add the IPV6 support to 3.3.1.
> > >
> > > https://issues.apache.org/jira/browse/HADOOP-11890
> > >
> > > Thanks
> > > D M Murali Krishna Reddy
> > >
> > > On Wed, Feb 24, 2021 at 9:13 AM Muralikrishna Dmmkr <
> > > muralikrishna.dm...@gmail.com> wrote:
> > >
> > > > Hi Brahma,
> > > >
> > > > Can we have this new feature "YARN Registry based AM discovery with
> > retry
> > > > and in-flight task persistent via JHS" in the upcoming 3.1.1
> release. I
> > > > have also attached a test-report in the below jira.
> > > >
> > > > https://issues.apache.org/jira/browse/MAPREDUCE-6726
> > > >
> > > >
> > > > Thanks,
> > > > D M Murali Krishna Reddy
> > > >
> > > > On Tue, Feb 23, 2021 at 10:11 AM Brahma Reddy Battula <
> > bra...@apache.org
> > > >
> > > > wrote:
> > > >
> > > >> Hi Bilwa,
> > > >>
> > > >> I have commented on the jira's you mentioned. Based on the stability
> > we
> > > >> can
> > > >> plan this.But needs to be merged ASAP.
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Feb 19, 2021 at 5:20 PM bilwa st  wrote:
> > > >>
> > > >> > Hi Brahma,
> > > >> >
> > > >> > Can we have below features in 3.3.1 release? We have been using
> > these
> > > >> > features for a long time. They are stable and tested in bigger
> > > clusters.
> > > >> >
> > > >> > 1. Container reuse -
> > > >> https://issues.apache.org/jira/browse/MAPREDUCE-6749
> > > >> > 2. Speculative attempts should not run on the same node -
> > > >> > https://issues.apache.org/jira/browse/MAPREDUCE-7169
> > > >> >
> > > >> > Thanks,
> > > >> > Bilwa
> > > >> >
> > > >> > On Thu, Feb 18, 2021, 1:49 PM Brahma Reddy Battula <
> > bra...@apache.org
> > > >
> > > >> > wrote:
> > > >> >
> > > >> >> Sorry for the late reply..
> > > >> >>
> > > >> >> I will come up with a plan.. Please let me know if anybody has
> some
> > > >> >> features/improvements/bugs that need to be included.
> > > >> >>
> > > >> >> On Mon, Feb 15, 2021 at 9:39 PM Sunil Govindan <
> sun...@apache.org>
> > > >> wrote:
> > > >> >>
> > > >> >> > Hi Wei-Chiu,
> > > >> >> >
> > > >> >> > What will be the next steps here for 3.3.1 planning?
> > > >> >> >
> > > >> >> > Thanks
> > > >> >> > Sunil
> > > >> >> >
> > > >> >> > On Mon, Feb 8, 2021 at 11:56 PM Stack 
> wrote:
> > > >> >> >
> > > >> >> > > On Wed, Feb 3, 2021 at 6:41 AM Steve Loughran
> > > >> >> >  > > >> >> > > >
> > > >> >> > > wrote:
> > > >> >> > >
> > > >> >> > > >
> > > >> >> > > > Regarding blockers &c: how about we have a little hackathon
> > > >> where we
> > > >> >> > try
> > > >> >> > > > and get things in. This means a promise of review time from
> > the
> > > >> >> people
> > > >> >> > > with
> > > >> >> > > > commit rights and other people who understand the code
> > (Stack?)
> > > >> >> > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> > > I'm up for helping get 3.3.1 out (reviewing, hackathon,
> > testing).
> > > >> >> > > Thanks,
> > > >> >> > > S
> > > >> >> > >
> > > >> >> > >
> > > >> >> > >
> > > >> >> > >
> > > >> >> > > > -steve
> > > >> >> > > >
> > > >> >> > > > On Thu, 28 Jan 2021 at 06:48, Ayush Saxena <
> > ayush...@gmail.com
> > > >
> > > >> >> wrote:
> > > >> >> > > >
> > > >> >> > > > > +1
> > > >> >> > > > > Just to mention we would need to release
> hadoop-thirdparty
> > > too
> > > >> >> > before.
> > > >> >> > > > > Presently we are using the snapshot version of it.
> > > >> >> > > > >
> > > >> >> > > > > -Ayush
> > > >> >> > > > >
> > > >> >> > > > > > On 28-Jan-2021, at 6:59 AM, Wei-Chiu Chuang <
> > > >> weic...@apache.org
> > > >> >> >
> > > >> >> > > > wrote:
> > > >> >> > > > > >
> > > >> >> > > > > > Hi all,
> > > >> >> > > > > >
> > > >> >> > > > > > Hadoop 3.3.0 was released half a year ago, and as of
> now
> > > >> we've
> > > >> >> > > > > accumulated
> > > >> >> > > > > > more than 400 c

Re: Wire compatibility between Hadoop 3.x client and 2.x server

[Vinayakumar B]

Thanks Jianliang,

I saw jira assigned to you.  Are you planning to provide a fix as well for
this?

If not, would you mind assigning to me?

-Vinay

On Wed, 28 Oct 2020 at 3:46 PM, Wu,Jianliang(vip.com) <
jianliang...@vipshop.com> wrote:

> Hi  VinayaKumar and Wei-Chiu
>
> I filed jira https://issues.apache.org/jira/browse/HDFS-15660 for details
>
>
> 2020年10月28日 17:08，Vinayakumar B  写道：
>
> Hi Wu,Jianliang,
> Have you created the Jira for the issue you mentioned due to
> getContentSummary?
>
> I might have a fix for this. Of-course it needs to be applied both client
> and server side.
>
> Let me know.
>
> -Vinay
>
>
> On Wed, Oct 14, 2020 at 12:26 PM Wu,Jianliang(vip.com) <
> jianliang...@vipshop.com> wrote:
>
> Ok, I will file a HDFS jira to report this issue.
>
>
> 2020年10月13日 20:43，Wei-Chiu Chuang  写道：
>
> Thanks Jialiang for reporting the issue.
> That sounds bad and should've not happened. Could you file a HDFS jira
>
> and
>
> fill in more details?
>
> On Mon, Oct 12, 2020 at 8:59 PM Wu,Jianliang(vip.com) <
> jianliang...@vipshop.com> wrote:
>
> In our case, when nn has upgraded to 3.1.3 and dn’s version was still
> 2.6,  we found hive to call getContentSummary method , the client and
> server was not compatible  because of hadoop3 added new PROVIDED storage
> type.
>
> 2020年10月13日 06:41，Chao Sun 
> sunc...@apache.org>>
>
> 写道：
>
>
>
>
>
> 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！
>
> This communication is intended only for the addressee(s) and may contain
> information that is privileged and confidential. You are hereby notified
> that, if you are not an intended recipient listed above, or an
>
> authorized
>
> employee or agent of an addressee of this communication responsible for
> delivering e-mail messages to an intended recipient, any dissemination,
> distribution or reproduction of this communication (including any
> attachments hereto) is strictly prohibited. If you have received this
> communication in error, please notify us immediately by a reply e-mail
> addressed to the sender and permanently delete the original e-mail
> communication and any attachments from all storage devices without
>
> making
>
> or otherwise retaining a copy.
>
>
>
> 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！
> This communication is intended only for the addressee(s) and may contain
> information that is privileged and confidential. You are hereby notified
> that, if you are not an intended recipient listed above, or an authorized
> employee or agent of an addressee of this communication responsible for
> delivering e-mail messages to an intended recipient, any dissemination,
> distribution or reproduction of this communication (including any
> attachments hereto) is strictly prohibited. If you have received this
> communication in error, please notify us immediately by a reply e-mail
> addressed to the sender and permanently delete the original e-mail
> communication and any attachments from all storage devices without making
> or otherwise retaining a copy.
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> 
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> 
>
>
> 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！
> This communication is intended only for the addressee(s) and may contain
> information that is privileged and confidential. You are hereby notified
> that, if you are not an intended recipient listed above, or an authorized
> employee or agent of an addressee of this communication responsible for
> delivering e-mail messages to an intended recipient, any dissemination,
> distribution or reproduction of this communication (including any
> attachments hereto) is strictly prohibited. If you have received this
> communication in error, please notify us immediately by a reply e-mail
> addressed to the sender and permanently delete the original e-mail
> communication and any attachments from all storage devices without making
> or otherwise retaining a copy.
>
-- 
-Vinay

Re: Wire compatibility between Hadoop 3.x client and 2.x server

[Vinayakumar B]

Hi Wu,Jianliang,
Have you created the Jira for the issue you mentioned due to
getContentSummary?

I might have a fix for this. Of-course it needs to be applied both client
and server side.

Let me know.

-Vinay


On Wed, Oct 14, 2020 at 12:26 PM Wu,Jianliang(vip.com) <
jianliang...@vipshop.com> wrote:

> Ok, I will file a HDFS jira to report this issue.
>
>
> > 2020年10月13日 20:43，Wei-Chiu Chuang  写道：
> >
> > Thanks Jialiang for reporting the issue.
> > That sounds bad and should've not happened. Could you file a HDFS jira
> and
> > fill in more details?
> >
> > On Mon, Oct 12, 2020 at 8:59 PM Wu,Jianliang(vip.com) <
> > jianliang...@vipshop.com> wrote:
> >
> >> In our case, when nn has upgraded to 3.1.3 and dn’s version was still
> >> 2.6,  we found hive to call getContentSummary method , the client and
> >> server was not compatible  because of hadoop3 added new PROVIDED storage
> >> type.
> >>
> >> 2020年10月13日 06:41，Chao Sun  sunc...@apache.org>>
> >> 写道：
> >>
> >>
> >>
> >>
> 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！
> >> This communication is intended only for the addressee(s) and may contain
> >> information that is privileged and confidential. You are hereby notified
> >> that, if you are not an intended recipient listed above, or an
> authorized
> >> employee or agent of an addressee of this communication responsible for
> >> delivering e-mail messages to an intended recipient, any dissemination,
> >> distribution or reproduction of this communication (including any
> >> attachments hereto) is strictly prohibited. If you have received this
> >> communication in error, please notify us immediately by a reply e-mail
> >> addressed to the sender and permanently delete the original e-mail
> >> communication and any attachments from all storage devices without
> making
> >> or otherwise retaining a copy.
> >>
>
> 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！
> This communication is intended only for the addressee(s) and may contain
> information that is privileged and confidential. You are hereby notified
> that, if you are not an intended recipient listed above, or an authorized
> employee or agent of an addressee of this communication responsible for
> delivering e-mail messages to an intended recipient, any dissemination,
> distribution or reproduction of this communication (including any
> attachments hereto) is strictly prohibited. If you have received this
> communication in error, please notify us immediately by a reply e-mail
> addressed to the sender and permanently delete the original e-mail
> communication and any attachments from all storage devices without making
> or otherwise retaining a copy.
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>

Re: Collect feedback for HDFS-15638

[Vinayakumar B]

IIUC, hive renames are from hive’s staging directory during write to final
destination within table.

Why not set the default ACLs of staging directory to whatever expected, and
then continue write remaining files.

In this way even after rename you will have expected ACLs on the final
files.

Setting default ACLs on staging directory can be done using single RPC.

-Vinay

On Sat, 17 Oct 2020 at 8:08 AM, Xinli shang  wrote:

> Thanks Owen for your reply! As mentioned in the Jira, default ACLs don't
> apply to rename. Any idea how rename can work without setting ACLs per
> file?
>
> On Fri, Oct 16, 2020 at 7:25 PM Owen O'Malley 
> wrote:
>
> > I'm very -1 on adding these semantics.
> >
> > When you create the table's directory, set the default ACL. That will
> have
> > exactly the effect that you are looking for without creating additional
> > semantics.
> >
> > .. Owen
> >
> > On Fri, Oct 16, 2020 at 7:02 PM Xinli shang 
> > wrote:
> >
> > > Hi all,
> > >
> > > I opened https://issues.apache.org/jira/browse/HDFS-15638 and want to
> > > collect feedback from the community. I know whenever changing the
> > > permission model that follows POSIX model is never a trivial change. So
> > > please comment on if you have concerns. For reading convenience, here
> is
> > a
> > > copy of the ticket.
> > >
> > > *Problem*: Currently, when a user tries to accesses a file he/she needs
> > the
> > > permissions of it's parent and ancestors and the permission of that
> file.
> > > This is correct generally, but for Hive tables directories/files, all
> the
> > > files under a partition or even a table usually have the same
> permissions
> > > for the same set of ACL groups. Although the permissions and ACL groups
> > are
> > > the same, the writer still need to call setfacl() for every file to add
> > > LDAP groups. This results in a huge amount of RPC calls to NN. HDFS has
> > > default ACL to solve that but that only applies to create and copy, but
> > not
> > > apply to rename. However, in Hive ETL, rename is very common.
> > >
> > > *Proposal*: Add a 1-bit flag to directory inodes to indicate whether or
> > not
> > > it is a Hive table directory. If that flag is set, then all the
> > > sub-directory and files under it will just use it's permission and ACL
> > > groups settings. By doing this way, Hive ETL doesn't need to set
> > > permissions at the file level. If that flag is not set(by default),
> work
> > as
> > > before. To set/unset that flag, it would require admin privilege.
> > >
> > > --
> > > Xinli Shang
> > >
> >
>
>
> --
> Xinli Shang
>
-- 
-Vinay

Re: [VOTE] Release Apache Hadoop 2.10.1 (RC0)

[Vinayakumar B]

Apologies for delayed voting.

+1 (Binding)

1. Verified the signature. Signatures are good.
2. Verified the sha512 checksum
   (I tried to use the sha512sum tool to verify. But ".sha512"
files doesn't have the expected format. I guess these files are regenerated
using "gpg --print-md", not the one used by createrelease scripts,
'sha512sum --tag'. It would be easy to verify using tool if the .sha512
have the proper format)
3. src tar have the "patchprocess" directory, may be need to backport the
fix in trunk to branch-2.10 to avoid this in future.
4. Deployed 3 node docker cluster and ran basic jobs. All-Ok.

-Vinay


On Mon, Sep 21, 2020 at 5:24 AM Wei-Chiu Chuang  wrote:

> +1 (binding)
>
> I did a security scan for the 2.10.1 RC0 and it looks fine to me.
>
> Checked recent critical/blocker HDFS issues that are not in 2.10.1. It
> looks mostly fine. Most of them are Hadoop 3.x features (EC, ... etc) but
> there is one worth attention:
>
>
>1. HDFS-14674  [SBN
>read] Got an unexpected txid when tail editlog.
>2.
>   1. But looking at the jira, it doesn't apply to 2.x so I think we are
>   good there.
>   2.
>   3.
>   4. I wanted to do an API compat check but didn't finish it yet. If
>   someone can do it quickly that would be great. (Does anyone know
> of a cloud
>   service that we can quickly do a Java API compat check?)
>
> Cheers,
> Wei-Chiu
>
> On Sun, Sep 20, 2020 at 9:25 AM Sunil Govindan  wrote:
>
> > +1 (binding)
> >
> > - verified checksum and sign. Shows as a Good signature from "Masatake
> > Iwasaki (CODE SIGNING KEY) "
> > - built from source
> > - ran basic MR job and looks good
> > - UI also seems fine
> >
> > Thanks,
> > Sunil
> >
> > On Sun, Sep 20, 2020 at 11:38 AM Masatake Iwasaki <
> > iwasak...@oss.nttdata.co.jp> wrote:
> >
> > > The RC0 got 2 binding +1's and 2 non-binging +1's [1].
> > >
> > > Based on the discussion about release vote [2],
> > > bylaws[3] defines the periods in minimum terms.
> > > We can extend it if there is not enough activity.
> > >
> > > I would like to extend the period to 7 days,
> > > until Monday September 21 at 10:00 am PDT.
> > >
> > > I will appreciate additional votes.
> > >
> > > Thanks,
> > > Masatake Iwasaki
> > >
> > > [1]
> > >
> >
> https://lists.apache.org/thread.html/r16a7f36315a0673c7d522c41065e7ef9c9ee15c76ffcb5db80931002%40%3Ccommon-dev.hadoop.apache.org%3E
> > > [2]
> > >
> >
> https://lists.apache.org/thread.html/e392b902273ee0c14ba34d72c44630e05f54cb3976109af510592ea2%401403330080%40%3Ccommon-dev.hadoop.apache.org%3E
> > > [3] https://hadoop.apache.org/bylaws.html
> > >
> > > On 2020/09/15 2:59, Masatake Iwasaki wrote:
> > > > Hi folks,
> > > >
> > > > This is the first release candidate for the second release of Apache
> > > Hadoop 2.10.
> > > > It contains 218 fixes/improvements since 2.10.0 [1].
> > > >
> > > > The RC0 artifacts are at:
> > > > http://home.apache.org/~iwasakims/hadoop-2.10.1-RC0/
> > > >
> > > > RC tag is release-2.10.1-RC0:
> > > > https://github.com/apache/hadoop/tree/release-2.10.1-RC0
> > > >
> > > > The maven artifacts are hosted here:
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1279/
> > > >
> > > > My public key is available here:
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > > The vote will run for 5 days, until Saturday, September 19 at 10:00
> am
> > > PDT.
> > > >
> > > > [1]
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%202.10.1
> > > >
> > > > Thanks,
> > > > Masatake Iwasaki
> > > >
> > > > -
> > > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> > > >
> > >
> > > -
> > > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
> > >
> > >
> >
>

[ANNOUNCE] New Apache Hadoop PMC member : Ayush Saxena

[Vinayakumar B]

Hi all,

I am very glad to announce that Ayush Saxena was voted to join Apache
Hadoop PMC.

Congratulations Ayush! Well deserved and thank you for your dedication to
the project. Please keep up the good work.

Thanks,
-Vinay

Re: [VOTE] Release Apache Hadoop 3.3.0 - RC0

[Vinayakumar B]

+1 (Binding)

-Verified all checksums and Signatures.
-Verified site, Release notes and Change logs
  + May be changelog and release notes could be grouped based on the
project at second level for better look (this needs to be supported from
yetus)
-Tested in x86 local 3-node docker cluster.
  + Built from source with OpenJdk 8 and Ubuntu 18.04
  + Deployed 3 node docker cluster
  + Ran various Jobs (wordcount, Terasort, Pi, etc)

No Issues reported.

-Vinay

On Fri, Jul 10, 2020 at 1:19 PM Sheng Liu  wrote:

> +1 (non-binding)
>
> - checkout the "3.3.0-aarch64-RC0" binaries packages
>
> - started a clusters with 3 nodes VMs of Ubuntu 18.04 ARM/aarch64,
> openjdk-11-jdk
>
> - checked some web UIs (NN, DN, RM, NM)
>
> - Executed a wordcount, TeraGen, TeraSort and TeraValidate
>
> - Executed a TestDFSIO job
>
> - Executed a Pi job
>
> BR,
> Liusheng
>
> Zhenyu Zheng  于2020年7月10日周五 下午3:45写道：
>
> > +1 (non-binding)
> >
> > - Verified all hashes and checksums
> > - Tested on ARM platform for the following actions:
> >   + Built from source on Ubuntu 18.04, OpenJDK 8
> >   + Deployed a pseudo cluster
> >   + Ran some example jobs(grep, wordcount, pi)
> >   + Ran teragen/terasort/teravalidate
> >   + Ran TestDFSIO job
> >
> > BR,
> >
> > Zhenyu
> >
> > On Fri, Jul 10, 2020 at 2:40 PM Akira Ajisaka 
> wrote:
> >
> > > +1 (binding)
> > >
> > > - Verified checksums and signatures.
> > > - Built from the source with CentOS 7 and OpenJDK 8.
> > > - Successfully upgraded HDFS to 3.3.0-RC0 in our development cluster
> > (with
> > > RBF, security, and OpenJDK 11) for end-users. No issues reported.
> > > - The document looks good.
> > > - Deployed pseudo cluster and ran some MapReduce jobs.
> > >
> > > Thanks,
> > > Akira
> > >
> > >
> > > On Tue, Jul 7, 2020 at 7:27 AM Brahma Reddy Battula  >
> > > wrote:
> > >
> > > > Hi folks,
> > > >
> > > > This is the first release candidate for the first release of Apache
> > > > Hadoop 3.3.0
> > > > line.
> > > >
> > > > It contains *1644[1]* fixed jira issues since 3.2.1 which include a
> lot
> > > of
> > > > features and improvements(read the full set of release notes).
> > > >
> > > > Below feature additions are the highlights of the release.
> > > >
> > > > - ARM Support
> > > > - Enhancements and new features on S3a,S3Guard,ABFS
> > > > - Java 11 Runtime support and TLS 1.3.
> > > > - Support Tencent Cloud COS File System.
> > > > - Added security to HDFS Router.
> > > > - Support non-volatile storage class memory(SCM) in HDFS cache
> > directives
> > > > - Support Interactive Docker Shell for running Containers.
> > > > - Scheduling of opportunistic containers
> > > > - A pluggable device plugin framework to ease vendor plugin
> development
> > > >
> > > > *The RC0 artifacts are at*:
> > > > http://home.apache.org/~brahma/Hadoop-3.3.0-RC0/
> > > >
> > > > *First release to include ARM binary, Have a check.*
> > > > *RC tag is *release-3.3.0-RC0.
> > > >
> > > >
> > > > *The maven artifacts are hosted here:*
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1271/
> > > >
> > > > *My public key is available here:*
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > > The vote will run for 5 weekdays, until Tuesday, July 13 at 3:50 AM
> > IST.
> > > >
> > > >
> > > > I have done a few testing with my pseudo cluster. My +1 to start.
> > > >
> > > >
> > > >
> > > > Regards,
> > > > Brahma Reddy Battula
> > > >
> > > >
> > > > 1. project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.3.0)
> > > AND
> > > > fixVersion not in (3.2.0, 3.2.1, 3.1.3) AND status = Resolved ORDER
> BY
> > > > fixVersion ASC
> > > >
> > >
> >
>

Re: [NOTICE] Removal of protobuf classes from Hadoop Token's public APIs' signature

[Vinayakumar B]

che.hadoop.thirdparty.protobuf.Descriptors
> >   [ERROR]   class file for
> > org.apache.hadoop.thirdparty.protobuf.Descriptors not found
> >   [ERROR]
> >
> /Users/weichiu/sandbox/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutput.java:[323,64]
> > incompatible types:
> > org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.PipelineAckProto
> > cannot be converted to com.google.protobuf.MessageLite
> >   [ERROR]
> >
> /Users/weichiu/sandbox/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/SyncReplicationWALProvider.java:[209,68]
> > invalid method reference
> >   [ERROR]   non-static method get() cannot be referenced from a
> > static context
> >
> >
> > Ozone:
> >
> > 17:01:19 2020/04/16 00:01:19 INFO: [ERROR] COMPILATION ERROR :
> > 17:01:19 2020/04/16 00:01:19 INFO: [INFO]
> > -
> > 17:01:19 2020/04/16 00:01:19 INFO: [ERROR]
> >
> >
> /grid/0/jenkins/workspace/workspace/CDH-CANARY-parallel-centos7/SOURCES/ozone/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/ScmBlockLocationProtocolClientSideTranslatorPB.java:[110,47]
> > incompatible types: com.google.protobuf.ServiceException cannot be
> > converted to org.apache.hadoop.thirdparty.protobuf.ServiceException
> > 17:01:19 2020/04/16 00:01:19 INFO: [ERROR]
> >
> >
> /grid/0/jenkins/workspace/workspace/CDH-CANARY-parallel-centos7/SOURCES/ozone/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java:[116,47]
> > incompatible types: com.google.protobuf.ServiceException cannot be
> > converted to org.apache.hadoop.thirdparty.protobuf.ServiceException
> > 17:01:19 2020/04/16 00:01:19 INFO: [INFO] 2 errors
> >
> >
> > There's another error where Ozone uses the Hadoop RPC framework which
> uses
> > the hadoop.thirdparty protobuf.
> >
> > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
> > (default-testCompile) on project hadoop-hdds-container-service:
> Compilation
> > failure
> > [ERROR]
> >
> >
> /Users/weichiu/sandbox/ozone/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/SCMTestUtils.java:[103,41]
> > incompatible types: com.google.protobuf.BlockingService cannot be
> converted
> > to org.apache.hadoop.thirdparty.protobuf.BlockingService
> >
> > BlockingService scmDatanodeService =
> > StorageContainerDatanodeProtocolService.
> > newReflectiveBlockingService(
> > new StorageContainerDatanodeProtocolServerSideTranslatorPB(
> > server, Mockito.mock(ProtocolMessageMetrics.class)));
> >
> >
> >
> > Ratis probably breaks as well since it depends on the Hadoop RPC
> framework
> > too.
> >
> > On Tue, Apr 28, 2020 at 10:58 PM Vinayakumar B 
> > wrote:
> >
> > > hi Wei-Chiu,
> > >
> > > Can you elaborate on what failures you are facing related to relocated
> > > protobuf classes.. ?
> > >
> > > IFAIK, if the issue with location of protobuf classes, still old jar
> > > protobuf-2.5.0.jar will be available in classpath. So downstream
> > depending
> > > on 2.5.0 version of protobuf still be able to access them.
> > >
> > > -vinay
> > >
> > > On Wed, 29 Apr 2020, 11:17 am Wei-Chiu Chuang, 
> > > wrote:
> > >
> > >> I'm sorry for coming to this late. I missed this message. It should
> have
> > >> been a DISCUSS thread rather than NOTICE.
> > >>
> > >> Looks like this is inevitable. But we should make the downstream
> > >> developers aware & make the update easier. As long as it is stated
> > clearly
> > >> how to update the code to support Hadoop 3.3, I am okay with that.
> > >>
> > >> Here's what I suggest:
> > >> (1) label the jira incompatible (just updated the jira) and updated
> the
> > >> release note to tell app developer how to update.
> > >> (2) declare ProtobufHelper a public API HADOOP-17019
> > >> <https://issues.apache.org/jira/browse/HADOOP-17019>
> > >>
> > >> Tez doesn't use the removed Token API, but there's code that breaks
> with
> > >> the relocated protobuf class. The ProtobufHelper API will make this
> > >> transition much easie

Re: [DISCUSS] making Ozone a separate Apache project

[Vinayakumar B]

+1

-Vinay

On Thu, 14 May 2020, 12:20 pm Surendra Singh Lilhore, <
surendralilh...@gmail.com> wrote:

> +1
>
> -Surendra
>
>
> On Thu, 14 May, 2020, 9:30 am yisheng lien,  wrote:
>
> > +1
> >
> > Thank you Marton for the detailed description.
> >
> >
> > Wanqiang Ji  於 2020年5月14日 週四 上午11:27 寫道：
> >
> > > +1
> > >
> > > On Wed, May 13, 2020 at 3:53 PM Elek, Marton  wrote:
> > >
> > > >
> > > >
> > > > I would like to start a discussion to make a separate Apache project
> > for
> > > > Ozone
> > > >
> > > >
> > > >
> > > > ### HISTORY [1]
> > > >
> > > >   * Apache Hadoop Ozone development started on a feature branch of
> > > > Hadoop repository (HDFS-7240)
> > > >
> > > >   * In the October of 2017 a discussion has been started to merge it
> to
> > > > the Hadoop main branch
> > > >
> > > >   * After a long discussion it's merged to Hadoop trunk at the March
> of
> > > > 2018
> > > >
> > > >   * During the discussion of the merge, it was suggested multiple
> times
> > > > to create a separated project for the Ozone. But at that time:
> > > >  1). Ozone was tightly integrated with Hadoop/HDFS
> > > >  2). There was an active plan to use Block layer of Ozone (HDDS
> or
> > > > HDSL at that time) as the block level of HDFS
> > > >  3). The community of Ozone was a subset of the HDFS community
> > > >
> > > >   * The first beta release of Ozone was just released. Seems to be a
> > > > good time before the first GA to make a decision about the future.
> > > >
> > > >
> > > >
> > > > ### WHAT HAS BEEN CHANGED
> > > >
> > > >   During the last years Ozone became more and more independent both
> at
> > > > the community and code side. The separation has been suggested again
> > and
> > > > again (for example by Owen [2] and Vinod [3])
> > > >
> > > >
> > > >
> > > >   From COMMUNITY point of view:
> > > >
> > > >
> > > >* Fortunately more and more new contributors are helping Ozone.
> > > > Originally the Ozone community was a subset of HDFS project. But now
> a
> > > > bigger and bigger part of the community is related to Ozone only.
> > > >
> > > >* It seems to be easier to _build_ the community as a separated
> > > project.
> > > >
> > > >* A new, younger project might have different practices
> > > > (communication, commiter criteria, development style) compared to
> old,
> > > > mature project
> > > >
> > > >* It's easier to communicate (and improve) these standards in a
> > > > separated projects with clean boundaries
> > > >
> > > >* Separated project/brand can help to increase the adoption rate
> and
> > > > attract more individual contributor (AFAIK it has been seen in
> > Submarine
> > > > after a similar move)
> > > >
> > > >   * Contribution process can be communicated more easily, we can make
> > > > first time contribution more easy
> > > >
> > > >
> > > >
> > > >   From CODE point of view Ozone became more and more independent:
> > > >
> > > >
> > > >   * Ozone has different release cycle
> > > >
> > > >   * Code is already separated from Hadoop code base
> > > > (apache/hadoop-ozone.git)
> > > >
> > > >   * It has separated CI (github actions)
> > > >
> > > >   * Ozone uses different (more strict) coding style (zero toleration
> of
> > > > unit test / checkstyle errors)
> > > >
> > > >   * The code itself became more and more independent from Hadoop on
> > > > Maven level. Originally it was compiled together with the in-tree
> > latest
> > > > Hadoop snapshot. Now it depends on released Hadoop artifacts (RPC,
> > > > Configuration...)
> > > >
> > > >   * It starts to use multiple version of Hadoop (on client side)
> > > >
> > > >   * Volume of resolved issues are already very high on Ozone side
> > (Ozone
> > > > had slightly more resolved issues than HDFS/YARN/MAPREDUCE/COMMON all
> > > > together in the last 2-3 months)
> > > >
> > > >
> > > > Summary: Before the first Ozone GA release, It seems to be a good
> time
> > > > to discuss the long-term future of Ozone. Managing it as a separated
> > TLP
> > > > project seems to have more benefits.
> > > >
> > > >
> > > > Please let me know what your opinion is...
> > > >
> > > > Thanks a lot,
> > > > Marton
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > [1]: For more details, see:
> > > > https://github.com/apache/hadoop-ozone/blob/master/HISTORY.md
> > > >
> > > > [2]:
> > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/0d0253f6e5fa4f609bd9b917df8e1e4d8848e2b7fdb3099b730095e6%40%3Cprivate.hadoop.apache.org%3E
> > > >
> > > > [3]:
> > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/8be74421ea495a62e159f2b15d74627c63ea1f67a2464fa02c85d4aa%40%3Chdfs-dev.hadoop.apache.org%3E
> > > >
> > > > -
> > > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> > > >
> > > >
> > >
> >
>

Re: [NOTICE] Removal of protobuf classes from Hadoop Token's public APIs' signature

[Vinayakumar B]

hi Wei-Chiu,

Can you elaborate on what failures you are facing related to relocated
protobuf classes.. ?

IFAIK, if the issue with location of protobuf classes, still old jar
protobuf-2.5.0.jar will be available in classpath. So downstream depending
on 2.5.0 version of protobuf still be able to access them.

-vinay

On Wed, 29 Apr 2020, 11:17 am Wei-Chiu Chuang,  wrote:

> I'm sorry for coming to this late. I missed this message. It should have
> been a DISCUSS thread rather than NOTICE.
>
> Looks like this is inevitable. But we should make the downstream
> developers aware & make the update easier. As long as it is stated clearly
> how to update the code to support Hadoop 3.3, I am okay with that.
>
> Here's what I suggest:
> (1) label the jira incompatible (just updated the jira) and updated the
> release note to tell app developer how to update.
> (2) declare ProtobufHelper a public API HADOOP-17019
> <https://issues.apache.org/jira/browse/HADOOP-17019>
>
> Tez doesn't use the removed Token API, but there's code that breaks with
> the relocated protobuf class. The ProtobufHelper API will make this
> transition much easier.
>
> Other downstreamers that break with the relocated protobuf include: Ozone
> and HBase. but neither of them use the removed Token API.
>
>
> On Wed, Jan 8, 2020 at 4:40 AM Vinayakumar B 
> wrote:
>
>> Hi All,
>>
>>This mail is to notify about the Removal of following public APIs from
>> Hadoop Common.
>>
>>  ClassName: org.apache.hadoop.security.token.Token
>>  APIs:
>>  public Token(TokenProto tokenPB);
>>  public TokenProto toTokenProto();
>>
>>Reason: These APIs are having Generated protobuf classes in the
>> signature. Right now due to protobuf upgrade in trunk (soon to be 3.3.0
>> release) these APIs are breaking the downstream builds, even though
>> downstreams dont use these APIs (just Loading Token class). Downstreams
>> are
>> still referencing having older version (2.5.0) of protobuf, hence build is
>> being broken.
>>
>> These APIs were added for the internal purpose(HADOOP-12563), to
>> support serializing tokens using protobuf in UGI Credentials.
>> Same purpose can be achieved using the Helper classes without introducing
>> protobuf classes in API signatures.
>>
>> Token.java is marked as Evolving, so I believe APIs can be changed
>> whenever
>> absolute necessary.
>>
>> Jira https://issues.apache.org/jira/browse/HADOOP-16621 has been
>> reported to solve downstream build failure.
>>
>> So since this API was added for internal purpose easy approach to solve
>> this is to remove APIs and use helper classes. Otherwise, as mentioned in
>> HADOOP-16621, workaround will add unnecessary codes to be maintained.
>>
>> If anyone using these APIs outside hadoop project accidentally, please
>> reply to this mail immediately.
>>
>> If no objection by next week, will go ahead with removal of above said
>> APIs
>> in HADOOP-16621.
>>
>> -Vinay
>>
>

Re: [RESULT] [VOTE] Release Apache Hadoop Thirdparty 1.0.0 - RC1

[Vinayakumar B]

Thanks brahma.
So total 5 +1s (4 binding, 1 non-binding).

Done with pushing bits. Should get mirrored in next 24 hours.

-Vinay


On Thu, Mar 19, 2020 at 12:55 AM Brahma Reddy Battula 
wrote:

>
> My late +1.. I missed this..
>
>
> On Thu, 19 Mar 2020 at 12:43 AM, Vinayakumar B 
> wrote:
>
>> Vote passes with 4 +1s (3 binding, 1 non-binding) and no -1s
>>
>> Thanks everyone for verification.
>> I will push out bits soon.
>>
>> -Vinay
>>
>>
>> On Sun, Mar 15, 2020 at 5:25 PM Surendra Singh Lilhore <
>> surendralilh...@gmail.com> wrote:
>>
>> > +1
>> >
>> > -Built trunk with -Dhadoop-thirdparty-protobuf.version=1.0.0
>> > -Verified artifacts available in repo.
>> >
>> >
>> > Thanks
>> > Surendra
>> >
>> >
>> > On Fri, Mar 13, 2020 at 9:51 AM Akira Ajisaka 
>> wrote:
>> >
>> >> +1
>> >>
>> >> - Verified signatures and checksums
>> >> - Built jars and docs from source
>> >> - Built hadoop trunk with hadoop-thirdparty 1.0.0
>> >> - Checked rat files and documents
>> >> - Checked LICENSE and NOTICE files
>> >>
>> >> Thanks,
>> >> Akira
>> >>
>> >> On Thu, Mar 12, 2020 at 5:26 AM Vinayakumar B > >
>> >> wrote:
>> >>
>> >> > Hi folks,
>> >> >
>> >> > Thanks to everyone's help on this release.
>> >> >
>> >> > I have re-created a release candidate (RC1) for Apache Hadoop
>> Thirdparty
>> >> > 1.0.0.
>> >> >
>> >> > RC Release artifacts are available at :
>> >> >
>> >> >
>> >>
>> http://home.apache.org/~vinayakumarb/release/hadoop-thirdparty-1.0.0-RC1/
>> >> >
>> >> > Maven artifacts are available in staging repo:
>> >> >
>> >> >
>> >>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1261/
>> >> >
>> >> > The RC tag in git is here:
>> >> > https://github.com/apache/hadoop-thirdparty/tree/release-1.0.0-RC1
>> >> >
>> >> > And my public key is at:
>> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >> >
>> >> > *This vote will run for 5 days, ending on March 18th 2020 at 11:59 pm
>> >> IST.*
>> >> >
>> >> > For the testing, I have verified Hadoop trunk compilation with
>> >> >"-DdistMgmtSnapshotsUrl=
>> >> >
>> >>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1261/
>> >> >  -Dhadoop-thirdparty-protobuf.version=1.0.0"
>> >> >
>> >> > My +1 to start.
>> >> >
>> >> > -Vinay
>> >> >
>> >>
>> >
>>
> --
>
>
>
> --Brahma Reddy Battula
>

[RESULT] [VOTE] Release Apache Hadoop Thirdparty 1.0.0 - RC1

[Vinayakumar B]

Vote passes with 4 +1s (3 binding, 1 non-binding) and no -1s

Thanks everyone for verification.
I will push out bits soon.

-Vinay


On Sun, Mar 15, 2020 at 5:25 PM Surendra Singh Lilhore <
surendralilh...@gmail.com> wrote:

> +1
>
> -Built trunk with -Dhadoop-thirdparty-protobuf.version=1.0.0
> -Verified artifacts available in repo.
>
>
> Thanks
> Surendra
>
>
> On Fri, Mar 13, 2020 at 9:51 AM Akira Ajisaka  wrote:
>
>> +1
>>
>> - Verified signatures and checksums
>> - Built jars and docs from source
>> - Built hadoop trunk with hadoop-thirdparty 1.0.0
>> - Checked rat files and documents
>> - Checked LICENSE and NOTICE files
>>
>> Thanks,
>> Akira
>>
>> On Thu, Mar 12, 2020 at 5:26 AM Vinayakumar B 
>> wrote:
>>
>> > Hi folks,
>> >
>> > Thanks to everyone's help on this release.
>> >
>> > I have re-created a release candidate (RC1) for Apache Hadoop Thirdparty
>> > 1.0.0.
>> >
>> > RC Release artifacts are available at :
>> >
>> >
>> http://home.apache.org/~vinayakumarb/release/hadoop-thirdparty-1.0.0-RC1/
>> >
>> > Maven artifacts are available in staging repo:
>> >
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1261/
>> >
>> > The RC tag in git is here:
>> > https://github.com/apache/hadoop-thirdparty/tree/release-1.0.0-RC1
>> >
>> > And my public key is at:
>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >
>> > *This vote will run for 5 days, ending on March 18th 2020 at 11:59 pm
>> IST.*
>> >
>> > For the testing, I have verified Hadoop trunk compilation with
>> >"-DdistMgmtSnapshotsUrl=
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1261/
>> >  -Dhadoop-thirdparty-protobuf.version=1.0.0"
>> >
>> > My +1 to start.
>> >
>> > -Vinay
>> >
>>
>

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Vinayakumar B]

Making ARM artifact optional, makes the release process simpler for RM  and
unblocks release process (if there is unavailability of ARM resources).

Still there are possible options to collaborate with RM ( as brahma
mentioned earlier) and provide ARM artifact may be before or after vote.
If feasible RM can decide to add ARM artifact by collaborating with @Brahma
Reddy Battula  or me to get the ARM artifact.

-Vinay

On Tue, Mar 17, 2020 at 11:39 PM Arpit Agarwal
 wrote:

> Thanks for the clarification Brahma. Can you update the proposal to state
> that it is optional (it may help to put the proposal on cwiki)?
>
> Also if we go ahead then the RM documentation should be clear this is an
> optional step.
>
>
> > On Mar 17, 2020, at 11:06 AM, Brahma Reddy Battula 
> wrote:
> >
> > Sure, we can't make mandatory while voting and we can upload to downloads
> > once release vote is passed.
> >
> > On Tue, 17 Mar 2020 at 11:24 PM, Arpit Agarwal
> >  wrote:
> >
> >>> Sorry,didn't get you...do you mean, once release voting is
> >>> processed and upload by RM..?
> >>
> >> Yes, that is what I meant. I don’t want us to make more mandatory work
> for
> >> the release manager because the job is hard enough already.
> >>
> >>
> >>> On Mar 17, 2020, at 10:46 AM, Brahma Reddy Battula 
> >> wrote:
> >>>
> >>> Sorry,didn't get you...do you mean, once release voting is processed
> and
> >>> upload by RM..?
> >>>
> >>> FYI. There is docker image for ARM also which support all scripts
> >>> (createrelease, start-build-env.sh, etc ).
> >>>
> >>> https://issues.apache.org/jira/browse/HADOOP-16797
> >>>
> >>> On Tue, Mar 17, 2020 at 10:59 PM Arpit Agarwal
> >>>  wrote:
> >>>
>  Can ARM binaries be provided after the fact? We cannot increase the
> RM’s
>  burden by asking them to generate an extra set of binaries.
> 
> 
> > On Mar 17, 2020, at 10:23 AM, Brahma Reddy Battula <
> bra...@apache.org>
>  wrote:
> >
> > + Dev mailing list.
> >
> > -- Forwarded message -
> > From: Brahma Reddy Battula 
> > Date: Tue, Mar 17, 2020 at 10:31 PM
> > Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
> > To: junping_du 
> >
> >
> > thanks junping for your reply.
> >
> > bq.  I think most of us in Hadoop community doesn't want to have
>  biased
> > on ARM or any other platforms.
> >
> > Yes, release voting will be based on the source code.AFAIK,Binary we
> >> are
> > providing for user to easy to download and verify.
> >
> > bq. The only thing I try to understand is how much complexity get
> > involved for our RM work. Does that potentially become a blocker for
>  future
> > releases? And how we can get rid of this risk.
> >
> > As I mentioned earlier, RM need to access the ARM machine(it will be
> > donated and current qbt also using one ARM machine) and build tar
> using
>  the
> > keys. As it can be common machine, RM can delete his keys once
> release
> > approved.
> > Can be sorted out as I mentioned earlier.(For accessing the ARM
> >> machine)
> >
> > bq.   If you can list the concrete work that RM need to do extra
> >> for
> > ARM release, that would help us to better understand.
> >
> > I can write and update for future reference.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Mar 17, 2020 at 10:41 AM 俊平堵  wrote:
> >
> >> Hi Brahma,
> >>   I think most of us in Hadoop community doesn't want to have biased
>  on
> >> ARM or any other platforms.
> >>   The only thing I try to understand is how much complexity get
> >> involved for our RM work. Does that potentially become a blocker for
>  future
> >> releases? And how we can get rid of this risk.
> >>If you can list the concrete work that RM need to do extra for
> ARM
> >> release, that would help us to better understand.
> >>
> >> Thanks,
> >>
> >> Junping
> >>
> >> Akira Ajisaka  于2020年3月13日周五 上午12:34写道：
> >>
> >>> If you can provide ARM release for future releases, I'm fine with
> >> that.
> >>>
> >>> Thanks,
> >>> Akira
> >>>
> >>> On Thu, Mar 12, 2020 at 9:41 PM Brahma Reddy Battula <
>  bra...@apache.org>
> >>> wrote:
> >>>
>  thanks Akira.
> 
>  Currently only problem is dedicated ARM for future RM.This i want
> to
> >>> sort
>  out like below,if you've some other,please let me know.
> 
>  i) Single machine and share cred to future RM ( as we can delete
> >> keys
> >>> once
>  release is over).
>  ii) Creating the jenkins project ( may be we need to discuss in
> the
>  board..)
>  iii) I can provide ARM release for future releases.
> 
> 
> 
> 
> 
> 
> 
>  On Thu, Mar 12, 2020 at 5:1

[VOTE] Release Apache Hadoop Thirdparty 1.0.0 - RC1

[Vinayakumar B]

Hi folks,

Thanks to everyone's help on this release.

I have re-created a release candidate (RC1) for Apache Hadoop Thirdparty
1.0.0.

RC Release artifacts are available at :
  http://home.apache.org/~vinayakumarb/release/hadoop-thirdparty-1.0.0-RC1/

Maven artifacts are available in staging repo:
https://repository.apache.org/content/repositories/orgapachehadoop-1261/

The RC tag in git is here:
https://github.com/apache/hadoop-thirdparty/tree/release-1.0.0-RC1

And my public key is at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

*This vote will run for 5 days, ending on March 18th 2020 at 11:59 pm IST.*

For the testing, I have verified Hadoop trunk compilation with
   "-DdistMgmtSnapshotsUrl=
https://repository.apache.org/content/repositories/orgapachehadoop-1261/
 -Dhadoop-thirdparty-protobuf.version=1.0.0"

My +1 to start.

-Vinay

Re: [VOTE] Release Apache Hadoop Thirdparty 1.0.0

[Vinayakumar B]

Hello everyone.

RC0 has been canceled.

Since issues mentioned above are already fixed now, soon I will create RC1
and re-create VOTE thread.

Thanks for trying out RC,
-Vinay


On Tue, Mar 3, 2020 at 11:41 AM Ayush Saxena  wrote:

> Hi Vinay
> Thanx for driving the release.
> Verified checksums and tried building from source.
> Everything seems to be working fine.
> But I feel the concerns regarding licences are valid.
> IMO we should fix them and include HADOOP-16895 too in the release
>
> -Ayush
>
> > On 29-Feb-2020, at 1:45 AM, Vinayakumar B 
> wrote:
> >
> > https://issues.apache.org/jira/browse/HADOOP-16895 jira created for
> > handling LICENCE and NOTICEs
> > PR also has been raised for a proposal. Please validate
> > https://github.com/apache/hadoop-thirdparty/pull/6
> >
> > -Vinay
> >
> >
> >> On Fri, Feb 28, 2020 at 11:48 PM Vinayakumar B  >
> >> wrote:
> >>
> >> Thanks Elek for detailed verification.
> >>
> >> Please find inline replies.
> >>
> >> -Vinay
> >>
> >>
> >>> On Fri, Feb 28, 2020 at 7:49 PM Elek, Marton  wrote:
> >>>
> >>>
> >>> Thank you very much to work on this release Vinay, 1.0.0 is always a
> >>> hard work...
> >>>
> >>>
> >>> 1. I downloaded it and I can build it from the source
> >>>
> >>> 2. Checked the signature and the sha512 of the src package and they are
> >>> fine
> >>>
> >>> 3. Yetus seems to be included in the source package. I am not sure if
> >>> it's intentional but I would remove the patchprocess directory from the
> >>> tar file.
> >>>
> >>> Since dev-support/create-release script and assembly file is copied
> from
> >> hadoop-repo,  I can find this issue exits in hadoop source release
> packages
> >> as well. ex: I checked 3.1.2 and 2.10 src packages.
> >> I will raise a Jira and fix this for both hadoop and thirdparty.
> >>
> >> 4. NOTICE.txt seems to be outdated (I am not sure, but I think the
> >>> Export Notice is unnecessary, especially for the source release, also
> >>> the note about the bouncycastle and Yarn server is unnecessary).
> >>>
> >>> Again, NOTICE.txt was copied from Hadoop and kept as is. I will create
> a
> >> jira to decide about NOTICE and LICENSEs
> >>
> >> 5. NOTICE-binary and LICENSE-binary seems to be unused (and they contain
> >>> unrelated entries, especially the NOTICE). IMHO
> >>>
> >>> We can decide in the Jira whether NOTICE-binary and LICENSE-binary to
> be
> >> used or not.
> >>
> >> 6. As far as I understand the binary release in this case is the maven
> >>> artifact. IANAL but the original protobuf license seems to be missing
> >>> from "unzip -p hadoop-shaded-protobuf_3_7-1.0.0.jar
> META-INF/LICENSE.txt"
> >>>
> >>
> >> I observed that there is one more file "META-INF/DEPENDENCIES" generated
> >> by shade plugin, which have reference to shaded artifacts and poniting
> to
> >> link of the original artifact LICENSE. I think this should be sufficient
> >> about protobuf's original license.
> >> IMO, "META-INF/LICENSE.txt" should point to current project's LICENSE,
> >> which in-turn can have contents/pointers of dependents' licenses.
> Siimilar
> >> approach followed in hadoop-shaded-client jars.
> >>
> >> hadoop's artifacts also will be uploaded to maven repo during release,
> >> which doesnot carry all LICENSE files in artifacts. It just says "See
> >> licenses/ for text of these licenses" which doesnot exist in artifact.
> May
> >> be we need to fix this too.
> >>
> >> 7. Minor nit: I would suggest to use only the filename in the sha512
> >>> files (instead of having the /build/source/target prefix). It would
> help
> >>> to use `sha512 -c` command to validate the checksum.
> >>>
> >>>
> >> Again, this is from create-release  script. will update the script.
> >>
> >> Thanks again to work on this,
> >>> Marton
> >>>
> >>> ps: I am not experienced with licensing enough to judge which one of
> >>> these are blocking and I might be wrong.
> >>>
> >>>
> >> IMO, none of these should be blocking and can be handled before next
> >> release. Still i

Re: [VOTE] Release Apache Hadoop Thirdparty 1.0.0

[Vinayakumar B]

https://issues.apache.org/jira/browse/HADOOP-16895 jira created for
handling LICENCE and NOTICEs
PR also has been raised for a proposal. Please validate
https://github.com/apache/hadoop-thirdparty/pull/6

-Vinay


On Fri, Feb 28, 2020 at 11:48 PM Vinayakumar B 
wrote:

> Thanks Elek for detailed verification.
>
> Please find inline replies.
>
> -Vinay
>
>
> On Fri, Feb 28, 2020 at 7:49 PM Elek, Marton  wrote:
>
>>
>> Thank you very much to work on this release Vinay, 1.0.0 is always a
>> hard work...
>>
>>
>> 1. I downloaded it and I can build it from the source
>>
>> 2. Checked the signature and the sha512 of the src package and they are
>> fine
>>
>> 3. Yetus seems to be included in the source package. I am not sure if
>> it's intentional but I would remove the patchprocess directory from the
>> tar file.
>>
>> Since dev-support/create-release script and assembly file is copied from
> hadoop-repo,  I can find this issue exits in hadoop source release packages
> as well. ex: I checked 3.1.2 and 2.10 src packages.
> I will raise a Jira and fix this for both hadoop and thirdparty.
>
> 4. NOTICE.txt seems to be outdated (I am not sure, but I think the
>> Export Notice is unnecessary, especially for the source release, also
>> the note about the bouncycastle and Yarn server is unnecessary).
>>
>> Again, NOTICE.txt was copied from Hadoop and kept as is. I will create a
> jira to decide about NOTICE and LICENSEs
>
> 5. NOTICE-binary and LICENSE-binary seems to be unused (and they contain
>> unrelated entries, especially the NOTICE). IMHO
>>
>> We can decide in the Jira whether NOTICE-binary and LICENSE-binary to be
> used or not.
>
> 6. As far as I understand the binary release in this case is the maven
>> artifact. IANAL but the original protobuf license seems to be missing
>> from "unzip -p hadoop-shaded-protobuf_3_7-1.0.0.jar META-INF/LICENSE.txt"
>>
>
> I observed that there is one more file "META-INF/DEPENDENCIES" generated
> by shade plugin, which have reference to shaded artifacts and poniting to
> link of the original artifact LICENSE. I think this should be sufficient
> about protobuf's original license.
> IMO, "META-INF/LICENSE.txt" should point to current project's LICENSE,
> which in-turn can have contents/pointers of dependents' licenses. Siimilar
> approach followed in hadoop-shaded-client jars.
>
> hadoop's artifacts also will be uploaded to maven repo during release,
> which doesnot carry all LICENSE files in artifacts. It just says "See
> licenses/ for text of these licenses" which doesnot exist in artifact. May
> be we need to fix this too.
>
> 7. Minor nit: I would suggest to use only the filename in the sha512
>> files (instead of having the /build/source/target prefix). It would help
>> to use `sha512 -c` command to validate the checksum.
>>
>>
> Again, this is from create-release  script. will update the script.
>
> Thanks again to work on this,
>> Marton
>>
>> ps: I am not experienced with licensing enough to judge which one of
>> these are blocking and I might be wrong.
>>
>>
> IMO, none of these should be blocking and can be handled before next
> release. Still if someone feels this should be fixed and RC should be cut
> again, I am open to it.
>
> Thanks.
>
>>
>>
>> On 2/25/20 8:17 PM, Vinayakumar B wrote:
>> > Hi folks,
>> >
>> > Thanks to everyone's help on this release.
>> >
>> > I have created a release candidate (RC0) for Apache Hadoop Thirdparty
>> 1.0.0.
>> >
>> > RC Release artifacts are available at :
>> >
>> http://home.apache.org/~vinayakumarb/release/hadoop-thirdparty-1.0.0-RC0/
>> >
>> > Maven artifacts are available in staging repo:
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1258/
>> >
>> > The RC tag in git is here:
>> > https://github.com/apache/hadoop-thirdparty/tree/release-1.0.0-RC0
>> >
>> > And my public key is at:
>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >
>> > *This vote will run for 5 days, ending on March 1st 2020 at 11:59 pm
>> IST.*
>> >
>> > For the testing, I have verified Hadoop trunk compilation with
>> > "-DdistMgmtSnapshotsUrl=
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1258/
>> > -Dhadoop-thirdparty-protobuf.version=1.0.0"
>> >
>> > My +1 to start.
>> >
>> > -Vinay
>> >
>>
>

Re: [VOTE] Release Apache Hadoop Thirdparty 1.0.0

[Vinayakumar B]

Thanks Elek for detailed verification.

Please find inline replies.

-Vinay


On Fri, Feb 28, 2020 at 7:49 PM Elek, Marton  wrote:

>
> Thank you very much to work on this release Vinay, 1.0.0 is always a
> hard work...
>
>
> 1. I downloaded it and I can build it from the source
>
> 2. Checked the signature and the sha512 of the src package and they are
> fine
>
> 3. Yetus seems to be included in the source package. I am not sure if
> it's intentional but I would remove the patchprocess directory from the
> tar file.
>
> Since dev-support/create-release script and assembly file is copied from
hadoop-repo,  I can find this issue exits in hadoop source release packages
as well. ex: I checked 3.1.2 and 2.10 src packages.
I will raise a Jira and fix this for both hadoop and thirdparty.

4. NOTICE.txt seems to be outdated (I am not sure, but I think the
> Export Notice is unnecessary, especially for the source release, also
> the note about the bouncycastle and Yarn server is unnecessary).
>
> Again, NOTICE.txt was copied from Hadoop and kept as is. I will create a
jira to decide about NOTICE and LICENSEs

5. NOTICE-binary and LICENSE-binary seems to be unused (and they contain
> unrelated entries, especially the NOTICE). IMHO
>
> We can decide in the Jira whether NOTICE-binary and LICENSE-binary to be
used or not.

6. As far as I understand the binary release in this case is the maven
> artifact. IANAL but the original protobuf license seems to be missing
> from "unzip -p hadoop-shaded-protobuf_3_7-1.0.0.jar META-INF/LICENSE.txt"
>

I observed that there is one more file "META-INF/DEPENDENCIES" generated by
shade plugin, which have reference to shaded artifacts and poniting to link
of the original artifact LICENSE. I think this should be sufficient about
protobuf's original license.
IMO, "META-INF/LICENSE.txt" should point to current project's LICENSE,
which in-turn can have contents/pointers of dependents' licenses. Siimilar
approach followed in hadoop-shaded-client jars.

hadoop's artifacts also will be uploaded to maven repo during release,
which doesnot carry all LICENSE files in artifacts. It just says "See
licenses/ for text of these licenses" which doesnot exist in artifact. May
be we need to fix this too.

7. Minor nit: I would suggest to use only the filename in the sha512
> files (instead of having the /build/source/target prefix). It would help
> to use `sha512 -c` command to validate the checksum.
>
>
Again, this is from create-release  script. will update the script.

Thanks again to work on this,
> Marton
>
> ps: I am not experienced with licensing enough to judge which one of
> these are blocking and I might be wrong.
>
>
IMO, none of these should be blocking and can be handled before next
release. Still if someone feels this should be fixed and RC should be cut
again, I am open to it.

Thanks.

>
>
> On 2/25/20 8:17 PM, Vinayakumar B wrote:
> > Hi folks,
> >
> > Thanks to everyone's help on this release.
> >
> > I have created a release candidate (RC0) for Apache Hadoop Thirdparty
> 1.0.0.
> >
> > RC Release artifacts are available at :
> >
> http://home.apache.org/~vinayakumarb/release/hadoop-thirdparty-1.0.0-RC0/
> >
> > Maven artifacts are available in staging repo:
> >
> https://repository.apache.org/content/repositories/orgapachehadoop-1258/
> >
> > The RC tag in git is here:
> > https://github.com/apache/hadoop-thirdparty/tree/release-1.0.0-RC0
> >
> > And my public key is at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > *This vote will run for 5 days, ending on March 1st 2020 at 11:59 pm
> IST.*
> >
> > For the testing, I have verified Hadoop trunk compilation with
> > "-DdistMgmtSnapshotsUrl=
> > https://repository.apache.org/content/repositories/orgapachehadoop-1258/
> > -Dhadoop-thirdparty-protobuf.version=1.0.0"
> >
> > My +1 to start.
> >
> > -Vinay
> >
>

[VOTE] Release Apache Hadoop Thirdparty 1.0.0

[Vinayakumar B]

Hi folks,

Thanks to everyone's help on this release.

I have created a release candidate (RC0) for Apache Hadoop Thirdparty 1.0.0.

RC Release artifacts are available at :
  http://home.apache.org/~vinayakumarb/release/hadoop-thirdparty-1.0.0-RC0/

Maven artifacts are available in staging repo:
https://repository.apache.org/content/repositories/orgapachehadoop-1258/

The RC tag in git is here:
https://github.com/apache/hadoop-thirdparty/tree/release-1.0.0-RC0

And my public key is at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

*This vote will run for 5 days, ending on March 1st 2020 at 11:59 pm IST.*

For the testing, I have verified Hadoop trunk compilation with
   "-DdistMgmtSnapshotsUrl=
https://repository.apache.org/content/repositories/orgapachehadoop-1258/
-Dhadoop-thirdparty-protobuf.version=1.0.0"

My +1 to start.

-Vinay

[DISCUSS] hadoop-thirdparty 1.0.0 release

[Vinayakumar B]

Hi All,

Since Hadoop-3.3.0 release is around the corner, its time to release
hadoop-thirdparty's first ever release, without which hadoop-3.3.0 cannot
proceed for release.

Below are the tentative date for RC and release. Since there is no much
activity in this repo (other than the opentracing related one, which I just
merged), Keeping the plan little aggressive.
Please let me know any concerns regarding the same.

 RC-0 : 25-Feb-2020
Release : 03-Mar-2020 (after 7 days Voting)

-Vinay

Re: [DISCUSS] About creation of Hadoop Thirdparty repository for shaded artifacts

[Vinayakumar B]

PR has been merged.

Thanks everyone for discussions.

-Vinay


On Thu, Jan 9, 2020 at 4:47 PM Ayush Saxena  wrote:

> Hi All,
> FYI :
> We will be going ahead with the present approach, will merge by tomorrow
> EOD. Considering no one has objections.
> Thanx Everyone!!!
>
> -Ayush
>
> > On 07-Jan-2020, at 9:22 PM, Brahma Reddy Battula 
> wrote:
> >
> > Hi Sree vaddi,Owen,stack,Duo Zhang,
> >
> > We can move forward based on your comments, just waiting for your
> > reply.Hope all of your comments answered..(unification we can think
> > parallel thread as Vinay mentioned).
> >
> >
> >
> > On Mon, 6 Jan 2020 at 6:21 PM, Vinayakumar B 
> > wrote:
> >
> >> Hi Sree,
> >>
> >>> apache/hadoop-thirdparty, How would it fit into ASF ? As an Incubating
> >> Project ? Or as a TLP ?
> >>> Or as a new project definition ?
> >> As already mentioned by Ayush, this will be a subproject of Hadoop.
> >> Releases will be voted by Hadoop PMC as per ASF process.
> >>
> >>
> >>> The effort to streamline and put in an accepted standard for the
> >> dependencies that require shading,
> >>> seems beyond the siloed efforts of hadoop, hbase, etc
> >>
> >>> I propose, we bring all the decision makers from all these artifacts in
> >> one room and decide best course of action.
> >>> I am looking at, no projects should ever had to shade any artifacts
> >> except as an absolute necessary alternative.
> >>
> >> This is the ideal proposal for any project. But unfortunately some
> projects
> >> takes their own course based on need.
> >>
> >> In the current case of protobuf in Hadoop,
> >>Protobuf upgrade from 2.5.0 (which is already EOL) was not taken up
> to
> >> avoid downstream failures. Since Hadoop is a platform, its dependencies
> >> will get added to downstream projects' classpath. So any change in
> Hadoop's
> >> dependencies will directly affect downstreams. Hadoop strictly follows
> >> backward compatibility as far as possible.
> >>Though protobuf provides wire compatibility b/w versions, it doesnt
> >> provide compatibility for generated sources.
> >>Now, to support ARM protobuf upgrade is mandatory. Using shading
> >> technique, In Hadoop internally can upgrade to shaded protobuf 3.x and
> >> still have 2.5.0 protobuf (deprecated) for downstreams.
> >>
> >> This shading is necessary to have both versions of protobuf supported.
> >> (2.5.0 (non-shaded) for downstream's classpath and 3.x (shaded) for
> >> hadoop's internal usage).
> >> And this entire work to be done before 3.3.0 release.
> >>
> >> So, though its ideal to make a common approach for all projects, I
> suggest
> >> for Hadoop we can go ahead as per current approach.
> >> We can also start the parallel effort to address these problems in a
> >> separate discussion/proposal. Once the solution is available we can
> revisit
> >> and adopt new solution accordingly in all such projects (ex: HBase,
> Hadoop,
> >> Ratis).
> >>
> >> -Vinay
> >>
> >>> On Mon, Jan 6, 2020 at 12:39 AM Ayush Saxena 
> wrote:
> >>>
> >>> Hey Sree
> >>>
> >>>> apache/hadoop-thirdparty, How would it fit into ASF ? As an Incubating
> >>>> Project ? Or as a TLP ?
> >>>> Or as a new project definition ?
> >>>>
> >>> A sub project of Apache Hadoop, having its own independent release
> >> cycles.
> >>> May be you can put this into the same column as ozone or as
> >>> submarine(couple of months ago).
> >>>
> >>> Unifying for all, seems interesting but each project is independent and
> >> has
> >>> its own limitations and way of thinking, I don't think it would be an
> >> easy
> >>> task to bring all on the same table and get them agree to a common
> stuff.
> >>>
> >>> I guess this has been into discussion since quite long, and there
> hasn't
> >>> been any other alternative suggested. Still we can hold up for a week,
> if
> >>> someone comes up with a better solution, else we can continue in the
> >>> present direction.
> >>>
> >>> -Ayush
> >>>
> >>>
> >>>
> >>> On Sun, 5 Jan 2020 at 05:03, Sree Vaddi  >> .invalid>
> >>> wrote:

Re: [DISCUSS] About creation of Hadoop Thirdparty repository for shaded artifacts

[Vinayakumar B]

Hi Sree,

> apache/hadoop-thirdparty, How would it fit into ASF ? As an Incubating
Project ? Or as a TLP ?
> Or as a new project definition ?
As already mentioned by Ayush, this will be a subproject of Hadoop.
Releases will be voted by Hadoop PMC as per ASF process.


> The effort to streamline and put in an accepted standard for the
dependencies that require shading,
> seems beyond the siloed efforts of hadoop, hbase, etc

>I propose, we bring all the decision makers from all these artifacts in
one room and decide best course of action.
> I am looking at, no projects should ever had to shade any artifacts
except as an absolute necessary alternative.

This is the ideal proposal for any project. But unfortunately some projects
takes their own course based on need.

In the current case of protobuf in Hadoop,
Protobuf upgrade from 2.5.0 (which is already EOL) was not taken up to
avoid downstream failures. Since Hadoop is a platform, its dependencies
will get added to downstream projects' classpath. So any change in Hadoop's
dependencies will directly affect downstreams. Hadoop strictly follows
backward compatibility as far as possible.
Though protobuf provides wire compatibility b/w versions, it doesnt
provide compatibility for generated sources.
Now, to support ARM protobuf upgrade is mandatory. Using shading
technique, In Hadoop internally can upgrade to shaded protobuf 3.x and
still have 2.5.0 protobuf (deprecated) for downstreams.

This shading is necessary to have both versions of protobuf supported.
(2.5.0 (non-shaded) for downstream's classpath and 3.x (shaded) for
hadoop's internal usage).
And this entire work to be done before 3.3.0 release.

So, though its ideal to make a common approach for all projects, I suggest
for Hadoop we can go ahead as per current approach.
We can also start the parallel effort to address these problems in a
separate discussion/proposal. Once the solution is available we can revisit
and adopt new solution accordingly in all such projects (ex: HBase, Hadoop,
Ratis).

-Vinay

On Mon, Jan 6, 2020 at 12:39 AM Ayush Saxena  wrote:

> Hey Sree
>
> > apache/hadoop-thirdparty, How would it fit into ASF ? As an Incubating
> > Project ? Or as a TLP ?
> > Or as a new project definition ?
> >
> A sub project of Apache Hadoop, having its own independent release cycles.
> May be you can put this into the same column as ozone or as
> submarine(couple of months ago).
>
> Unifying for all, seems interesting but each project is independent and has
> its own limitations and way of thinking, I don't think it would be an easy
> task to bring all on the same table and get them agree to a common stuff.
>
> I guess this has been into discussion since quite long, and there hasn't
> been any other alternative suggested. Still we can hold up for a week, if
> someone comes up with a better solution, else we can continue in the
> present direction.
>
> -Ayush
>
>
>
> On Sun, 5 Jan 2020 at 05:03, Sree Vaddi 
> wrote:
>
> > apache/hadoop-thirdparty, How would it fit into ASF ? As an Incubating
> > Project ? Or as a TLP ?
> > Or as a new project definition ?
> >
> > The effort to streamline and put in an accepted standard for the
> > dependencies that require shading,seems beyond the siloed efforts of
> > hadoop, hbase, etc
> >
> > I propose, we bring all the decision makers from all these artifacts in
> > one room and decide best course of action.I am looking at, no projects
> > should ever had to shade any artifacts except as an absolute necessary
> > alternative.
> >
> >
> > Thank you./Sree
> >
> >
> >
> > On Saturday, January 4, 2020, 7:49:18 AM PST, Vinayakumar B <
> > vinayakum...@apache.org> wrote:
> >
> >  Hi,
> > Sorry for the late reply,.
> > >>> To be exact, how can we better use the thirdparty repo? Looking at
> > HBase as an example, it looks like everything that are known to break a
> lot
> > after an update get shaded into the hbase-thirdparty artifact: guava,
> > netty, ... etc.
> > Is it the purpose to isolate these naughty dependencies?
> > Yes, shading is to isolate these naughty dependencies from downstream
> > classpath and have independent control on these upgrades without breaking
> > downstreams.
> >
> > First PR https://github.com/apache/hadoop-thirdparty/pull/1 to create
> the
> > protobuf shaded jar is ready to merge.
> >
> > Please take a look if anyone interested, will be merged may be after two
> > days if no objections.
> >
> > -Vinay
> >
> >
> > On Thu, Oct 10, 2019 at 3:30 AM Wei-Chiu Chuang 
> > wrote:
> >
> > > Hi I am la

Re: [DISCUSS] About creation of Hadoop Thirdparty repository for shaded artifacts

[Vinayakumar B]

Hi,
Sorry for the late reply,.
>>> To be exact, how can we better use the thirdparty repo? Looking at
HBase as an example, it looks like everything that are known to break a lot
after an update get shaded into the hbase-thirdparty artifact: guava,
netty, ... etc.
Is it the purpose to isolate these naughty dependencies?
Yes, shading is to isolate these naughty dependencies from downstream
classpath and have independent control on these upgrades without breaking
downstreams.

First PR https://github.com/apache/hadoop-thirdparty/pull/1 to create the
protobuf shaded jar is ready to merge.

Please take a look if anyone interested, will be merged may be after two
days if no objections.

-Vinay


On Thu, Oct 10, 2019 at 3:30 AM Wei-Chiu Chuang  wrote:

> Hi I am late to this but I am keen to understand more.
>
> To be exact, how can we better use the thirdparty repo? Looking at HBase
> as an example, it looks like everything that are known to break a lot after
> an update get shaded into the hbase-thirdparty artifact: guava, netty, ...
> etc.
> Is it the purpose to isolate these naughty dependencies?
>
> On Wed, Oct 9, 2019 at 12:38 PM Vinayakumar B 
> wrote:
>
>> Hi All,
>>
>> I have updated the PR as per @Owen O'Malley 
>> 's suggestions.
>>
>> i. Renamed the module to 'hadoop-shaded-protobuf37'
>> ii. Kept the shaded package to 'o.a.h.thirdparty.protobuf37'
>>
>> Please review!!
>>
>> Thanks,
>> -Vinay
>>
>>
>> On Sat, Sep 28, 2019 at 10:29 AM 张铎(Duo Zhang) 
>> wrote:
>>
>> > For HBase we have a separated repo for hbase-thirdparty
>> >
>> > https://github.com/apache/hbase-thirdparty
>> >
>> > We will publish the artifacts to nexus so we do not need to include
>> > binaries in our git repo, just add a dependency in the pom.
>> >
>> >
>> >
>> https://mvnrepository.com/artifact/org.apache.hbase.thirdparty/hbase-shaded-protobuf
>> >
>> >
>> > And it has its own release cycles, only when there are special
>> requirements
>> > or we want to upgrade some of the dependencies. This is the vote thread
>> for
>> > the newest release, where we want to provide a shaded gson for jdk7.
>> >
>> >
>> >
>> https://lists.apache.org/thread.html/f12c589baabbc79c7fb2843422d4590bea982cd102e2bd9d21e9884b@%3Cdev.hbase.apache.org%3E
>> >
>> >
>> > Thanks.
>> >
>> > Vinayakumar B  于2019年9月28日周六 上午1:28写道：
>> >
>> > > Please find replies inline.
>> > >
>> > > -Vinay
>> > >
>> > > On Fri, Sep 27, 2019 at 10:21 PM Owen O'Malley <
>> owen.omal...@gmail.com>
>> > > wrote:
>> > >
>> > > > I'm very unhappy with this direction. In particular, I don't think
>> git
>> > is
>> > > > a good place for distribution of binary artifacts. Furthermore, the
>> PMC
>> > > > shouldn't be releasing anything without a release vote.
>> > > >
>> > > >
>> > > Proposed solution doesnt release any binaries in git. Its actually a
>> > > complete sub-project which follows entire release process, including
>> VOTE
>> > > in public. I have mentioned already that release process is similar to
>> > > hadoop.
>> > > To be specific, using the (almost) same script used in hadoop to
>> generate
>> > > artifacts, sign and deploy to staging repository. Please let me know
>> If I
>> > > am conveying anything wrong.
>> > >
>> > >
>> > > > I'd propose that we make a third party module that contains the
>> > *source*
>> > > > of the pom files to build the relocated jars. This should
>> absolutely be
>> > > > treated as a last resort for the mostly Google projects that
>> regularly
>> > > > break binary compatibility (eg. Protobuf & Guava).
>> > > >
>> > > >
>> > > Same has been implemented in the PR
>> > > https://github.com/apache/hadoop-thirdparty/pull/1. Please check and
>> let
>> > > me
>> > > know If I misunderstood. Yes, this is the last option we have AFAIK.
>> > >
>> > >
>> > > > In terms of naming, I'd propose something like:
>> > > >
>> > > > org.apache.hadoop.thirdparty.protobuf2_5
>> > > > org.apache.hadoop.thirdparty.guava28
>> > > >
>> > > > 

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Vinayakumar B]

Thanks Zhenyu Zheng for the offer to donate machines for ARM CI.

We will definitely make use of it. Let me check how we can set up Jenkins
job for ARM.

-Thanks
Vinay


On Tue, 22 Oct 2019, 1:04 pm Zhenyu Zheng, 
wrote:

> Hi All,
>
> Some updates for ARM CI, our team has succesfully donated ARM resources
> and setup an ARM CI for Apache Spark:
> https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm/
> it will set to periodic job and then PR trigger when we think it is stable
> enough.
>
> I really hope we can do the same for Hadoop.
>
> BR,
>
> On Fri, Sep 6, 2019 at 6:11 AM Vinayakumar B 
> wrote:
>
>> Thanks @Anu
>>
>> I understand the concern. I took it in different manner.
>>
>> Anyway, since protobuf upgrade looks huge, and need everyone's eyes on
>> changes as early as possible, its better to do it trunk itself.
>>
>> I was able to come to successfull attempt of upgrading protobuf as per
>> suggestion of stack in Jira
>> https://issues.apache.org/jira/browse/HADOOP-13363?focusedCommentId=15958253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15958253
>>  .
>>
>> Have created the PR. Please review.  Changes looks huge, because all
>> references of "com.google.protobuf" relocated to
>> "o.a.h.shaded.com.google.protobuf".
>> Otherwise changes are reasonable.
>>
>> This change is with still keeping the current 2.5.0 dependency, for
>> downstream builds. So essentially nothing should be changed for downstreams.
>>
>> -Vinay
>>
>>
>> On Thu, Sep 5, 2019 at 9:56 PM Anu Engineer 
>> wrote:
>>
>>> Yes, I think that is what Sunil and I are trying to suggest; the complex
>>> dependencies like Protobuf, if you do it in the trunk you have a better
>>> change of getting it done. Otherwise, at merge point random downstream
>>> applications which you have never heard of will object, and Hadoop
>>> compatibility rules are very clear so you cannot fix it.
>>>
>>> With that said, even doing this in the trunk is complex; It might be
>>> good for you to host a meeting and get some feedback. I have openly said it
>>> is a great idea like "belling the cat", but the effort is in getting the
>>> community to agree and align. Solve that, most of your technical problems
>>> will be easier to solve.
>>>
>>> If you go into a branch, it might be that the community might forget
>>> about your work; and when you come in to merge you will see issues which
>>> you did not think about.
>>>
>>> So, Here is what would be great if you can make this happen; for ARM
>>> work, get a list of dependencies that needed to be upgraded; see if you can
>>> get the community aligned with this goal; since ARM might not be in the
>>> target for many users. You need to convince users that even without ARM,
>>> this is a great idea.
>>>
>>> If you like we can get together during one of the HDFS meetups hosted by
>>> Wei-chiu.
>>>
>>> Thanks
>>> Anu
>>>
>>>
>>>
>>> On Thu, Sep 5, 2019 at 3:19 AM Vinayakumar B 
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Thanks for the response.
>>>> As I see, protobuf upgrade is long pending and most awaited one.
>>>>
>>>> @Sunil
>>>> Protobuf upgrade looks to be a non-trivial task.
>>>> Thanks @Duo Zhang for the suggestion of
>>>> 'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the
>>>> problem
>>>> of dependency on build environment.
>>>> However more problem lies in upgrade protobuf without breaking the
>>>> downstream builds.
>>>> Reason is many downstream projects directly refers server specific jars
>>>> and
>>>> they expect protobuf-2.5.0 jar to get added to classpath by transitive
>>>> dependency.
>>>>
>>>> There are some historical discussions and suggestions on
>>>> https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
>>>> upgrade.
>>>> Recommended path for solution is, try to upgrade protobuf using shading
>>>> of
>>>> latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
>>>> dependencies for downstreams.
>>>> I am trying out ideas on this and if it gets completed within time, may
>>>> be
>>>> we can target trunk itself for the protobuf up

Re: [DISCUSS] About creation of Hadoop Thirdparty repository for shaded artifacts

[Vinayakumar B]

Hi All,

I have updated the PR as per @Owen O'Malley 
's suggestions.

i. Renamed the module to 'hadoop-shaded-protobuf37'
ii. Kept the shaded package to 'o.a.h.thirdparty.protobuf37'

Please review!!

Thanks,
-Vinay


On Sat, Sep 28, 2019 at 10:29 AM 张铎(Duo Zhang) 
wrote:

> For HBase we have a separated repo for hbase-thirdparty
>
> https://github.com/apache/hbase-thirdparty
>
> We will publish the artifacts to nexus so we do not need to include
> binaries in our git repo, just add a dependency in the pom.
>
>
> https://mvnrepository.com/artifact/org.apache.hbase.thirdparty/hbase-shaded-protobuf
>
>
> And it has its own release cycles, only when there are special requirements
> or we want to upgrade some of the dependencies. This is the vote thread for
> the newest release, where we want to provide a shaded gson for jdk7.
>
>
> https://lists.apache.org/thread.html/f12c589baabbc79c7fb2843422d4590bea982cd102e2bd9d21e9884b@%3Cdev.hbase.apache.org%3E
>
>
> Thanks.
>
> Vinayakumar B  于2019年9月28日周六 上午1:28写道：
>
> > Please find replies inline.
> >
> > -Vinay
> >
> > On Fri, Sep 27, 2019 at 10:21 PM Owen O'Malley 
> > wrote:
> >
> > > I'm very unhappy with this direction. In particular, I don't think git
> is
> > > a good place for distribution of binary artifacts. Furthermore, the PMC
> > > shouldn't be releasing anything without a release vote.
> > >
> > >
> > Proposed solution doesnt release any binaries in git. Its actually a
> > complete sub-project which follows entire release process, including VOTE
> > in public. I have mentioned already that release process is similar to
> > hadoop.
> > To be specific, using the (almost) same script used in hadoop to generate
> > artifacts, sign and deploy to staging repository. Please let me know If I
> > am conveying anything wrong.
> >
> >
> > > I'd propose that we make a third party module that contains the
> *source*
> > > of the pom files to build the relocated jars. This should absolutely be
> > > treated as a last resort for the mostly Google projects that regularly
> > > break binary compatibility (eg. Protobuf & Guava).
> > >
> > >
> > Same has been implemented in the PR
> > https://github.com/apache/hadoop-thirdparty/pull/1. Please check and let
> > me
> > know If I misunderstood. Yes, this is the last option we have AFAIK.
> >
> >
> > > In terms of naming, I'd propose something like:
> > >
> > > org.apache.hadoop.thirdparty.protobuf2_5
> > > org.apache.hadoop.thirdparty.guava28
> > >
> > > In particular, I think we absolutely need to include the version of the
> > > underlying project. On the other hand, since we should not be shading
> > > *everything* we can drop the leading com.google.
> > >
> > >
> > IMO, This naming convention is easy for identifying the underlying
> project,
> > but  it will be difficult to maintain going forward if underlying project
> > versions changes. Since thirdparty module have its own releases, each of
> > those release can be mapped to specific version of underlying project.
> Even
> > the binary artifact can include a MANIFEST with underlying project
> details
> > as per Steve's suggestion on HADOOP-13363.
> > That said, if you still prefer to have project number in artifact id, it
> > can be done.
> >
> > The Hadoop project can make releases of  the thirdparty module:
> > >
> > > 
> > >   org.apache.hadoop
> > >   hadoop-thirdparty-protobuf25
> > >   1.0
> > > 
> > >
> > >
> > Note that the version has to be the hadoop thirdparty release number,
> which
> > > is part of why you need to have the underlying version in the artifact
> > > name. These we can push to maven central as new releases from Hadoop.
> > >
> > >
> > Exactly, same has been implemented in the PR. hadoop-thirdparty module
> have
> > its own releases. But in HADOOP Jira, thirdparty versions can be
> > differentiated using prefix "thirdparty-".
> >
> > Same solution is being followed in HBase. May be people involved in HBase
> > can add some points here.
> >
> > Thoughts?
> > >
> > > .. Owen
> > >
> > > On Fri, Sep 27, 2019 at 8:38 AM Vinayakumar B  >
> > > wrote:
> > >
> > >> Hi All,
> > >>
> > >>I wanted to discuss about the separate repo for thirdparty
>

Re: Daily Builds Getting Aborted Due To Timeout

[Vinayakumar B]

Thanks Ayush,

I think possible other ways are improving the test runtime.

1. try "parallel-tests" profile if applicable on all modules whichever is
okay with tests running in parallel. This may reduce the total unit tests
runtime drastically. ex: I can see hadoop-hdfs-rbf takes around ~22 min.
Here parallel-tests can be implemented (Also need to run and verify that
all tests are using local filesystem path which doesnot collide with other
tests running in parallel.

2. Right now all modules of HDDS and OZone also running as part of
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/
, But
I can also see there was a QBT Job created for Ozone alone. Can we make
that work and exclude HDDS and OZone related executions from
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1271/ ?
  In above aborted builds, it was running unit tests in HDDS/Ozone
module. That means all other modules have finished.

3. If above options are not possible or not resulting in any improvements
then we can definitely go for timeout increase.

-Vinay


On Fri, Sep 27, 2019 at 8:54 PM Ayush Saxena  wrote:

> Hi All,
> Just to bring into notice the hadoop daily builds are getting aborted due
> to timeout( Configured to be 900 Minutes).
>
> > Build timed out (after 900 minutes). Marking the build as aborted.
> > Build was aborted
> > [CHECKSTYLE] Skipping publisher since build result is ABORTED
> > [FINDBUGS] Skipping publisher since build result is ABORTED
> >
> > Recording test results
> > No emails were triggered.
> > Finished: ABORTED
> >
> >
> Reference :
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1271/
>
> Checked with the infra team, The only resolution told was to increase the
> configured time of 900 mins or make the build take less time.
>
> Someone with access to change the config can probably increase the time.
> (Probably people in https://whimsy.apache.org/roster/group/hudson-jobadmin
> have access)
> *Link To Change Configured Time* :
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/configure
> <
> https://slack-redir.net/link?url=https%3A%2F%2Fbuilds.apache.org%2Fjob%2Fhadoop-qbt-trunk-java8-linux-x86%2Fconfigure&v=3
> >
>
>
> -Ayush
>

Re: [DISCUSS] About creation of Hadoop Thirdparty repository for shaded artifacts

[Vinayakumar B]

Please find replies inline.

-Vinay

On Fri, Sep 27, 2019 at 10:21 PM Owen O'Malley 
wrote:

> I'm very unhappy with this direction. In particular, I don't think git is
> a good place for distribution of binary artifacts. Furthermore, the PMC
> shouldn't be releasing anything without a release vote.
>
>
Proposed solution doesnt release any binaries in git. Its actually a
complete sub-project which follows entire release process, including VOTE
in public. I have mentioned already that release process is similar to
hadoop.
To be specific, using the (almost) same script used in hadoop to generate
artifacts, sign and deploy to staging repository. Please let me know If I
am conveying anything wrong.


> I'd propose that we make a third party module that contains the *source*
> of the pom files to build the relocated jars. This should absolutely be
> treated as a last resort for the mostly Google projects that regularly
> break binary compatibility (eg. Protobuf & Guava).
>
>
Same has been implemented in the PR
https://github.com/apache/hadoop-thirdparty/pull/1. Please check and let me
know If I misunderstood. Yes, this is the last option we have AFAIK.


> In terms of naming, I'd propose something like:
>
> org.apache.hadoop.thirdparty.protobuf2_5
> org.apache.hadoop.thirdparty.guava28
>
> In particular, I think we absolutely need to include the version of the
> underlying project. On the other hand, since we should not be shading
> *everything* we can drop the leading com.google.
>
>
IMO, This naming convention is easy for identifying the underlying project,
but  it will be difficult to maintain going forward if underlying project
versions changes. Since thirdparty module have its own releases, each of
those release can be mapped to specific version of underlying project. Even
the binary artifact can include a MANIFEST with underlying project details
as per Steve's suggestion on HADOOP-13363.
That said, if you still prefer to have project number in artifact id, it
can be done.

The Hadoop project can make releases of  the thirdparty module:
>
> 
>   org.apache.hadoop
>   hadoop-thirdparty-protobuf25
>   1.0
> 
>
>
Note that the version has to be the hadoop thirdparty release number, which
> is part of why you need to have the underlying version in the artifact
> name. These we can push to maven central as new releases from Hadoop.
>
>
Exactly, same has been implemented in the PR. hadoop-thirdparty module have
its own releases. But in HADOOP Jira, thirdparty versions can be
differentiated using prefix "thirdparty-".

Same solution is being followed in HBase. May be people involved in HBase
can add some points here.

Thoughts?
>
> .. Owen
>
> On Fri, Sep 27, 2019 at 8:38 AM Vinayakumar B 
> wrote:
>
>> Hi All,
>>
>>I wanted to discuss about the separate repo for thirdparty dependencies
>> which we need to shaded and include in Hadoop component's jars.
>>
>>Apologies for the big text ahead, but this needs clear explanation!!
>>
>>Right now most needed such dependency is protobuf. Protobuf dependency
>> was not upgraded from 2.5.0 onwards with the fear that downstream builds,
>> which depends on transitive dependency protobuf coming from hadoop's jars,
>> may fail with the upgrade. Apparently protobuf does not guarantee source
>> compatibility, though it guarantees wire compatibility between versions.
>> Because of this behavior, version upgrade may cause breakage in known and
>> unknown (private?) downstreams.
>>
>>So to tackle this, we came up the following proposal in HADOOP-13363.
>>
>>Luckily, As far as I know, no APIs, either public to user or between
>> Hadoop processes, is not directly using protobuf classes in signatures.
>> (If
>> any exist, please let us know).
>>
>>Proposal:
>>
>>
>>1. Create a artifact(s) which contains shaded dependencies. All such
>> shading/relocation will be with known prefix
>> **org.apache.hadoop.thirdparty.**.
>>2. Right now protobuf jar (ex: o.a.h.thirdparty:hadoop-shaded-protobuf)
>> to start with, all **com.google.protobuf** classes will be relocated as
>> **org.apache.hadoop.thirdparty.com.google.protobuf**.
>>3. Hadoop modules, which needs protobuf as dependency, will add this
>> shaded artifact as dependency (ex:
>> o.a.h.thirdparty:hadoop-shaded-protobuf).
>>4. All previous usages of "com.google.protobuf" will be relocated to
>> "org.apache.hadoop.thirdparty.com.google.protobuf" in the code and will be
>> committed. Please note, this replacement is One-Time directly in source
>> code, NOT during co

[DISCUSS] About creation of Hadoop Thirdparty repository for shaded artifacts

[Vinayakumar B]

Hi All,

   I wanted to discuss about the separate repo for thirdparty dependencies
which we need to shaded and include in Hadoop component's jars.

   Apologies for the big text ahead, but this needs clear explanation!!

   Right now most needed such dependency is protobuf. Protobuf dependency
was not upgraded from 2.5.0 onwards with the fear that downstream builds,
which depends on transitive dependency protobuf coming from hadoop's jars,
may fail with the upgrade. Apparently protobuf does not guarantee source
compatibility, though it guarantees wire compatibility between versions.
Because of this behavior, version upgrade may cause breakage in known and
unknown (private?) downstreams.

   So to tackle this, we came up the following proposal in HADOOP-13363.

   Luckily, As far as I know, no APIs, either public to user or between
Hadoop processes, is not directly using protobuf classes in signatures. (If
any exist, please let us know).

   Proposal:
   

   1. Create a artifact(s) which contains shaded dependencies. All such
shading/relocation will be with known prefix
**org.apache.hadoop.thirdparty.**.
   2. Right now protobuf jar (ex: o.a.h.thirdparty:hadoop-shaded-protobuf)
to start with, all **com.google.protobuf** classes will be relocated as
**org.apache.hadoop.thirdparty.com.google.protobuf**.
   3. Hadoop modules, which needs protobuf as dependency, will add this
shaded artifact as dependency (ex: o.a.h.thirdparty:hadoop-shaded-protobuf).
   4. All previous usages of "com.google.protobuf" will be relocated to
"org.apache.hadoop.thirdparty.com.google.protobuf" in the code and will be
committed. Please note, this replacement is One-Time directly in source
code, NOT during compile and package.
   5. Once all usages of "com.google.protobuf" is relocated, then hadoop
dont care about which version of original  "protobuf-java" is in dependency.
   6. Just keep "protobuf-java:2.5.0" in dependency tree not to break the
downstreams. But hadoop will be originally using the latest protobuf
present in "o.a.h.thirdparty:hadoop-shaded-protobuf".

   7. Coming back to separate repo, Following are most appropriate reasons
of keeping shaded dependency artifact in separate repo instead of submodule.

  7a. These artifacts need not be built all the time. It needs to be
built only when there is a change in the dependency version or the build
process.
  7b. If added as "submodule in Hadoop repo", maven-shade-plugin:shade
will execute only in package phase. That means, "mvn compile" or "mvn
test-compile" will not be failed as this artifact will not have relocated
classes, instead it will have original classes, resulting in compilation
failure. Workaround, build thirdparty submodule first and exclude
"thirdparty" submodule in other executions. This will be a complex process
compared to keeping in a separate repo.

  7c. Separate repo, will be a subproject of Hadoop, using the same
HADOOP jira project, with different versioning prefixed with "thirdparty-"
(ex: thirdparty-1.0.0).
  7d. Separate will have same release process as Hadoop.


HADOOP-13363 (https://issues.apache.org/jira/browse/HADOOP-13363) is an
umbrella jira tracking the changes to protobuf upgrade.

PR (https://github.com/apache/hadoop-thirdparty/pull/1) has been raised
for separate repo creation in (HADOOP-16595 (
https://issues.apache.org/jira/browse/HADOOP-16595)

Please provide your inputs for the proposal and review the PR to
proceed with the proposal.


   -Thanks,
Vinay

On Fri, Sep 27, 2019 at 11:54 AM Vinod Kumar Vavilapalli 
wrote:

> Moving the thread to the dev lists.
>
> Thanks
> +Vinod
>
> > On Sep 23, 2019, at 11:43 PM, Vinayakumar B 
> wrote:
> >
> > Thanks Marton,
> >
> > Current created 'hadoop-thirdparty' repo is empty right now.
> > Whether to use that repo  for shaded artifact or not will be monitored in
> > HADOOP-13363 umbrella jira. Please feel free to join the discussion.
> >
> > There is no existing codebase is being moved out of hadoop repo. So I
> think
> > right now we are good to go.
> >
> > -Vinay
> >
> > On Mon, Sep 23, 2019 at 11:38 PM Marton Elek  wrote:
> >
> >>
> >> I am not sure if it's defined when is a vote required.
> >>
> >> https://www.apache.org/foundation/voting.html
> >>
> >> Personally I think it's a big enough change to send a notification to
> the
> >> dev lists with a 'lazy consensus'  closure
> >>
> >> Marton
> >>
> >> On 2019/09/23 17:46:37, Vinayakumar B  wrote:
> >>> Hi,
> >>>
> >>> As discussed in HADOOP-13363, protobuf 3.x jar (and may be more in
> >> f

Re: [NOTICE] Building trunk needs protoc 3.7.1

[Vinayakumar B]

Thanks Steve.

Idea is not to shade all artifacts.
Instead maintain one artifact ( hadoop-thirdparty) which have all such
dependencies ( com.google.* may be), add  this artifact as dependency in
hadoop modules. Use shaded classes directly in the code of hadoop modules
instead of shading at package phase.

Hbase, ozone and ratis already following this way. The artifact (
hadoop-thirdparty) with shaded dependencies can be maintained in a separate
repo as suggested by stack on HADOOP-13363 or could be maintained as a
separate module in Hadoop repo. If maintained in separate repo, need to
build this only when there are changes related to shaded dependencies.


-Vinay

On Sun, 22 Sep 2019, 10:11 pm Steve Loughran,  wrote:

>
>
> On Sun, Sep 22, 2019 at 3:22 PM Vinayakumar B 
> wrote:
>
>>Protobuf provides Wire compatibility between releases.. but not
>> guarantees the source compatibility in generated sources. There will be a
>> problem in compatibility if anyone uses generated protobuf message outside
>> of Hadoop modules. Which ideally shouldn't be as generated sources are not
>> public APIs.
>>
>>There should not be any compatibility problems between releases in
>> terms
>> of communication provided both uses same syntax (proto2) of proto message.
>> This I have verified by communication between protobuf 2.5.0 client with
>> protobuf 3.7.1 server.
>>
>>To avoid the downstream transitive dependency classpath problem, who
>> might be using protobuf 2.5.0 classes, planning to shade the 3.7.1 classes
>> and its usages in all hadoop modules.. and keep 2.5.0 jar back in hadoop
>> classpath.
>>
>> Hope I have answered your question.
>>
>> -Vinay
>>
>>
> While I support the move and CP isolation, this is going to (finally)
> force us to make shaded versions of all artifacts which we publish with the
> intent of them being loaded on the classpath of other applications
>

Re: [NOTICE] Building trunk needs protoc 3.7.1

[Vinayakumar B]

   Protobuf provides Wire compatibility between releases.. but not
guarantees the source compatibility in generated sources. There will be a
problem in compatibility if anyone uses generated protobuf message outside
of Hadoop modules. Which ideally shouldn't be as generated sources are not
public APIs.

   There should not be any compatibility problems between releases in terms
of communication provided both uses same syntax (proto2) of proto message.
This I have verified by communication between protobuf 2.5.0 client with
protobuf 3.7.1 server.

   To avoid the downstream transitive dependency classpath problem, who
might be using protobuf 2.5.0 classes, planning to shade the 3.7.1 classes
and its usages in all hadoop modules.. and keep 2.5.0 jar back in hadoop
classpath.

Hope I have answered your question.

-Vinay

On Sun, 22 Sep 2019, 7:38 pm Vinod Kumar Vavilapalli, 
wrote:

> Quick question, being lazy here, lots of JIRA updates on HADOOP-13363 over
> the years not helping either.
>
> Does anyone know what this upgrade will mean w.r.t compatibility for the
> Hadoop releases themselves? Remember that trunk is still 3.x.
>
> Thanks
> +Vinod
>
> > On Sep 21, 2019, at 9:55 AM, Vinayakumar B 
> wrote:
> >
> > @Wangda Tan  ,
> > Sorry for the confusion. HADOOP-13363 is umbrella jira to track multiple
> > stages of protobuf upgrade in subtasks. (jar upgrade, Docker update,
> plugin
> > upgrade, shading, etc).
> > Right now, first task of jar upgrade is done. So need to update the
> protoc
> > executable in the in build environments.
> >
> > @张铎(Duo Zhang)  ,
> > Sorry for the inconvenience. Yes, indeed plugin update before jar upgrde
> > was possible. Sorry I missed it.
> >
> > Plugin update needed to be done for whole project, for which precommit
> > jenkins will need more time complete end-to-end runs.
> > So plugin update is planned in stages in further subtasks. It could be
> done
> > in 2-3 days.
> >
> > -Vinay
> >
> > On Sat, 21 Sep 2019, 5:55 am 张铎(Duo Zhang), 
> wrote:
> >
> >> I think this one is alread in place so we have to upgrade...
> >>
> >> https://issues.apache.org/jira/browse/HADOOP-16557
> >>
> >> Wangda Tan  于2019年9月21日周六 上午7:19写道：
> >>
> >>> Hi Vinay,
> >>>
> >>> A bit confused, I saw the HADOOP-13363 is still pending. Do we need to
> >>> upgrade protobuf version to 3.7.1 NOW or once HADOOP-13363 is
> completed?
> >>>
> >>> Thanks,
> >>> Wangda
> >>>
> >>> On Fri, Sep 20, 2019 at 8:11 AM Vinayakumar B  >
> >>> wrote:
> >>>
> >>>> Hi All,
> >>>>
> >>>> A very long pending task, protobuf upgrade is happening in
> >> HADOOP-13363.
> >>> As
> >>>> part of that protobuf version is upgraded to 3.7.1.
> >>>>
> >>>> Please update your build environments to have 3.7.1 protobuf version.
> >>>>
> >>>> BUILIDING.txt has been updated with latest instructions.
> >>>>
> >>>> This pre-requisite to update protoc dependecy manually is required
> >> until
> >>>> 'hadoop-maven-plugin' is replaced with 'protobuf-mavem-plugin' to
> >>>> dynamically resolve required protoc exe.
> >>>>
> >>>> Dockerfile is being updated to have latest 3.7.1 as default protoc for
> >>> test
> >>>> environments.
> >>>>
> >>>> Thanks,
> >>>> -Vinay
> >>>>
> >>>
> >>
>
>

Re: [NOTICE] Building trunk needs protoc 3.7.1

[Vinayakumar B]

@Wangda Tan  ,
Sorry for the confusion. HADOOP-13363 is umbrella jira to track multiple
stages of protobuf upgrade in subtasks. (jar upgrade, Docker update, plugin
upgrade, shading, etc).
Right now, first task of jar upgrade is done. So need to update the protoc
executable in the in build environments.

@张铎(Duo Zhang)  ,
Sorry for the inconvenience. Yes, indeed plugin update before jar upgrde
was possible. Sorry I missed it.

Plugin update needed to be done for whole project, for which precommit
jenkins will need more time complete end-to-end runs.
So plugin update is planned in stages in further subtasks. It could be done
in 2-3 days.

-Vinay

On Sat, 21 Sep 2019, 5:55 am 张铎(Duo Zhang),  wrote:

> I think this one is alread in place so we have to upgrade...
>
> https://issues.apache.org/jira/browse/HADOOP-16557
>
> Wangda Tan  于2019年9月21日周六 上午7:19写道：
>
> > Hi Vinay,
> >
> > A bit confused, I saw the HADOOP-13363 is still pending. Do we need to
> > upgrade protobuf version to 3.7.1 NOW or once HADOOP-13363 is completed?
> >
> > Thanks,
> > Wangda
> >
> > On Fri, Sep 20, 2019 at 8:11 AM Vinayakumar B 
> > wrote:
> >
> > > Hi All,
> > >
> > > A very long pending task, protobuf upgrade is happening in
> HADOOP-13363.
> > As
> > > part of that protobuf version is upgraded to 3.7.1.
> > >
> > > Please update your build environments to have 3.7.1 protobuf version.
> > >
> > > BUILIDING.txt has been updated with latest instructions.
> > >
> > > This pre-requisite to update protoc dependecy manually is required
> until
> > > 'hadoop-maven-plugin' is replaced with 'protobuf-mavem-plugin' to
> > > dynamically resolve required protoc exe.
> > >
> > > Dockerfile is being updated to have latest 3.7.1 as default protoc for
> > test
> > > environments.
> > >
> > > Thanks,
> > > -Vinay
> > >
> >
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Vinayakumar B]

Thanks @Anu

I understand the concern. I took it in different manner.

Anyway, since protobuf upgrade looks huge, and need everyone's eyes on
changes as early as possible, its better to do it trunk itself.

I was able to come to successfull attempt of upgrading protobuf as per
suggestion of stack in Jira
https://issues.apache.org/jira/browse/HADOOP-13363?focusedCommentId=15958253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15958253
 .

Have created the PR. Please review.  Changes looks huge, because all
references of "com.google.protobuf" relocated to
"o.a.h.shaded.com.google.protobuf".
Otherwise changes are reasonable.

This change is with still keeping the current 2.5.0 dependency, for
downstream builds. So essentially nothing should be changed for downstreams.

-Vinay


On Thu, Sep 5, 2019 at 9:56 PM Anu Engineer  wrote:

> Yes, I think that is what Sunil and I are trying to suggest; the complex
> dependencies like Protobuf, if you do it in the trunk you have a better
> change of getting it done. Otherwise, at merge point random downstream
> applications which you have never heard of will object, and Hadoop
> compatibility rules are very clear so you cannot fix it.
>
> With that said, even doing this in the trunk is complex; It might be good
> for you to host a meeting and get some feedback. I have openly said it is a
> great idea like "belling the cat", but the effort is in getting the
> community to agree and align. Solve that, most of your technical problems
> will be easier to solve.
>
> If you go into a branch, it might be that the community might forget about
> your work; and when you come in to merge you will see issues which you did
> not think about.
>
> So, Here is what would be great if you can make this happen; for ARM work,
> get a list of dependencies that needed to be upgraded; see if you can get
> the community aligned with this goal; since ARM might not be in the target
> for many users. You need to convince users that even without ARM, this is a
> great idea.
>
> If you like we can get together during one of the HDFS meetups hosted by
> Wei-chiu.
>
> Thanks
> Anu
>
>
>
> On Thu, Sep 5, 2019 at 3:19 AM Vinayakumar B 
> wrote:
>
>> Hi all,
>>
>> Thanks for the response.
>> As I see, protobuf upgrade is long pending and most awaited one.
>>
>> @Sunil
>> Protobuf upgrade looks to be a non-trivial task.
>> Thanks @Duo Zhang for the suggestion of
>> 'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the
>> problem
>> of dependency on build environment.
>> However more problem lies in upgrade protobuf without breaking the
>> downstream builds.
>> Reason is many downstream projects directly refers server specific jars
>> and
>> they expect protobuf-2.5.0 jar to get added to classpath by transitive
>> dependency.
>>
>> There are some historical discussions and suggestions on
>> https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
>> upgrade.
>> Recommended path for solution is, try to upgrade protobuf using shading of
>> latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
>> dependencies for downstreams.
>> I am trying out ideas on this and if it gets completed within time, may be
>> we can target trunk itself for the protobuf upgrade.
>>
>> separate branch idea is suggested for the overall ARM support including
>> protobuf upgrade and other problems mentioned in the discussion above.
>>
>> I dont expect separate branch to have a huge changes, apart from bug
>> fixes,
>> since there are no separate features specific to ARM is being planned.
>> So timely rebase of separate branch would reduce the overhead on branch
>> review/merge task.
>>
>> Still, if the solution to protobuf upgrade winds up early, without any
>> side
>> effects, I am more than happy to land it in trunk itself.
>>
>> Thanks,
>> Vinay
>> On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan  wrote:
>>
>> > Thanks Vinay for starting the thread.
>> >
>> > I agree to Anu's view point related to protobuf. And with the suggestion
>> > pointed out by Duo Zhang, if we can make use
>> > of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to
>> 3.0.0
>> > of protobuf will also be more easier.
>> >
>> > However i think its better to do this effort in trunk itself.
>> > In offline talks, few members were interested to start 3.3.0 release.
>> And
>> > given that happens soon, I feel its better
>> > we do this task in trunk itself as branch

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Vinayakumar B]

essor. Glad to see that
>> the
>> >> > hadoop community is also willing to solve the problem.
>> >> >
>> >> > Anu Engineer  于2019年9月3日周二 上午1:23写道：
>> >> >
>> >> >> +1, for the branch idea. Just FYI, Your biggest problem is proving
>> that
>> >> >> Hadoop and the downstream projects work correctly after you upgrade
>> >> core
>> >> >> components like Protobuf.
>> >> >> So while branching and working on a branch is easy, merging back
>> after
>> >> you
>> >> >> upgrade some of these core components is insanely hard. You might
>> want
>> >> to
>> >> >> make sure that community buys into upgrading these components in the
>> >> trunk.
>> >> >> That way we will get testing and downstream components will notice
>> when
>> >> >> things break.
>> >> >>
>> >> >> That said, I have lobbied for the upgrade of Protobuf for a really
>> long
>> >> >> time; I have argued that 2.5 is out of support and we cannot stay on
>> >> that
>> >> >> branch forever; or we need to take ownership of the Protobuf 2.5
>> code
>> >> base.
>> >> >> It has been rightly pointed to me that while all the arguments I
>> make
>> >> is
>> >> >> correct; it is a very complicated task to upgrade Protobuf, and the
>> >> worst
>> >> >> part is we will not even know what breaks until downstream projects
>> >> pick up
>> >> >> these changes and work against us.
>> >> >>
>> >> >> If we work off the Hadoop version 3 — and assume that we have
>> >> "shading" in
>> >> >> place for all deployments; it might be possible to get there; still
>> a
>> >> >> daunting task.
>> >> >>
>> >> >> So best of luck with the branch approach — But please remember,
>> Merging
>> >> >> back will be hard, Just my 2 cents.
>> >> >>
>> >> >> — Anu
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
>> zhengzhenyul...@gmail.com
>> >> >
>> >> >> wrote:
>> >> >>
>> >> >>> Hi,
>> >> >>>
>> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
>> >> separate
>> >> >>> branch with it's own ARM CI seems a really good idea.
>> >> >>> By doing this we won't break any of the undergoing development in
>> >> trunk
>> >> >> and
>> >> >>> a CI can be a very good way to show what are the
>> >> >>> current problems and what have been fixed, it will also provide a
>> very
>> >> >> good
>> >> >>> view for contributors that are intrested to working on
>> >> >>> this. We can finally merge back the branch to trunk until the
>> >> community
>> >> >>> thinks it is good enough and stable enough. We can donate
>> >> >>> ARM machines to the existing CI system for the job.
>> >> >>>
>> >> >>> I wonder if this approch possible?
>> >> >>>
>> >> >>> BR,
>> >> >>>
>> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
>> liusheng2...@gmail.com>
>> >> >>> wrote:
>> >> >>>
>> >> >>>> Hi,
>> >> >>>>
>> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
>> community
>> >> >>>> mentioned by Vinay. I am working on building and
>> >> >>>> testing Hadoop components on aarch64 server these days, besides
>> the
>> >> >>> missing
>> >> >>>> dependices of ARM platform issues #1 #2 #3
>> >> >>>> mentioned by Vinay, other similar issue has also be found, such as
>> >> the
>> >> >>>> "PhantomJS" dependent package also missing for aarch64.
>> >> >>>>
>> >> >>>> To promote the ARM support for Hadoop, we have discussed and
>>

Re: [VOTE] Moving Submarine to a separate Apache project proposal

[Vinayakumar B]

+1

Thanks for the proposal.
Its very interesting project and looks very promising one by looking at the
participations from various companies and the speed of development.

I would also like to participate in the project.
Please add me as well.

Thanks,
-Vinay

On Tue, 3 Sep 2019, 12:38 pm Rakesh Radhakrishnan, 
wrote:

> +1, Thanks for the proposal.
>
> I am interested to participate in this project. Please include me as well
> in the project.
>
> Thanks,
> Rakesh
>
> On Tue, Sep 3, 2019 at 11:59 AM zhankun tang 
> wrote:
>
> > +1
> >
> > Thanks for Wangda's proposal.
> >
> > The submarine project is born within Hadoop, but not limited to Hadoop.
> It
> > began with a trainer on YARN but it quickly realized that only a trainer
> is
> > not enough to meet the AI platform requirements. But now there's no
> > user-friendly open-source solution covers the whole AI pipeline like data
> > engineering, training, and serving. And the underlying data
> infrastructure
> > itself is also evolving, for instance, many people love k8s. Not
> mentioning
> > there're many AI domain problems in this area to be solved.
> > It's almost for sure that building such an ML platform would utilize
> > various other open-source components taking ML into consideration
> > initially.
> >
> > I see submarine grows rapidly towards an enterprise-grade ML platform
> which
> > could potentially enable AI ability for data engineer and scientist. This
> > is an exciting thing for both the community and the industry.
> >
> > BR,
> > Zhankun
> >
> >
> > On Tue, 3 Sep 2019 at 13:34, Xun Liu  wrote:
> >
> > > +1
> > >
> > > Hello everyone, I am a member of the submarine development team.
> > > I have been contributing to submarine for more than a year.
> > > I have seen the progress of submarine development very fast.
> > > In more than a year, there are 9 long-term developers of different
> > > companies. Contributing,
> > > submarine cumulative code has more than 200,000 lines of code, is
> growing
> > > very fast,
> > > and is used in the production environment of multiple companies.
> > >
> > > In the submarine development group, there are 5 PMCs and 7committer
> > members
> > > from Hadoop, spark, zeppelin projects.
> > > They are very familiar with the development process and specifications
> of
> > > the apache community,
> > > and can well grasp the project development progress and project
> quality.
> > > So I recommend submarine to be a TLP project directly.
> > >
> > > We will continue to contribute to the submarine project. :-)
> > >
> > > Xun Liu
> > > Regards
> > >
> > > On Tue, 3 Sep 2019 at 12:01, Devaraj K  wrote:
> > >
> > > > +1
> > > >
> > > > Thanks Wangda for the proposal.
> > > > I would like to participate in this project, Please add me also to
> the
> > > > project.
> > > >
> > > > Regards
> > > > Devaraj K
> > > >
> > > > On Mon, Sep 2, 2019 at 8:50 PM zac yuan 
> wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Submarine will be a complete solution for AI service development.
> It
> > > can
> > > > > take advantage of two best cluster systems: yarn and k8s, which
> will
> > > help
> > > > > more and more people get AI ability. To be a separate Apache
> project,
> > > > will
> > > > > accelerate the procedure of development apparently.
> > > > >
> > > > > Look forward to a big success in submarine project~
> > > > >
> > > > > 朱林浩  于2019年9月3日周二 上午10:38写道：
> > > > >
> > > > > > +1,
> > > > > > Hopefully, that will become the top project,
> > > > > >
> > > > > > I also hope to make more contributions to this project.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > At 2019-09-03 09:26:53, "Naganarasimha Garla" <
> > > > > naganarasimha...@apache.org>
> > > > > > wrote:
> > > > > > >+ 1,
> > > > > > >I would also like start participate in this project, hope to
> > get
> > > > > > myself
> > > > > > >added to the project.
> > > > > > >
> > > > > > >Thanks and Regards,
> > > > > > >+ Naga
> > > > > > >
> > > > > > >On Tue, Sep 3, 2019 at 8:35 AM Wangda Tan 
> > > > wrote:
> > > > > > >
> > > > > > >> Hi Sree,
> > > > > > >>
> > > > > > >> I put it to the proposal, please let me know what you think:
> > > > > > >>
> > > > > > >> The traditional path at Apache would have been to create an
> > > > incubator
> > > > > > >> > project, but the code is already being released by Apache
> and
> > > most
> > > > > of
> > > > > > the
> > > > > > >> > developers are familiar with Apache rules and guidelines. In
> > > > > > particular,
> > > > > > >> > the proposed PMC has 2 Apache TLP PMCs and proposed initial
> > > > > committers
> > > > > > >> > have 4 Apache TLP PMCs (from 3 different companies). They
> will
> > > > > provide
> > > > > > >> > oversight and guidance for the developers that are less
> > > > experienced
> > > > > in
> > > > > > >> the
> > > > > > >> > Apache Way. Therefore, the Submarine project would like to
> > > propose
> > > > > > >> becoming
> > > > > 

[DISCUSS] ARM/aarch64 support for Hadoop

[Vinayakumar B]

Hi Folks,

ARM is becoming famous lately in its processing capability and has got the
potential to run Bigdata workloads.
Many users have been moving to ARM machines due to its low cost.

In the past there were attempts to compile Hadoop on ARM (Rasberry PI) for
experimental purposes. Today ARM architecture is taking some of the
serverside processing as well. So there will be/is a real need of Hadoop to
support ARM architecture as well.

There are bunch of users who are trying out building Hadoop on ARM, trying
to add ARM CI to hadoop and facing issues[1]. Also some

As of today, Hadoop does not compile on ARM due to below issues, found from
testing done in openlab in [2].

1. Protobuf :
---
 Hadoop project (also some downstream projects) stuck to protobuf 2.5.0
version, due to backward compatibility reasons. Protobuf-2.5.0 is not being
maintained in the community. While protobuf 3.x is being actively adopted
widely, still protobuf 3.x provides wire compatibility for proto2 messages.
Due to some compilation issues in the generated java code, which can induce
problems in downstream. Due to this reason protobuf upgrade from 2.5.0 was
not taken up.
In 3.0.0 onwards, hadoop supports shading of libraries to avoid classpath
problem in downstream projects.
There are patches available to fix compilation in Hadoop. But need to
find a way to upgrade protobuf to latest version and still maintain the
downstream's classpath using shading feature of Hadoop build.

 There is a Jira for protobuf upgrade[3] created even before shade
support was added to Hadoop. Now need to revisit the Jira and continue
explore possibilities.

2. leveldbjni:
---
Current leveldbjni used in YARN doesnot support ARM architecture, need
to check whether any of the future versions support ARM and can hadoop
upgrade to that version.


3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
-
'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by default in
the maven repository. Workaround is to build it locally and keep in local
maven repository.
Need to check whether any future versions of 'protoc-gen-grpc-java' is
having ARM executable and whether hadoop-yarn-csi can upgrade it?


Once the compilation issues are solved, then there might be many native
code related issues due to different architectures.
So to explore everything, need to join hands together and proceed.


Let us discuss and check, whether any body else out there who also need the
support of Hadoop on ARM architectures and ready to lend their hands and
time in this work.


[1] https://issues.apache.org/jira/browse/HADOOP-16358
[2]
https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
[3] https://issues.apache.org/jira/browse/HADOOP-13363

-Vinay

[ANNOUNCE] New Apache Hadoop Committer - Ayush Saxena

[Vinayakumar B]

In bcc: general@, please bcc: (and not cc:) general@ if you want to include

It's my pleasure to announce that Ayush Saxena has been elected as
committer
on the Apache Hadoop project recognising his continued contributions to the
project.

Please join me in congratulating him.

Hearty Congratulations & Welcome aboard Ayush!


Vinayakumar B
(On behalf of the Hadoop PMC)

Re: [VOTE]: Support for RBF data locality Solution

[Vinayakumar B]

+1 for approach A.

On Thu, 11 Apr 2019, 12:23 pm Akira Ajisaka,  wrote:

> The Approach A looks good to me.
>
> Thanks,
> Akira
>
> On Thu, Apr 11, 2019 at 2:30 PM Xiaoqiao He  wrote:
> >
> > Hi forks,
> >
> > The current implementation of RBF is not sensitive about data locality,
> > since NameNode could not get real client hostname by invoke
> > Server#getRemoteAddress when RPC request forward by Router to NameNode.
> > Therefore, it will lead to several challenges, for instance,
> >
> >- a. Client could have to go for remote read instead of local read,
> >Short-Circuit could not be used in most cases.
> >- b. Block placement policy could not run as except based on defined
> >rack aware. Thus it will loss local node write.
> >
> > There are some different solutions to solve data locality issue after
> > discussion, some of them will change RPC protocol, so we look forward to
> > furthermore suggestions and votes. HDFS-13248 is tracking the issue.
> >
> >- Approach A: Changing IPC/RPC layer protocol
> (IpcConnectionContextProto
> >or RpcHeader#RpcRequestHeaderProto) and add extra field about client
> >hostname. Of course the new field is optional, only input by Router
> and
> >parse by Namenode in generally. This approach is compatibility and
> Client
> >should do nothing after changing.
> >- Approach B: Changing ClientProtocol and add extra interface
> >create/append/getBlockLocations with additional parameter about client
> >hostname. As approach A, it is input by Router and parse by Namenode,
> and
> >also is compatibility.
> >- Approach C: Solve write and read locality separately based on
> current
> >interface and no changes, for write, hack client hostname as one of
> favor
> >nodes for addBlocks, for read, reorder targets at Router after
> Namenode
> >returns result to Router.
> >
> > As discussion and evaluation in HDFS-13248, we prefer to change IPC/RPC
> > layer protocol to support RPC data locality. We welcome more suggestions,
> > votes or just give us feedback to push forward this feature. Thanks.
> >
> > Best Regards,
> > Hexiaoqiao
> >
> > reference
> > [1] https://issues.apache.org/jira/browse/HDFS-13248
> > [2] https://issues.apache.org/jira/browse/HDFS-10467
> >
> > [3] https://issues.apache.org/jira/browse/HDFS-12615
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Re: Including Original Author in git commits.

[Vinayakumar B]

So.. if we started doing that already.. we can encourage contributors to
attach formatted patch.. or create PRs.

And update wiki to follow exact steps to contribute and commit.

-Vinay


On Thu, 14 Feb 2019, 4:54 pm Steve Loughran  I've been trying to do that recently, though as it forces me to go to the
> command line rather than using Atlassian Sourcetree, I've been getting
> other things wrong. To those people who have been dealing with commits I've
> managed to mess up: apologies.
>
> 1. Once someone is down as an author you don't need to add their email
> address; the first time you will need to get their email address
> 2. Akira, Aaron and I also use the -S option to GPG sign the commits. We
> should all be doing that, as it is the way to show who really committed the
> patch. Add --show-signature to the end of any git log to command to see
> those.
> 3. note that if you cherry-pick a patch into a different branch, you have
> to use -S in the git cherry-pick command to resign it.
>
> we should all have our GPG keys in the KEYS file, and co-sign the others in
> there, so that we have that mutual trust.
>
> -Steve
>
> ps: one flaw in the GPG process: if you ever revoke the key then all
> existing commits are considered untrusted
>
> http://steveloughran.blogspot.com/2017/10/roca-breaks-my-commit-process.html
>
>
>
>
> On Thu, Feb 14, 2019 at 9:12 AM Akira Ajisaka  wrote:
>
> > Hi Vinay,
> >
> > I'm already doing this if I can get the original author name and the
> > email address in some way.
> > If the patch is created by git format-patch command, smart-apply-patch
> > --committer option can do this automatically.
> >
>
> Never knew that
>

Including Original Author in git commits.

[Vinayakumar B]

Hi all,

Right now we are committing with commitrer name as author and adding
contributor name in commit message. This way of committing will not reflect
in contributors git profile.

I believe current way of crediting.. is based on SVN days..

Since now all apache repos are following git.. we can use git
functionalities (already some are using PRs) and use original author name
in commits and use signed of message for committers.


Any thoughts?

-Vinay

Re: [VOTE] Release Apache Hadoop 3.1.2 - RC1

[Vinayakumar B]

Sorry for being late.

Thanks everyone for the Release efforts.

+1 (Binding)

-Built from tag on Ubuntu-18.04
-Successful Native build.
-Deployed a 3-Node docker cluster with kerberos Security Enabled on Centos7
images.
-Verified basic HDFS Commands
-Ran basic wordcount Job.

-Vinay

On Wed, Feb 6, 2019 at 2:38 AM Wangda Tan  wrote:

> Thanks everyone for voting, we have 6 binding votes and 6 non-binding
> votes, no vetos. The vote passed. We will go ahead to stage this release.
>
> - Wangda and Sunil.
>
> On Tue, Feb 5, 2019 at 12:05 PM Billie Rinaldi 
> wrote:
>
> > Thanks Sunil, the new source tarball matches the RC tag and its checksum
> > and signature look good.
> >
> > Billie
> >
> > On Tue, Feb 5, 2019 at 10:50 AM Sunil G  wrote:
> >
> > > Thanks Billie for pointing out.
> > > I have updated source by removing patchprocess and extra line create
> > > release.
> > >
> > > Also updated checksum as well.
> > >
> > > @bil...@apache.org   @Wangda Tan  >
> > > please help to verify this changed bit once.
> > >
> > > Thanks
> > > Sunil
> > >
> > > On Tue, Feb 5, 2019 at 5:23 AM Billie Rinaldi <
> billie.rina...@gmail.com>
> > > wrote:
> > >
> > >> Hey Sunil and Wangda, thanks for the RC. The source tarball has a
> > >> patchprocess directory with some yetus code in it. Also, the file
> > >> dev-support/bin/create-release file has the following line added:
> > >>   export GPG_AGENT_INFO="/home/sunilg/.gnupg/S.gpg-agent:$(pgrep
> > >> gpg-agent):1"
> > >>
> > >> I think we are probably due for an overall review of LICENSE and
> NOTICE.
> > >> I saw some idiosyncrasies there but nothing that looked like a
> blocker.
> > >>
> > >> On Mon, Jan 28, 2019 at 10:20 PM Sunil G  wrote:
> > >>
> > >>> Hi Folks,
> > >>>
> > >>> On behalf of Wangda, we have an RC1 for Apache Hadoop 3.1.2.
> > >>>
> > >>> The artifacts are available here:
> > >>> http://home.apache.org/~sunilg/hadoop-3.1.2-RC1/
> > >>>
> > >>> The RC tag in git is release-3.1.2-RC1:
> > >>> https://github.com/apache/hadoop/commits/release-3.1.2-RC1
> > >>>
> > >>> The maven artifacts are available via repository.apache.org at
> > >>>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1215
> > >>>
> > >>> This vote will run 5 days from now.
> > >>>
> > >>> 3.1.2 contains 325 [1] fixed JIRA issues since 3.1.1.
> > >>>
> > >>> We have done testing with a pseudo cluster and distributed shell job.
> > >>>
> > >>> My +1 to start.
> > >>>
> > >>> Best,
> > >>> Wangda Tan and Sunil Govindan
> > >>>
> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> > (3.1.2)
> > >>> ORDER BY priority DESC
> > >>>
> > >>
> >
>

Re: [DISCUSS] Move to gitbox

[Vinayakumar B]

+1

-Vinay

On Mon, 10 Dec 2018, 1:22 pm Elek, Marton 
> Thanks Akira,
>
> +1 (non-binding)
>
> I think it's better to do it now at a planned date.
>
> If I understood well the only bigger task here is to update all the
> jenkins jobs. (I am happy to help/contribute what I can do)
>
>
> Marton
>
> On 12/8/18 6:25 AM, Akira Ajisaka wrote:
> > Hi all,
> >
> > Apache Hadoop git repository is in git-wip-us server and it will be
> > decommissioned.
> > If there are no objection, I'll file a JIRA ticket with INFRA to
> > migrate to https://gitbox.apache.org/ and update documentation.
> >
> > According to ASF infra team, the timeframe is as follows:
> >
> >> - December 9th 2018 -> January 9th 2019: Voluntary (coordinated)
> relocation
> >> - January 9th -> February 6th: Mandated (coordinated) relocation
> >> - February 7th: All remaining repositories are mass migrated.
> >> This timeline may change to accommodate various scenarios.
> >
> > If we got consensus by January 9th, I can file a ticket with INFRA and
> > migrate it.
> > Even if we cannot got consensus, the repository will be migrated by
> > February 7th.
> >
> > Regards,
> > Akira
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (HDFS-13983) TestOfflineImageViewer crashes in windows

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-13983:


 Summary: TestOfflineImageViewer crashes in windows
 Key: HDFS-13983
 URL: https://issues.apache.org/jira/browse/HDFS-13983
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Vinayakumar B


TestOfflineImageViewer crashes in windows because, OfflineImageViewer 
REVERSEXML tries to delete the outputfile and re-create the same stream which 
is already created.

Also there are unclosed RAF for input files which blocks from files being 
deleted.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: HADOOP-14163 proposal for new hadoop.apache.org

[Vinayakumar B]

+1, New site looks great.

Just one nit in README
'--refresh' flag for hugo server no longer available.

-Vinay

On Mon, 3 Sep 2018, 10:21 am Shashikant Banerjee, 
wrote:

> +1
>
> Thanks
> Shashi
>
> On 9/3/18, 9:23 AM, "Mukul Kumar Singh"  wrote:
>
> +1, Thanks for working on this Marton.
>
> -Mukul
>
> On 03/09/18, 9:02 AM, "John Zhuge"  wrote:
>
> +1 Like the new site.
>
> On Sun, Sep 2, 2018 at 7:02 PM Weiwei Yang 
> wrote:
>
> > That's really nice, +1.
> >
> > --
> > Weiwei
> >
> > On Sat, Sep 1, 2018 at 4:36 AM Wangda Tan 
> wrote:
> >
> > > +1, thanks for working on this, Marton!
> > >
> > > Best,
> > > Wangda
> > >
> > > On Fri, Aug 31, 2018 at 11:24 AM Arpit Agarwal <
> aagar...@hortonworks.com
> > >
> > > wrote:
> > >
> > > > +1
> > > >
> > > > Thanks for initiating this Marton.
> > > >
> > > >
> > > > On 8/31/18, 1:07 AM, "Elek, Marton"  wrote:
> > > >
> > > > Bumping this thread at last time.
> > > >
> > > > I have the following proposal:
> > > >
> > > > 1. I will request a new git repository hadoop-site.git
> and import
> > the
> > > > new site to there (which has exactly the same content as
> the
> > existing
> > > > site).
> > > >
> > > > 2. I will ask infra to use the new repository as the
> source of
> > > > hadoop.apache.org
> > > >
> > > > 3. I will sync manually all of the changes in the next
> two months
> > > back
> > > > to the svn site from the git (release announcements, new
> > committers)
> > > >
> > > > IN CASE OF ANY PROBLEM we can switch back to the svn
> without any
> > > > problem.
> > > >
> > > > If no-one objects within three days, I'll assume lazy
> consensus and
> > > > start with this plan. Please comment if you have
> objections.
> > > >
> > > > Again: it allows immediate fallback at any time as svn
> repo will be
> > > > kept
> > > > as is (+ I will keep it up-to-date in the next 2 months)
> > > >
> > > > Thanks,
> > > > Marton
> > > >
> > > >
> > > > On 06/21/2018 09:00 PM, Elek, Marton wrote:
> > > > >
> > > > > Thank you very much to bump up this thread.
> > > > >
> > > > >
> > > > > About [2]: (Just for the clarification) the content of
> the
> > proposed
> > > > > website is exactly the same as the old one.
> > > > >
> > > > > About [1]. I believe that the "mvn site" is perfect
> for the
> > > > > documentation but for website creation there are more
> simple and
> > > > > powerful tools.
> > > > >
> > > > > Hugo has more simple compared to jekyll. Just one
> binary, without
> > > > > dependencies, works everywhere (mac, linux, windows)
> > > > >
> > > > > Hugo has much more powerful compared to "mvn site".
> Easier to
> > > > create/use
> > > > > more modern layout/theme, and easier to handle the
> content (for
> > > > example
> > > > > new release announcements could be generated as part
> of the
> > release
> > > > > process)
> > > > >
> > > > > I think it's very low risk to try out a new approach
> for the site
> > > > (and
> > > > > easy to rollback in case of problems)
> > > > >
> > > > > Marton
> > > > >
> > > > > ps: I just updated the patch/preview site with the
> recent
> > releases:
> > > > >
> > > > > ***
> > > > > * http://hadoop.anzix.net *
> > > > > ***
> > > > >
> > > > > On 06/21/2018 01:27 AM, Vinod Kumar Vavilapalli wrote:
> > > > >> Got pinged about this offline.
> > > > >>
> > > > >> Thanks for keeping at it, Marton!
> > > > >>
> > > > >> I think there are two road-blocks here
> > > > >>   (1) Is the mechanism using which the website is
> built good
> > > enough
> > > > -
> > > > >> mvn-site / hugo etc?
> > > > >>   (2) Is the new website good enough?
> > > > >>
> > > > >> For (1), I just think we need more committer
> attention and get
> > > > >> feedback rapidly and get it in.
> > > > >>
> > > > >> For (2), how about we do it in a different way in the
> interest
> > of
>   

[jira] [Created] (HDFS-13816) dfs.getQuotaUsage() throws NPE on non-existent dir instead of FileNotFoundException

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-13816:


 Summary: dfs.getQuotaUsage() throws NPE on non-existent dir 
instead of FileNotFoundException
 Key: HDFS-13816
 URL: https://issues.apache.org/jira/browse/HDFS-13816
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Reporter: Vinayakumar B


{{dfs.getQuotaUsage()}} on non-existent path should throw FileNotFoundException.

{noformat}
java.lang.NullPointerException
at 
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getQuotaUsageInt(FSDirStatAndListingOp.java:573)
at 
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getQuotaUsage(FSDirStatAndListingOp.java:554)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getQuotaUsage(FSNamesystem.java:3221)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getQuotaUsage(NameNodeRpcServer.java:1404)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getQuotaUsage(ClientNamenodeProtocolServerSideTranslatorPB.java:1861)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: 回复：[VOTE] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature branch to trunk

[Vinayakumar B]

+1

Great work guys.
-Vinay

On Tue, 7 Aug 2018, 7:20 pm Anu Engineer,  wrote:

> +1, Sorry for the late vote. Thanks for the perseverance and seeing this
> thru.
>
> --Anu
>
>
> On 8/7/18, 1:04 AM, "郑锴(铁杰)"  wrote:
>
> +1 for the work to be in. Thanks Uma and folks for the hard taking!
>
> When it's in, I'd suggest we use a more general name for the new
> daemon service. It'd be good to evolve and support more self-running admin
> functionalities incubated first there before doing it directly in NameNode.
>
> Regards,
> Kai
> --
> 发件人：Uma Maheswara Rao G 
> 发送时间：2018年8月1日(星期三) 14:38
> 收件人：hdfs-dev 
> 主 题：[VOTE] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature
> branch to trunk
>
> Hi All,
>
>
>
>  From the positive responses from JIRA discussion and no objections
> from
> below DISCUSS thread [1], I am converting it to voting thread.
>
>
>
>  Last couple of weeks we spent time on testing the feature and so far
> it is
> working fine. Surendra uploaded a test report at HDFS-10285:  [2]
>
>
>
>  In this phase, we provide to run SPS outside of Namenode only and as a
> next phase we continue to discuss and work on to enable it as Internal
> SPS
> as explained below. We have got clean QA report on branch and if there
> are
> any static tool comments triggered later while running this thread, we
> will
> make sure to fix them before merge. We committed and continue to
> improve
> the code on trunk. Please refer to HDFS-10285 for discussion details.
>
>
>
>  This has been a long effort and we're grateful for the support we've
> received from the community. In particular, thanks to Andrew Wang,
> Anoop
> Sam John, Anu Engineer, Chris Douglas, Daryn Sharp, Du Jingcheng , Ewan
> Higgs, Jing Zhao, Kai Zheng,  Rakesh R, Ramkrishna , Surendra Singh
> Lilhore
> , Thomas Demoor, Uma Maheswara Rao G, Vinayakumar, Virajith,  Wei Zhou,
> Yuanbo Liu. Without these members effort, this feature might not have
> reached to this state.
>
>
>
> To start with, here is my +1
>
> It will end on 6th Aug.
>
>
>
> Regards,
>
> Uma
>
> [1]  https://s.apache.org/bhyu
> [2]  https://s.apache.org/AXvL
>
>
> On Wed, Jun 27, 2018 at 3:21 PM, Uma Maheswara Rao G <
> hadoop@gmail.com>
> wrote:
>
> > Hi All,
> >
> >   After long discussions(offline and on JIRA) on SPS, we came to a
> > conclusion on JIRA(HDFS-10285) that, we will go ahead with External
> SPS
> > merge in first phase. In this phase process will not be running
> inside
> > Namenode.
> >   We will continue discussion on Internal SPS. Current code base
> supports
> > both internal and external option. We have review comments for
> Internal
> > which needs some additional works for analysis and testing etc. We
> will
> > move Internal SPS work to under HDFS-12226 (Follow-on work for SPS
> in NN)
> > We are working on cleanup task HDFS-13076 for the merge. .
> > For more clarity on Internal and External SPS proposal thoughts,
> please
> > refer to JIRA HDFS-10285.
> >
> > If there are no objections with this, I will go ahead for voting
> soon.
> >
> > Regards,
> > Uma
> >
> > On Fri, Nov 17, 2017 at 3:16 PM, Uma Maheswara Rao G <
> hadoop@gmail.com
> > > wrote:
> >
> >> Update: We worked on the review comments and additional JIRAs above
> >> mentioned.
> >>
> >> >1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we
> >> planned to take up the support for recursive API support.
> HDFS-12291<
> >> https://issues.apache.org/jira/browse/HDFS-12291>
> >>
> >> We provided the recursive API support now.
> >>
> >> >2. Xattr optimizations HDFS-12225 >> he.org/jira/browse/HDFS-12225>
> >> Improved this portion as well
> >>
> >> >3. Few other review comments already fixed and committed
> HDFS-12214<
> >> https://issues.apache.org/jira/browse/HDFS-12214>
> >> Fixed the comments.
> >>
> >> We are continuing to test the feature and working so far well. Also
> we
> >> uploaded a combined patch and got the good QA report.
> >>
> >> If there are no further objections, we would like to go for merge
> vote
> >> tomorrow. Please by default this feature will be disabled.
> >>
> >> Regards,
> >> Uma
> >>
> >> On Fri, Aug 18, 2017 at 11:27 PM, Gangumalla, Uma <
> >> uma.ganguma...@intel.com> wrote:
> >>
> >>> Hi Andrew,
> >>>
> >>> >Great to hear. It'd be nice to define which use cases are met by
> the
> >>> current version of SPS, and which will be handled after the merge.
> >>> After the discussions in JIRA, we planned to support recursive API
> as
> >>> well. The primary use cases we planned was for Hbase. Please check
> next

Re: [VOTE] Adopt HDSL as a new Hadoop subproject

[Vinayakumar B]

+1 for the subproject of HDSL

Instead of maintaining directly as part of main repo itself, may be we can
explore 'git submodules'

-Vinay


On 23 Mar 2018 11:47 am, "Takanobu Asanuma"  wrote:

+1 (non-binding).

Thanks,
-Takanobu Asanuma


> All,
>
> Following our discussions on the previous thread (Merging branch HDFS-7240
> to trunk), I'd like to propose the following:
>
> * HDSL become a subproject of Hadoop.
> * HDSL will release separately from Hadoop. Hadoop releases will not
> contain HDSL and vice versa.
> * HDSL will get its own jira instance so that the release tags stay
> separate.
> * On trunk (as opposed to release branches) HDSL will be a separate module
> in Hadoop's source tree. This will enable the HDSL to work on their trunk
> and the Hadoop trunk without making releases for every change.
> * Hadoop's trunk will only build HDSL if a non-default profile is enabled.
> * When Hadoop creates a release branch, the RM will delete the HDSL module
> from the branch.
> * HDSL will have their own Yetus checks and won't cause failures in the
> Hadoop patch check.
>
> I think this accomplishes most of the goals of encouraging HDSL
development
> while minimizing the potential for disruption of HDFS development.
>
> The vote will run the standard 7 days and requires a lazy 2/3 vote. PMC
> votes are binding, but everyone is encouraged to vote.
>
> +1 (binding)
>
> .. Owen

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Merging HDFS-8707 (C++ HDFS client) to trunk

[Vinayakumar B]

Definitely this would be great addition. Kudos to everyone's contributions.

I am not a C++ expert. So cannot vote on code.

  ---A libhdfs/libhdfs3 compatible C API that allows libhdfs++ to serve as
a drop-in replacement for clients that only need read support (until libhdfs++
also supports writes).

Wouldn't it be nice to have write support as well before merge...?
If everyone feels its okay to have read alone for now, I am okay anyway.

On 1 Mar 2018 11:35 pm, "Jim Clampffer"  wrote:

> Thanks for the feedback Chris and Kai!
>
> Chris, do you mean potentially landing this in its current state and
> handling some of the rough edges after?  I could see this working just
> because there's no impact on any existing code.
>
> With regards to your questions Kai:
> There isn't a good doc for the internal architecture yet; I just reassigned
> HDFS-9115 to myself to handle that.  Are there any specific areas you'd
> like to know about so I can prioritize those?
> Here's some header files that include a lot of comments that should help
> out for now:
> -hdfspp.h - main header for the C++ API
> -filesystem.h and filehandle.h - describes some rules about object
> lifetimes and threading from the API point of view (most classes have
> comments describing any restrictions on threading, locking, and lifecycle).
> -rpc_engine.h and rpc_connection.h begin getting into the async RPC
> implementation.
>
>
> 1) Yes, it's a reimplementation of the entire client in C++.  Using
> libhdfs3 as a reference helps a lot here but it's still a lot of work.
> 2) EC isn't supported now, though that'd be great to have, and I agree that
> it's going to be take a lot of effort to implement.  Right now if you tried
> to read an EC file I think you'd get some unhelpful error out of the block
> reader but I don't have an EC enabled cluster set up to test.  Adding an
> explicit not supported message would be straightforward.
> 3) libhdfs++ reuses all of the minidfscluster tests that libhdfs already
> had so we get consistency checks on the C API.  There's a few new tests
> that also get run on both libhdfs and libhdfs++ and make sure the expected
> output is the same too.
> 4) I agree, I just haven't had a chance to look into the distribution build
> to see how to do it.  HDFS-9465 is tracking this.
> 5) Not yet (HDFS-8765).
>
> Regards,
> James
>
>
>
>
> On Thu, Mar 1, 2018 at 4:28 AM, 郑锴(铁杰) 
> wrote:
>
> > The work sounds solid and great! + to have this.
> >
> > Is there any quick doc to take a glance at? Some quick questions to be
> > familiar with:
> > 1. Seems the client is all implemented in c++ without any Java codes (so
> > no JVM overhead), which means lots of work, rewriting HDFS client. Right?
> > 2.  Guess erasure coding feature isn't supported, as it'd involve
> > significant development, right? If yes, what will it say when read
> erasure
> > coded file?
> > 3. Is there any building/testing mechanism to enforce the consistency
> > between the c++ part and Java part?
> > 4. I thought the public header and lib should be exported when building
> > the distribution package, otherwise hard to use the new C api.
> > 5. Is the short-circuit read supported?
> >
> > Thanks.
> >
> >
> > Regards,
> > Kai
> >
> > --
> > 发件人：Chris Douglas 
> > 发送时间：2018年3月1日(星期四) 05:08
> > 收件人：Jim Clampffer 
> > 抄 送：Hdfs-dev 
> > 主 题：Re: [DISCUSS] Merging HDFS-8707 (C++ HDFS client) to trunk
> >
> > +1
> >
> > Let's get this done. We've had many false starts on a native HDFS
> > client. This is a good base to build on. -C
> >
> > On Wed, Feb 28, 2018 at 9:55 AM, Jim Clampffer
> >  wrote:
> > > Hi everyone,
> > >
> > > I'd like to start a thread to discuss merging the HDFS-
> > 8707 aka libhdfs++
> > > into trunk.  I sent originally sent a similar
> > email out last October but it
> > > sounds like it was buried by discussions about other feature merges
> that
> > > were going on at the time.
> > >
> > > libhdfs++ is an HDFS client written in C++ designed to be used in
> > > applications that are written in non-JVM based
> > languages.  In its current
> > > state it supports kerberos authenticated reads from HDFS
> > and has been used
> > > in production clusters for over a year so it has had a
> > significant amount
> > > of burn-in time.  The HDFS-8707 branch has been around for about 2
> years
> > > now so I'd like to know people's thoughts on what it would take to
> merge
> > > current branch and handling writes and encrypted reads in a new one.
> > >
> > > Current notable features:
> > >   -A libhdfs/libhdfs3 compatible C API that allows
> > libhdfs++ to serve as a
> > > drop-in replacement for clients that only need read support (until
> > > libhdfs++ also supports writes).
> > >   -An asynchronous C++ API with synchronous shims on top if the client
> > > application wants to do blocking operations.  Internally a single
> thread
> > > (optionally more) uses select/epoll by way of boost::a

Re: [VOTE] Release Apache Hadoop 3.0.1 (RC0)

[Vinayakumar B]

+1 Binding,

1. Checked out code from tag.
2. Built and deployed a 3-node docker cluster in secure mode (HDFS HA,
Non-HA YARN). Docker script available in [1], if anyone need,
3. Ran wordcount job successfully.
4. Verified basic webhdfs commands using curl.
5. Verified basic HDFS commands


[1] https://github.com/vinayakumarb/docker-cluster-deployer

Thanks,
-Vinay

On Wed, Feb 21, 2018 at 5:41 AM, Lei Xu  wrote:

> Dear PMC and committers
>
> Please help to verify and vote this release. I will extend the voting
> period to Friday (Feb 23th) of this week.
>
> Thanks!
>
> On Thu, Feb 15, 2018 at 3:36 PM, Lei Xu  wrote:
> > Hi, all
> >
> > I've created release candidate 0 for Apache Hadoop 3.0.1
> >
> > Apache Hadoop 3.0.1 will be the first bug fix release for Apache
> > Hadoop 3.0 release. It includes 49 bug fixes, which include 10
> > blockers and 8 are critical.
> >
> > Please note:
> > * HDFS-12990. Change default NameNode RPC port back to 8020. It makes
> > incompatible changes to Hadoop 3.0.0.  After 3.0.1 releases, Apache
> > Hadoop 3.0.0 will be deprecated due to this change.
> >
> > The release page is:
> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release
> >
> > New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC0/
> >
> > The git tag is release-3.0.1-RC0, and the latest commit is
> > 494d075055b52b0cc922bc25237e231bb3771c90
> >
> > The maven artifacts are available:
> > https://repository.apache.org/content/repositories/orgapachehadoop-1078/
> >
> > Please try the release and vote; the vote will run for the usual 5
> > days, ending on 2/20/2017 6pm PST time.
> >
> > Thanks!
> >
> > --
> > Lei (Eddy) Xu
>
>
>
> --
> Lei (Eddy) Xu
> Software Engineer, Cloudera
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: does it make sense to get remaining space by sum all the ones of the datanode

[Vinayakumar B]

in the latest code this is is not applicable. each storage is chosen and
space of that storage is checked.

On 29 Jan 2018 3:53 pm, "Xie Gang"  wrote:

> 2.4 and 2.6:
>
> public long getRemaining(StorageType t) {
>   long remaining = 0;
>   for(DatanodeStorageInfo s : getStorageInfos()) {
> if (s.getStorageType() == t) {
>   remaining += s.getRemaining();
> }
>   }
>   return remaining;
> }
>
>
> On Mon, Jan 29, 2018 at 6:12 PM, Vinayakumar B 
> wrote:
>
>> in which version of Hadoop you are seeing this?
>>
>> -Vinay
>>
>> On 29 Jan 2018 3:26 pm, "Xie Gang"  wrote:
>>
>> Hello,
>>
>> We recently hit a issue that almost all the disk of the datanode got full
>> even we configured the du .reserved.
>>
>> After tracking down the code, found that when we choose a target datanode
>> and check if it's good candidate for block allocation (isGoodTarget()), it
>> only checks if the total left space of all the volumes(the same type), not
>> each volume. This logic makes the reservation of each volume useless.
>> Is this a problem or do I have any misunderstanding?
>>
>> final long remaining = node.getRemaining(storage.getStorageType());
>> if (requiredSize > remaining - scheduledSize) {
>>   logNodeIsNotChosen(storage, "the node does not have enough "
>>   + storage.getStorageType() + " space"
>>   + " (required=" + requiredSize
>>   + ", scheduled=" + scheduledSize
>>   + ", remaining=" + remaining + ")");
>>   stats.incrOverScheduled();
>>   return false;
>> }
>>
>>
>>
>> --
>> Xie Gang
>>
>
>
>
> --
> Xie Gang
>

Re: does it make sense to get remaining space by sum all the ones of the datanode

[Vinayakumar B]

in which version of Hadoop you are seeing this?

-Vinay

On 29 Jan 2018 3:26 pm, "Xie Gang"  wrote:

Hello,

We recently hit a issue that almost all the disk of the datanode got full
even we configured the du .reserved.

After tracking down the code, found that when we choose a target datanode
and check if it's good candidate for block allocation (isGoodTarget()), it
only checks if the total left space of all the volumes(the same type), not
each volume. This logic makes the reservation of each volume useless.
Is this a problem or do I have any misunderstanding?

final long remaining = node.getRemaining(storage.getStorageType());
if (requiredSize > remaining - scheduledSize) {
  logNodeIsNotChosen(storage, "the node does not have enough "
  + storage.getStorageType() + " space"
  + " (required=" + requiredSize
  + ", scheduled=" + scheduledSize
  + ", remaining=" + remaining + ")");
  stats.incrOverScheduled();
  return false;
}



--
Xie Gang

[jira] [Created] (HDFS-13027) Handle NPE due to deleted blocks in race condition

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-13027:


 Summary: Handle NPE due to deleted blocks in race condition
 Key: HDFS-13027
 URL: https://issues.apache.org/jira/browse/HDFS-13027
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Reporter: Vinayakumar B


Since File deletions and Block removal from BlocksMap done in separate locks, 
there are possibilities of NPE due to calls of 
{{blockManager.getBlockCollection(block)}} returning null.

Handle all possibilities of NPEs due to this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13011) Support replacing multiple nodes during pipeline recovery and append

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-13011:


 Summary: Support replacing multiple nodes during pipeline recovery 
and append
 Key: HDFS-13011
 URL: https://issues.apache.org/jira/browse/HDFS-13011
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs-client
Reporter: Vinayakumar B


During pipeline recovery only one additional node will be asked and will be 
replaced with failed node.
But if initial pipeline size is less than replication, then extra nodes could 
be added during pipeline recovery to satisfy the replication during write 
itself.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: When permission is disabled, why setOwner() && setPermission() still check the permission?

[Vinayakumar B]

Hi Brahma,

Thanks for bringing this up.

According to history of git, there is no historical reasons found for me.
May be someone with branch-1 experience, can answer this ..?

-Vinay

On Tue, Aug 1, 2017 at 6:50 PM, Brahma Reddy Battula <
brahmareddy.batt...@huawei.com> wrote:

> Hi All
>
> why  the "dfs.permissions.enabled" flag was not considered for this
> setOwner() and setPermission() check..?
>
> and why extra super user required for setOwner()? is the check required
> even in case of permissions disabled?
>
> Any idea on this..?
>
>
>
> Thanks
> Brahma Reddy Battula
>
>

[jira] [Created] (HDFS-12212) Options.Rename.To_TRASH is considered even when Options.Rename.NONE is specified

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-12212:


 Summary: Options.Rename.To_TRASH is considered even when 
Options.Rename.NONE is specified
 Key: HDFS-12212
 URL: https://issues.apache.org/jira/browse/HDFS-12212
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Reporter: Vinayakumar B


HDFS-8312 introduced {{Options.Rename.TO_TRASH}} to differentiate the movement 
to trash and other renames for permission checks.

When Options.Rename.NONE is passed also TO_TRASH is considered for rename and 
wrong permissions are checked for rename.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12157) Do fsyncDirectory(..) outside of FSDataset lock

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-12157:


 Summary: Do fsyncDirectory(..) outside of FSDataset lock
 Key: HDFS-12157
 URL: https://issues.apache.org/jira/browse/HDFS-12157
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode
Reporter: Vinayakumar B
Priority: Critical






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12120) Use new block for pre-RollingUpgrade files` append requests

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-12120:


 Summary: Use new block for pre-RollingUpgrade files` append 
requests
 Key: HDFS-12120
 URL: https://issues.apache.org/jira/browse/HDFS-12120
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Vinayakumar B


After the RollingUpgrade prepare, append on pre-RU files will re-open the same 
last block and makes changes to it (appending extra data, changing genstamp 
etc).
These changes to the block will not be tracked in Datanodes (either in trash or 
via hardlinks)

This creates problem if RollingUpgrade.Rollback is called.
Since block state and size both changed, after rollback block will be marked 
corrupted.

To avoid this, first time append on pre-RU files can be forced to write to new 
block itself.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11898) DFSClient#isHedgedReadsEnabled() should be per client flag

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-11898:


 Summary: DFSClient#isHedgedReadsEnabled() should be per client 
flag 
 Key: HDFS-11898
 URL: https://issues.apache.org/jira/browse/HDFS-11898
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs-client
Reporter: Vinayakumar B
Assignee: Vinayakumar B


DFSClient#isHedgedReadsEnabled() returns value based on static 
{{HEDGED_READ_THREAD_POOL}}. 
Hence if any of the client initialized this in JVM, all remaining client reads 
will be going through hedged read itself.
This flag should be per client value.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: RollingUpgrade Rollback openfiles issue

[Vinayakumar B]

>>What do you think about truncate? Unlike append, it is utilizing the
block recovery /commitBlockSynchronization facility.  Then what about
regular block recoveries?

I did check about truncate. It saves a copy in trash before truncating..
after restore from trash original data is restored. So not a problem.

About regular block recoveries I didnt check in particular. I think it's
similar to first openfiles case I mentioned.

-Vinay


On 27 May 2017 01:29, "Kihwal Lee"  wrote:

>So I think, it’s better to revisit the RU solution considering these
usecases.
Yes, I agree. Any use case that mutates the block content or gen stamp
needs to be revisited.

I remember talking about the append case, but cannot find the relevant
code. We might have not actually done it. We could save a copy of the
existing block & meta in the data storage trash, up on append while in a RU.

What do you think about truncate? Unlike append, it is utilizing the block
recovery /commitBlockSynchronization facility.  Then what about regular
block recoveries?

Kihwal




________
From: Vinayakumar B 
To: Kihwal Lee ; "hdfs-dev@hadoop.apache.org" <
hdfs-dev@hadoop.apache.org>
Sent: Friday, May 26, 2017 5:05 AM
Subject: RE: RollingUpgrade Rollback openfiles issue



Thanks Kihwal,

Found another case, which looks more dangerous.

1. File written and closed. FINALIZED block in DN.
2. Rolling upgrade started.
3. File re-opened for append, and appended some bytes. BUT not closed. i.e.
BLOCK moved from FINALIZED to RBW in DN with updated genstamp.
4. RU Rollback Done.

After rollback, file will be in CLOSED state, where as block with updated
genstamp will be RWR state in DN. So this will be considered as CORRUPT in
Namenode.

After rollback, file will have only corrupted locations to read from.

So I think, it’s better to revisit the RU solution considering these
usecases.

Any thoughts?

-Vinay
From:Kihwal Lee [mailto:kih...@yahoo-inc.com]
Sent: 25 May 2017 22:16
To: Vinayakumar B ; hdfs-dev@hadoop.apache.org
Subject: Re: RollingUpgrade Rollback openfiles issue

Hi Vinay,

If I rephrase the question,

Does a RU rollback snapshot provide a consistent snapshot of the
distributed file system?

I don't think we aimed it to be a completely consistent snapshot. It is
meant to be a safe place to go back with the old version of software.  This
is normally used as a last resort. By design, the datanodes will have extra
blocks on rollback, which will be invalidated quickly.  But the short
presence of blocks with "future" block ids still can interfere with block
allocations after rolling back, if the cluster is used right away.  As you
pointed out, the under-construction block length is not recorded either.

>But now, extra bytes are seen after rollback. Is this correct?
I think it is a reasonable compromise.  If you can make a general argument
against it, we can revisit the design and try to fix it.

Kihwal





From:Vinayakumar B 
To: "hdfs-dev@hadoop.apache.org" 
Sent: Thursday, May 25, 2017 8:10 AM
Subject: RollingUpgrade Rollback openfiles issue


Hi all,

Have a doubt with expected behavior in case of RollingUpgrade and rollback.

Scenario:

1. file was being written before rolling upgrade started and written some
bytes, say X, with hsync().
2. Rolling upgrade done and writer continued to write and added some more
bytes and file closed with X+Y bytes.
3. Now rollback done.

i. Current state of the File is UNDERCONSTRUCTION.
Ii. getFileStatus() returns with size X. BUT in replicas there is a
FINALIZED replica with size X+Y.
iii. recoverLease() on the file closes the file with X+Y bytes.

Question:
  What should be the size here after rollback + recoverLease()?
  Since user always writes with hsync(), application might have some other
track of how much bytes written. But now, extra bytes are seen after
rollback. Is this correct?

-vinay

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

RE: RollingUpgrade Rollback openfiles issue

[Vinayakumar B]

Thanks Kihwal,

Found another case, which looks more dangerous.

1. File written and closed. FINALIZED block in DN.
2. Rolling upgrade started.
3. File re-opened for append, and appended some bytes. BUT not closed. i.e. 
BLOCK moved from FINALIZED to RBW in DN with updated genstamp.
4. RU Rollback Done.

After rollback, file will be in CLOSED state, where as block with updated 
genstamp will be RWR state in DN. So this will be considered as CORRUPT in 
Namenode.

After rollback, file will have only corrupted locations to read from.

So I think, it’s better to revisit the RU solution considering these usecases.

Any thoughts?

-Vinay
From: Kihwal Lee [mailto:kih...@yahoo-inc.com]
Sent: 25 May 2017 22:16
To: Vinayakumar B ; hdfs-dev@hadoop.apache.org
Subject: Re: RollingUpgrade Rollback openfiles issue

Hi Vinay,

If I rephrase the question,

Does a RU rollback snapshot provide a consistent snapshot of the distributed 
file system?

I don't think we aimed it to be a completely consistent snapshot. It is meant 
to be a safe place to go back with the old version of software.  This is 
normally used as a last resort. By design, the datanodes will have extra blocks 
on rollback, which will be invalidated quickly.  But the short presence of 
blocks with "future" block ids still can interfere with block allocations after 
rolling back, if the cluster is used right away.  As you pointed out, the 
under-construction block length is not recorded either.

>But now, extra bytes are seen after rollback. Is this correct?
I think it is a reasonable compromise.  If you can make a general argument 
against it, we can revisit the design and try to fix it.

Kihwal

____
From: Vinayakumar B 
mailto:vinayakumar...@huawei.com>>
To: "hdfs-dev@hadoop.apache.org<mailto:hdfs-dev@hadoop.apache.org>" 
mailto:hdfs-dev@hadoop.apache.org>>
Sent: Thursday, May 25, 2017 8:10 AM
Subject: RollingUpgrade Rollback openfiles issue


Hi all,

Have a doubt with expected behavior in case of RollingUpgrade and rollback.

Scenario:

1. file was being written before rolling upgrade started and written some 
bytes, say X, with hsync().
2. Rolling upgrade done and writer continued to write and added some more bytes 
and file closed with X+Y bytes.
3. Now rollback done.

i. Current state of the File is UNDERCONSTRUCTION.
Ii. getFileStatus() returns with size X. BUT in replicas there is a FINALIZED 
replica with size X+Y.
iii. recoverLease() on the file closes the file with X+Y bytes.

Question:
  What should be the size here after rollback + recoverLease()?
  Since user always writes with hsync(), application might have some other 
track of how much bytes written. But now, extra bytes are seen after rollback. 
Is this correct?

-vinay

[jira] [Created] (HDFS-11889) Datanodes should skip reading replicas from cache during Rollback

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-11889:


 Summary: Datanodes should skip reading replicas from cache during 
Rollback
 Key: HDFS-11889
 URL: https://issues.apache.org/jira/browse/HDFS-11889
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode, rolling upgrades
Reporter: Vinayakumar B






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

RollingUpgrade Rollback openfiles issue

[Vinayakumar B]

Hi all,

Have a doubt with expected behavior in case of RollingUpgrade and rollback.

Scenario:

1. file was being written before rolling upgrade started and written some 
bytes, say X, with hsync().
2. Rolling upgrade done and writer continued to write and added some more bytes 
and file closed with X+Y bytes.
3. Now rollback done.

i. Current state of the File is UNDERCONSTRUCTION.
Ii. getFileStatus() returns with size X. BUT in replicas there is a FINALIZED 
replica with size X+Y.
iii. recoverLease() on the file closes the file with X+Y bytes.

Question:
   What should be the size here after rollback + recoverLease()?
   Since user always writes with hsync(), application might have some other 
track of how much bytes written. But now, extra bytes are seen after rollback. 
Is this correct?

-vinay

Need Help with review

[Vinayakumar B]

Hi,

Please help in reviewing below Jiras

https://issues.apache.org/jira/browse/HDFS-5042
https://issues.apache.org/jira/browse/HDFS-11875
https://issues.apache.org/jira/browse/HDFS-11856
https://issues.apache.org/jira/browse/HDFS-11708
https://issues.apache.org/jira/browse/HDFS-11738
https://issues.apache.org/jira/browse/HDFS-11067
https://issues.apache.org/jira/browse/HDFS-10614


Thanks,

-Vinay

[jira] [Created] (HDFS-11875) Sort last block's locations for append

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-11875:


 Summary: Sort last block's locations for append
 Key: HDFS-11875
 URL: https://issues.apache.org/jira/browse/HDFS-11875
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Reporter: Vinayakumar B
Assignee: Vinayakumar B


Last block's locations are not sorted in shortest node first manner as done for 
new block allocation.
Sort the nodes before giving lastblock for append.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11856) Ability to re-add Upgrading Nodes (remote) to pipeline for future pipeline updates

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-11856:


 Summary: Ability to re-add Upgrading Nodes (remote) to pipeline 
for future pipeline updates
 Key: HDFS-11856
 URL: https://issues.apache.org/jira/browse/HDFS-11856
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs-client, rolling upgrades
Affects Versions: 2.7.3
Reporter: Vinayakumar B
Assignee: Vinayakumar B


During rolling upgrade if the DN gets restarted, then it will send special 
OOB_RESTART status to all streams opened for write.
1. Local clients will wait for 30 seconds to datanode to come back.
2. Remote clients will consider these nodes as bad nodes and continue with 
pipeline recoveries and write. These restarted nodes will be considered as bad, 
and will be excluded for lifetime of stream.

In case of small cluster, where total nodes itself is 3, each time a remote 
node restarts for upgrade, it will be excluded.
So a stream writing to 3 nodes initial, will end-up writing to only one node at 
the end, there are no other nodes to replace.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11738) hdged pread takes more time when block moved from initial locations

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-11738:


 Summary: hdged pread takes more time when block moved from initial 
locations
 Key: HDFS-11738
 URL: https://issues.apache.org/jira/browse/HDFS-11738
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs-client
Reporter: Vinayakumar B
Assignee: Vinayakumar B


Scenario : 
Same as HDFS-11708.

During Hedge read, 
1. First two locations fails to read the data in hedged mode.
2. chooseData refetches locations and adds a future to read from DN3.
3. after adding future to DN3, main thread goes for refetching locations in 
loop and stucks there till all 3  retries to fetch locations exhausted, which 
consumes ~20 seconds with exponential retry time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11710) hadoop-hdfs-native-client build fails in trunk after HDFS-11529

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-11710:


 Summary: hadoop-hdfs-native-client build fails in trunk after 
HDFS-11529
 Key: HDFS-11710
 URL: https://issues.apache.org/jira/browse/HDFS-11710
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: native
Affects Versions: 3.0.0-alpha3
Reporter: Vinayakumar B
Priority: Blocker


HDFS-11529 used 'hdfsThreadDestructor()' in jni_helper.c.
But this function is implemented in only "posix/thread_local_storage.c" NOT in 
"windows/thread_local_storage.c"

Fails with following errors
{noformat}
 [exec]   hdfs.dir\RelWithDebInfo\thread_local_storage.obj  /machine:x64 
/debug 
 [exec]  Creating library 
D:/hadoop/work/hadoop-hdfs-project/hadoop-hdfs-native-client/target/native/bin/RelWithDebInfo/hdfs.lib
 and object 
D:/hadoop/work/hadoop-hdfs-project/hadoop-hdfs-native-client/target/native/bin/RelWithDebInfo/hdfs.exp
 [exec] jni_helper.obj : error LNK2019: unresolved external symbol 
hdfsThreadDestructor referenced in function getJNIEnv 
[D:\hadoop\work\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\hdfs.vcxproj]
 [exec] 
D:\hadoop\work\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\bin\RelWithDebInfo\hdfs.dll
 : fatal error LNK1120: 1 unresolved externals 
[D:\hadoop\work\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\hdfs.vcxproj]
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11708) positional read will fail if replicas moved to different DNs after stream is opened

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-11708:


 Summary: positional read will fail if replicas moved to different 
DNs after stream is opened
 Key: HDFS-11708
 URL: https://issues.apache.org/jira/browse/HDFS-11708
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs-client
Affects Versions: 2.7.3
Reporter: Vinayakumar B
Assignee: Vinayakumar B
Priority: Critical



Scenario:
1. File was written to DN1, DN2 with RF=2
2. File stream opened to read and kept. Block Locations are [DN1,DN2]
3. One of the replica (DN2) moved to another datanode (DN3) due to datanode 
dead/balancing/etc.
4. Latest block locations in NameNode will be DN1 and DN3.
5. DN1 went down, but not yet detected as dead in NameNode.
6. Client start reading using positional read api "read(pos, buf[], offset, 
length)"



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11674) reserveSpaceForReplicas is not released if append request failed due to mirror down and replica recovered

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-11674:


 Summary: reserveSpaceForReplicas is not released if append request 
failed due to mirror down and replica recovered
 Key: HDFS-11674
 URL: https://issues.apache.org/jira/browse/HDFS-11674
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode
Reporter: Vinayakumar B
Assignee: Vinayakumar B


Scenario:
1. 3 Node cluster with 
"dfs.client.block.write.replace-datanode-on-failure.policy"  as DEFAULT
Block is written with x data.
2. One of the Datanode, NOT the first DN, is down
3. Client tries to append data to block and fails since one DN is down.
4. calls recoverLease() on the file.
5. Successfull recovery happens.

Issue:
1. DNs which were connected from client before encountering mirror down, will 
have the reservedSpaceForReplicas incremented, BUT never decremented. 
2. So in long run DN's all space will be in reservedSpaceForReplicas resulting 
OutOfSpace errors.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11621) Rename 'seq.io.sort.mb' and 'seq.io.sort.factor' with prefix 'io.seqfile'

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-11621:


 Summary: Rename 'seq.io.sort.mb' and 'seq.io.sort.factor' with 
prefix 'io.seqfile'
 Key: HDFS-11621
 URL: https://issues.apache.org/jira/browse/HDFS-11621
 Project: Hadoop HDFS
  Issue Type: Bug
        Reporter: Vinayakumar B
        Assignee: Vinayakumar B


HADOOP-6801 introduced new configs 'seq.io.sort.mb' and 'seq.io.sort.factor' .
These can be renamed to have prefix 'io.seqfile' to be consistent with other 
configs related to sequence file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11224) Lifeline message should be ignored for dead nodes

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-11224:


 Summary: Lifeline message should be ignored for dead nodes
 Key: HDFS-11224
 URL: https://issues.apache.org/jira/browse/HDFS-11224
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Vinayakumar B
Priority: Critical


Lifeline messages should be ignored for dead nodes in NameNode.
Otherwise, cluster level stats such as capacity, used, etc will be doubled, 
after re-registration of node.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11213) FilterFileSystem should override rename(.., options) to take effect of Rename options called via FilterFileSystem implementations

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-11213:


 Summary: FilterFileSystem should override rename(.., options) to 
take effect of Rename options called via FilterFileSystem implementations
 Key: HDFS-11213
 URL: https://issues.apache.org/jira/browse/HDFS-11213
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Vinayakumar B


HDFS-8312 Added Rename.TO_TRASH option to add a security check before moving to 
trash.

But for FilterFileSystem implementations since this rename(..options) is not 
overridden, it uses default FileSystem implementation where Rename.TO_TRASH 
option is not delegated to NameNode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11212) FilterFileSystem should override rename(.., options) to take effect of Rename options called via FilterFileSystem implementations

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-11212:


 Summary: FilterFileSystem should override rename(.., options) to 
take effect of Rename options called via FilterFileSystem implementations
 Key: HDFS-11212
 URL: https://issues.apache.org/jira/browse/HDFS-11212
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Vinayakumar B


HDFS-8312 Added Rename.TO_TRASH option to add a security check before moving to 
trash.

But for FilterFileSystem implementations since this rename(..options) is not 
overridden, it uses default FileSystem implementation where Rename.TO_TRASH 
option is not delegated to NameNode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11098) Datanode in tests cannot start in Windows after HDFS-10368

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-11098:


 Summary: Datanode in tests cannot start in Windows after HDFS-10368
 Key: HDFS-11098
 URL: https://issues.apache.org/jira/browse/HDFS-11098
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode
Affects Versions: 3.0.0-alpha2
Reporter: Vinayakumar B
Assignee: Vinayakumar B
Priority: Blocker


After HDFS-10368
Starting datanode's in MiniDFSCluster in windows throws below exception
{noformat}java.lang.IllegalArgumentException: URI: 
file:/D:/code/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1
 is not in the expected format
at 
org.apache.hadoop.hdfs.server.datanode.StorageLocation.(StorageLocation.java:68)
at 
org.apache.hadoop.hdfs.server.datanode.StorageLocation.parse(StorageLocation.java:123)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.getStorageLocations(DataNode.java:2561)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2545)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1613)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:860)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:491)
at 
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:450)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11067) DFS#listStatusIterator(..) should throw FileNotFoundException if the directory deleted before fetching next batch of entries

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-11067:


 Summary: DFS#listStatusIterator(..) should throw 
FileNotFoundException if the directory deleted before fetching next batch of 
entries
 Key: HDFS-11067
 URL: https://issues.apache.org/jira/browse/HDFS-11067
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs-client
Reporter: Vinayakumar B
Assignee: Vinayakumar B


DFS#listStatusIterator(..) currently stops iterating silently when the 
directory gets deleted before fetching the next batch of entries.

It should throw FileNotFoundException() and let user know that file is deleted 
in the middle.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10957) Retire BKJM from trunk

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-10957:


 Summary: Retire BKJM from trunk
 Key: HDFS-10957
 URL: https://issues.apache.org/jira/browse/HDFS-10957
 Project: Hadoop HDFS
  Issue Type: Task
Reporter: Vinayakumar B
Assignee: Vinayakumar B


Based on discussions done in mailing list 
[here|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201607.mbox/%3c1288296033.5942327.1469727453010.javamail.ya...@mail.yahoo.com%3E]
 BKJM is no longer required as QJM supports the required functionality with 
minimum deployment overhead.

This Jira is for tracking purpose.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10901) QJM should not consider stale/failed txn available in any one of JNs.

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-10901:


 Summary: QJM should not consider stale/failed txn available in any 
one of JNs.
 Key: HDFS-10901
 URL: https://issues.apache.org/jira/browse/HDFS-10901
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: qjm
Reporter: Vinayakumar B
Assignee: Vinayakumar B
Priority: Critical


In one of our cluster faced an issue, where NameNode restart failed due to a 
stale/failed txn available in one JN but not others. 

Scenario is:
1. Full cluster restart
2. startLogSegment Txn(195222) synced in Only one JN but failed to others, 
because they were shutting down. Only editlog file was created but txn was not 
synced in others, so after restart they were marked as empty.
3. Cluster restarted. During failover, this new logSegment missed the recovery 
because this JN was slow in responding to this call.
4. Other JNs recover was successfull, as there was no in-progress files.
5. editlog.openForWrite() detected that (195222) was already available, and 
failed the failover.

Same steps repeated until that stale editlog in JN was manually deleted.

Since QJM is a quorum of JNs, txn is considered successfull, if its written min 
quorum. Otherwise it will be failed.
So, same case should be applied while selecting streams for reading also.
Stale/failed txns available in only less JNs should not be considered for 
reading.

HDFS-10519, does similar work to consider 'durable' txns based on 
'committedTxnId'. But updating 'committedTxnId' for every flush with one more 
RPC seems tobe problematic to performance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10902) QJM should not consider stale/failed txn available in any one of JNs.

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-10902:


 Summary: QJM should not consider stale/failed txn available in any 
one of JNs.
 Key: HDFS-10902
 URL: https://issues.apache.org/jira/browse/HDFS-10902
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: qjm
Reporter: Vinayakumar B
Assignee: Vinayakumar B
Priority: Critical


In one of our cluster faced an issue, where NameNode restart failed due to a 
stale/failed txn available in one JN but not others. 

Scenario is:
1. Full cluster restart
2. startLogSegment Txn(195222) synced in Only one JN but failed to others, 
because they were shutting down. Only editlog file was created but txn was not 
synced in others, so after restart they were marked as empty.
3. Cluster restarted. During failover, this new logSegment missed the recovery 
because this JN was slow in responding to this call.
4. Other JNs recover was successfull, as there was no in-progress files.
5. editlog.openForWrite() detected that (195222) was already available, and 
failed the failover.

Same steps repeated until that stale editlog in JN was manually deleted.

Since QJM is a quorum of JNs, txn is considered successfull, if its written min 
quorum. Otherwise it will be failed.
So, same case should be applied while selecting streams for reading also.
Stale/failed txns available in only less JNs should not be considered for 
reading.

HDFS-10519, does similar work to consider 'durable' txns based on 
'committedTxnId'. But updating 'committedTxnId' for every flush with one more 
RPC seems tobe problematic to performance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[DISCUSS] Retire BKJM from trunk?

[Vinayakumar B]

Hi All,

   BKJM was Active and made much stable when the NameNode HA was implemented 
and there was no QJM implemented.
   Now QJM is present and is much stable which is adopted by many production 
environment.
   I wonder whether it would be a good time to retire BKJM from trunk?

   Are there any users of BKJM exists?

-Vinay

RE: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Vinayakumar B]

+1 (binding)

1. Downloaded and Built from branch-2.7.3
2. Started up HDFS and YARN in Single Node cluster.
3. Ran WordCount job multiple times and Success.
4. Verified the "Release Notes" available at the URL mentioned by Vinod.


Apart from that, 
Faced same issues as Andrew wang, while running the WordCount job first time in 
my new Ubuntu installation, without 'configuring the shuffle handler properly'. 
Whole session got logged by closing all other applications open. After 
configuring the shuffle handler properly, job was successful though.

-Vinay

-Original Message-
From: Andrew Wang [mailto:andrew.w...@cloudera.com] 
Sent: 26 July 2016 00:22
To: Karthik Kambatla 
Cc: larry mccay ; Vinod Kumar Vavilapalli 
; common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

I'll also add that, as a YARN newbie, I did hit two usability issues. These are 
very unlikely to be regressions, and I can file JIRAs if they seem fixable.

* I didn't have SSH to localhost set up (new laptop), and when I tried to run 
the Pi job, it'd exit my window manager session. I feel there must be a more 
developer-friendly solution here.
* If you start the NodeManager and not the RM, the NM has a handler for SIGTERM 
and SIGINT that blocked my Ctrl-C and kill attempts during startup.
I had to kill -9 it.

On Mon, Jul 25, 2016 at 11:44 AM, Andrew Wang 
wrote:

> I got asked this off-list, so as a reminder, only PMC votes are 
> binding on releases. Everyone is encouraged to vote on releases though!
>
> +1 (binding)
>
> * Downloaded source, built
> * Started up HDFS and YARN
> * Ran Pi job which as usual returned 4, and a little teragen
>
> On Mon, Jul 25, 2016 at 11:08 AM, Karthik Kambatla 
> 
> wrote:
>
>> +1 (binding)
>>
>> * Downloaded and build from source
>> * Checked LICENSE and NOTICE
>> * Pseudo-distributed cluster with FairScheduler
>> * Ran MR and HDFS tests
>> * Verified basic UI
>>
>> On Sun, Jul 24, 2016 at 1:07 PM, larry mccay  wrote:
>>
>> > +1 binding
>> >
>> > * downloaded and built from source
>> > * checked LICENSE and NOTICE files
>> > * verified signatures
>> > * ran standalone tests
>> > * installed pseudo-distributed instance on my mac
>> > * ran through HDFS and mapreduce tests
>> > * tested credential command
>> > * tested webhdfs access through Apache Knox
>> >
>> >
>> > On Fri, Jul 22, 2016 at 10:15 PM, Vinod Kumar Vavilapalli < 
>> > vino...@apache.org> wrote:
>> >
>> > > Hi all,
>> > >
>> > > I've created a release candidate RC0 for Apache Hadoop 2.7.3.
>> > >
>> > > As discussed before, this is the next maintenance release to 
>> > > follow up 2.7.2.
>> > >
>> > > The RC is available for validation at:
>> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ < 
>> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>> > >
>> > > The RC tag in git is: release-2.7.3-RC0
>> > >
>> > > The maven artifacts are available via repository.apache.org < 
>> > > http://repository.apache.org/> at
>> > >
>> https://repository.apache.org/content/repositories/orgapachehadoop-10
>> 40/
>> > <
>> > >
>> https://repository.apache.org/content/repositories/orgapachehadoop-10
>> 40/
>> > >
>> > >
>> > > The release-notes are inside the tar-balls at location 
>> > > hadoop-common-project/hadoop-common/src/main/docs/releasenotes.ht
>> > > ml. I hosted this at 
>> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.htm
>> > > l < 
>> > > http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.h
>> > > tml>
>> > for
>> > > your quick perusal.
>> > >
>> > > As you may have noted, a very long fix-cycle for the License & 
>> > > Notice issues (HADOOP-12893) caused 2.7.3 (along with every other 
>> > > Hadoop
>> > release)
>> > > to slip by quite a bit. This release's related discussion thread 
>> > > is
>> > linked
>> > > below: [1].
>> > >
>> > > Please try the release and vote; the vote will run for the usual 
>> > > 5
>> days.
>> > >
>> > > Thanks,
>> > > Vinod
>> > >
>> > > [1]: 2.7.3 release plan:
>> > >
>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.ht
>> ml
>> > <
>> > > http://markmail.org/thread/6yv2fyrs4jlepmmr>
>> >
>>
>
>

[jira] [Created] (HDFS-10614) Appended blocks can be closed even before IBRs from DataNodes

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-10614:


 Summary: Appended blocks can be closed even before IBRs from 
DataNodes
 Key: HDFS-10614
 URL: https://issues.apache.org/jira/browse/HDFS-10614
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Vinayakumar B


Scenario:
   1. Open the file for append()
   2. Trigger append pipeline setup by adding some data.
   3. Consider RECEIVING IBRs of DNs reaches NN first.
   4. updatePipeline() rpc sent to namenode to update the pipeline.
   5. Now, if complete() is called on the file even before closing the 
pipeline, then block will be COMPLETE, even before block is actually FINALIZED 
at DN side and file will be closed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10570) Netty-all jar should be first in class path while running tests in eclipse

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-10570:


 Summary: Netty-all jar should be first in class path while running 
tests in eclipse
 Key: HDFS-10570
 URL: https://issues.apache.org/jira/browse/HDFS-10570
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Reporter: Vinayakumar B
Assignee: Vinayakumar B
Priority: Minor


While debugging tests in eclipse, Cannot access DN http url. 
Also WebHdfs tests cannot run in eclipse due to classes loading from old 
version of netty jars instead of netty-all jar.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10384) MiniDFSCluster doesnt support multiple HTTPS server instances

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-10384:


 Summary: MiniDFSCluster doesnt support multiple HTTPS server 
instances
 Key: HDFS-10384
 URL: https://issues.apache.org/jira/browse/HDFS-10384
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Reporter: Vinayakumar B
Assignee: Vinayakumar B


MiniDFSCluster doesnot support Multiple instances of HTTPS servers. Because it 
will not use the ephemeral port for HTTPS addresses of NameNode and DataNode. 
So second instance will get PortAlreadyInUse exception.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10274) Move NameSystem#isInStartupSafeMode() to BlockManagerSafeMode

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-10274:


 Summary: Move NameSystem#isInStartupSafeMode() to 
BlockManagerSafeMode
 Key: HDFS-10274
 URL: https://issues.apache.org/jira/browse/HDFS-10274
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Vinayakumar B


To reduce the number of methods in Namesystem interface and for clean looking 
refactor, its better to move {{isInStartupSafeMode()}} to BlockManager and 
BlockManagerSafeMode, as most of the callers are in BlockManager. So one more 
interface overhead can be reduced.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-10273) Remove duplicate logSync() and log message in FSN#enterSafemode()

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-10273:


 Summary: Remove duplicate logSync() and log message in 
FSN#enterSafemode()
 Key: HDFS-10273
 URL: https://issues.apache.org/jira/browse/HDFS-10273
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Vinayakumar B
Priority: Minor


Remove duplicate logSync() and log message in FSN#enterSafemode()
{code:title=FSN#enterSafemode(..)}
  // Before Editlog is in OpenForWrite mode, editLogStream will be null. So,
  // logSyncAll call can be called only when Edlitlog is in OpenForWrite 
mode
  if (isEditlogOpenForWrite) {
getEditLog().logSyncAll();
  }
  setManualAndResourceLowSafeMode(!resourcesLow, resourcesLow);
  NameNode.stateChangeLog.info("STATE* Safe mode is ON.\n" +
  getSafeModeTip());
  if (isEditlogOpenForWrite) {
getEditLog().logSyncAll();
  }
  NameNode.stateChangeLog.info("STATE* Safe mode is ON" + getSafeModeTip());
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-10256) Use GenericTestUtils.getTestDir method in tests for temporary directories

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-10256:


 Summary: Use GenericTestUtils.getTestDir method in tests for 
temporary directories
 Key: HDFS-10256
 URL: https://issues.apache.org/jira/browse/HDFS-10256
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: build, test
Reporter: Vinayakumar B
Assignee: Vinayakumar B






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-10252) Is DataNode aware of the name of the file that it is going to store?

[Vinayakumar B (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HDFS-10252.
--
Resolution: Invalid

> Is DataNode aware of the name of the file that it is going to store?
> 
>
> Key: HDFS-10252
> URL: https://issues.apache.org/jira/browse/HDFS-10252
> Project: Hadoop HDFS
>  Issue Type: Test
>  Components: datanode, namenode
>Reporter: Dimitrios Sarigiannis
>Priority: Minor
>
> I am going through the HDFS Namenode and Datanode code and I am trying to see 
> if the DataNode is aware of the names of the files that are stored in it (and 
> other metadata as well).
> Assuming that we have the most simple case: 
> 1 NameNode
> 1 DataNode
> 1 single machine running HDFS with replication factor 1. 
> and considering the way HDFS works a use case could be: 
> A client requests to write a file from local to HDFS (for example: "hdfs dfs 
> -put file /file")
> He first communicates with NameNode and gets where this file should be stored.
> Then, after receiving an answer, he requests to the DataNode to store that 
> file.
> (At that point I am going to be a little more specific about the code)
> The DataNode has a DataXceiverServer class which runs and waits for requests. 
> When a request comes, it starts a DataXceiver thread and try to serve that 
> request. What I would like to know is, if at that specific point the DataNode 
> knows the name of the file that it is going to store. I spent hours of 
> debugging but I could not find it. Is it somewhere there, or only the 
> NameNode knows the name of that file?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-8782) Upgrade to block ID-based DN storage layout delays DN registration

[Vinayakumar B (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HDFS-8782.
-
Resolution: Duplicate

Resolved as dup of HDFS-8578.
Feel free to re-open if it doesnt serve the purpose.

> Upgrade to block ID-based DN storage layout delays DN registration
> --
>
> Key: HDFS-8782
> URL: https://issues.apache.org/jira/browse/HDFS-8782
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Haohui Mai
>Priority: Critical
>
> We have seen multiple incidents at production sites that there are long 
> delays for DNs to register to the NN when upgrading to post 2.6 release.
> Further investigation shows that the DN is blocked when upgrading the storage 
> layout introduced in HDFS-6482. The new storage layout requires making up to 
> 64k directories in the underlying file system. Unfortunately the current 
> implementation calls {{mkdirs()}} sequentially and upgrades each volume in 
> sequential order.
> As a result, upgrading a DN with a lot of disks or with blocks that have 
> random block ID takes a long time (usually in hours), and the DN won't 
> register to the NN unless it finishes upgrading all the storage directory. 
> The excessive delays confuse operations and break the assumption of rolling 
> upgrades.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

RE: Rise of the Dragon

[Vinayakumar B]

I think, he cloned main HDFS-7285 jira.
By default, "clone all subtasks" will be enabled. 
So in one shot all 206 subtasks also cloned, Creating huge mess in mails.

Anyway I have deleted the cloned Jira, along with subtasks.


-Vinay 

-Original Message-
From: Zhe Zhang [mailto:z...@apache.org] 
Sent: 17 March 2016 05:11
To: hdfs-dev@hadoop.apache.org
Subject: Re: Rise of the Dragon

Seems like someone tried to "clone" all erasure coding JIRAs.

On Wed, Mar 16, 2016 at 12:30 PM Colin P. McCabe  wrote:

> Who is creating all these dragon JIRAs?
>
> Colin
>

[jira] [Created] (HDFS-9952) Expose FSNamesystem lock wait time as metrics

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-9952:
---

 Summary: Expose FSNamesystem lock wait time as metrics
 Key: HDFS-9952
 URL: https://issues.apache.org/jira/browse/HDFS-9952
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Vinayakumar B
Assignee: Vinayakumar B


Expose FSNameSystem's readlock() and writeLock() wait time as metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-8578) On upgrade, Datanode should process all storage/data dirs in parallel

[Vinayakumar B (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HDFS-8578.
-
Resolution: Fixed

Fixed the compilation in branch-2.7

> On upgrade, Datanode should process all storage/data dirs in parallel
> -
>
> Key: HDFS-8578
> URL: https://issues.apache.org/jira/browse/HDFS-8578
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Reporter: Raju Bairishetti
>        Assignee: Vinayakumar B
>Priority: Critical
> Fix For: 2.7.3
>
> Attachments: HDFS-8578-01.patch, HDFS-8578-02.patch, 
> HDFS-8578-03.patch, HDFS-8578-04.patch, HDFS-8578-05.patch, 
> HDFS-8578-06.patch, HDFS-8578-07.patch, HDFS-8578-08.patch, 
> HDFS-8578-09.patch, HDFS-8578-10.patch, HDFS-8578-11.patch, 
> HDFS-8578-12.patch, HDFS-8578-13.patch, HDFS-8578-14.patch, 
> HDFS-8578-15.patch, HDFS-8578-16.patch, HDFS-8578-17.patch, 
> HDFS-8578-branch-2.6.0.patch, HDFS-8578-branch-2.7-001.patch, 
> HDFS-8578-branch-2.7-002.patch, HDFS-8578-branch-2.7-003.patch, 
> h8578_20151210.patch, h8578_20151211.patch, h8578_20151211b.patch, 
> h8578_20151212.patch, h8578_20151213.patch, h8578_20160117.patch, 
> h8578_20160128.patch, h8578_20160128b.patch, h8578_20160216.patch, 
> h8578_20160218-branch-2.7-addendum.patch, h8578_20160218.patch
>
>
> Right now, during upgrades datanode is processing all the storage dirs 
> sequentially. Assume it takes ~20 mins to process a single storage dir then  
> datanode which has ~10 disks will take around 3hours to come up.
> *BlockPoolSliceStorage.java*
> {code}
>for (int idx = 0; idx < getNumStorageDirs(); idx++) {
>   doTransition(datanode, getStorageDir(idx), nsInfo, startOpt);
>   assert getCTime() == nsInfo.getCTime() 
>   : "Data-node and name-node CTimes must be the same.";
> }
> {code}
> It would save lots of time during major upgrades if datanode process all 
> storagedirs/disks parallelly.
> Can we make datanode to process all storage dirs parallelly?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-9783) DataNode deadlocks after running hdfs mover

[Vinayakumar B (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HDFS-9783.
-
Resolution: Duplicate

Resolving as duplicate.
Feel free to re-open if fix in HDFS-9661 doesnt work for you.

> DataNode deadlocks after running hdfs mover
> ---
>
> Key: HDFS-9783
> URL: https://issues.apache.org/jira/browse/HDFS-9783
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 2.6.0
>Reporter: David Watzke
> Attachments: datanode.stack.txt
>
>
> We're running CDH 5.4.4 (with Hadoop 2.6.0). Recently we've added 800T of 
> ARCHIVE storage, marked some data (16T * 2 repl. factor) as "COLD" and ran 
> the hdfs mover to enforce the storage policy.
> After a few minutes, the mover hangs because one or more datanodes hang as 
> well. Please check out the deadlock revealed by jstack. Also, here's what 
> appeared in DN log:
> 2016-02-09 15:58:15,676 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Not able to copy block 1157230686 to /5.45.60.142:40144 because threads quota 
> is exceeded.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9776) Truncate block recovery will never happen if the Namenode goes down immediately after truncate

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-9776:
---

 Summary: Truncate block recovery will never happen if the Namenode 
goes down immediately after truncate
 Key: HDFS-9776
 URL: https://issues.apache.org/jira/browse/HDFS-9776
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Vinayakumar B


Initial analysys of Recent test failure in 
{{TestHAAppend#testMultipleAppendsDuringCatchupTailing}}
[here|https://builds.apache.org/job/PreCommit-HDFS-Build/14420/testReport/org.apache.hadoop.hdfs.server.namenode.ha/TestHAAppend/testMultipleAppendsDuringCatchupTailing/]
 
has found that, if the Active NameNode goes down immediately after truncate 
operation, but before BlockRecovery command sent to datanode,

Then this block will never be truncated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9536) OOM errors during parallel upgrade to Block-ID based layout

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-9536:
---

 Summary: OOM errors during parallel upgrade to Block-ID based 
layout
 Key: HDFS-9536
 URL: https://issues.apache.org/jira/browse/HDFS-9536
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Vinayakumar B


This is a follow-up jira for the OOM errors observed during parallel upgrade to 
Block-ID based datanode layout using HDFS-8578 fix.

more clue 
[here|https://issues.apache.org/jira/browse/HDFS-8578?focusedCommentId=15042012&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15042012]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-9127) Re-replication for files with enough replicas in single rack

[Vinayakumar B (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HDFS-9127.
-
Resolution: Invalid

This problem doesnt exist on latest code.
Feel free to re-open if found again.

> Re-replication for files with enough replicas in single rack
> 
>
> Key: HDFS-9127
> URL: https://issues.apache.org/jira/browse/HDFS-9127
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>
> Found while debugging testcases in HDFS-8647
>  *Scenario:* 
> ===
> Start a cluster with Single rack with three DN's
> write a file with RF=3
> adde two Nodes with different racks
> As per blockplacement policy ([Rack 
> Awareness|http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/RackAwareness.html])
>  atleast one replica needs to replicate to newly added rack.But it is not 
> happening..Because of following reason.
> {color:blue}
> when cluster was single rack,block will be removed from 
> {{neededReplications}} after 3 replicas.
> later, after adding new rack, only replications will happen which are present 
> in {{neededReplications}}
> So for the blocks which already have enough replicas, new rack replications 
> will not take place..
> {color}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

RE: [VOTE] Release Apache Hadoop 2.7.2 RC0

[Vinayakumar B]

I think one recent critical issue
HDFS-9413, getContentSummary() on standby should throw StandbyException,
Found by Brahma, is a good candidate for 2.7.2 as well.

-vinay
On Nov 14, 2015 5:20 AM, "Naganarasimha G R (Naga)" <
garlanaganarasi...@huawei.com> wrote:

> Hi Vinod,
>
> Can you also  look at YARN-2859, if its important then we might have to
> revert that one and fix it for 2.7.2.
>
> + Naga
> 
> From: Vinod Kumar Vavilapalli [vino...@apache.org]
> Sent: Friday, November 13, 2015 23:56
> To: Jason Lowe
> Cc: mapreduce-...@hadoop.apache.org; common-...@hadoop.apache.org;
> hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.7.2 RC0
>
> Thanks for reporting this Jason!
>
> Everyone, I am canceling this RC given the feedback, we will go again
> after addressing the open issues.
>
> Thanks
> +Vinod
>
> > On Nov 13, 2015, at 7:57 AM, Jason Lowe  wrote:
> >
> > -1 (binding)
> >
> > Ran into public localization issues and filed YARN-4354 <
> https://issues.apache.org/jira/browse/YARN-4354>. We need that resolved
> before the release is ready.  We will either need a timely fix or may have
> to revert YARN-2902 to unblock the release if my root-cause analysis is
> correct.  I'll dig into this more today.
> >
> > Jason
> >
> > From: Vinod Kumar Vavilapalli 
> > To: common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
> > Cc: vino...@apache.org
> > Sent: Wednesday, November 11, 2015 10:31 PM
> > Subject: [VOTE] Release Apache Hadoop 2.7.2 RC0
> >
> > Hi all,
> >
> >
> > I've created a release candidate RC0 for Apache Hadoop 2.7.2.
> >
> >
> > As discussed before, this is the next maintenance release to follow up
> > 2.7.1.
> >
> >
> > The RC is available for validation at:
> >
> > *http://people.apache.org/~vinodkv/hadoop-2.7.2-RC0/ <
> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC0/>
> >
> >  http://people.apache.org/~vinodkv/hadoop-2.7.2-RC0/>>*
> >
> >
> > The RC tag in git is: release-2.7.2-RC0
> >
> >
> > The maven artifacts are available via repository.apache.org at
> >
> > *
> https://repository.apache.org/content/repositories/orgapachehadoop-1023/ <
> https://repository.apache.org/content/repositories/orgapachehadoop-1023/>
> >
> > <
> https://repository.apache.org/content/repositories/orgapachehadoop-1023/ <
> https://repository.apache.org/content/repositories/orgapachehadoop-1023/
> >>*
> >
> >
> > As you may have noted, an unusually long 2.6.3 release caused 2.7.2 to
> slip
> > by quite a bit. This release's related discussion threads are linked
> below:
> > [1] and [2].
> >
> >
> > Please try the release and vote; the vote will run for the usual 5 days.
> >
> >
> > Thanks,
> >
> > Vinod
> >
> >
> > [1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes <
> http://markmail.org/message/oozq3gvd4nhzsaes>
> >
> > [2]: Planning Apache Hadoop 2.7.2
> > http://markmail.org/message/iktqss2qdeykgpqk <
> http://markmail.org/message/iktqss2qdeykgpqk>
> >
> >

[jira] [Created] (HDFS-9411) HDFS ZoneLabel support

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-9411:
---

 Summary: HDFS ZoneLabel support
 Key: HDFS-9411
 URL: https://issues.apache.org/jira/browse/HDFS-9411
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Vinayakumar B
Assignee: Vinayakumar B


HDFS currently stores data blocks on different datanodes chosen by 
BlockPlacement Policy. These datanodes are random within the 
scope(local-rack/different-rack/nodegroup) of network topology. 
In Multi-tenant (Tenant can be user/service) scenario, blocks of any tenant can 
be on any datanodes.
 Based on applications of different tenant, sometimes datanode might get busy 
making the other tenant's application to slow down. It would be better if 
admin's have a provision to logically divide the cluster among multi-tenants.

ZONE_LABELS can logically divide the cluster datanodes into multiple Zones.

High level design doc to follow soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

RE: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

[Vinayakumar B]

+1 for the idea.
On Nov 3, 2015 07:22, "Zheng, Kai"  wrote:

> Sounds good to me. When it's determined to include EC in 2.9 release, it
> may be good to have a rough release date as Zhe asked, so accordingly the
> scope of EC can be discussed out. We still have quite a few of things as
> Phase I follow-on tasks to do before EC can be deployed in a production
> system. Phase II to develop non-striping EC for cold data would possibly be
> started after that. We might consider to include only Phase I and leave
> Phase II for next release according to the rough release date.
>
> Regards,
> Kai
>
> -Original Message-
> From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com]
> Sent: Tuesday, November 03, 2015 5:41 AM
> To: hdfs-dev@hadoop.apache.org
> Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285
> (erasure coding) branch to trunk]
>
> +1 for EC to go into 2.9. Yes, 3.x would be long way to go when we plan
> +to
> have 2.8 and 2.9 releases.
>
> Regards,
> Uma
>
> On 11/2/15, 11:49 AM, "Vinod Vavilapalli"  wrote:
>
> >Forking the thread. Started looking at the 2.8 list, various features¹
> >status and arrived here.
> >
> >While I understand the pervasive nature of EC and a need for a
> >significant bake-in, moving this to a 3.x release is not a good idea.
> >We will surely get a 2.8 out this year and, as needed, I can even spend
> >time getting started on a 2.9. OTOH, 3.x is long ways off, and given
> >all the incompatibilities there, it would be a while before users can
> >get their hands on EC if it were to be only on 3.x. At best, this may
> >force sites that want EC to backport the entire EC feature to older
> >releases, at worst this will be repeat the mess of 0.20 security release
> forks.
> >
> >If we think adding this to 2.8 (even if it switched off) is too much
> >risk per our original plan, let¹s move this to 2.9, there by leaving
> >enough time for stability, integration testing and bake-in, and a
> >realistic chance of having it end up on users¹ clusters soonish.
> >
> >+Vinod
> >
> >> On Oct 19, 2015, at 1:44 PM, Andrew Wang 
> >>wrote:
> >>
> >> I think our plan thus far has been to target this for 3.0. I'm okay
> >>with  putting it in branch-2 if we've given a hard look at
> >>compatibility, but  I'll note though that 2.8 is already looking like
> >>quite a large release,  and our release bandwidth has been focused on
> >>the 2.6 and 2.7 maintenance  releases. Adding another multi-hundred
> >>JIRAs to 2.8 might make it too  unwieldy to get out the door. If we
> >>bump EC past that, 3.0 might very well  be our next release vehicle. I
> >>do plan to revive the 3.0 schedule some time  next year. With EC and
> >>JDK8 in a good spot, the only big feature remaining  is classpath
> >>isolation.
> >>
> >> EC is also a pretty fundamental change to HDFS. Even if it's
> >>compatible, in  terms of size and impact it might best belong in a new
> >>major release.
> >>
> >> Best,
> >> Andrew
> >>
> >> On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B <
> >> vinayakumarb.apa...@gmail.com> wrote:
> >>
> >>> Is anyone else also thinks that feature is ready to goto branch-2
> >>>as well?
> >>>
> >>> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since
> >>>then and  ready to go in branch-2.
> >>>
> >>> -Vinay
> >>> On Oct 6, 2015 12:51 AM, "Zhe Zhang"  wrote:
> >>>
> >>>> Thanks Vinay for capturing the issue and Uma for offering the help.
> >>>>
> >>>> ---
> >>>> Zhe Zhang
> >>>>
> >>>> On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma <
> >>> uma.ganguma...@intel.com
> >>>>>
> >>>> wrote:
> >>>>
> >>>>> Vinay,
> >>>>>
> >>>>>
> >>>>> I would merge them as part of HDFS-9182.
> >>>>>
> >>>>> Thanks,
> >>>>> Uma
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 10/5/15, 12:48 AM, "Vinayakumar B" 
> >>>>>wrote:
> >>>>>
> >>>>>> Hi Andrew,
> >>>>>> I see CHANGES.txt entries not yet merged from
> >>> CHANGES-HDFS-EC-7285.txt.
> >>>>>>
> >>>&

[jira] [Resolved] (HDFS-9324) DFSClient got deadlock when close file and failed to renew lease

[Vinayakumar B (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HDFS-9324.
-
Resolution: Duplicate

duplicate of HDFS-9294

> DFSClient got deadlock when close file and failed to renew lease
> 
>
> Key: HDFS-9324
> URL: https://issues.apache.org/jira/browse/HDFS-9324
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs-client
>    Reporter: Vinayakumar B
>        Assignee: Vinayakumar B
> Attachments: deadlock.log
>
>
> this issue is from one of the user mailing list query from daniedeng(邓飞)
> there is possibility of dead lock between lease renewer and DFSOutputstream 
> when file is getting closed and lease renewal also failed.
> stack trace will be attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)