Re: Thoughts about moving submarine to a separate git repo?

[Wangda Tan]

>
> One major point from my end is that the commits to new repo ideally should
> happen from committers or branch committers.


Definitely agree!

On Thu, Aug 22, 2019 at 12:32 PM Sunil Govindan  wrote:

> Thanks Wangda for sharing the thoughts.
>
> I agree with the idea of new repo for more cleaner code base and remove
> additional dependencies, jenkins etc.
> One major point from my end is that the commits to new repo ideally should
> happen from committers or branch committers.
>
> As hadoop community, we could help in this case. @Wangda Tan
>   your thoughts?
>
> Thanks
> Sunil
>
> On Wed, Aug 21, 2019 at 9:47 AM Wangda Tan  wrote:
>
>> Hi Xun,
>>
>> Thanks for starting this thread. I'm glad to see the existing momentum
>> made
>> by Submarine community, and I like the proposal to make it be a separate
>> Git repo.
>>
>> Some suggestions:
>>
>> 1) For options mentioned by Sunil: I think it's better to be a separate
>> Git
>> repo instead of a branch. To me, the branch is targeted for a
>> diverged codebase instead of a new codebase. Since Submarine needs a clean
>> root source code directory. I think moving to a new Git repo makes more
>> sense.
>>
>> 2) For the Submarine external code, when we pulling them in, I think we
>> need to make sure license, iCLA, code comply with Apache standard. Which
>> means we need to do some additional reviews, etc. for patches being pulled
>> in. (instead of as-is).
>>
>> 3) Can we address comments from @Wei-Chiu Chuang  ,
>> to
>> give some extra time for existing Hadoop committers/contributors who have
>> interests to review the code? Waiting for at least 1 day for a big patch
>> and 6 hours for a minor fix might be a good rule to follow. And @Wei-Chiu
>> Chuang  please let Submarine community know if you
>> have
>> anything interested to review so developers can ping you when they have
>> any
>> patches.
>>
>> Best,
>> Wangda
>>
>> On Tue, Aug 20, 2019 at 9:54 PM Wei-Chiu Chuang 
>> wrote:
>>
>> > >
>> > >
>> > >
>> > > Submarine dev community has a total of 8 developers and submits an
>> > average
>> > > of 4 to 5 PR per day.
>> > > But there are a limited number of Hadoop committer actively help
>> review
>> > and
>> > > merge patches, which causes development progress delays.
>> > >
>> > > I just want to point this out that this is concerning -- I wanted to
>> help
>> > review patches, but it wasn't obvious the patches were raised as PRs in
>> a
>> > non-apache git repo.
>> > Please be more inclusive.
>> >
>>
>

[jira] [Created] (HADOOP-16527) Add a whitelist of endpoints to allow simple authentication even if security is enabled

[Akira Ajisaka (Jira)]

Akira Ajisaka created HADOOP-16527:
--

 Summary: Add a whitelist of endpoints to allow simple 
authentication even if security is enabled
 Key: HADOOP-16527
 URL: https://issues.apache.org/jira/browse/HADOOP-16527
 Project: Hadoop Common
  Issue Type: New Feature
  Components: security
Reporter: Akira Ajisaka


HADOOP-15707 added /isActive servlet for load balancers and HADOOP-16398 added 
/prom servlet for prometheus server. However, prometheus server and most load 
balancers do not support kerberos authentication. Therefore if kerberos 
authentication is enabled, prometheus server or load balancers cannot access to 
the endpoints.

We would like to propose a whitelist of endpoints to avoid kerberos 
authentication.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Thoughts about moving submarine to a separate git repo?

[Sunil Govindan]

Thanks Wangda for sharing the thoughts.

I agree with the idea of new repo for more cleaner code base and remove
additional dependencies, jenkins etc.
One major point from my end is that the commits to new repo ideally should
happen from committers or branch committers.

As hadoop community, we could help in this case. @Wangda Tan
  your thoughts?

Thanks
Sunil

On Wed, Aug 21, 2019 at 9:47 AM Wangda Tan  wrote:

> Hi Xun,
>
> Thanks for starting this thread. I'm glad to see the existing momentum made
> by Submarine community, and I like the proposal to make it be a separate
> Git repo.
>
> Some suggestions:
>
> 1) For options mentioned by Sunil: I think it's better to be a separate Git
> repo instead of a branch. To me, the branch is targeted for a
> diverged codebase instead of a new codebase. Since Submarine needs a clean
> root source code directory. I think moving to a new Git repo makes more
> sense.
>
> 2) For the Submarine external code, when we pulling them in, I think we
> need to make sure license, iCLA, code comply with Apache standard. Which
> means we need to do some additional reviews, etc. for patches being pulled
> in. (instead of as-is).
>
> 3) Can we address comments from @Wei-Chiu Chuang  , to
> give some extra time for existing Hadoop committers/contributors who have
> interests to review the code? Waiting for at least 1 day for a big patch
> and 6 hours for a minor fix might be a good rule to follow. And @Wei-Chiu
> Chuang  please let Submarine community know if you
> have
> anything interested to review so developers can ping you when they have any
> patches.
>
> Best,
> Wangda
>
> On Tue, Aug 20, 2019 at 9:54 PM Wei-Chiu Chuang 
> wrote:
>
> > >
> > >
> > >
> > > Submarine dev community has a total of 8 developers and submits an
> > average
> > > of 4 to 5 PR per day.
> > > But there are a limited number of Hadoop committer actively help review
> > and
> > > merge patches, which causes development progress delays.
> > >
> > > I just want to point this out that this is concerning -- I wanted to
> help
> > review patches, but it wasn't obvious the patches were raised as PRs in a
> > non-apache git repo.
> > Please be more inclusive.
> >
>

[jira] [Resolved] (HADOOP-16061) Update Apache Yetus to 0.10.0

[Akira Ajisaka (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka resolved HADOOP-16061.

Fix Version/s: 3.1.3
   3.2.1
   3.3.0
   Resolution: Fixed

Committed to trunk, branch-3.2, and branch-3.1. Closing.

> Update Apache Yetus to 0.10.0
> -
>
> Key: HADOOP-16061
> URL: https://issues.apache.org/jira/browse/HADOOP-16061
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
>
> Yetus 0.10.0 is out. Let's upgrade.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16526) Support LDAP authenticaition (bind) via GSSAPI

[Todd Lipcon (Jira)]

Todd Lipcon created HADOOP-16526:


 Summary: Support LDAP authenticaition (bind) via GSSAPI
 Key: HADOOP-16526
 URL: https://issues.apache.org/jira/browse/HADOOP-16526
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Reporter: Todd Lipcon


Currently the LDAP group mapping provider only supports simple (user/password) 
authentication. In some cases it's more convenient to use GSSAPI (kerberos) 
authentication here, particularly when the server doing the mapping is already 
using a keytab provided by the same instance (eg IPA or AD). We should provide 
a configuration to turn on GSSAPI and put the right UGI 'doAs' calls in place 
to ensure an appropriate Subject in those calls.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Hadoop storage community online sync

[Aaron Fabbri]

Thank you Wei-Chiu for organizing this and sending out notes!

On Wed, Aug 21, 2019 at 1:10 PM Wei-Chiu Chuang  wrote:

> We had a great turnout today, thanks to Konstantin for leading the
> discussion of the NameNode Fine-Grained Locking proposal.
>
> There were at least 16 participants joined the call.
>
> Today's summary can be found here:
>
> https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit#
>
> 8/19/2019
>
> We are moving the sync to 10AM US PDT!
>
> NameNode Fine-Grained Locking via InMemory Namespace Partitioning
>
> Attendee:
>
> Konstantin, Chen, Weichiu, Xiaoyu, Anu, Matt, pljeliazkov, Chao Sun, Clay,
> Bharat Viswanadham, Matt, Craig Condit, Matthew Sharp, skumpf, Artem
> Ervits, Mohammad J Khan, Nanda, Alex Moundalexis.
>
> Konstantin lead the discussion of HDFS-14703
> .
>
> There are three important parts:
>
> (1) Partition namespace into multiple GSet, different part of namespace can
> be processed in parallel.
>
> (2) INode Key
>
> (3) Latch lock
>
> How to support snapshot —> should be able to get partitioned similarly.
>
> Balance partition strategies: several possible ways. Dynamic partition
> strategy, Static partitioning strategy —> no need a higher level navigation
> lock.
>
> Dynamic strategy: starting with 1, and grow.
>
> And: why does the design doc use static partitioning? determining the size
> of partitions is hard. what about starting with 1024 partitions.
>
> Hotspot problem
>
> A related task, HDFS-14617
>  (Improve fsimage load
> time by writing sub-sections to the fsimage index) writes multiple inode
> sections and inode directory sections, and load sections in parallel. It
> sounds like we can combine it with the fine-grained locking and partition
> inode/inode directory sections by the namespace partitions.
>
> Anu: snapshot complicates design. Renames. Copy on write?
>
> Anu: suggest to implement this feature without snapshot support to simplify
> design and implementation.
>
> Konstantin: will develop in a feature branch. Feel free to pick up jiras or
> share thoughts.
>
> FoldedTreeSet implemented in HDFS-9260
>  is relevant. Need to fix
> or revert before developing the namespace partitioning feature.
>
> On Mon, Aug 19, 2019 at 2:55 PM Wei-Chiu Chuang 
> wrote:
>
> > For this week,
> > We will have Konstantin and the LinkedIn folks to discuss a recent
> project
> > that's been baking for quite a while. This is an exciting project as it
> has
> > the potential to improve NameNode's throughput by 40%.
> >
> > HDFS-14703  NameNode
> > Fine-Grained Locking
> >
> > Access instruction, and the past sync notes are available here:
> >
> https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit?usp=sharing
> >
> > Reminder: We have Bi-weekly Hadoop storage online sync every other
> > Wednesday.
> > If there are no objections, I'd like to move the time to 10AM US pacific
> > time (GMT-8)
> >
>

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Masatake Iwasaki]

+1

Thanks,
Masatake Iwasaki

On 8/21/19 12:02, Wangda Tan wrote:

Hi all,

This is a vote thread to mark any versions smaller than 2.7 (inclusive),
and 3.0 EOL. This is based on discussions of [1]

This discussion runs for 7 days and will conclude on Aug 28 Wed.

Please feel free to share your thoughts.

Thanks,
Wangda

[1]
http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAD++eC=ou-tit1faob-dbecqe6ht7ede7t1dyra2p1yinpe...@mail.gmail.com%3e
,




-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16525) LDAP group mapping should include primary posix group

[Todd Lipcon (Jira)]

Todd Lipcon created HADOOP-16525:


 Summary: LDAP group mapping should include primary posix group
 Key: HADOOP-16525
 URL: https://issues.apache.org/jira/browse/HADOOP-16525
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Todd Lipcon
Assignee: Todd Lipcon


When configuring LdapGroupsMapping against FreeIPA, the current implementation 
searches for groups which have the user listed as a member. This catches all 
"secondary" groups but misses the user's primary group (typically the same name 
as their username). We should include a search for a group matching the user's 
primary gidNumber in the group search.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Hadoop storage community online sync

[Wei-Chiu Chuang]

We had a great turnout today, thanks to Konstantin for leading the
discussion of the NameNode Fine-Grained Locking proposal.

There were at least 16 participants joined the call.

Today's summary can be found here:
https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit#

8/19/2019

We are moving the sync to 10AM US PDT!

NameNode Fine-Grained Locking via InMemory Namespace Partitioning

Attendee:

Konstantin, Chen, Weichiu, Xiaoyu, Anu, Matt, pljeliazkov, Chao Sun, Clay,
Bharat Viswanadham, Matt, Craig Condit, Matthew Sharp, skumpf, Artem
Ervits, Mohammad J Khan, Nanda, Alex Moundalexis.

Konstantin lead the discussion of HDFS-14703
.

There are three important parts:

(1) Partition namespace into multiple GSet, different part of namespace can
be processed in parallel.

(2) INode Key

(3) Latch lock

How to support snapshot —> should be able to get partitioned similarly.

Balance partition strategies: several possible ways. Dynamic partition
strategy, Static partitioning strategy —> no need a higher level navigation
lock.

Dynamic strategy: starting with 1, and grow.

And: why does the design doc use static partitioning? determining the size
of partitions is hard. what about starting with 1024 partitions.

Hotspot problem

A related task, HDFS-14617
 (Improve fsimage load
time by writing sub-sections to the fsimage index) writes multiple inode
sections and inode directory sections, and load sections in parallel. It
sounds like we can combine it with the fine-grained locking and partition
inode/inode directory sections by the namespace partitions.

Anu: snapshot complicates design. Renames. Copy on write?

Anu: suggest to implement this feature without snapshot support to simplify
design and implementation.

Konstantin: will develop in a feature branch. Feel free to pick up jiras or
share thoughts.

FoldedTreeSet implemented in HDFS-9260
 is relevant. Need to fix
or revert before developing the namespace partitioning feature.

On Mon, Aug 19, 2019 at 2:55 PM Wei-Chiu Chuang 
wrote:

> For this week,
> We will have Konstantin and the LinkedIn folks to discuss a recent project
> that's been baking for quite a while. This is an exciting project as it has
> the potential to improve NameNode's throughput by 40%.
>
> HDFS-14703  NameNode
> Fine-Grained Locking
>
> Access instruction, and the past sync notes are available here:
> https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit?usp=sharing
>
> Reminder: We have Bi-weekly Hadoop storage online sync every other
> Wednesday.
> If there are no objections, I'd like to move the time to 10AM US pacific
> time (GMT-8)
>

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Subru Krishnan]

+1

On Wed, Aug 21, 2019 at 8:52 AM Kihwal Lee 
wrote:

> +1
>
> Kihwal
>
> On Tue, Aug 20, 2019 at 10:03 PM Wangda Tan  wrote:
>
> > Hi all,
> >
> > This is a vote thread to mark any versions smaller than 2.7 (inclusive),
> > and 3.0 EOL. This is based on discussions of [1]
> >
> > This discussion runs for 7 days and will conclude on Aug 28 Wed.
> >
> > Please feel free to share your thoughts.
> >
> > Thanks,
> > Wangda
> >
> > [1]
> >
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAD++eC=ou-tit1faob-dbecqe6ht7ede7t1dyra2p1yinpe...@mail.gmail.com%3e
> > ,
> >
>

[jira] [Created] (HADOOP-16524) Automatic keystore reloading for HttpServer2

[Kihwal Lee (Jira)]

Kihwal Lee created HADOOP-16524:
---

 Summary: Automatic keystore reloading for HttpServer2
 Key: HADOOP-16524
 URL: https://issues.apache.org/jira/browse/HADOOP-16524
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Kihwal Lee


Jetty 9 simplified reloading of keystore. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Haibo Chen]

+1

Haibo

From: Erik Krogen 
Sent: Wednesday, August 21, 2019 08:28
To: Jonathan Eagles ; Sunil Govindan 
Cc: Wangda Tan ; yarn-dev ; 
Hdfs-dev ; mapreduce-dev 
; Hadoop Common 
Subject: Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

+1 thanks for initiating an official vote Wangda!

From: Jonathan Eagles 
Sent: Wednesday, August 21, 2019 8:22 AM
To: Sunil Govindan 
Cc: Wangda Tan ; yarn-dev ; 
Hdfs-dev ; mapreduce-dev 
; Hadoop Common 
Subject: Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

+1.

On Wed, Aug 21, 2019 at 10:15 AM Sunil Govindan  wrote:

> +1
>
> - Sunil
>
> On Wed, Aug 21, 2019 at 8:33 AM Wangda Tan  wrote:
>
> > Hi all,
> >
> > This is a vote thread to mark any versions smaller than 2.7 (inclusive),
> > and 3.0 EOL. This is based on discussions of [1]
> >
> > This discussion runs for 7 days and will conclude on Aug 28 Wed.
> >
> > Please feel free to share your thoughts.
> >
> > Thanks,
> > Wangda
> >
> > [1]
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail-archives.apache.org%2Fmod_mbox%2Fhadoop-yarn-dev%2F201908.mbox%2F%253cCAD%2B%2BeC%3DOu-tit1FAoB-DbeCqe6hT7eDe7T1dYRa2p1yinpEC2A%40mail.gmail.com%253edata=02%7C01%7Chachen%40linkedin.com%7Cd38c56074f614dc4ced608d7264c49ee%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637019981409842905sdata=iEpnkNwZ36d9c6wTRLWu4z%2FqUgL5HHHzjOtsKG4EXdk%3Dreserved=0
> > ,
> >
>

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Kihwal Lee]

+1

Kihwal

On Tue, Aug 20, 2019 at 10:03 PM Wangda Tan  wrote:

> Hi all,
>
> This is a vote thread to mark any versions smaller than 2.7 (inclusive),
> and 3.0 EOL. This is based on discussions of [1]
>
> This discussion runs for 7 days and will conclude on Aug 28 Wed.
>
> Please feel free to share your thoughts.
>
> Thanks,
> Wangda
>
> [1]
>
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAD++eC=ou-tit1faob-dbecqe6ht7ede7t1dyra2p1yinpe...@mail.gmail.com%3e
> ,
>

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Erik Krogen]

+1 thanks for initiating an official vote Wangda!

From: Jonathan Eagles 
Sent: Wednesday, August 21, 2019 8:22 AM
To: Sunil Govindan 
Cc: Wangda Tan ; yarn-dev ; 
Hdfs-dev ; mapreduce-dev 
; Hadoop Common 
Subject: Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

+1.

On Wed, Aug 21, 2019 at 10:15 AM Sunil Govindan  wrote:

> +1
>
> - Sunil
>
> On Wed, Aug 21, 2019 at 8:33 AM Wangda Tan  wrote:
>
> > Hi all,
> >
> > This is a vote thread to mark any versions smaller than 2.7 (inclusive),
> > and 3.0 EOL. This is based on discussions of [1]
> >
> > This discussion runs for 7 days and will conclude on Aug 28 Wed.
> >
> > Please feel free to share your thoughts.
> >
> > Thanks,
> > Wangda
> >
> > [1]
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail-archives.apache.org%2Fmod_mbox%2Fhadoop-yarn-dev%2F201908.mbox%2F%253cCAD%2B%2BeC%3DOu-tit1FAoB-DbeCqe6hT7eDe7T1dYRa2p1yinpEC2A%40mail.gmail.com%253edata=02%7C01%7Cekrogen%40linkedin.com%7Cf7cda6a57de04eb1e6c008d7264b6d17%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637019977701030286sdata=I8EABq7OZfDXTQ8hSQ0MaSpBXnT4B4MXWqeFMCK7swM%3Dreserved=0
> > ,
> >
>

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Jonathan Eagles]

+1.

On Wed, Aug 21, 2019 at 10:15 AM Sunil Govindan  wrote:

> +1
>
> - Sunil
>
> On Wed, Aug 21, 2019 at 8:33 AM Wangda Tan  wrote:
>
> > Hi all,
> >
> > This is a vote thread to mark any versions smaller than 2.7 (inclusive),
> > and 3.0 EOL. This is based on discussions of [1]
> >
> > This discussion runs for 7 days and will conclude on Aug 28 Wed.
> >
> > Please feel free to share your thoughts.
> >
> > Thanks,
> > Wangda
> >
> > [1]
> >
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAD++eC=ou-tit1faob-dbecqe6ht7ede7t1dyra2p1yinpe...@mail.gmail.com%3e
> > ,
> >
>

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Sunil Govindan]

+1

- Sunil

On Wed, Aug 21, 2019 at 8:33 AM Wangda Tan  wrote:

> Hi all,
>
> This is a vote thread to mark any versions smaller than 2.7 (inclusive),
> and 3.0 EOL. This is based on discussions of [1]
>
> This discussion runs for 7 days and will conclude on Aug 28 Wed.
>
> Please feel free to share your thoughts.
>
> Thanks,
> Wangda
>
> [1]
>
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAD++eC=ou-tit1faob-dbecqe6ht7ede7t1dyra2p1yinpe...@mail.gmail.com%3e
> ,
>

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Eric Badger]

+1

On Wed, Aug 21, 2019 at 9:40 AM Sean Busbey 
wrote:

> +1
>
> On Tue, Aug 20, 2019 at 10:03 PM Wangda Tan  wrote:
> >
> > Hi all,
> >
> > This is a vote thread to mark any versions smaller than 2.7 (inclusive),
> > and 3.0 EOL. This is based on discussions of [1]
> >
> > This discussion runs for 7 days and will conclude on Aug 28 Wed.
> >
> > Please feel free to share your thoughts.
> >
> > Thanks,
> > Wangda
> >
> > [1]
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAD++eC=ou-tit1faob-dbecqe6ht7ede7t1dyra2p1yinpe...@mail.gmail.com%3e
> > ,
>
>
>
> --
> busbey
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Sean Busbey]

+1

On Tue, Aug 20, 2019 at 10:03 PM Wangda Tan  wrote:
>
> Hi all,
>
> This is a vote thread to mark any versions smaller than 2.7 (inclusive),
> and 3.0 EOL. This is based on discussions of [1]
>
> This discussion runs for 7 days and will conclude on Aug 28 Wed.
>
> Please feel free to share your thoughts.
>
> Thanks,
> Wangda
>
> [1]
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAD++eC=ou-tit1faob-dbecqe6ht7ede7t1dyra2p1yinpe...@mail.gmail.com%3e
> ,



-- 
busbey

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Hadoop Community Sync Up Schedule

[epa...@apache.org]

Let's go with bi-weekly (every 2 weeks). Sometimes this gives us 3 sync-ups in 
one month, which I think is fine.
-Eric Payne

On Wednesday, August 21, 2019, 5:01:52 AM CDT, Wangda Tan  
wrote: 
> 
> For folks in other US time zones: how about 11am PDT, is it better or 10am
> PDT will be better? I will be fine with both.
> 
> Hi Matt,
> 
> Thanks for mentioning this issue, this is the exactly issue I saw 藍.
> 
> Basically there’re two options:
> 
> - a. weekly, bi-weekly (for odd/even week) and every four months.
> - b. weekly, 1st/3rd week or 2nd/4th week, x-th week monthly.
> 
> I’m not sure which one is easier for people to understand as the issue you
> mentioned.
> 
> After thinking about it. I prefer a. since it is more consistent for
> audience and not disrupted because of calendar.
> 
> If we choose a. I will redo the proposal and make it aligns with a.
> 
> Thoughts?
> 
> Thanks,
> Wangda


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/420/

[Aug 20, 2019 5:38:52 PM] (weichiu) HDFS-14311. Multi-threading conflict at 
layoutVersion when loading block




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml
 
   hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml 
   hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml
 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client
 
   Boxed value is unboxed and then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:[line 335] 

Failed junit tests :

   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency 
   hadoop.hdfs.qjournal.client.TestQuorumJournalManager 
   hadoop.mapred.gridmix.TestLoadJob 
   hadoop.mapred.gridmix.TestGridmixSubmission 
   hadoop.yarn.client.cli.TestRMAdminCLI 
   hadoop.registry.secure.TestSecureLogins 
   hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor 
   hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/420/artifact/out/diff-compile-cc-root-jdk1.7.0_95.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/420/artifact/out/diff-compile-javac-root-jdk1.7.0_95.txt
  [328K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/420/artifact/out/diff-compile-cc-root-jdk1.8.0_222.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/420/artifact/out/diff-compile-javac-root-jdk1.8.0_222.txt
  [308K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/420/artifact/out/diff-checkstyle-root.txt
  [16M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/420/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/420/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/420/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/420/artifact/out/diff-patch-shellcheck.txt
  [72K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/420/artifact/out/diff-patch-shelldocs.txt
  [8.0K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/420/artifact/out/whitespace-eol.txt
  [12M]
   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/420/artifact/out/whitespace-tabs.txt
  [1.2M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/420/artifact/out/xml.txt
  [12K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/420/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/420/artifact/out/diff-javadoc-javadoc-root-jdk1.7.0_95.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/420/artifact/out/diff-javadoc-javadoc-root-jdk1.8.0_222.txt
  [1.1M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/420/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [320K]
   

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Dinesh Chitlangia]

+1

-Dinesh




On Wed, Aug 21, 2019 at 4:22 AM Akira Ajisaka  wrote:

> +1
>
> -Akira
>
> On Wed, Aug 21, 2019 at 12:10 PM Jonathan Hung 
> wrote:
> >
> > +1. Thanks!
> >
> > Jonathan Hung
> >
> >
> > On Tue, Aug 20, 2019 at 8:03 PM Wangda Tan  wrote:
> >
> > > Hi all,
> > >
> > > This is a vote thread to mark any versions smaller than 2.7
> (inclusive),
> > > and 3.0 EOL. This is based on discussions of [1]
> > >
> > > This discussion runs for 7 days and will conclude on Aug 28 Wed.
> > >
> > > Please feel free to share your thoughts.
> > >
> > > Thanks,
> > > Wangda
> > >
> > > [1]
> > >
> > >
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAD++eC=ou-tit1faob-dbecqe6ht7ede7t1dyra2p1yinpe...@mail.gmail.com%3e
> > > ,
> > >
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
>

Re: Hadoop Community Sync Up Schedule

[Wangda Tan]

For folks in other US time zones: how about 11am PDT, is it better or 10am
PDT will be better? I will be fine with both.

Hi Matt,

Thanks for mentioning this issue, this is the exactly issue I saw 藍.

Basically there’re two options:

- a. weekly, bi-weekly (for odd/even week) and every four months.
- b. weekly, 1st/3rd week or 2nd/4th week, x-th week monthly.

I’m not sure which one is easier for people to understand as the issue you
mentioned.

After thinking about it. I prefer a. since it is more consistent for
audience and not disrupted because of calendar.

If we choose a. I will redo the proposal and make it aligns with a.

Thoughts?

Thanks,
Wangda



On Wed, Aug 21, 2019 at 4:11 AM Matt Foley  wrote:

> Hi Wangda, thanks for this. A question about the schedule correction:
>
> > > 1) In the proposal, repeats are not properly. (I used bi-weekly
> instead of
> > 2nd/4th week as repeat frequency). I'd like to fix the frequency on Thu
> and
> > it will take effect starting next week.
>
> I understand that “bi-weekly” may cause 3 meetings in some months, and
> that’s not the intent.
> Is it your intent to schedule “Wednesday of the 2nd and 4th weeks of each
> month” or “2nd and 4th Wednesday of each month”? — which of course are not
> the same, for months that start on Thu-Sat...
>
> As a side note, I find that my calendar program cannot express “Wednesday
> of the second week of the month”, but does know how to do “second Wednesday
> of the month”.
>
> But I’ll schedule them one-by-one if I have to, I just wanted clarity. :-)
>
> Thanks,
> —Matt
>
>
> On Aug 19, 2019, at 8:31 PM, Wangda Tan  wrote:
>
> Hi folks,
>
> We have run community sync up for 1.5 months. I spoke to folks offline and
> got some feedback. Here's a summary of what I've observed from sync ups and
> talked to organizers.
>
> Following sync ups have very good participants (sometimes 10+ folks
> joined):
> - YARN/MR monthly sync up in APAC (Mandarin)
> - HDFS monthly sync up in APAC (Mandarin).
> - Submarine weekly sync up in APAC (Mandarin).
>
> Following sync up have OK-ish participants: (3-5 folks joined).
> - Storage monthly sync up in APAC (English)
> - Storage bi-weekly sync up in US (English)
> - YARN bi-weekly sync up in US (English).
>
> Following sync ups don't have good participants: (Skipped a couple of
> times).
> - YARN monthly sync up in APAC (English).
> - Submarine bi-weekly sync up in US (English).
>
> *So I'd like to propose the following changes and fixes of the schedule: *
> 1) Cancel the YARN/MR monthly sync up in APAC (English). Folks from APAC
> who speak English can choose to join the US session.
> 2) Cancel the Submarine bi-weekly sync up in US (English). Now Submarine
> developers and users are fast-growing in Mandarin-speaking areas. We can
> resume the sync if we do see demands from English-speaking areas.
> 3) Update the US sync up time from 9AM to 10AM PDT. 9AM is too early for
> most of the west-cost folks.
>
> *Following are fixes for the schedule:  *
> 1) In the proposal, repeats are not properly. (I used bi-weekly instead of
> 2nd/4th week as repeat frequency). I'd like to fix the frequency on Thu and
> it will take effect starting next week.
>
> Overall, thanks for everybody who participated in the sync ups. I do see
> community contributions grow in the last one month!
>
> Any thoughts about the proposal?
>
> Thanks,
> Wangda
>
>
>
>
> On Thu, Jul 25, 2019 at 11:53 AM 俊平堵  wrote:
>
> > Hi Folks,
> >
> > Kindly remind that we have YARN+MR APAC sync today, and you are
> > welcome to join:
> >
> >
> > Time and Date：07/25 1:00 pm (CST Time)
> >
> > Zoom link：Zoom | https://cloudera.zoom.us/j/880548968
> >
> > Summary：
> >
> >
> https://docs.google.com/document/d/1GY55sXrekVd-aDyRY7uzaX0hMDPyh3T-AL1kUY2TI5M
> >
> >
> > Thanks,
> >
> >
> > Junping
> >
> >
> >
> > Wangda Tan  于2019年6月28日周五 上午2:57写道：
> >
> >> Hi folks,
> >>
> >> Here's the Hadoop Community Sync Up proposal/schedule:
> >>
> >
> https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#heading=h.xh4zfwj8ppmn
> >>
> >> And here's calendar file:
> >>
> >>
> >>
> >
> https://calendar.google.com/calendar/ical/hadoop.community.sync.up%40gmail.com/public/basic.ics
> >>
> >> We gave it a try this week for YARN+MR and Submarine sync, feedbacks
> from
> >> participants seems pretty good, lots of new information shared during
> > sync
> >> up, and companies are using/developing Hadoop can better know each
> other.
> >>
> >> Next week there're 4 community sync-ups (Two Submarine for different
> >> timezones, one YARN+MR, one storage), please join to whichever you're
> >> interested:
> >>
> >> [image: image.png]
> >>
> >> Zoom info and notes can be found in the Google calendar invitation.
> >>
> >> Thanks,
> >> Wangda
> >>
> >
>
> --
Sent from Gmail Mobile

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Akira Ajisaka]

+1

-Akira

On Wed, Aug 21, 2019 at 12:10 PM Jonathan Hung  wrote:
>
> +1. Thanks!
>
> Jonathan Hung
>
>
> On Tue, Aug 20, 2019 at 8:03 PM Wangda Tan  wrote:
>
> > Hi all,
> >
> > This is a vote thread to mark any versions smaller than 2.7 (inclusive),
> > and 3.0 EOL. This is based on discussions of [1]
> >
> > This discussion runs for 7 days and will conclude on Aug 28 Wed.
> >
> > Please feel free to share your thoughts.
> >
> > Thanks,
> > Wangda
> >
> > [1]
> >
> > http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAD++eC=ou-tit1faob-dbecqe6ht7ede7t1dyra2p1yinpe...@mail.gmail.com%3e
> > ,
> >

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

RE: [DISCUSS] Hadoop-3.2.1 release proposal

[Bibinchundatt]

Hi Rohith 

Thank you for initiating this

Few critical/blocker jira's we could consider

YARN-9714
YARN-9642
YARN-9640

Regards
Bibin
-Original Message-
From: Rohith Sharma K S [mailto:rohithsharm...@apache.org] 
Sent: 21 August 2019 11:42
To: Wei-Chiu Chuang 
Cc: Hdfs-dev ; yarn-dev 
; mapreduce-dev ; 
Hadoop Common 
Subject: Re: [DISCUSS] Hadoop-3.2.1 release proposal

On Tue, 20 Aug 2019 at 22:28, Wei-Chiu Chuang  wrote:

> Hi Rohith,
> Thanks for initiating this.
> I want to bring up one blocker issue: HDFS-13596 
>  (NN restart fails 
> after RollingUpgrade from 2.x to 3.x)
>

> This should be a blocker for all active Hadoop 3.x releases: 3.3.0, 
> 3.2.1, 3.1.3. Hopefully we can get this fixed within this week.
> Additionally, HDFS-14396
>  (Failed to load 
> image from FSImageFile when downgrade from 3.x to 2.x).Probably not a 
> blocker but nice to have.
>

 Please set target version so that I don't miss in blockers/critical
list for 3.2.1 https://s.apache.org/7yjh5.


>
> On Tue, Aug 20, 2019 at 3:22 AM Rohith Sharma K S < 
> rohithsharm...@apache.org> wrote:
>
>> Hello folks,
>>
>> It's been more than six month Hadoop-3.2.0 is released i.e 16th Jan,2019.
>> We have several important fixes landed in branch-3.2 (around 48 
>> blockers/critical https://s.apache.org/ozd6o).
>>
>> I am planning to do a maintenance release of 3.2.1 in next few weeks 
>> i.e around 1st week of September.
>>
>> So far I don't see any blockers/critical in 3.2.1. I see few pending 
>> issues on 3.2.1 are https://s.apache.org/ni6v7.
>>
>> *Proposal*:
>> Code Freezing Date:  30th August 2019 Release Date : 7th Sept 2019
>>
>> Please let me know if you have any thoughts or comments on this plan.
>>
>> Thanks & Regards
>> Rohith Sharma K S
>>
>

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] Hadoop-3.2.1 release proposal

[Rohith Sharma K S]

On Tue, 20 Aug 2019 at 22:28, Wei-Chiu Chuang  wrote:

> Hi Rohith,
> Thanks for initiating this.
> I want to bring up one blocker issue: HDFS-13596
>  (NN restart fails
> after RollingUpgrade from 2.x to 3.x)
>

> This should be a blocker for all active Hadoop 3.x releases: 3.3.0, 3.2.1,
> 3.1.3. Hopefully we can get this fixed within this week.
> Additionally, HDFS-14396
>  (Failed to load image
> from FSImageFile when downgrade from 3.x to 2.x).Probably not a blocker but
> nice to have.
>

 Please set target version so that I don't miss in blockers/critical
list for 3.2.1 https://s.apache.org/7yjh5.


>
> On Tue, Aug 20, 2019 at 3:22 AM Rohith Sharma K S <
> rohithsharm...@apache.org> wrote:
>
>> Hello folks,
>>
>> It's been more than six month Hadoop-3.2.0 is released i.e 16th Jan,2019.
>> We have several important fixes landed in branch-3.2 (around 48
>> blockers/critical https://s.apache.org/ozd6o).
>>
>> I am planning to do a maintenance release of 3.2.1 in next few weeks i.e
>> around 1st week of September.
>>
>> So far I don't see any blockers/critical in 3.2.1. I see few pending
>> issues
>> on 3.2.1 are https://s.apache.org/ni6v7.
>>
>> *Proposal*:
>> Code Freezing Date:  30th August 2019
>> Release Date : 7th Sept 2019
>>
>> Please let me know if you have any thoughts or comments on this plan.
>>
>> Thanks & Regards
>> Rohith Sharma K S
>>
>