Re: [DISCUSS] Feature Branch Merge and Security Audits

[larry mccay]

Thanks for the examples, Mike.

I think some of those should actually just be added to the checklist in
other places as they are best practices.
Which raises an interesting point that some of those items can be enabled
by default and maybe indicating so throughout the list makes sense.

Then we can ask for a description of any other Secure by Default
considerations at the end.

I will work on a new revision this morning.


On Wed, Oct 25, 2017 at 4:56 PM, Michael Yoder  wrote:

> #8 is a great topic - given that Hadoop is insecure by default.
>> Actual movement to Secure by Default would be a challenge both
>> technically (given the need for kerberos) and discussion-wise.
>> Asking whether you have considered any settings of configurations that
>> can be secure by default is an interesting idea.
>>
>> Can you provide an example though?
>>
>
> It's tough, I admit - kerberos requires a KDC, TLS requires certificates,
> etc.  But here are some ideas:
>
> - Default to only listen for network traffic on the loopback interface.
> The admin would have to take specific action to listen on a non-loopback
> address. Hence secure by default. I've known web servers that ship like
> this. The counter argument to this is that this is a "useless by default"
> setting for a distributed system... which does have some validity.
> - A more constrained version of the above is to not bind to any network
> interface that has an internet-routable ip address. (That is, not in the
> ranges  192.168.x.x,
> 172.16.x.x, and 10.x).  The idea is that we wouldn't want to risk traffic
> that's obviously headed towards the open internet.  Sure this isn't
> perfect, but it would catch some cases. The admin could provide a specific
> flag to override.  (I got this one from discussion with the Kudu folks.)
> - The examples don't have to be big. Another example would be... if using
> TLS, and if the certificate authority used to sign the certificate is in
> the default certificate store, turn on HSTS automatically.
> - Always turn off TLSv1 and TLSv1.1
> - Forbid single-DES and RC4 encryption algorithms
>
> You get the idea.
> -Mike
>
>
>
>>
>>
>> On Wed, Oct 25, 2017 at 2:14 PM, Michael Yoder 
>> wrote:
>>
>>> On Sat, Oct 21, 2017 at 8:47 AM, larry mccay  wrote:
>>>
 New Revision...

>>>
>>> These lists are wonderful. I appreciate the split between the Tech
>>> Preview and the GA Readiness lists, with the emphasis on the former being
>>> "don't enable by default" or at least "don't enable if security is on".  I
>>> don't have any comments on that part.
>>>
>>> Additions inline below. If some of the additions are items covered by
>>> existing frameworks that any code would use, please forgive my ignorance.
>>> Also, my points aren't as succinct as yours. Feel free to reword.
>>>
>>> *GA Readiness Security Audit*
 At this point, we are merging full or partial security model
 implementations.
 Let's inventory what is covered by the model at this point and whether
 there are future merges required to be full.

 *1. UIs*

 1.1. What sort of validation is being done on any accepted user input?
 (pointers to code would be appreciated)
 1.2. What explicit protections have been built in for (pointers to code
 would be appreciated):
   1.2.1. cross site scripting
   1.2.2. cross site request forgery
   1.2.3. click jacking (X-Frame-Options)

>>>
>>> 1.2.4 If using cookies, is the secure flag for cookies
>>>  turned on?
>>>
>>>
 1.3. What sort of authentication is required for access to the UIs?
   1.3.1. Kerberos
 1.3.1.1. has TGT renewal been accounted for
 1.3.1.2. SPNEGO support?
 1.3.1.3. Delegation token?
   1.3.2. Proxy User ACL?
 1.4. What authorization is available for determining who can access
 what capabilities of the UIs for either viewing, modifying data and/or
 related processes?
 1.5. Is there any input that will ultimately be persisted in
 configuration for executing shell commands or processes?
 1.6. Do the UIs support the trusted proxy pattern with doas
 impersonation?
 1.7. Is there TLS/SSL support?

>>>
>>> 1.7.1 Is it possible to configure TLS protocols and cipher suites?
>>> 1.7.2 Is it possible to configure support for HTTP Strict Transport
>>> Security
>>> 
>>> (HSTS)?
>>> 1.8 Are accesses to the UI audited? ("User X logged into Y from IP
>>> address Z", etc)
>>>
>>>
 *2. REST APIs*

 2.1. Do the REST APIs support the trusted proxy pattern with doas
 impersonation capabilities?
 2.2. What explicit protections have been built in for:
   2.2.1. cross site scripting (XSS)
   2.2.2. cross site request forgery (CSRF)
   2.2.3. XML External Entity (XXE)
 2.3. What is being used for authentication 

Re: [DISCUSS] Feature Branch Merge and Security Audits

[larry mccay]

New revision...

I have incorporated additions from Mike and added a [DEFAULT] tag to those
items that should be considered for Secure by Default settings.
I am hoping that we can close down on the actual lists shortly and move to
discussing the meta points on how/when to require the completion of the
checklists and whether and how they should be included as docs for the
feature moving forward.

Some comments that I have gotten offline have included concern that
targeting merge requests would only capture a subset of new features and
may actually affect the decision to use branches or not. This is certainly
something that we wouldn't want to do. At the same time, we don't want to
be so intrusive in the development cycles to bog down those patches that
just fix bugs.

At any rate, let's close down on the checklists here first.

Thanks!

*Tech Preview Security Audit*
For features that are being merged without full security model coverage,
there need to be a base line of assurances that they do not introduce new
attack vectors in deployments that are from actual releases or even just
built from trunk.

*1. UIs*

1.1. Are there new UIs added with this merge?
1.2. Are they enabled/accessible by default?
1.3. Are they hosted in existing processes or as part of a new
process/server?
1.4. If new process/server, is it launched by default?

*2. APIs*

2.1. Are there new REST APIs added with this merge?
2.2. Are they enabled by default?
2.3. Are there RPC based APIs added with this merge?
2.4. Are they enabled by default?

*3. Secure Clusters*

3.1. Is this feature disabled completely in secure deployments?
3.2. If not, is there some justification as to why it should be available?

*4. CVEs*

4.1. Have all dependencies introduced by this merge been checked for known
issues?




--


*GA Readiness Security Audit*
At this point, we are merging full or partial security model
implementations.
Let's inventory what is covered by the model at this point and whether
there are future merges required to be full.

*1. UIs*

1.1. What sort of validation is being done on any accepted user input?
[DEFAULT] (pointers to code would be appreciated)
1.2. What explicit protections have been built in for (pointers to code
would be appreciated):
  1.2.1. cross site scripting [DEFAULT]
  1.2.2. cross site request forgery [DEFAULT]
  1.2.3. click jacking (X-Frame-Options) [DEFAULT]
  1.2.4 If using cookies, is the secure flag for cookies turned on?
[DEFAULT]
  1.2.5 If using cookies, is the HTTPOnly flag turned on? [DEFAULT]
1.3. What sort of authentication is required for access to the UIs?
[DEFAULT]
  1.3.1. Kerberos
1.3.1.1. has TGT renewal been accounted for
1.3.1.2. SPNEGO support?
1.3.1.3. Delegation token?
  1.3.2. Proxy User ACL?
1.4. What authorization is available for determining who can access what
capabilities of the UIs for either viewing, modifying data and/or related
processes? [DEFAULT]
1.5. Is there any input that will ultimately be persisted in configuration
for executing shell commands or processes?
  1.5.1 If so, how is it validated before persistence? [DEFAULT]
1.6. Do the UIs support the trusted proxy pattern with doas impersonation?
1.7. Is there TLS/SSL support? [DEFAULT]
  1.7.1 Is it possible to configure TLS protocols and cipher suites?
  1.7.2 Is it possible to configure support for HTTP Strict Transport
Security (HSTS)?
1.8 Are accesses to the UIs audited? ("User X logged into Y from IP address
Z", etc) [DEFAULT]

*2. REST APIs*

2.1. Do the REST APIs support the trusted proxy pattern with doas
impersonation capabilities?
2.2. What explicit protections have been built in for:
  2.2.1. cross site scripting (XSS) [DEFAULT]
  2.2.2. cross site request forgery (CSRF) [DEFAULT]
  2.2.3. XML External Entity (XXE) [DEFAULT]
2.3. What is being used for authentication - Hadoop Auth Module? [DEFAULT]
2.4. Are there separate processes for the HTTP resources (UIs and REST
endpoints) or are they part of existing processes?
2.5. Is there TLS/SSL support? [DEFAULT]
  2.5.1 Is it possible to configure TLS protocols and cipher suites?
  2.5.2 Is it possible to configure support for HTTP Strict Transport
Security (HSTS)? [DEFAULT]
2.6. Are there new CLI commands and/or clients for accessing the REST APIs?
2.7. What authorization enforcement points are there within the REST APIs?
2.8 Are accesses to the REST APIs audited? ("User X accessed resource Y
from IP address Z", etc) [DEFAULT]

*3. Encryption*

3.1. Is there any support for encryption of persisted data?
3.2. If so, is KMS and the hadoop key command used for key management?
3.3. KMS interaction with Proxy Users?
3.4 Cryptography is hard. There are more obscure pitfalls in crypto than
any other in computer science. Standard cryptographic libraries should
always be used. Does this work attempt to create an encryption sche

Re: Access to Confluence Wiki

[Arpit Agarwal]

Can you please grant me write access too?

Thanks.


On 10/30/17, 11:01 PM, "Akira Ajisaka"  wrote:

Done. Welcome!

-Akira

On 2017/10/28 3:26, Hanisha Koneru wrote:
> Hi,
> 
> Can I please get access to the Confluence Hadoop Wiki. My confluence id 
is “hanishakoneru”.
> 
> Thanks,
> Hanisha
> 

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Access to Confluence Wiki

[Hanisha Koneru]

Thank you, Akira.

-Hanisha







On 10/30/17, 11:01 PM, "Akira Ajisaka"  wrote:

>Done. Welcome!
>
>-Akira
>
>On 2017/10/28 3:26, Hanisha Koneru wrote:
>> Hi,
>> 
>> Can I please get access to the Confluence Hadoop Wiki. My confluence id is 
>> “hanishakoneru”.
>> 
>> Thanks,
>> Hanisha
>> 
>

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/575/

[Oct 30, 2017 4:49:22 PM] (cdouglas) HADOOP-14992. Upgrade Avro patch version. 
Contributed by Bharat
[Oct 30, 2017 6:04:22 PM] (templedf) YARN-6927. Add support for individual 
resource types requests in
[Oct 30, 2017 7:41:28 PM] (templedf) YARN-7336. Unsafe cast from long to int 
Resource.hashCode() method
[Oct 30, 2017 10:16:51 PM] (junping_du) HADOOP-14990. Clean up jdiff xml files 
added for 2.8.2 release.
[Oct 31, 2017 4:49:15 AM] (aajisaka) HADOOP-14980. [JDK9] Upgrade 
maven-javadoc-plugin to 3.0.0-M1.
[Oct 31, 2017 4:50:28 AM] (aajisaka) Revert "HADOOP-14980. [JDK9] Upgrade 
maven-javadoc-plugin to 3.0.0-M1.
[Oct 31, 2017 4:51:26 AM] (aajisaka) HADOOP-14980. [JDK9] Upgrade 
maven-javadoc-plugin to 3.0.0-M1.




-1 overall


The following subsystems voted -1:
asflicense findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 
   
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.policies.DominantResourceFairnessPolicy$DominantResourceFairnessComparator2
 implements Comparator but not Serializable At 
DominantResourceFairnessPolicy.java:Serializable At 
DominantResourceFairnessPolicy.java:[lines 367-508] 
   
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.policies.DominantResourceFairnessPolicy$DominantResourceFairnessComparatorN
 implements Comparator but not Serializable At 
DominantResourceFairnessPolicy.java:Serializable At 
DominantResourceFairnessPolicy.java:[lines 167-357] 

FindBugs :

   module:hadoop-tools/hadoop-resourceestimator 
   Dead store to jobHistory in 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.getHistoryResourceSkyline(String,
 String) At 
ResourceEstimatorService.java:org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.getHistoryResourceSkyline(String,
 String) At ResourceEstimatorService.java:[line 196] 
   Incorrect lazy initialization and update of static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.skylineStore
 in new org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() 
At ResourceEstimatorService.java:of static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.skylineStore
 in new org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() 
At ResourceEstimatorService.java:[lines 78-82] 
   Write to static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.config 
from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:[line 80] 
   Write to static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.gson from 
instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:[line 106] 
   Write to static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.logParser 
from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:[line 86] 
   Write to static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.rleType 
from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:[line 108] 
   Write to static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.skylineStore
 from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:[line 82] 
   Write to static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.skylineStoreType
 from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:from instance method new 
org.apache.hadoop.resourceestimator

Re: Access to Confluence Wiki

[Akira Ajisaka]

Done!

-Akira

On 2017/11/01 1:56, Arpit Agarwal wrote:

Can you please grant me write access too?

Thanks.


On 10/30/17, 11:01 PM, "Akira Ajisaka"  wrote:

 Done. Welcome!
 
 -Akira
 
 On 2017/10/28 3:26, Hanisha Koneru wrote:

 > Hi,
 >
 > Can I please get access to the Confluence Hadoop Wiki. My confluence id 
is “hanishakoneru”.
 >
 > Thanks,
 > Hanisha
 >
 
 -

 To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
 For additional commands, e-mail: common-dev-h...@hadoop.apache.org
 
 



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14998) Make AuthenticationFilter @Public

[Robert Kanter (JIRA)]

Robert Kanter created HADOOP-14998:
--

 Summary: Make AuthenticationFilter @Public
 Key: HADOOP-14998
 URL: https://issues.apache.org/jira/browse/HADOOP-14998
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 3.0.0
Reporter: Robert Kanter


{{org.apache.hadoop.security.authentication.server.AuthenticationFilter}} is 
currently marked as {{\@Private}} and {{\@Unstable}}.  
{code:java}
@InterfaceAudience.Private
@InterfaceStability.Unstable
public class AuthenticationFilter implements Filter {
{code}

However, many other projects (e.g. Oozie, Hive, Solr, HBase, etc) have been 
using it for quite some time without having any compatibility issues AFAIK.  It 
doesn't seem to have had any breaking changes in quite some time.  On top of 
that, it implements {{javax.servlet.Filter}}, so it can't change too widely 
anyway.  {{AuthenticationFilter}} provides a lot of useful code for dealing 
with tokens, Kerberos, etc, and we should encourage related projects to re-use 
this code instead of rolling their own.

I propose we change it to {{\@Public}} and {{\@Evolving}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-14555) document how to run wasb tests in azure docs site/testing.md

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang reopened HADOOP-14555:
--

> document how to run wasb tests in azure docs site/testing.md
> 
>
> Key: HADOOP-14555
> URL: https://issues.apache.org/jira/browse/HADOOP-14555
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>
> There's no single (current) documentation on running the azure tests
> * There's some in site/index.md, but iit looks potentially out of date 
> (refers to an older azure SDK version)
> * There's a file 
> {{src/test/org/apache/hadoop/fs/azure/RunningLiveWasbTests.txt}}  which 
> refers to a nonexistent doc {{hadoop-tools/hadoop-azure/README.txt }} for 
> instructions.
> Proposed: 
> # move testing docs out of main azure doc page, with link from there. 
> # bring up to date with SDK, move of tests to ITests.
> # purge all other references, including bits of test javadocs which are no 
> longer correct.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14555) document how to run wasb tests in azure docs site/testing.md

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang resolved HADOOP-14555.
--
   Resolution: Duplicate
Fix Version/s: (was: 3.0.0)

> document how to run wasb tests in azure docs site/testing.md
> 
>
> Key: HADOOP-14555
> URL: https://issues.apache.org/jira/browse/HADOOP-14555
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>
> There's no single (current) documentation on running the azure tests
> * There's some in site/index.md, but iit looks potentially out of date 
> (refers to an older azure SDK version)
> * There's a file 
> {{src/test/org/apache/hadoop/fs/azure/RunningLiveWasbTests.txt}}  which 
> refers to a nonexistent doc {{hadoop-tools/hadoop-azure/README.txt }} for 
> instructions.
> Proposed: 
> # move testing docs out of main azure doc page, with link from there. 
> # bring up to date with SDK, move of tests to ITests.
> # purge all other references, including bits of test javadocs which are no 
> longer correct.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Cutting branch-2.9

[Arun Suresh]

Hello folks

We just cut branch-2.9 since all the critical/blocker issues are now
resolved.
We plan to perform some sanity checks for the rest of the week and cut
branch-2.9.0 and push out an RC0 by the end of the week.

Kindly refrain from committing to branch-2.9 without giving us a heads up.

@Junping, lets move the jdiff conversation to separate thread.

Cheers
-Arun/Subru

On Mon, Oct 30, 2017 at 12:39 PM, Subru Krishnan  wrote:

> We want to give heads up that we are going to cut branch-2.9 tomorrow
> morning.
>
> We are down to 3 blockers and they all are close to being committed
> (thanks everyone):
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release
>
> There are 4 other non-blocker JIRAs that are targeted for 2.9.0 which are
> close to completion.
>
> Folks who are working/reviewing these, kindly prioritize accordingly so
> that we can make the release on time.
> https://issues.apache.org/jira/browse/YARN-7398?filter=12342468
>
> Thanks in advance!
>
> -Subru/Arun
>
>
>

Jdiff for 2.9.0 (was: Cutting branch-2.9)

[Arun Suresh]

Hi Junping / Wangda,

We just ran jdiff on branch-2.9.
I have copied the output here : http://home.apache.org/~asuresh/2.9.0-jdiff/

We are going thru it but it would be great if you guys can please take a
look and let us know if you spot anything amiss.

Thanks
-Arun

On Mon, Oct 30, 2017 at 4:52 PM, Arun Suresh  wrote:

> Hello Jumping
>
> Thanks for chiming in.. Appreciate the offer to help. I did run jdiff on
> branch-2 today.. did not find any red flags. Will post the report here
> shortly for review.
>
> Cheers
> -Arun
>
> On Oct 30, 2017 4:44 PM, "Junping Du"  wrote:
>
>> Hi Subru and Arun,
>>  Thanks for moving forward with 2.9 release. Is the first cut of 2.9
>> release supposed to be a stable version or just an alpha version? If it is
>> supposed to be a stable version, we should run jdiff test and check for API
>> compatibility before releasing out. Please let me know if you need any help
>> here.
>>
>> Thanks,
>>
>> Junping
>> 
>> From: Subru Krishnan 
>> Sent: Monday, October 30, 2017 12:39 PM
>> To: common-dev@hadoop.apache.org; yarn-...@hadoop.apache.org;
>> hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
>> Cc: Arun Suresh
>> Subject: Cutting branch-2.9
>>
>> We want to give heads up that we are going to cut branch-2.9 tomorrow
>> morning.
>>
>> We are down to 3 blockers and they all are close to being committed
>> (thanks
>> everyone):
>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release
>>
>> There are 4 other non-blocker JIRAs that are targeted for 2.9.0 which are
>> close to completion.
>>
>> Folks who are working/reviewing these, kindly prioritize accordingly so
>> that we can make the release on time.
>> https://issues.apache.org/jira/browse/YARN-7398?filter=12342468
>>
>> Thanks in advance!
>>
>> -Subru/Arun
>>
>>
>>
>> -
>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>>
>>

2017-10-31 Hadoop 3 release status update

[Andrew Wang]

https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3+release+status+
updates

2017-10-31

Lots of progress towards GA, we look on track for cutting RC0 this week. I
ran the versions script to check the branch matches up with JIRA and fixed
things up, and also checked that the changelog and release notes look
reasonable.

Highlights:

   - Resource types vote has passed and will be merged with branch-3.0
   shortly.
   - Down to three blockers on the dashboard, all being actively revved.

Red flags:

   - Still need to validate that resource types is ready to go once it's
   merged.

Previous tracked GA blockers that have been resolved or dropped:

   - Change of ExecutionType
  - YARN-7178
   - Add
  documentation for Container Update API RESOLVED : Arun got the patch
  in with reviews from Wangda and Haibo.
   - ReservationSystem
  - YARN-4827
  
- Document
  configuration of ReservationSystem for FairScheduler RESOLVED: Yufei
  and Subru got this in.
   - Rolling upgrade
  - YARN-6142
   - Support
  rolling upgrade between 2.x and 3.x RESOLVED : Ray resolved this
  since we think it's sufficiently complete.
   - Erasure coding
  - HDFS-12686
  
- Erasure
  coding system policy state is not correctly saved and loaded during real
  cluster restart RESOLVED: Resolved this one to incorporate it in
  HDFS-12682

GA blockers:

   - Rolling upgrade
  - HDFS-11096
  
- Support
  rolling upgrade between 2.x and 3.xPATCH AVAILABLE: I asked Sean if
  we can downgrade this from blocker
   - Erasure coding
  - HDFS-12682
  
- ECAdmin
  -listPolicies will always show SystemErasureCodingPolicies state
as DISABLED
   PATCH AVAILABLE: Actively being worked on and reviewed, should be in
  soon
  - HDFS-11467
  
- Support
  ErasureCoding section in OIV XML/ReverseXMLPATCH AVAILABLE: Waiting
  on HDFS-12682, I asked if we can work concurrently

Features merged for GA:

   - Erasure coding
  - Testing is still ongoing at Cloudera, no new bugs found recently
  - Closing on remaining blockers for GA
   - Classpath isolation (HADOOP-11656)
   - HADOOP-13916
  
- Document
  how downstream clients should make use of the new shaded client artifacts
   OPEN: Seems unlikely to make it
   - Compat guide (HADOOP-13714
   )
  - HADOOP-14876
  
- Create
  downstream developer docs from the compatibility guidelines PATCH
  AVAILABLE: Patch is being actively revved and reviewed, Robert +1'd,
  Anu posted a big review
  - HADOOP-14875
  
- Create
  end user documentation from the compatibility guidelines PATCH
  AVAILABLE: No patch yet
   - TSv2 alpha 2
   - This was merged, no problems thus far [image: (smile)]
   - API-based scheduler configuration YARN-5734
    - OrgQueue
   for easy CapacityScheduler queue configuration management RESOLVED
  - Merged, no problems thus far [image: (smile)]
   - HDFS router-based configuration HDFS-10467
    -
Router-based
   HDFS federation RESOLVED
  - Merged, no problems thus far [image: (smile)]
   - Resource types YARN-3926
    - Extend
   the YARN resource model for easier resource-type management and profiles
   RESOLVED
  - Vote has passed, Daniel is currently doing the mechanics of merging
  - Need to also perform final validation post-merge

Dropping the "unmerged features" section since we're not letting in
anything else at this point.

Re: Cutting branch-2.9

[俊平堵]

Hi Arun/Subru,
Thanks for updating on 2.9.0 release progress. A quick question here:
are we planning to release from branch-2.9 directly?
I doubt this as it seems against our current branch release practice (
https://wiki.apache.org/hadoop/HowToRelease#Branching). To get rid of any
confusion, I would suggest to cut-off branch-2.9.0 for 2.9.0 release work.
In the mean time, branch-2.9 should be reserved for next 2.9 point release
(2.9.1) and branch-2 should be reserved for next minor release (2.10.0 or
whatever name it is). Thoughts?

bq. @Junping, lets move the jdiff conversation to separate thread.
Sure. I will reply jdiff in separated thread.

Thanks,

Junping

2017-10-31 13:44 GMT-07:00 Arun Suresh :

> Hello folks
>
> We just cut branch-2.9 since all the critical/blocker issues are now
> resolved.
> We plan to perform some sanity checks for the rest of the week and cut
> branch-2.9.0 and push out an RC0 by the end of the week.
>
> Kindly refrain from committing to branch-2.9 without giving us a heads up.
>
> @Junping, lets move the jdiff conversation to separate thread.
>
> Cheers
> -Arun/Subru
>
> On Mon, Oct 30, 2017 at 12:39 PM, Subru Krishnan  wrote:
>
> > We want to give heads up that we are going to cut branch-2.9 tomorrow
> > morning.
> >
> > We are down to 3 blockers and they all are close to being committed
> > (thanks everyone):
> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release
> >
> > There are 4 other non-blocker JIRAs that are targeted for 2.9.0 which are
> > close to completion.
> >
> > Folks who are working/reviewing these, kindly prioritize accordingly so
> > that we can make the release on time.
> > https://issues.apache.org/jira/browse/YARN-7398?filter=12342468
> >
> > Thanks in advance!
> >
> > -Subru/Arun
> >
> >
> >
>

Re: Cutting branch-2.9

[Subru Krishnan]

Hi Junping,

We are planning to cut branch-2.9.0 on Thursday as there are couple of
non-blocking bug fixes coming in which we hope to get in tomorrow. Hope
that answers your question.

Thanks,
Subru

On Tue, Oct 31, 2017 at 2:35 PM, 俊平堵  wrote:

> Hi Arun/Subru,
> Thanks for updating on 2.9.0 release progress. A quick question here:
> are we planning to release from branch-2.9 directly?
> I doubt this as it seems against our current branch release practice (
> https://wiki.apache.org/hadoop/HowToRelease#Branching). To get rid of any
> confusion, I would suggest to cut-off branch-2.9.0 for 2.9.0 release work.
> In the mean time, branch-2.9 should be reserved for next 2.9 point release
> (2.9.1) and branch-2 should be reserved for next minor release (2.10.0 or
> whatever name it is). Thoughts?
>
> bq. @Junping, lets move the jdiff conversation to separate thread.
> Sure. I will reply jdiff in separated thread.
>
> Thanks,
>
> Junping
>
> 2017-10-31 13:44 GMT-07:00 Arun Suresh :
>
>> Hello folks
>>
>> We just cut branch-2.9 since all the critical/blocker issues are now
>> resolved.
>> We plan to perform some sanity checks for the rest of the week and cut
>> branch-2.9.0 and push out an RC0 by the end of the week.
>>
>> Kindly refrain from committing to branch-2.9 without giving us a heads up.
>>
>> @Junping, lets move the jdiff conversation to separate thread.
>>
>> Cheers
>> -Arun/Subru
>>
>> On Mon, Oct 30, 2017 at 12:39 PM, Subru Krishnan 
>> wrote:
>>
>> > We want to give heads up that we are going to cut branch-2.9 tomorrow
>> > morning.
>> >
>> > We are down to 3 blockers and they all are close to being committed
>> > (thanks everyone):
>> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release
>> >
>> > There are 4 other non-blocker JIRAs that are targeted for 2.9.0 which
>> are
>> > close to completion.
>> >
>> > Folks who are working/reviewing these, kindly prioritize accordingly so
>> > that we can make the release on time.
>> > https://issues.apache.org/jira/browse/YARN-7398?filter=12342468
>> >
>> > Thanks in advance!
>> >
>> > -Subru/Arun
>> >
>> >
>> >
>>
>
>

Re: Cutting branch-2.9

[Arun Suresh]

Hi Junping,

> In the mean time, branch-2.9 should be reserved for next 2.9 point
release (2.9.1) and branch-2 should be reserved for next minor release
(2.10.0 or whatever name it is). Thoughts?
Yup, That is the current plan. We've updated mvn versions in branch-2 to
point to 2.10.0-SNAPSHOT.
Once we cut branch-2.9.0, we will update mvn versions on branch-2.9.0 to
2.9.0 and set versions on branch-2.9 to 2.9.1-SNAPSHOT (We plan to do that
by Thursday)

Cheers
-Arun

On Tue, Oct 31, 2017 at 2:35 PM, 俊平堵  wrote:

> Hi Arun/Subru,
> Thanks for updating on 2.9.0 release progress. A quick question here:
> are we planning to release from branch-2.9 directly?
> I doubt this as it seems against our current branch release practice (
> https://wiki.apache.org/hadoop/HowToRelease#Branching). To get rid of any
> confusion, I would suggest to cut-off branch-2.9.0 for 2.9.0 release work.
> In the mean time, branch-2.9 should be reserved for next 2.9 point release
> (2.9.1) and branch-2 should be reserved for next minor release (2.10.0 or
> whatever name it is). Thoughts?
>
> bq. @Junping, lets move the jdiff conversation to separate thread.
> Sure. I will reply jdiff in separated thread.
>
> Thanks,
>
> Junping
>
> 2017-10-31 13:44 GMT-07:00 Arun Suresh :
>
>> Hello folks
>>
>> We just cut branch-2.9 since all the critical/blocker issues are now
>> resolved.
>> We plan to perform some sanity checks for the rest of the week and cut
>> branch-2.9.0 and push out an RC0 by the end of the week.
>>
>> Kindly refrain from committing to branch-2.9 without giving us a heads up.
>>
>> @Junping, lets move the jdiff conversation to separate thread.
>>
>> Cheers
>> -Arun/Subru
>>
>> On Mon, Oct 30, 2017 at 12:39 PM, Subru Krishnan 
>> wrote:
>>
>> > We want to give heads up that we are going to cut branch-2.9 tomorrow
>> > morning.
>> >
>> > We are down to 3 blockers and they all are close to being committed
>> > (thanks everyone):
>> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release
>> >
>> > There are 4 other non-blocker JIRAs that are targeted for 2.9.0 which
>> are
>> > close to completion.
>> >
>> > Folks who are working/reviewing these, kindly prioritize accordingly so
>> > that we can make the release on time.
>> > https://issues.apache.org/jira/browse/YARN-7398?filter=12342468
>> >
>> > Thanks in advance!
>> >
>> > -Subru/Arun
>> >
>> >
>> >
>>
>
>

[jira] [Created] (HADOOP-14999) AliyunOSS: provide block-based output stream to support large file (> 5GB)

[Genmao Yu (JIRA)]

Genmao Yu created HADOOP-14999:
--

 Summary: AliyunOSS: provide block-based output stream to support 
large file (> 5GB)
 Key: HADOOP-14999
 URL: https://issues.apache.org/jira/browse/HADOOP-14999
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/oss
Reporter: Genmao Yu
Assignee: Genmao Yu
Priority: Major






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org