[jira] [Created] (HADOOP-14804) correct wrong parameters format order in core-default.xml

[Chen Hongfei (JIRA)]

Chen Hongfei created HADOOP-14804:
-

 Summary: correct wrong parameters format order in core-default.xml
 Key: HADOOP-14804
 URL: https://issues.apache.org/jira/browse/HADOOP-14804
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 3.0.0-alpha3
Reporter: Chen Hongfei
Priority: Trivial
 Fix For: 3.0.0-alpha3


descriptions of "HTTP CORS" parameters is before the names:  

   Comma separated list of headers that are allowed for web
services needing cross-origin (CORS) support.
  hadoop.http.cross-origin.allowed-headers
  X-Requested-With,Content-Type,Accept,Origin
 
..
but they should be following value as others.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Merge HADOOP-13345 (S3Guard feature branch)

[Steve Loughran]

On 23 Aug 2017, at 19:21, Aaron Fabbri 
mailto:fab...@cloudera.com>> wrote:


On Tue, Aug 22, 2017 at 10:24 AM, Steve Loughran 
mailto:ste...@hortonworks.com>> wrote:
video being processed:  
https://www.youtube.com/watch?v=oIe5Zl2YsLE&feature=youtu.be


Awesome demo Steve, thanks for doing this.  Particularly glad to see folks 
using and extending the failure injection client.

The HADOOP-13786 iteration turns on throttle event generation. All the new 
committer stuff is ready for it, but all the existing S3A FS ops react to a 
throttle exception by failing, when they need to just back off a bit. This 
complicates testing as I have to explicitly turn off fault injection for setup 
& teardown


Demoing the CLI tool was great as well.


I'm going to have to do another iteration on that CLI tool post-merge, as I had 
one big problem: working out if the bucket and all the binding settings meant 
it was "guarded". I think we'll need to track what issues like that crop up in 
the field and add the diagnostics/other options.

+I think another one that'd be useful would be to enum all s3guard DDB tables 
in a region/globally & list their allocated IOPs. I know the AWS UI can list 
tables by region, but you need to look around every region to find out if 
you've accidentally created one. If you enum all table & look for a s3guard 
version marker, then you can identify tables.

Wanted to mention two things:

1. Authoritative mode is not fully implemented yet with Dynamo (it needs to 
persist an extra bit for directories).  I do have an auth-mode patch (done for 
a hackathon) that I need to post which shows large performance improvements 
over what S3Guard has today.  As you said, we don't consider authoritative mode 
ready for production yet: we want to play with it more and improve the prune 
algorithm first.  Authoritative mode can be thought of as a nice bonus in the 
future: The main goal of S3Guard v1 is to fix the get / list consistency issues 
you mentioned, which it does well.


we need to call that out in the release notes.

2. Also wanted to thank Lei (Eddy) Xu, he was very active during early design 
and contributed some patches as well.


good point. Lei: you will get a special mention the next time I do the demo


Again, great demo, enjoyed it!

-AF


its actually quite hard to show any benefits of s3guard on the command line, so 
I've ended up showing some scala tests where I turn on the (bundled) 
inconsistent AWS client to show how you then need to enable s3guard to make the 
stack traces go away


On 22 Aug 2017, at 11:17, Steve Loughran 
mailto:ste...@hortonworks.com>>>
 wrote:

+1 (binding)

I'm happy with it; it's a great piece of work by (in no particular order): 
Chris Nauroth, Aaron Fabbri, Sean McRory & Mingliang Liu. plus a few bits in 
the corners where I got to break things while they were all asleep. Also 
deserving a mention: Thomas Demoor & Ewan Higgs @ WDC for consultancy on the 
corners of S3, everyone who tested in (including our QA team), Sanjay Radia, & 
others.

I've already done a couple of iterations of fixing checksyles & code reviews, 
so I think it is ready. I also have a branch-2 patch based on earlier work by 
Mingliang, for people who want that.




On 17 Aug 2017, at 23:07, Aaron Fabbri 
mailto:fab...@cloudera.com>>>
 wrote:

Hello,

I'd like to open a vote (7 days, ending August 24 at 3:10 PST) to merge the
HADOOP-13345 feature branch into trunk.

This branch contains the new S3Guard feature which adds metadata
consistency features to the S3A client.  Formatted site documentation can
be found here:

https://github.com/apache/hadoop/blob/HADOOP-13345/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md

The current patch against trunk is posted here:

https://issues.apache.org/jira/browse/HADOOP-13998

The branch modifies the s3a portion of the hadoop-tools/hadoop-aws module:

- The feature is off by default, and care has been taken to insure it has
no impact when disabled.
- S3Guard can be enabled with the production database which is backed by
DynamoDB, or with a local, in-memory implementation that facilitates
integration testing without having to pay for a database.
- getFileStatus() as well as directory listing consistency has been
implemented and thoroughly tested, including delete tracking.
- Convenient Maven profiles for testing with and without S3Guard.
- New failure injection code and integration tests that exercise it.  We
use timers and a wrapper around the Amazon SDK client object to force
consistency delays to occur.  This allows us to assert that S3Guard works
as advertised.  This will be extended with more types of failure injection
to continue hardening the S3A client.

Outside of hadoop-tools/hadoop-aws's s3a directory there are some minor
changes:

- core-default.xml defaults and documentation for s3guard parameters.
- A

[jira] [Resolved] (HADOOP-14007) cli to list info about a bucket (S3guard or not)

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HADOOP-14007.
-
Resolution: Duplicate

HADOOP-14220 covers same issue, although it's newer it's got a patch in progress

> cli to list info about a bucket (S3guard or not)
> 
>
> Key: HADOOP-14007
> URL: https://issues.apache.org/jira/browse/HADOOP-14007
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Steve Loughran
>
> It turns out to be useful during dev to find things out about a bucket:
> * whether it uses s3guard
> * location of table
> * table size
> * version & create timestamp
> * maybe: which auth mechanism worked (though we may need to add some more 
> tracking in our providers there, and tread carefully w.r.t security)
> this could be added to some s3 cli, e.g. "hadoop s3 info s3a://bucket"
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

testers of new FTP FS needed

[Steve Loughran]

Not many people seem aware of the fact that thers's a new ftp fs client in 
progress

https://issues.apache.org/jira/browse/HADOOP-1


Can anyone who likes to use ftp:// test against their own FTP endpointsit's 
a diverse enough platform that it'd be good to know how well it works, and so, 
how ready this is for replacing the original ftp client

Re: [VOTE] Merge HADOOP-13345 (S3Guard feature branch)

[Kihwal Lee]

+1 (binding)
Great work guys!

On Thu, Aug 24, 2017 at 5:01 AM, Steve Loughran 
wrote:

>
> On 23 Aug 2017, at 19:21, Aaron Fabbri mailto:fa
> b...@cloudera.com>> wrote:
>
>
> On Tue, Aug 22, 2017 at 10:24 AM, Steve Loughran  mailto:ste...@hortonworks.com>> wrote:
> video being processed:  https://www.youtube.com/watch?
> v=oIe5Zl2YsLE&feature=youtu.be
>
>
> Awesome demo Steve, thanks for doing this.  Particularly glad to see folks
> using and extending the failure injection client.
>
> The HADOOP-13786 iteration turns on throttle event generation. All the new
> committer stuff is ready for it, but all the existing S3A FS ops react to a
> throttle exception by failing, when they need to just back off a bit. This
> complicates testing as I have to explicitly turn off fault injection for
> setup & teardown
>
>
> Demoing the CLI tool was great as well.
>
>
> I'm going to have to do another iteration on that CLI tool post-merge, as
> I had one big problem: working out if the bucket and all the binding
> settings meant it was "guarded". I think we'll need to track what issues
> like that crop up in the field and add the diagnostics/other options.
>
> +I think another one that'd be useful would be to enum all s3guard DDB
> tables in a region/globally & list their allocated IOPs. I know the AWS UI
> can list tables by region, but you need to look around every region to find
> out if you've accidentally created one. If you enum all table & look for a
> s3guard version marker, then you can identify tables.
>
> Wanted to mention two things:
>
> 1. Authoritative mode is not fully implemented yet with Dynamo (it needs
> to persist an extra bit for directories).  I do have an auth-mode patch
> (done for a hackathon) that I need to post which shows large performance
> improvements over what S3Guard has today.  As you said, we don't consider
> authoritative mode ready for production yet: we want to play with it more
> and improve the prune algorithm first.  Authoritative mode can be thought
> of as a nice bonus in the future: The main goal of S3Guard v1 is to fix the
> get / list consistency issues you mentioned, which it does well.
>
>
> we need to call that out in the release notes.
>
> 2. Also wanted to thank Lei (Eddy) Xu, he was very active during early
> design and contributed some patches as well.
>
>
> good point. Lei: you will get a special mention the next time I do the demo
>
>
> Again, great demo, enjoyed it!
>
> -AF
>
>
> its actually quite hard to show any benefits of s3guard on the command
> line, so I've ended up showing some scala tests where I turn on the
> (bundled) inconsistent AWS client to show how you then need to enable
> s3guard to make the stack traces go away
>
>
> On 22 Aug 2017, at 11:17, Steve Loughran  ste...@hortonworks.com>>> wrote:
>
> +1 (binding)
>
> I'm happy with it; it's a great piece of work by (in no particular order):
> Chris Nauroth, Aaron Fabbri, Sean McRory & Mingliang Liu. plus a few bits
> in the corners where I got to break things while they were all asleep. Also
> deserving a mention: Thomas Demoor & Ewan Higgs @ WDC for consultancy on
> the corners of S3, everyone who tested in (including our QA team), Sanjay
> Radia, & others.
>
> I've already done a couple of iterations of fixing checksyles & code
> reviews, so I think it is ready. I also have a branch-2 patch based on
> earlier work by Mingliang, for people who want that.
>
>
>
>
> On 17 Aug 2017, at 23:07, Aaron Fabbri mailto:fa
> b...@cloudera.com>>>
> wrote:
>
> Hello,
>
> I'd like to open a vote (7 days, ending August 24 at 3:10 PST) to merge the
> HADOOP-13345 feature branch into trunk.
>
> This branch contains the new S3Guard feature which adds metadata
> consistency features to the S3A client.  Formatted site documentation can
> be found here:
>
> https://github.com/apache/hadoop/blob/HADOOP-13345/
> hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
>
> The current patch against trunk is posted here:
>
> https://issues.apache.org/jira/browse/HADOOP-13998
>
> The branch modifies the s3a portion of the hadoop-tools/hadoop-aws module:
>
> - The feature is off by default, and care has been taken to insure it has
> no impact when disabled.
> - S3Guard can be enabled with the production database which is backed by
> DynamoDB, or with a local, in-memory implementation that facilitates
> integration testing without having to pay for a database.
> - getFileStatus() as well as directory listing consistency has been
> implemented and thoroughly tested, including delete tracking.
> - Convenient Maven profiles for testing with and without S3Guard.
> - New failure injection code and integration tests that exercise it.  We
> use timers and a wrapper around the Amazon SDK client object to force
> consistency delays to occur.  This allows us to assert that S3Guard works
> as advertised.  This will be e

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Sunil G]

Thank you very much Vrushali, Rohith, Varun and other folks who made this
happen. Great work, really appreciate the same!!

+1 (binding) from my side:

# Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
# Accessed new YARN UI which shows various flows/flow activity etc. Seems
fine.
# Based on code, looks like all apis are compatible.
# REST api docs looks fine as well, I guess we could improve that a bit
more post merge as well.
# Adding to additional thoughts which are discussed here, native service
also could publish events to atsv2. I think that work is also happened in
branch.

Looking forward to a much wider adoption of ATSv2 with more projects.

Thanks
Sunil


On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
vrushalic2...@gmail.com> wrote:

> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/502/

[Aug 23, 2017 4:56:20 PM] (wangda) YARN-6251. Do async container release to 
prevent deadlock during
[Aug 23, 2017 6:37:18 PM] (jzhuge) HADOOP-14251. Credential provider should 
handle property key
[Aug 23, 2017 10:22:24 PM] (rchiang) HADOOP-14649. Update aliyun-sdk-oss 
version to 2.8.1. (Genmao Yu via
[Aug 24, 2017 12:06:16 AM] (xiao) HDFS-10899. Add functionality to re-encrypt 
EDEKs.
[Aug 24, 2017 1:06:29 AM] (junping_du) YARN-7090. 
testRMRestartAfterNodeLabelDisabled get failed when




-1 overall


The following subsystems voted -1:
findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 
   Hard coded reference to an absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:[line 490] 

Failed junit tests :

   hadoop.security.TestRaceWhenRelogin 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 
   hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 
   hadoop.hdfs.server.datanode.TestDataNodeMetrics 
   hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem 
   hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation 
   hadoop.yarn.sls.appmaster.TestAMSimulator 
   hadoop.yarn.sls.nodemanager.TestNMSimulator 

Timed out junit tests :

   org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands 
   
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore 
   
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA 
   
org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/502/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/502/artifact/out/diff-compile-javac-root.txt
  [292K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/502/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/502/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/502/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/502/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/502/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/502/artifact/out/whitespace-tabs.txt
  [1.2M]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/502/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/502/artifact/out/diff-javadoc-javadoc-root.txt
  [1.9M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/502/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [148K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/502/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [304K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/502/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt
  [32K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/502/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [64K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/502/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt
  [16K]

Powered by Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: com

[jira] [Created] (HADOOP-14805) Upgrade to zstd 1.3.1

[Andrew Wang (JIRA)]

Andrew Wang created HADOOP-14805:


 Summary: Upgrade to zstd 1.3.1
 Key: HADOOP-14805
 URL: https://issues.apache.org/jira/browse/HADOOP-14805
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 3.0.0-alpha2, 2.9.0
Reporter: Andrew Wang






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: zstd compression

[Andrew Wang]

Happily, the zstd maintainer responded to my issue, and it's now dual
licensed with BSD and GPLv2. We need to upgrade to zstd 1.3.1 to pull this
in. I filed this JIRA for tracking:

https://issues.apache.org/jira/browse/HADOOP-14805

On Tue, Jul 25, 2017 at 10:27 AM, Andrew Wang 
wrote:

> Thanks Sean, Owen. I've opened an issue on their github here:
>
> https://github.com/facebook/zstd/issues/775
>
> I figure it doesn't hurt to ask, particularly if they intend for zstd to
> be a replacement for the commonly-embedded zlib.
>
> On Tue, Jul 25, 2017 at 6:17 AM, Owen O'Malley 
> wrote:
>
>> I'd support asking Facebook to change it with both my hadoop and orc hats
>> on.
>>
>> .. Owen
>>
>> > On Jul 24, 2017, at 23:43, Sean Busbey  wrote:
>> >
>> > Nope. Once I found out HBase's use was compliant as an optional runtime
>> > dependency I stopped looking.
>> >
>> >> On Jul 24, 2017 7:22 PM, "Andrew Wang" 
>> wrote:
>> >>
>> >> I think it'd still be worth asking FB to relicense zstandard. Being
>> able
>> >> to bundle it in the release would make it easier to use, since I doubt
>> >> there are zstandard packages in the default OS repos.
>> >>
>> >> Sean, have you already filed an issue with zstandard?
>> >>
>> >> On Mon, Jul 17, 2017 at 1:30 PM, Jason Lowe
>> 
>> >> wrote:
>> >>
>> >>> I think we are OK to leave support for the zstd codec in the Hadoop
>> code
>> >>> base.  I asked Chris Mattman for clarification, noting that the
>> support for
>> >>> the zstd codec requires the user to install the zstd headers and
>> libraries
>> >>> and then configure it to be included in the native Hadoop build.  The
>> >>> Hadoop releases are not shipping any zstd code (e.g.: headers or
>> libraries)
>> >>> nor does it require zstd as a mandatory dependency.  Here's what he
>> said:
>> >>>
>> >>>
>> >>> On Monday, July 17, 2017 11:07 AM, Chris Mattmann <
>> mattm...@apache.org>
>> >>> wrote:
>> >>>
>>  Hi Jason,
>> 
>>  This sounds like an optional dependency on a Cat-X software. This
>> isn’t
>> >>> the only type of compression
>>  that is allowed within Hadoop, correct? If it is truly optional and
>> you
>> >>> have gone to that level of detail
>>  below to make the user opt in, and if we are not shipping zstd with
>> our
>> >>> products (source code releases),
>>  then this is an acceptable usage.
>> 
>>  Cheers,
>>  Chris
>> >>>
>> >>>
>> >>> So I think we are in the clear with respect to zstd usage as long as
>> we
>> >>> keep it as an optional codec where the user needs to get the headers
>> and
>> >>> libraries for zstd and configure it into the native Hadoop build.
>> >>>
>> >>> Jason
>> >>>
>> >>> On Monday, July 17, 2017 9:44 AM, Sean Busbey 
>> >>> wrote:
>> >>>
>> >>>
>> >>>
>> >>> I know that the HBase community is also looking at what to do about
>> >>>
>> >>> our inclusion of zstd. We've had it in releases since late 2016. My
>> >>>
>> >>> plan was to request that they relicense it.
>> >>>
>> >>>
>> >>> Perhaps the Hadoop PMC could join HBase in the request?
>> >>>
>> >>>
>> >>> On Sun, Jul 16, 2017 at 8:11 PM, Allen Wittenauer
>> >>>
>> >>>  wrote:
>> >>>
>> 
>> >>>
>> It looks like HADOOP-13578 added Facebook's zstd compression
>> >>> codec.  Unfortunately, that codec is using the same 3-clause BSD
>> (LICENSE
>> >>> file) + patent grant license (PATENTS file) that React is using and
>> RocksDB
>> >>> was using.
>> >>>
>> 
>> >>>
>> Should that code get reverted?
>> >>>
>> 
>> >>>
>> 
>> >>>
>> 
>> >>>
>>  
>> -
>> >>>
>>  To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> >>>
>>  For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> >>>
>> 
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>>
>> >>> busbey
>> >>>
>> >>>
>> >>> -
>> >>>
>> >>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> >>>
>> >>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> >>>
>> >>> -
>> >>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> >>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> >>>
>> >>>
>> >>
>>
>
>

Re: [VOTE] Merge YARN-3926 (resource profile) to trunk

[Sunil G]

Thank you very much Varun Vasudev, Wangda Tan, Daniel and all the folks who
helped in getting this feature in this level.

Starting with my +1 (binding).


# Tested a 5 node cluster with resource profiles enabled/disabled (feature
is disabled by default)

# All apis added are marked as Unstable/Evolving (very few)

# There is no compatibility break with older versions (we have added UT
cases also to ensure same)

# Performance tests were done using SLS and also with some tight loops unit
tests. There is no much regression with current trunk.

# Latest jenkins +1 on YARN-7013 for whole branch code.

# Verified old RM UI and new YARN UI (newly added resources could be seen
easily)


Once again thanks all the folks who helped in getting this feature. Kudos!


Thanks

- Sunil


On Thu, Aug 24, 2017 at 12:20 AM Wangda Tan  wrote:

>  Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-3926 (Resource profile) to trunk. The vote will run for
> 7 days and will end August 30 10:00 AM PDT.
>
> Briefly, YARN-3926 can extend resource model of YARN to support resource
> types other than CPU and memory, so it will be a cornerstone of features
> like GPU support (YARN-6223), disk scheduling/isolation (YARN-2139), FPGA
> support (YARN-5983), network IO scheduling/isolation (YARN-2140). In
> addition to that, YARN-3926 allows admin to preconfigure resource profiles
> in the cluster, for example, m3.large means <2 vcores, 8 GB memory, 64 GB
> disk>, so applications can request "m3.large" profile instead of specifying
> all resource types’s values.
>
> There are 32 subtasks that were completed as part of this effort.
>
> This feature needs to be explicitly turned on before use. We paid close
> attention to compatibility, performance, and scalability of this feature,
> mentioned in [1], we didn't see observable performance regression in large
> scale SLS (scheduler load simulator) executions and saw less than 5%
> performance regression by using micro benchmark added by YARN-6775.
>
> This feature works from end-to-end (including UI/CLI/application/server),
> we have setup a cluster with this feature turned on runs for several weeks,
> we didn't see any issues by far.
>
> Merge JIRA: YARN-7013 (Jenkins gave +1 already).
> Documentation: YARN-7056
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort including design discussion/development/reviews, etc.: Varun
> Vasudev, Sunil Govind, Daniel Templeton, Vinod Vavilapalli, Yufei Gu,
> Karthik Kambatla, Jason Lowe, Arun Suresh.
>
> Regards,
> Wangda Tan
>
> [1]
>
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201708.mbox/%3CCAD%2B%2BeCnjEHU%3D-M33QdjnND0ZL73eKwxRua4%3DBbp4G8inQZmaMg%40mail.gmail.com%3E
>

Re: [VOTE] Merge HADOOP-13345 (S3Guard feature branch)

[Andrew Wang]

FYI that committer +1s are binding on merges, so Sean and Mingliang's +1s
can be upgraded to binding.

On Thu, Aug 24, 2017 at 6:09 AM, Kihwal Lee  wrote:

> +1 (binding)
> Great work guys!
>
> On Thu, Aug 24, 2017 at 5:01 AM, Steve Loughran 
> wrote:
>
> >
> > On 23 Aug 2017, at 19:21, Aaron Fabbri mailto:fa
> > b...@cloudera.com>> wrote:
> >
> >
> > On Tue, Aug 22, 2017 at 10:24 AM, Steve Loughran  <
> > mailto:ste...@hortonworks.com>> wrote:
> > video being processed:  https://www.youtube.com/watch?
> > v=oIe5Zl2YsLE&feature=youtu.be
> >
> >
> > Awesome demo Steve, thanks for doing this.  Particularly glad to see
> folks
> > using and extending the failure injection client.
> >
> > The HADOOP-13786 iteration turns on throttle event generation. All the
> new
> > committer stuff is ready for it, but all the existing S3A FS ops react
> to a
> > throttle exception by failing, when they need to just back off a bit.
> This
> > complicates testing as I have to explicitly turn off fault injection for
> > setup & teardown
> >
> >
> > Demoing the CLI tool was great as well.
> >
> >
> > I'm going to have to do another iteration on that CLI tool post-merge, as
> > I had one big problem: working out if the bucket and all the binding
> > settings meant it was "guarded". I think we'll need to track what issues
> > like that crop up in the field and add the diagnostics/other options.
> >
> > +I think another one that'd be useful would be to enum all s3guard DDB
> > tables in a region/globally & list their allocated IOPs. I know the AWS
> UI
> > can list tables by region, but you need to look around every region to
> find
> > out if you've accidentally created one. If you enum all table & look for
> a
> > s3guard version marker, then you can identify tables.
> >
> > Wanted to mention two things:
> >
> > 1. Authoritative mode is not fully implemented yet with Dynamo (it needs
> > to persist an extra bit for directories).  I do have an auth-mode patch
> > (done for a hackathon) that I need to post which shows large performance
> > improvements over what S3Guard has today.  As you said, we don't consider
> > authoritative mode ready for production yet: we want to play with it more
> > and improve the prune algorithm first.  Authoritative mode can be thought
> > of as a nice bonus in the future: The main goal of S3Guard v1 is to fix
> the
> > get / list consistency issues you mentioned, which it does well.
> >
> >
> > we need to call that out in the release notes.
> >
> > 2. Also wanted to thank Lei (Eddy) Xu, he was very active during early
> > design and contributed some patches as well.
> >
> >
> > good point. Lei: you will get a special mention the next time I do the
> demo
> >
> >
> > Again, great demo, enjoyed it!
> >
> > -AF
> >
> >
> > its actually quite hard to show any benefits of s3guard on the command
> > line, so I've ended up showing some scala tests where I turn on the
> > (bundled) inconsistent AWS client to show how you then need to enable
> > s3guard to make the stack traces go away
> >
> >
> > On 22 Aug 2017, at 11:17, Steve Loughran  > ste...@hortonworks.com> ste...@hortonworks.com>>> wrote:
> >
> > +1 (binding)
> >
> > I'm happy with it; it's a great piece of work by (in no particular
> order):
> > Chris Nauroth, Aaron Fabbri, Sean McRory & Mingliang Liu. plus a few bits
> > in the corners where I got to break things while they were all asleep.
> Also
> > deserving a mention: Thomas Demoor & Ewan Higgs @ WDC for consultancy on
> > the corners of S3, everyone who tested in (including our QA team), Sanjay
> > Radia, & others.
> >
> > I've already done a couple of iterations of fixing checksyles & code
> > reviews, so I think it is ready. I also have a branch-2 patch based on
> > earlier work by Mingliang, for people who want that.
> >
> >
> >
> >
> > On 17 Aug 2017, at 23:07, Aaron Fabbri mailto:fa
> > b...@cloudera.com> >>>
> > wrote:
> >
> > Hello,
> >
> > I'd like to open a vote (7 days, ending August 24 at 3:10 PST) to merge
> the
> > HADOOP-13345 feature branch into trunk.
> >
> > This branch contains the new S3Guard feature which adds metadata
> > consistency features to the S3A client.  Formatted site documentation can
> > be found here:
> >
> > https://github.com/apache/hadoop/blob/HADOOP-13345/
> > hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
> >
> > The current patch against trunk is posted here:
> >
> > https://issues.apache.org/jira/browse/HADOOP-13998
> >
> > The branch modifies the s3a portion of the hadoop-tools/hadoop-aws
> module:
> >
> > - The feature is off by default, and care has been taken to insure it has
> > no impact when disabled.
> > - S3Guard can be enabled with the production database which is backed by
> > DynamoDB, or with a local, in-memory implementation that facilitates
> > integration testing without having to pay for a database.
> > - getFileStatus() as well 

[jira] [Created] (HADOOP-14806) "BlockVerificationFailures" and "BlocksVerified" show up as 0 in Datanode JMX

[Sai Nukavarapu (JIRA)]

Sai Nukavarapu created HADOOP-14806:
---

 Summary: "BlockVerificationFailures" and "BlocksVerified" show up 
as 0 in Datanode JMX
 Key: HADOOP-14806
 URL: https://issues.apache.org/jira/browse/HADOOP-14806
 Project: Hadoop Common
  Issue Type: Bug
  Components: metrics
Affects Versions: 2.7.1
Reporter: Sai Nukavarapu


"BlockVerificationFailures" and "BlocksVerified" show up as 0 in Datanode JMX
Looking at the code, i see below description.
{noformat}
`BlockVerificationFailures` | Total number of verifications failures | 
`BlocksVerified` | Total number of blocks verified | 
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Branch merges and 3.0.0-beta1 scope

[Andrew Wang]

Glad to see the discussion continued in my absence :)

>From a release management perspective, it's *extremely* reasonable to block
the inclusion of new features a month from the planned release date. A
typical software development lifecycle includes weeks of feature freeze and
weeks of code freeze. It is no knock on any developer or any feature to say
that we should not include something in 3.0.0.

I've been very open and clear about the goals, schedule, and scope of 3.0.0
over the last year plus. The point of the extended alpha process was to get
all our features in during alpha, and the alpha merge window has been open
for a year. I'm unmoved by arguments about how long a feature has been
worked on. None of these were not part of the original 3.0.0 scope, and our
users have been waiting even longer for big-ticket 3.0 items like JDK8 and
HDFS EC that were part of the discussed scope.

I see that two VOTEs have gone out since I was out. I still plan to follow
the proposal in my original email. This means I'll cut branch-3 and
branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will open
up development for Hadoop 3.1.0 and 4.0.0.

I'm reaching out to the lead contributor of each of these features
individually to discuss. We need to close on this quickly, and email is too
low bandwidth at this stage.

Best,
Andrew

Re: [VOTE] Merge HADOOP-13345 (S3Guard feature branch)

[Mingliang Liu]

Thanks Andrew. Arpit also told me about this but I forgot to bring it up here.

Best,

> On Aug 24, 2017, at 10:59 AM, Andrew Wang  wrote:
> 
> FYI that committer +1s are binding on merges, so Sean and Mingliang's +1s
> can be upgraded to binding.
> 
> On Thu, Aug 24, 2017 at 6:09 AM, Kihwal Lee  wrote:
> 
>> +1 (binding)
>> Great work guys!
>> 
>> On Thu, Aug 24, 2017 at 5:01 AM, Steve Loughran 
>> wrote:
>> 
>>> 
>>> On 23 Aug 2017, at 19:21, Aaron Fabbri mailto:fa
>>> b...@cloudera.com>> wrote:
>>> 
>>> 
>>> On Tue, Aug 22, 2017 at 10:24 AM, Steve Loughran > <
>>> mailto:ste...@hortonworks.com>> wrote:
>>> video being processed:  https://www.youtube.com/watch?
>>> v=oIe5Zl2YsLE&feature=youtu.be
>>> 
>>> 
>>> Awesome demo Steve, thanks for doing this.  Particularly glad to see
>> folks
>>> using and extending the failure injection client.
>>> 
>>> The HADOOP-13786 iteration turns on throttle event generation. All the
>> new
>>> committer stuff is ready for it, but all the existing S3A FS ops react
>> to a
>>> throttle exception by failing, when they need to just back off a bit.
>> This
>>> complicates testing as I have to explicitly turn off fault injection for
>>> setup & teardown
>>> 
>>> 
>>> Demoing the CLI tool was great as well.
>>> 
>>> 
>>> I'm going to have to do another iteration on that CLI tool post-merge, as
>>> I had one big problem: working out if the bucket and all the binding
>>> settings meant it was "guarded". I think we'll need to track what issues
>>> like that crop up in the field and add the diagnostics/other options.
>>> 
>>> +I think another one that'd be useful would be to enum all s3guard DDB
>>> tables in a region/globally & list their allocated IOPs. I know the AWS
>> UI
>>> can list tables by region, but you need to look around every region to
>> find
>>> out if you've accidentally created one. If you enum all table & look for
>> a
>>> s3guard version marker, then you can identify tables.
>>> 
>>> Wanted to mention two things:
>>> 
>>> 1. Authoritative mode is not fully implemented yet with Dynamo (it needs
>>> to persist an extra bit for directories).  I do have an auth-mode patch
>>> (done for a hackathon) that I need to post which shows large performance
>>> improvements over what S3Guard has today.  As you said, we don't consider
>>> authoritative mode ready for production yet: we want to play with it more
>>> and improve the prune algorithm first.  Authoritative mode can be thought
>>> of as a nice bonus in the future: The main goal of S3Guard v1 is to fix
>> the
>>> get / list consistency issues you mentioned, which it does well.
>>> 
>>> 
>>> we need to call that out in the release notes.
>>> 
>>> 2. Also wanted to thank Lei (Eddy) Xu, he was very active during early
>>> design and contributed some patches as well.
>>> 
>>> 
>>> good point. Lei: you will get a special mention the next time I do the
>> demo
>>> 
>>> 
>>> Again, great demo, enjoyed it!
>>> 
>>> -AF
>>> 
>>> 
>>> its actually quite hard to show any benefits of s3guard on the command
>>> line, so I've ended up showing some scala tests where I turn on the
>>> (bundled) inconsistent AWS client to show how you then need to enable
>>> s3guard to make the stack traces go away
>>> 
>>> 
>>> On 22 Aug 2017, at 11:17, Steve Loughran >> ste...@hortonworks.com>> ste...@hortonworks.com>>> wrote:
>>> 
>>> +1 (binding)
>>> 
>>> I'm happy with it; it's a great piece of work by (in no particular
>> order):
>>> Chris Nauroth, Aaron Fabbri, Sean McRory & Mingliang Liu. plus a few bits
>>> in the corners where I got to break things while they were all asleep.
>> Also
>>> deserving a mention: Thomas Demoor & Ewan Higgs @ WDC for consultancy on
>>> the corners of S3, everyone who tested in (including our QA team), Sanjay
>>> Radia, & others.
>>> 
>>> I've already done a couple of iterations of fixing checksyles & code
>>> reviews, so I think it is ready. I also have a branch-2 patch based on
>>> earlier work by Mingliang, for people who want that.
>>> 
>>> 
>>> 
>>> 
>>> On 17 Aug 2017, at 23:07, Aaron Fabbri mailto:fa
>>> b...@cloudera.com> 
>>> wrote:
>>> 
>>> Hello,
>>> 
>>> I'd like to open a vote (7 days, ending August 24 at 3:10 PST) to merge
>> the
>>> HADOOP-13345 feature branch into trunk.
>>> 
>>> This branch contains the new S3Guard feature which adds metadata
>>> consistency features to the S3A client.  Formatted site documentation can
>>> be found here:
>>> 
>>> https://github.com/apache/hadoop/blob/HADOOP-13345/
>>> hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
>>> 
>>> The current patch against trunk is posted here:
>>> 
>>> https://issues.apache.org/jira/browse/HADOOP-13998
>>> 
>>> The branch modifies the s3a portion of the hadoop-tools/hadoop-aws
>> module:
>>> 
>>> - The feature is off by default, and care has been taken to insure it has
>>> no impact when disabled.
>>> - S3Guard can be enable

[jira] [Created] (HADOOP-14807) should prevent the possibility of NPE about ReconfigurableBase.java

[hu xiaodong (JIRA)]

hu xiaodong created HADOOP-14807:


 Summary: should prevent the possibility of  NPE about 
ReconfigurableBase.java
 Key: HADOOP-14807
 URL: https://issues.apache.org/jira/browse/HADOOP-14807
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: hu xiaodong
Assignee: hu xiaodong
Priority: Minor


1.
{code:title=ReconfigurationThread.java|borderStyle=solid}  
  protected String reconfigurePropertyImpl(String property, String newVal)
  throws ReconfigurationException {
final DatanodeManager datanodeManager = namesystem.getBlockManager()
.getDatanodeManager();

if (property.equals(DFS_HEARTBEAT_INTERVAL_KEY)) {
  return reconfHeartbeatInterval(datanodeManager, property, newVal);
} else if (property.equals(DFS_NAMENODE_HEARTBEAT_RECHECK_INTERVAL_KEY)) {
  return reconfHeartbeatRecheckInterval(datanodeManager, property, newVal);
} else if (property.equals(FS_PROTECTED_DIRECTORIES)) {
  return reconfProtectedDirectories(newVal);
} else if (property.equals(HADOOP_CALLER_CONTEXT_ENABLED_KEY)) {
  return reconfCallerContextEnabled(newVal);
} else if (property.equals(ipcClientRPCBackoffEnable)) {
  return reconfigureIPCBackoffEnabled(newVal);
} 
   //===
   //here may throw a ReconfigurationException which getCause() is null
   //===
   else {
  throw new ReconfigurationException(property, newVal, getConf().get(
  property));
}
  }
{code}
2.
{code:title=ReconfigurationThread.java|borderStyle=solid}  
private static class ReconfigurationThread extends Thread {
private ReconfigurableBase parent;

ReconfigurationThread(ReconfigurableBase base) {
  this.parent = base;
}

// See {@link ReconfigurationServlet#applyChanges}
public void run() {
  LOG.info("Starting reconfiguration task.");
  final Configuration oldConf = parent.getConf();
  final Configuration newConf = parent.getNewConf();
  final Collection changes =
  parent.getChangedProperties(newConf, oldConf);
  Map> results = Maps.newHashMap();
  ConfigRedactor oldRedactor = new ConfigRedactor(oldConf);
  ConfigRedactor newRedactor = new ConfigRedactor(newConf);
  for (PropertyChange change : changes) {
String errorMessage = null;
String oldValRedacted = oldRedactor.redact(change.prop, change.oldVal);
String newValRedacted = newRedactor.redact(change.prop, change.newVal);
if (!parent.isPropertyReconfigurable(change.prop)) {
  LOG.info(String.format(
  "Property %s is not configurable: old value: %s, new value: %s",
  change.prop,
  oldValRedacted,
  newValRedacted));
  continue;
}
LOG.info("Change property: " + change.prop + " from \""
+ ((change.oldVal == null) ? "" : oldValRedacted)
+ "\" to \""
+ ((change.newVal == null) ? "" : newValRedacted)
+ "\".");
try {
  String effectiveValue =
  parent.reconfigurePropertyImpl(change.prop, change.newVal);
  if (change.newVal != null) {
oldConf.set(change.prop, effectiveValue);
  } else {
oldConf.unset(change.prop);
  }
} catch (ReconfigurationException e) {
  //===
  // here may occurs NPE,  because  e.getCause() may be null.
  //===
  errorMessage = e.getCause().getMessage();
}
results.put(change, Optional.fromNullable(errorMessage));
  }

  synchronized (parent.reconfigLock) {
parent.endTime = Time.now();
parent.status = Collections.unmodifiableMap(results);
parent.reconfigThread = null;
  }
}
  }
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Branch merges and 3.0.0-beta1 scope

[Rohith Sharma K S]

Hi Andrew

Thanks for update on release plan!

I would like to discuss specifically regarding compatibility of releases.
What is the compatibility to be maintained for GA if we don't merge to
beta1 release? IIUC, till now all the releases were alpha where
compatibility was not that important. All the public interfaces are
subjected to modifications. Once we release beta, compatibility would be a
matter.
During this gap i.e between beta-GA release, should we maintain
compatibility ?
If my understanding is right then TSv2 have to be merged with beta1
release. In TSv2 phase-2, we have compatibility changes from phase-1.


Thanks & Regards
Rohith Sharma K S

On 25 August 2017 at 02:03, Andrew Wang  wrote:

> Glad to see the discussion continued in my absence :)
>
> From a release management perspective, it's *extremely* reasonable to block
> the inclusion of new features a month from the planned release date. A
> typical software development lifecycle includes weeks of feature freeze and
> weeks of code freeze. It is no knock on any developer or any feature to say
> that we should not include something in 3.0.0.
>
> I've been very open and clear about the goals, schedule, and scope of 3.0.0
> over the last year plus. The point of the extended alpha process was to get
> all our features in during alpha, and the alpha merge window has been open
> for a year. I'm unmoved by arguments about how long a feature has been
> worked on. None of these were not part of the original 3.0.0 scope, and our
> users have been waiting even longer for big-ticket 3.0 items like JDK8 and
> HDFS EC that were part of the discussed scope.
>
> I see that two VOTEs have gone out since I was out. I still plan to follow
> the proposal in my original email. This means I'll cut branch-3 and
> branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will open
> up development for Hadoop 3.1.0 and 4.0.0.
>
> I'm reaching out to the lead contributor of each of these features
> individually to discuss. We need to close on this quickly, and email is too
> low bandwidth at this stage.
>
> Best,
> Andrew
>