Re: [Discuss] - HDDS-4 Branch merge

[Mukul Kumar Singh]

+1, This is a great feature addition to Ozone.

Thanks,
Mukul

On 1/9/19, 12:52 PM, "Jitendra Pandey"  wrote:

+1, This is a great effort and another milestone in making HDDS and Ozone 
ready for enterprises.

On 1/7/19, 5:10 PM, "Anu Engineer"  wrote:

Hi All,

I would like to propose a merge of HDDS-4 branch to the Hadoop trunk.
HDDS-4 branch implements the security work for HDDS and Ozone.

HDDS-4 branch contains the following features:
- Hadoop Kerberos and Tokens support
- A Certificate infrastructure used by Ozone and HDDS.
- Audit Logging and parsing support (Spread across trunk and HDDS-4)
- S3 Security Support - AWS Signature Support.
- Apache Ranger Support for Ozone

I will follow up with a formal vote later this week if I hear no
objections. AFAIK, the changes are isolated to HDDS/Ozone and should not
impact any other Hadoop project.

Thanks
Anu



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC1

[Wilfred Spiegelenburg]

+1 (non binding)

- build from source on MacOSX 10.14.2, 1.8.0u181
- successful native build on Ubuntu 16.04.3
- confirmed the checksum and signature
- deployed a single node cluster  (openjdk 1.8u191 / centos 7.5)
- uploaded the MR framework
- configured YARN with the FS
- ran multiple MR jobs

> On 8 Jan 2019, at 22:42, Sunil G  wrote:
> 
> Hi folks,
> 
> 
> Thanks to all of you who helped in this release [1] and for helping to vote
> for RC0. I have created second release candidate (RC1) for Apache Hadoop
> 3.2.0.
> 
> 
> Artifacts for this RC are available here:
> 
> http://home.apache.org/~sunilg/hadoop-3.2.0-RC1/
> 
> 
> RC tag in git is release-3.2.0-RC1.
> 
> 
> 
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1178/
> 
> 
> This vote will run 7 days (5 weekdays), ending on 14th Jan at 11:59 pm PST.
> 
> 
> 
> 3.2.0 contains 1092 [2] fixed JIRA issues since 3.1.0. Below feature
> additions
> 
> are the highlights of this release.
> 
> 1. Node Attributes Support in YARN
> 
> 2. Hadoop Submarine project for running Deep Learning workloads on YARN
> 
> 3. Support service upgrade via YARN Service API and CLI
> 
> 4. HDFS Storage Policy Satisfier
> 
> 5. Support Windows Azure Storage - Blob file system in Hadoop
> 
> 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
> 
> 7. Improvements in Router-based HDFS federation
> 
> 
> 
> Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
> 
> I have done few testing with my pseudo cluster. My +1 to start.
> 
> 
> 
> Regards,
> 
> Sunil
> 
> 
> 
> [1]
> 
> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
> 
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
> AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
> ORDER BY fixVersion ASC


Wilfred Spiegelenburg | Software Engineer
cloudera.com 

[jira] [Created] (HADOOP-16041) UserAgent string for ABFS

[Shweta (JIRA)]

Shweta created HADOOP-16041:
---

 Summary: UserAgent string for ABFS
 Key: HADOOP-16041
 URL: https://issues.apache.org/jira/browse/HADOOP-16041
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/azure
Affects Versions: 3.2.0
Reporter: Shweta
Assignee: Shweta
 Fix For: 3.3.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC1

[Eric Badger]

+1 (non-binding)

- Verified all hashes and checksums
- Built from source on RHEL 7
- Deployed a single node pseudo cluster
- Ran some example jobs
- Verified the Docker environment works with non-entrypoint mode

Eric

On Tue, Jan 8, 2019 at 5:42 AM Sunil G  wrote:

> Hi folks,
>
>
> Thanks to all of you who helped in this release [1] and for helping to vote
> for RC0. I have created second release candidate (RC1) for Apache Hadoop
> 3.2.0.
>
>
> Artifacts for this RC are available here:
>
> http://home.apache.org/~sunilg/hadoop-3.2.0-RC1/
>
>
> RC tag in git is release-3.2.0-RC1.
>
>
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1178/
>
>
> This vote will run 7 days (5 weekdays), ending on 14th Jan at 11:59 pm PST.
>
>
>
> 3.2.0 contains 1092 [2] fixed JIRA issues since 3.1.0. Below feature
> additions
>
> are the highlights of this release.
>
> 1. Node Attributes Support in YARN
>
> 2. Hadoop Submarine project for running Deep Learning workloads on YARN
>
> 3. Support service upgrade via YARN Service API and CLI
>
> 4. HDFS Storage Policy Satisfier
>
> 5. Support Windows Azure Storage - Blob file system in Hadoop
>
> 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
>
> 7. Improvements in Router-based HDFS federation
>
>
>
> Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
>
> I have done few testing with my pseudo cluster. My +1 to start.
>
>
>
> Regards,
>
> Sunil
>
>
>
> [1]
>
>
> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
> AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
> ORDER BY fixVersion ASC
>

[jira] [Resolved] (HADOOP-15934) ABFS: make retry policy configurable

[Da Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Da Zhou resolved HADOOP-15934.
--
Resolution: Won't Fix

> ABFS: make retry policy configurable
> 
>
> Key: HADOOP-15934
> URL: https://issues.apache.org/jira/browse/HADOOP-15934
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Da Zhou
>Assignee: Da Zhou
>Priority: Major
>  Labels: won't-fix
>
> Currently the retry policy parameter is hard coded, should make it 
> configurable for user.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: getting Yetus and github to be friends

[Steve Loughran]

> On 9 Jan 2019, at 18:51, Sean Busbey  wrote:
> 
> Unfortunately I think we're a little late to the game to have this be
> much more than "what does AW have to do".
> 
> The upcoming Yetus 0.9.0 release includes a pretty huge reworking of
> Yetus to make it work better in non-ASF settings; including getting
> the GitHub PR handling updated to work with projects that don't expect
> JIRA to be a thing. The subsystem has changed enough that I'd be
> surprised if users of the previous plugin finding problems would be
> helpful.
> 

yes, I've seen the 0.90 release will fix this, and AW has a PoC PR up: 
https://github.com/apache/hadoop/pull/459

There's a new github account, hadoop-yetus for this.

> That isn't to say more help wouldn't be appreciated. For my part I'm
> dedicating time this week to more reviews to help move AW's work along
> marginally faster. I'll also be pushing to get feedback to Yetus about
> how the new release  impacts the older projects here in ASF land. I'd
> be thrilled if anyone else in the Hadoop community wanted to do that
> work. Some of it'll be using the RC on local dev machines. Some will
> require ASF Jenkins access; but literally any committer can be granted
> access to that by the Hadoop PMC.

OK

> 
> On Thu, Jan 3, 2019 at 4:28 AM Steve Loughran  wrote:
>> 
>> 
>> 
>>> On 3 Jan 2019, at 01:22, lqjacklee  wrote:
>>> 
>>> Thanks Steve, I like the style PR from the GitHub. when should we start to
>>> change it ?
>> 
>> 
>> Step 1 is having Yetus review github PRs associated with a JIRA. Without 
>> that part of the process, we can't begin to use github for reviewing.
>> 
>> For a long time we were on yetus 0.5.0, which I believe lagged here. But 
>> since september, Hadoop has been on Yetus 0.8.0 -the latest release 
>> (HADOOP-14761). If there's more in there for github PR review we should 
>> enable it. And if we need more, well, its an ASF project: we all have the 
>> right to submit patches there too, the duty to test them first
>> 
>> -steve
>> 
>> 
>>> 
>>> On Thu, Jan 3, 2019 at 7:07 AM Steve Loughran 
>>> wrote:
>>> 
 
 The new gitbox repo apparently does 2 way linking from github: you can
 commit a PR there and it'll make its way back
 
 this could be really slick -and do a big change to our review process.
 
 Before we can go near it though, we need to get Yetus doing its review &
 test of github PRs, which is not working right now
 
 What will it take to do that? And that means not "what does AW have to
 do", but "how can we help get this done?"
 
 -steve
 
 
 -
 To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
 For additional commands, e-mail: common-dev-h...@hadoop.apache.org
 
 
>> 
>> 
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> 
> 
> 
> -- 
> busbey
> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: getting Yetus and github to be friends

[Sean Busbey]

Unfortunately I think we're a little late to the game to have this be
much more than "what does AW have to do".

The upcoming Yetus 0.9.0 release includes a pretty huge reworking of
Yetus to make it work better in non-ASF settings; including getting
the GitHub PR handling updated to work with projects that don't expect
JIRA to be a thing. The subsystem has changed enough that I'd be
surprised if users of the previous plugin finding problems would be
helpful.

That isn't to say more help wouldn't be appreciated. For my part I'm
dedicating time this week to more reviews to help move AW's work along
marginally faster. I'll also be pushing to get feedback to Yetus about
how the new release  impacts the older projects here in ASF land. I'd
be thrilled if anyone else in the Hadoop community wanted to do that
work. Some of it'll be using the RC on local dev machines. Some will
require ASF Jenkins access; but literally any committer can be granted
access to that by the Hadoop PMC.

On Thu, Jan 3, 2019 at 4:28 AM Steve Loughran  wrote:
>
>
>
> > On 3 Jan 2019, at 01:22, lqjacklee  wrote:
> >
> > Thanks Steve, I like the style PR from the GitHub. when should we start to
> > change it ?
>
>
> Step 1 is having Yetus review github PRs associated with a JIRA. Without that 
> part of the process, we can't begin to use github for reviewing.
>
> For a long time we were on yetus 0.5.0, which I believe lagged here. But 
> since september, Hadoop has been on Yetus 0.8.0 -the latest release 
> (HADOOP-14761). If there's more in there for github PR review we should 
> enable it. And if we need more, well, its an ASF project: we all have the 
> right to submit patches there too, the duty to test them first
>
> -steve
>
>
> >
> > On Thu, Jan 3, 2019 at 7:07 AM Steve Loughran 
> > wrote:
> >
> >>
> >> The new gitbox repo apparently does 2 way linking from github: you can
> >> commit a PR there and it'll make its way back
> >>
> >> this could be really slick -and do a big change to our review process.
> >>
> >> Before we can go near it though, we need to get Yetus doing its review &
> >> test of github PRs, which is not working right now
> >>
> >> What will it take to do that? And that means not "what does AW have to
> >> do", but "how can we help get this done?"
> >>
> >> -steve
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >>
> >>
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>


-- 
busbey

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16040) ABFS: Bug fix for tolerateOobAppends configuration

[Da Zhou (JIRA)]

Da Zhou created HADOOP-16040:


 Summary: ABFS: Bug fix for tolerateOobAppends configuration
 Key: HADOOP-16040
 URL: https://issues.apache.org/jira/browse/HADOOP-16040
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/azure
Affects Versions: 3.2.0
Reporter: Da Zhou
Assignee: Da Zhou


Cause: configuration for tolerateOobAppends is never used.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1011/

[Jan 8, 2019 5:51:30 AM] (aajisaka) Revert "HADOOP-14556. S3A to support 
Delegation Tokens."
[Jan 8, 2019 6:04:06 AM] (bharat) HDDS-926. Use Timeout rule for the the test 
methods in TestOzoneManager.
[Jan 8, 2019 6:30:53 AM] (wwei) YARN-9037. [CSI] Ignore volume resource in 
resource calculators based on
[Jan 8, 2019 11:57:57 AM] (stevel) HADOOP-16018. DistCp won't reassemble chunks 
when blocks per chunk > 0.
[Jan 8, 2019 1:54:38 PM] (elek) HDDS-968. Fix TestObjectPut failures. 
Contributed by Bharat Viswanadham.
[Jan 8, 2019 2:12:08 PM] (elek) HDDS-969. Fix TestOzoneManagerRatisServer test 
failure. Contributed by
[Jan 8, 2019 2:48:48 PM] (elek) HDDS-924. MultipartUpload: S3 APi for complete 
Multipart Upload.
[Jan 8, 2019 6:38:06 PM] (gifuma) HDFS-14189. Fix intermittent failure of 
TestNameNodeMetrics. Contributed
[Jan 8, 2019 9:27:19 PM] (elek) HDDS-965. Ozone: checkstyle improvements and 
code quality scripts.
[Jan 8, 2019 10:54:05 PM] (jlowe) YARN-6523. Optimize system credentials sent 
in node heartbeat responses.
[Jan 9, 2019 1:04:57 AM] (templedf) HDFS-14132. Add BlockLocation.isStriped() 
to determine if block is




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.util.TestDiskCheckerWithDiskIo 
   hadoop.util.TestReadWriteDiskValidator 
   hadoop.security.ssl.TestSSLFactory 
   hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.server.resourcemanager.TestResourceTrackerService 
   hadoop.yarn.server.resourcemanager.security.TestAMRMTokens 
   hadoop.mapreduce.security.ssl.TestEncryptedShuffle 
   hadoop.mapred.TestReduceFetchFromPartialMem 
   hadoop.mapreduce.TestYarnClientProtocolProvider 
   hadoop.mapred.TestMRIntermediateDataEncryption 
   hadoop.mapreduce.TestMapReduceLazyOutput 
   hadoop.mapreduce.v2.TestMRJobs 
   hadoop.mapred.TestReduceFetch 
   hadoop.mapred.TestLazyOutput 
   hadoop.mapred.TestClusterMapReduceTestCase 
   hadoop.mapreduce.v2.TestMiniMRProxyUser 
   hadoop.mapreduce.TestMRJobClient 
   hadoop.mapred.TestMerge 
   hadoop.mapreduce.security.TestMRCredentials 
   hadoop.mapreduce.security.TestBinaryTokenFile 
   hadoop.mapred.TestJobName 
   hadoop.yarn.service.TestServiceAM 
   hadoop.streaming.TestFileArgs 
   hadoop.streaming.TestMultipleCachefiles 
   hadoop.streaming.TestSymLink 
   hadoop.streaming.TestMultipleArchiveFiles 
   hadoop.mapred.gridmix.TestDistCacheEmulation 
   hadoop.mapred.gridmix.TestGridmixSubmission 
   hadoop.mapred.gridmix.TestSleepJob 
   hadoop.mapred.gridmix.TestLoadJob 
   hadoop.tools.TestDistCh 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1011/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1011/artifact/out/diff-compile-javac-root.txt
  [336K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1011/artifact/out/diff-checkstyle-root.txt
  [17M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1011/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1011/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1011/artifact/out/diff-patch-pylint.txt
  [60K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1011/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1011/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1011/artifact/out/whitespace-eol.txt
  [9.3M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1011/artifact/out/whitespace-tabs.txt
  [1.1M]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1011/artifact/out/branch-findbugs-hadoop-hdds_client.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1011/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1011/artifact/out/branch-findbugs-hadoop-hdds_framework.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1011/

[jira] [Created] (HADOOP-16039) backport HADOOP-15965, "Upgrade ADLS SDK to 2.3.3" to branch-2

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-16039:
---

 Summary: backport HADOOP-15965, "Upgrade ADLS SDK to 2.3.3" to 
branch-2
 Key: HADOOP-16039
 URL: https://issues.apache.org/jira/browse/HADOOP-16039
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs/azure
Affects Versions: 2.9.2
Reporter: Steve Loughran
Assignee: Steve Loughran


Backport the ADLS SDK 2.3.3 update to branch-2, retest. etc



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org