Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Junping Du]

Thanks all for response with verification work and vote!


Sounds like we are hitting several issues here, although none seems to be 
blockers so far. Given the large commit set - 2000+ commits first landed in 
branch-2 release, we may should follow 2.7.0 practice that to claim this 
release is not for production cluster, just like Vinod's suggestion in previous 
email. We should quickly come up with 2.8.1 release in next 1 or 2 month for 
production deployment.


We will close the vote in next 24 hours. For people who haven't vote, please 
keep on verification work and report any issues if founded - I will check if 
another round of RC is needed based on your findings. Thanks!


Thanks,


Junping



From: Kuhu Shukla 
Sent: Tuesday, March 21, 2017 3:17 PM
Cc: Junping Du; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)


+1 (non-binding)

- Verified signatures.
- Downloaded and built from source tar.gz.
- Deployed a pseudo-distributed cluster on Mac Sierra.
- Ran example Sleep job successfully.
- Deployed latest Apache Tez 0.9 and ran sample Tez orderedwordcount 
successfully.

Thank you Junping and everyone else who worked on getting this release out.

Warm Regards,
Kuhu
On Tuesday, March 21, 2017, 3:42:46 PM CDT, Eric Badger 
 wrote:
+1 (non-binding)

- Verified checksums and signatures of all files
- Built from source on MacOS Sierra via JDK 1.8.0 u65
- Deployed single-node cluster
- Successfully ran a few sample jobs

Thanks,

Eric

On Tuesday, March 21, 2017 2:56 PM, John Zhuge  wrote:



+1. Thanks for the great effort, Junping!


  - Verified checksums and signatures of the tarballs
  - Built source code with Java 1.8.0_66-b17 on Mac OS X 10.12.3
  - Built source and native code with Java 1.8.0_111 on Centos 7.2.1511
  - Cloud connectors:
  - s3a: integration tests, basic fs commands
  - adl: live unit tests, basic fs commands. See notes below.
  - Deployed a pseudo cluster, passed the following sanity tests in
  both insecure and SSL mode:
  - HDFS: basic dfs, distcp, ACL commands
  - KMS and HttpFS: basic tests
  - MapReduce wordcount
  - balancer start/stop


Needs the following JIRAs to pass all ADL tests:

  - HADOOP-14205. No FileSystem for scheme: adl. Contributed by John Zhuge.
  - HDFS-11132. Allow AccessControlException in contract tests when
  getFileStatus on subdirectory of existing files. Contributed by Vishwajeet
  Dusane
  - HADOOP-13928. TestAdlFileContextMainOperationsLive.testGetFileContext1
  runtime error. (John Zhuge via lei)


On Mon, Mar 20, 2017 at 10:31 AM, John Zhuge  wrote:

> Yes, it only affects ADL. There is a workaround of adding these 2
> properties to core-site.xml:
>
>  
>fs.adl.impl
>org.apache.hadoop.fs.adl.AdlFileSystem
>  
>
>  
>fs.AbstractFileSystem.adl.impl
>org.apache.hadoop.fs.adl.Adl
>  
>
> I have the initial patch ready but hitting these live unit test failures:
>
> Failed tests:
>
> TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
> testListStatus:257
> expected:<1> but was:<10>
>
> Tests in error:
>
> TestAdlFileContextMainOperationsLive>FileContextMainOperationsBaseTest.
> testMkdirsFailsForSubdirectoryOfExistingFile:254
> » AccessControl
>
> TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
> testMkdirsFailsForSubdirectoryOfExistingFile:190
> » AccessControl
>
>
> Stay tuned...
>
> John Zhuge
> Software Engineer, Cloudera
>
> On Mon, Mar 20, 2017 at 10:02 AM, Junping Du  wrote:
>
> > Thank you for reporting the issue, John! Does this issue only affect ADL
> > (Azure Data Lake) which is a new feature for 2.8 rather than other
> existing
> > FS? If so, I think we can leave the fix to 2.8.1 to fix given this is
> not a
> > regression and just a new feature get broken.?
> >
> >
> > Thanks,
> >
> >
> > Junping
> > --
> > *From:* John Zhuge 
> > *Sent:* Monday, March 20, 2017 9:07 AM
> > *To:* Junping Du
> > *Cc:* common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> > yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
> > *Subject:* Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
> >
> > Discovered https://issues.apache.org/jira/browse/HADOOP-14205 "No
> > FileSystem for scheme: adl".
> >
> > The issue were caused by backporting HADOOP-13037 to branch-2 and
> earlier.
> > HADOOP-12666 should not be backported, but some changes are needed:
> > property fs.adl.impl in core-default.xml and hadoop-tools-dist/pom.xml.
> >
> > I am working on a patch.
> >
> >
> > John Zhuge
> > Software Engineer, Cloudera
> >
> > On Fri, Mar 17, 2017 at 2:18 AM, Junping Du  wrote:
> >
> >> Hi all,
> >>  With fix of HDFS-11431 get in, I've 

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Kuhu Shukla]

+1 (non-binding)

- Verified signatures.
- Downloaded and built from source tar.gz.
- Deployed a pseudo-distributed cluster on Mac Sierra.
- Ran example Sleep job successfully.
- Deployed latest Apache Tez 0.9 and ran sample Tez orderedwordcount 
successfully.

Thank you Junping and everyone else who worked on getting this release out.

Warm Regards,
Kuhu
On Tuesday, March 21, 2017, 3:42:46 PM CDT, Eric Badger 
 wrote:
+1 (non-binding)

- Verified checksums and signatures of all files
- Built from source on MacOS Sierra via JDK 1.8.0 u65
- Deployed single-node cluster
- Successfully ran a few sample jobs

Thanks,

Eric

On Tuesday, March 21, 2017 2:56 PM, John Zhuge  wrote:



+1. Thanks for the great effort, Junping!


  - Verified checksums and signatures of the tarballs
  - Built source code with Java 1.8.0_66-b17 on Mac OS X 10.12.3
  - Built source and native code with Java 1.8.0_111 on Centos 7.2.1511
  - Cloud connectors:
      - s3a: integration tests, basic fs commands
      - adl: live unit tests, basic fs commands. See notes below.
  - Deployed a pseudo cluster, passed the following sanity tests in
  both insecure and SSL mode:
      - HDFS: basic dfs, distcp, ACL commands
      - KMS and HttpFS: basic tests
      - MapReduce wordcount
      - balancer start/stop


Needs the following JIRAs to pass all ADL tests:

  - HADOOP-14205. No FileSystem for scheme: adl. Contributed by John Zhuge.
  - HDFS-11132. Allow AccessControlException in contract tests when
  getFileStatus on subdirectory of existing files. Contributed by Vishwajeet
  Dusane
  - HADOOP-13928. TestAdlFileContextMainOperationsLive.testGetFileContext1
  runtime error. (John Zhuge via lei)


On Mon, Mar 20, 2017 at 10:31 AM, John Zhuge  wrote:

> Yes, it only affects ADL. There is a workaround of adding these 2
> properties to core-site.xml:
>
>  
>    fs.adl.impl
>    org.apache.hadoop.fs.adl.AdlFileSystem
>  
>
>  
>    fs.AbstractFileSystem.adl.impl
>    org.apache.hadoop.fs.adl.Adl
>  
>
> I have the initial patch ready but hitting these live unit test failures:
>
> Failed tests:
>
> TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
> testListStatus:257
> expected:<1> but was:<10>
>
> Tests in error:
>
> TestAdlFileContextMainOperationsLive>FileContextMainOperationsBaseTest.
> testMkdirsFailsForSubdirectoryOfExistingFile:254
> » AccessControl
>
> TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
> testMkdirsFailsForSubdirectoryOfExistingFile:190
> » AccessControl
>
>
> Stay tuned...
>
> John Zhuge
> Software Engineer, Cloudera
>
> On Mon, Mar 20, 2017 at 10:02 AM, Junping Du  wrote:
>
> > Thank you for reporting the issue, John! Does this issue only affect ADL
> > (Azure Data Lake) which is a new feature for 2.8 rather than other
> existing
> > FS? If so, I think we can leave the fix to 2.8.1 to fix given this is
> not a
> > regression and just a new feature get broken.​
> >
> >
> > Thanks,
> >
> >
> > Junping
> > --
> > *From:* John Zhuge 
> > *Sent:* Monday, March 20, 2017 9:07 AM
> > *To:* Junping Du
> > *Cc:* common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> > yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
> > *Subject:* Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
> >
> > Discovered https://issues.apache.org/jira/browse/HADOOP-14205 "No
> > FileSystem for scheme: adl".
> >
> > The issue were caused by backporting HADOOP-13037 to branch-2 and
> earlier.
> > HADOOP-12666 should not be backported, but some changes are needed:
> > property fs.adl.impl in core-default.xml and hadoop-tools-dist/pom.xml.
> >
> > I am working on a patch.
> >
> >
> > John Zhuge
> > Software Engineer, Cloudera
> >
> > On Fri, Mar 17, 2017 at 2:18 AM, Junping Du  wrote:
> >
> >> Hi all,
> >>      With fix of HDFS-11431 get in, I've created a new release candidate
> >> (RC3) for Apache Hadoop 2.8.0.
> >>
> >>      This is the next minor release to follow up 2.7.0 which has been
> >> released for more than 1 year. It comprises 2,900+ fixes, improvements,
> and
> >> new features. Most of these commits are released for the first time in
> >> branch-2.
> >>
> >>      More information about the 2.8.0 release plan can be found here:
> >> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
> >>
> >>      New RC is available at: http://home.apache.org/~junpin
> >> g_du/hadoop-2.8.0-RC3
> >>
> >>      The RC tag in git is: release-2.8.0-RC3, and the latest commit id
> >> is: 91f2b7a13d1e97be65db92ddabc627cc29ac0009
> >>
> >>      The maven artifacts are available via repository.apache.org at:
> >> https://repository.apache.org/content/repositories/orgapachehadoop-1057
> >>
> >>      Please try the release and vote; the vote will run for the usual 5
> >> days, ending on 03/22/2017 PDT time.

> >>
> >> Thanks,
> >>
> >> 

[jira] [Created] (YARN-6372) Add default value for NM disk validator

[Yufei Gu (JIRA)]

Yufei Gu created YARN-6372:
--

 Summary: Add default value for NM disk validator
 Key: YARN-6372
 URL: https://issues.apache.org/jira/browse/YARN-6372
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: nodemanager
Affects Versions: 3.0.0-alpha2, 2.7.3
Reporter: Yufei Gu
Assignee: Yufei Gu


YARN-5137 make DiskChecker pluggable in NodeManager. We should give a default 
value in case NM does't provide the configuration item.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-6371) NodeHealthCheckerService member vars should be final

[Daniel Templeton (JIRA)]

Daniel Templeton created YARN-6371:
--

 Summary: NodeHealthCheckerService member vars should be final
 Key: YARN-6371
 URL: https://issues.apache.org/jira/browse/YARN-6371
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: nodemanager
Affects Versions: 3.0.0-alpha2
Reporter: Daniel Templeton
Assignee: Daniel Templeton
Priority: Trivial


{code}
  private NodeHealthScriptRunner nodeHealthScriptRunner;
  private LocalDirsHandlerService dirsHandler;
{code}

They can both be final.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Eric Badger]

+1 (non-binding)

- Verified checksums and signatures of all files
- Built from source on MacOS Sierra via JDK 1.8.0 u65
- Deployed single-node cluster
- Successfully ran a few sample jobs

Thanks,

Eric

On Tuesday, March 21, 2017 2:56 PM, John Zhuge  wrote:



+1. Thanks for the great effort, Junping!


   - Verified checksums and signatures of the tarballs
   - Built source code with Java 1.8.0_66-b17 on Mac OS X 10.12.3
   - Built source and native code with Java 1.8.0_111 on Centos 7.2.1511
   - Cloud connectors:
  - s3a: integration tests, basic fs commands
  - adl: live unit tests, basic fs commands. See notes below.
   - Deployed a pseudo cluster, passed the following sanity tests in
   both insecure and SSL mode:
  - HDFS: basic dfs, distcp, ACL commands
  - KMS and HttpFS: basic tests
  - MapReduce wordcount
  - balancer start/stop


Needs the following JIRAs to pass all ADL tests:

   - HADOOP-14205. No FileSystem for scheme: adl. Contributed by John Zhuge.
   - HDFS-11132. Allow AccessControlException in contract tests when
   getFileStatus on subdirectory of existing files. Contributed by Vishwajeet
   Dusane
   - HADOOP-13928. TestAdlFileContextMainOperationsLive.testGetFileContext1
   runtime error. (John Zhuge via lei)


On Mon, Mar 20, 2017 at 10:31 AM, John Zhuge  wrote:

> Yes, it only affects ADL. There is a workaround of adding these 2
> properties to core-site.xml:
>
>   
> fs.adl.impl
> org.apache.hadoop.fs.adl.AdlFileSystem
>   
>
>   
> fs.AbstractFileSystem.adl.impl
> org.apache.hadoop.fs.adl.Adl
>   
>
> I have the initial patch ready but hitting these live unit test failures:
>
> Failed tests:
>
> TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
> testListStatus:257
> expected:<1> but was:<10>
>
> Tests in error:
>
> TestAdlFileContextMainOperationsLive>FileContextMainOperationsBaseTest.
> testMkdirsFailsForSubdirectoryOfExistingFile:254
> » AccessControl
>
> TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
> testMkdirsFailsForSubdirectoryOfExistingFile:190
> » AccessControl
>
>
> Stay tuned...
>
> John Zhuge
> Software Engineer, Cloudera
>
> On Mon, Mar 20, 2017 at 10:02 AM, Junping Du  wrote:
>
> > Thank you for reporting the issue, John! Does this issue only affect ADL
> > (Azure Data Lake) which is a new feature for 2.8 rather than other
> existing
> > FS? If so, I think we can leave the fix to 2.8.1 to fix given this is
> not a
> > regression and just a new feature get broken.​
> >
> >
> > Thanks,
> >
> >
> > Junping
> > --
> > *From:* John Zhuge 
> > *Sent:* Monday, March 20, 2017 9:07 AM
> > *To:* Junping Du
> > *Cc:* common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> > yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
> > *Subject:* Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
> >
> > Discovered https://issues.apache.org/jira/browse/HADOOP-14205 "No
> > FileSystem for scheme: adl".
> >
> > The issue were caused by backporting HADOOP-13037 to branch-2 and
> earlier.
> > HADOOP-12666 should not be backported, but some changes are needed:
> > property fs.adl.impl in core-default.xml and hadoop-tools-dist/pom.xml.
> >
> > I am working on a patch.
> >
> >
> > John Zhuge
> > Software Engineer, Cloudera
> >
> > On Fri, Mar 17, 2017 at 2:18 AM, Junping Du  wrote:
> >
> >> Hi all,
> >>  With fix of HDFS-11431 get in, I've created a new release candidate
> >> (RC3) for Apache Hadoop 2.8.0.
> >>
> >>  This is the next minor release to follow up 2.7.0 which has been
> >> released for more than 1 year. It comprises 2,900+ fixes, improvements,
> and
> >> new features. Most of these commits are released for the first time in
> >> branch-2.
> >>
> >>   More information about the 2.8.0 release plan can be found here:
> >> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
> >>
> >>   New RC is available at: http://home.apache.org/~junpin
> >> g_du/hadoop-2.8.0-RC3
> >>
> >>   The RC tag in git is: release-2.8.0-RC3, and the latest commit id
> >> is: 91f2b7a13d1e97be65db92ddabc627cc29ac0009
> >>
> >>   The maven artifacts are available via repository.apache.org at:
> >> https://repository.apache.org/content/repositories/orgapachehadoop-1057
> >>
> >>   Please try the release and vote; the vote will run for the usual 5
> >> days, ending on 03/22/2017 PDT time.

> >>
> >> Thanks,
> >>
> >> Junping
> >>
> >
> >
>



-- 
John

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Allen Wittenauer]

> On Mar 21, 2017, at 10:12 AM, Andrew Wang  wrote:
> 
> I poked around a bit. The 3.0.0-alpha2 binary tarball is only 246M and has
> more changes than 2.8.0.


Not to disclaim any other potential issues, but it's worth noting 3.x de-dupes 
jar files as part of the packaging process.  So it's not exactly an 
apples-to-apples comparison. (Although I think the new yarn-ui made the 
significant loss in excess jars moot.  Without that, I'd expect 3.x to be about 
half the size.)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/264/

[Mar 20, 2017 5:59:09 PM] (varunsaxena) YARN-6353. Clean up OrderingPolicy 
javadoc (Daniel Templeton via Varun
[Mar 20, 2017 6:29:23 PM] (arp) HADOOP-14059. typo in s3a rename(self, subdir) 
error message.
[Mar 20, 2017 9:25:00 PM] (templedf) YARN-6309. Fair scheduler docs should have 
the queue and
[Mar 20, 2017 11:31:24 PM] (liuml07) HADOOP-13945. Azure: Add Kerberos and 
Delegation token support to WASB
[Mar 21, 2017 12:15:13 AM] (cdouglas) HDFS-6648. Order of namenodes in 
ConfiguredFailoverProxyProvider is
[Mar 21, 2017 4:54:58 AM] (arp) HDFS-11545. Propagate DataNode's slow disks 
info to the NameNode via
[Mar 21, 2017 8:16:31 AM] (sunilg) YARN-6362. Use frontend-maven-plugin 0.0.22 
version for new yarn ui.
[Mar 21, 2017 9:44:17 AM] (yqlin) HDFS-11358. DiskBalancer: Report command 
supports reading nodes from




-1 overall


The following subsystems voted -1:
compile unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.security.token.delegation.TestZKDelegationTokenSecretManager 
   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.server.timeline.TestRollingLevelDB 
   hadoop.yarn.server.timeline.TestTimelineDataManager 
   hadoop.yarn.server.timeline.TestLeveldbTimelineStore 
   hadoop.yarn.server.timeline.webapp.TestTimelineWebServices 
   hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore 
   hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore 
   
hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer 
   hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.client.api.async.impl.TestNMClientAsync 
   hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore 
   hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient 
   hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.mapred.TestShuffleHandler 
   hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService 

Timed out junit tests :

   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean 
   org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache 
  

   compile:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/264/artifact/out/patch-compile-root.txt
  [132K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/264/artifact/out/patch-compile-root.txt
  [132K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/264/artifact/out/patch-compile-root.txt
  [132K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/264/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [136K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/264/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [432K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/264/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/264/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [52K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/264/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [68K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/264/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [324K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/264/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/264/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt
  [28K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/264/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/264/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-ui.txt
  [8.0K]
 

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[John Zhuge]

+1. Thanks for the great effort, Junping!


   - Verified checksums and signatures of the tarballs
   - Built source code with Java 1.8.0_66-b17 on Mac OS X 10.12.3
   - Built source and native code with Java 1.8.0_111 on Centos 7.2.1511
   - Cloud connectors:
  - s3a: integration tests, basic fs commands
  - adl: live unit tests, basic fs commands. See notes below.
   - Deployed a pseudo cluster, passed the following sanity tests in
   both insecure and SSL mode:
  - HDFS: basic dfs, distcp, ACL commands
  - KMS and HttpFS: basic tests
  - MapReduce wordcount
  - balancer start/stop


Needs the following JIRAs to pass all ADL tests:

   - HADOOP-14205. No FileSystem for scheme: adl. Contributed by John Zhuge.
   - HDFS-11132. Allow AccessControlException in contract tests when
   getFileStatus on subdirectory of existing files. Contributed by Vishwajeet
   Dusane
   - HADOOP-13928. TestAdlFileContextMainOperationsLive.testGetFileContext1
   runtime error. (John Zhuge via lei)


On Mon, Mar 20, 2017 at 10:31 AM, John Zhuge  wrote:

> Yes, it only affects ADL. There is a workaround of adding these 2
> properties to core-site.xml:
>
>   
> fs.adl.impl
> org.apache.hadoop.fs.adl.AdlFileSystem
>   
>
>   
> fs.AbstractFileSystem.adl.impl
> org.apache.hadoop.fs.adl.Adl
>   
>
> I have the initial patch ready but hitting these live unit test failures:
>
> Failed tests:
>
> TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
> testListStatus:257
> expected:<1> but was:<10>
>
> Tests in error:
>
> TestAdlFileContextMainOperationsLive>FileContextMainOperationsBaseTest.
> testMkdirsFailsForSubdirectoryOfExistingFile:254
> » AccessControl
>
> TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
> testMkdirsFailsForSubdirectoryOfExistingFile:190
> » AccessControl
>
>
> Stay tuned...
>
> John Zhuge
> Software Engineer, Cloudera
>
> On Mon, Mar 20, 2017 at 10:02 AM, Junping Du  wrote:
>
> > Thank you for reporting the issue, John! Does this issue only affect ADL
> > (Azure Data Lake) which is a new feature for 2.8 rather than other
> existing
> > FS? If so, I think we can leave the fix to 2.8.1 to fix given this is
> not a
> > regression and just a new feature get broken.​
> >
> >
> > Thanks,
> >
> >
> > Junping
> > --
> > *From:* John Zhuge 
> > *Sent:* Monday, March 20, 2017 9:07 AM
> > *To:* Junping Du
> > *Cc:* common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> > yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
> > *Subject:* Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
> >
> > Discovered https://issues.apache.org/jira/browse/HADOOP-14205 "No
> > FileSystem for scheme: adl".
> >
> > The issue were caused by backporting HADOOP-13037 to branch-2 and
> earlier.
> > HADOOP-12666 should not be backported, but some changes are needed:
> > property fs.adl.impl in core-default.xml and hadoop-tools-dist/pom.xml.
> >
> > I am working on a patch.
> >
> >
> > John Zhuge
> > Software Engineer, Cloudera
> >
> > On Fri, Mar 17, 2017 at 2:18 AM, Junping Du  wrote:
> >
> >> Hi all,
> >>  With fix of HDFS-11431 get in, I've created a new release candidate
> >> (RC3) for Apache Hadoop 2.8.0.
> >>
> >>  This is the next minor release to follow up 2.7.0 which has been
> >> released for more than 1 year. It comprises 2,900+ fixes, improvements,
> and
> >> new features. Most of these commits are released for the first time in
> >> branch-2.
> >>
> >>   More information about the 2.8.0 release plan can be found here:
> >> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
> >>
> >>   New RC is available at: http://home.apache.org/~junpin
> >> g_du/hadoop-2.8.0-RC3
> >>
> >>   The RC tag in git is: release-2.8.0-RC3, and the latest commit id
> >> is: 91f2b7a13d1e97be65db92ddabc627cc29ac0009
> >>
> >>   The maven artifacts are available via repository.apache.org at:
> >> https://repository.apache.org/content/repositories/orgapachehadoop-1057
> >>
> >>   Please try the release and vote; the vote will run for the usual 5
> >> days, ending on 03/22/2017 PDT time.
> >>
> >> Thanks,
> >>
> >> Junping
> >>
> >
> >
>



-- 
John

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Zhihai Xu]

Thanks Junping for creating release 2.8.0 (RC3).

+1 (non-binding)

* Downloaded and built from source
* Verified md5 checksums and signature
* Deployed a pseudo cluster
* verified basic HDFS operations and Pi job.
* Did a sanity check for RM and NM UI.

Thanks

zhihai

On Tue, Mar 21, 2017 at 12:02 PM, Steve Loughran 
wrote:

>
> I'm an =0 I'm afraid
>
> I think those extra files need to be pulled which triggers a repackage, at
> which point there's a couple of
>
> HADOOP-14205 No FileSystem for scheme: adl
> HADOOP-14204 S3A multipart commit failing,
>
> of the two, the ADL FS one is the most significant, though if there was
> going to be a re-roll, I'd like them in.
>
> I also worry about windows test coverage, as irrespective of whether
> people are planning to run it on a windows host in production, the hadoop
> libs get used client side in job submit, and more now in Spark .
> HADOOP-14201 showed up that (a) my new VM is a mess, so complicating
> testing and (b) gzip was failing. I Worry about gzip, as that could be a
> sign of native lib problems.. someone needs to look at it a bit more.
> However, I don't think it should really be a blocker, given the ASF doesn't
> officially redistribute windows binaries.
>
> so: some issues with packaging and a couple of patches which need to go
> in. Test coverage on windows is potentially longer, and I don't want it to
> hold things up.
>
> Anyhow: =0. If everyone else is happy to ship, then let's get it out the
> door, as that's the only way that we'll get all the in-the-field bugreps
> coming back in. We will need to release some new versions in the coming
> weeks to handle those
>
> -Steve
>
>
>
>
> > On 21 Mar 2017, at 18:43, Haibo Chen  wrote:
> >
> > Thanks Junping for working on the new release!
> >
> > +1 non-binding
> >
> > 1) Downloaded the source, verified the checksum
> > 2) Built natively from source, and deployed it to a pseudo-distributed
> > cluster
> > 3) Ran sleep and teragen job and checked both YARN and JHS web UI
> > 4) Played with yarn + mapreduce command lines
> >
> > Best,
> > Haibo Chen
> >
> > On Mon, Mar 20, 2017 at 11:18 AM, Junping Du 
> wrote:
> >
> >> ?Thanks for update, John. Then we should be OK with fixing this issue in
> >> 2.8.1.
> >>
> >> Mark the target version of HADOOP-14205 to 2.8.1 instead of 2.8.0 and
> bump
> >> up to blocker in case we could miss this in releasing 2.8.1. :)
> >>
> >>
> >> Thanks,
> >>
> >>
> >> Junping
> >>
> >> 
> >> From: John Zhuge 
> >> Sent: Monday, March 20, 2017 10:31 AM
> >> To: Junping Du
> >> Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> >> yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
> >> Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
> >>
> >> Yes, it only affects ADL. There is a workaround of adding these 2
> >> properties to core-site.xml:
> >>
> >>  
> >>fs.adl.impl
> >>org.apache.hadoop.fs.adl.AdlFileSystem
> >>  
> >>
> >>  
> >>fs.AbstractFileSystem.adl.impl
> >>org.apache.hadoop.fs.adl.Adl
> >>  
> >>
> >> I have the initial patch ready but hitting these live unit test
> failures:
> >>
> >> Failed tests:
> >>  TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
> testListStatus:257
> >> expected:<1> but was:<10>
> >>
> >> Tests in error:
> >>  TestAdlFileContextMainOperationsLive>FileContextMainOperationsBaseT
> est.
> >> testMkdirsFailsForSubdirectoryOfExistingFile:254 » AccessControl
> >>  TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
> >> testMkdirsFailsForSubdirectoryOfExistingFile:190 » AccessControl
> >>
> >>
> >> Stay tuned...
> >>
> >> John Zhuge
> >> Software Engineer, Cloudera
> >>
> >> On Mon, Mar 20, 2017 at 10:02 AM, Junping Du   >> j...@hortonworks.com>> wrote:
> >>
> >> Thank you for reporting the issue, John! Does this issue only affect ADL
> >> (Azure Data Lake) which is a new feature for 2.8 rather than other
> existing
> >> FS? If so, I think we can leave the fix to 2.8.1 to fix given this is
> not a
> >> regression and just a new feature get broken.?
> >>
> >>
> >> Thanks,
> >>
> >>
> >> Junping
> >>
> >> 
> >> From: John Zhuge >
> >> Sent: Monday, March 20, 2017 9:07 AM
> >> To: Junping Du
> >> Cc: common-...@hadoop.apache.org;
> >> hdfs-...@hadoop.apache.org;
> >> yarn-dev@hadoop.apache.org;
> >> mapreduce-...@hadoop.apache.org
> >> Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
> >>
> >> Discovered https://issues.apache.org/jira/browse/HADOOP-14205 "No
> >> FileSystem for scheme: adl".
> >>
> >> The issue were caused by backporting HADOOP-13037 to branch-2 and
> earlier.
> >> HADOOP-12666 

[jira] [Created] (YARN-6370) Properly handle rack requests for non-active subclusters in LocalityMulticastAMRMProxyPolicy

[Botong Huang (JIRA)]

Botong Huang created YARN-6370:
--

 Summary: Properly handle rack requests for non-active subclusters 
in LocalityMulticastAMRMProxyPolicy
 Key: YARN-6370
 URL: https://issues.apache.org/jira/browse/YARN-6370
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Botong Huang
Assignee: Botong Huang
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

[Vinod Kumar Vavilapalli]

Thanks for taking the mantle from me on 2.8.0 and some persistent work getting 
2.8.0 out the door, Junping!

Apologies for bringing this up late, but I’d like to add one comment.

We should repeat what we did for 2.7.0 and In line with our experience there, 
we should annotate this release as not ready for production use. See the 
releases page - 
http://hadoop.apache.org/releases.html#25+August%2C+2016%3A+Release+2.7.3+available
 for our messaging on 2.7.0.

The expectation is that more downstream projects pick up the bits, iron out any 
incompatibilities we might have missed, and production users then pick up a 
solid 2.8.1.

Thanks
+Vinod

> On Mar 14, 2017, at 1:41 AM, Junping Du  wrote:
> 
> Hi all,
> With several important fixes get merged last week, I've created a new 
> release candidate (RC2) for Apache Hadoop 2.8.0.
> 
> This is the next minor release to follow up 2.7.0 which has been released 
> for more than 1 year. It comprises 2,919 fixes, improvements, and new 
> features. Most of these commits are released for the first time in branch-2.
> 
>  More information about the 2.8.0 release plan can be found here: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
> 
>  Please note that RC0 and RC1 are not voted public because significant 
> issues are found just after RC tag getting published.
> 
>  The RC is available at: 
> http://home.apache.org/~junping_du/hadoop-2.8.0-RC2
> 
>  The RC tag in git is: release-2.8.0-RC2
> 
>  The maven artifacts are available via repository.apache.org at: 
> https://repository.apache.org/content/repositories/orgapachehadoop-1056
> 
>  Please try the release and vote; the vote will run for the usual 5 days, 
> ending on 03/20/2017 PDT time.
> 
> Thanks,
> 
> Junping


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Steve Loughran]

I'm an =0 I'm afraid

I think those extra files need to be pulled which triggers a repackage, at 
which point there's a couple of 

HADOOP-14205 No FileSystem for scheme: adl
HADOOP-14204 S3A multipart commit failing,

of the two, the ADL FS one is the most significant, though if there was going 
to be a re-roll, I'd like them in.

I also worry about windows test coverage, as irrespective of whether people are 
planning to run it on a windows host in production, the hadoop libs get used 
client side in job submit, and more now in Spark . HADOOP-14201 showed up that 
(a) my new VM is a mess, so complicating testing and (b) gzip was failing. I 
Worry about gzip, as that could be a sign of native lib problems.. someone 
needs to look at it a bit more. However, I don't think it should really be a 
blocker, given the ASF doesn't officially redistribute windows binaries.

so: some issues with packaging and a couple of patches which need to go in. 
Test coverage on windows is potentially longer, and I don't want it to hold 
things up.

Anyhow: =0. If everyone else is happy to ship, then let's get it out the door, 
as that's the only way that we'll get all the in-the-field bugreps coming back 
in. We will need to release some new versions in the coming weeks to handle 
those

-Steve




> On 21 Mar 2017, at 18:43, Haibo Chen  wrote:
> 
> Thanks Junping for working on the new release!
> 
> +1 non-binding
> 
> 1) Downloaded the source, verified the checksum
> 2) Built natively from source, and deployed it to a pseudo-distributed
> cluster
> 3) Ran sleep and teragen job and checked both YARN and JHS web UI
> 4) Played with yarn + mapreduce command lines
> 
> Best,
> Haibo Chen
> 
> On Mon, Mar 20, 2017 at 11:18 AM, Junping Du  wrote:
> 
>> ?Thanks for update, John. Then we should be OK with fixing this issue in
>> 2.8.1.
>> 
>> Mark the target version of HADOOP-14205 to 2.8.1 instead of 2.8.0 and bump
>> up to blocker in case we could miss this in releasing 2.8.1. :)
>> 
>> 
>> Thanks,
>> 
>> 
>> Junping
>> 
>> 
>> From: John Zhuge 
>> Sent: Monday, March 20, 2017 10:31 AM
>> To: Junping Du
>> Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
>> yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
>> Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
>> 
>> Yes, it only affects ADL. There is a workaround of adding these 2
>> properties to core-site.xml:
>> 
>>  
>>fs.adl.impl
>>org.apache.hadoop.fs.adl.AdlFileSystem
>>  
>> 
>>  
>>fs.AbstractFileSystem.adl.impl
>>org.apache.hadoop.fs.adl.Adl
>>  
>> 
>> I have the initial patch ready but hitting these live unit test failures:
>> 
>> Failed tests:
>>  
>> TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.testListStatus:257
>> expected:<1> but was:<10>
>> 
>> Tests in error:
>>  TestAdlFileContextMainOperationsLive>FileContextMainOperationsBaseTest.
>> testMkdirsFailsForSubdirectoryOfExistingFile:254 » AccessControl
>>  TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
>> testMkdirsFailsForSubdirectoryOfExistingFile:190 » AccessControl
>> 
>> 
>> Stay tuned...
>> 
>> John Zhuge
>> Software Engineer, Cloudera
>> 
>> On Mon, Mar 20, 2017 at 10:02 AM, Junping Du  j...@hortonworks.com>> wrote:
>> 
>> Thank you for reporting the issue, John! Does this issue only affect ADL
>> (Azure Data Lake) which is a new feature for 2.8 rather than other existing
>> FS? If so, I think we can leave the fix to 2.8.1 to fix given this is not a
>> regression and just a new feature get broken.?
>> 
>> 
>> Thanks,
>> 
>> 
>> Junping
>> 
>> 
>> From: John Zhuge >
>> Sent: Monday, March 20, 2017 9:07 AM
>> To: Junping Du
>> Cc: common-...@hadoop.apache.org;
>> hdfs-...@hadoop.apache.org;
>> yarn-dev@hadoop.apache.org;
>> mapreduce-...@hadoop.apache.org
>> Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
>> 
>> Discovered https://issues.apache.org/jira/browse/HADOOP-14205 "No
>> FileSystem for scheme: adl".
>> 
>> The issue were caused by backporting HADOOP-13037 to branch-2 and earlier.
>> HADOOP-12666 should not be backported, but some changes are needed:
>> property fs.adl.impl in core-default.xml and hadoop-tools-dist/pom.xml.
>> 
>> I am working on a patch.
>> 
>> 
>> John Zhuge
>> Software Engineer, Cloudera
>> 
>> On Fri, Mar 17, 2017 at 2:18 AM, Junping Du > u...@hortonworks.com>> wrote:
>> Hi all,
>> With fix of HDFS-11431 get in, I've created a new release candidate
>> (RC3) for Apache Hadoop 2.8.0.
>> 
>> This is the next minor release to follow up 2.7.0 which has been
>> released for more than 1 year. It comprises 

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/352/

[Mar 20, 2017 5:59:09 PM] (varunsaxena) YARN-6353. Clean up OrderingPolicy 
javadoc (Daniel Templeton via Varun
[Mar 20, 2017 6:29:23 PM] (arp) HADOOP-14059. typo in s3a rename(self, subdir) 
error message.
[Mar 20, 2017 9:25:00 PM] (templedf) YARN-6309. Fair scheduler docs should have 
the queue and
[Mar 20, 2017 11:31:24 PM] (liuml07) HADOOP-13945. Azure: Add Kerberos and 
Delegation token support to WASB
[Mar 21, 2017 12:15:13 AM] (cdouglas) HDFS-6648. Order of namenodes in 
ConfiguredFailoverProxyProvider is
[Mar 21, 2017 4:54:58 AM] (arp) HDFS-11545. Propagate DataNode's slow disks 
info to the NameNode via




-1 overall


The following subsystems voted -1:
asflicense unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.TestDistributedFileSystem 
   hadoop.hdfs.TestDecommission 
   hadoop.hdfs.TestEncryptionZones 
   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency 
   
hadoop.hdfs.server.datanode.metrics.TestDataNodeOutlierDetectionViaMetrics 
   hadoop.hdfs.server.namenode.TestEditLogRace 
   hadoop.hdfs.TestMaintenanceState 
   hadoop.hdfs.server.datanode.TestDataNodeUUID 
   hadoop.yarn.server.nodemanager.containermanager.TestContainerManager 
   hadoop.yarn.server.timeline.webapp.TestTimelineWebServices 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.client.api.impl.TestAMRMClient 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/352/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/352/artifact/out/diff-compile-javac-root.txt
  [180K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/352/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/352/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/352/artifact/out/diff-patch-shellcheck.txt
  [24K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/352/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/352/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/352/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/352/artifact/out/diff-javadoc-javadoc-root.txt
  [2.2M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/352/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [692K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/352/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [36K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/352/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/352/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [324K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/352/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [12K]

   asflicense:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/352/artifact/out/patch-asflicense-problems.txt
  [4.0K]

Powered by Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Haibo Chen]

Thanks Junping for working on the new release!

+1 non-binding

1) Downloaded the source, verified the checksum
2) Built natively from source, and deployed it to a pseudo-distributed
cluster
3) Ran sleep and teragen job and checked both YARN and JHS web UI
4) Played with yarn + mapreduce command lines

Best,
Haibo Chen

On Mon, Mar 20, 2017 at 11:18 AM, Junping Du  wrote:

> ?Thanks for update, John. Then we should be OK with fixing this issue in
> 2.8.1.
>
> Mark the target version of HADOOP-14205 to 2.8.1 instead of 2.8.0 and bump
> up to blocker in case we could miss this in releasing 2.8.1. :)
>
>
> Thanks,
>
>
> Junping
>
> 
> From: John Zhuge 
> Sent: Monday, March 20, 2017 10:31 AM
> To: Junping Du
> Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
>
> Yes, it only affects ADL. There is a workaround of adding these 2
> properties to core-site.xml:
>
>   
> fs.adl.impl
> org.apache.hadoop.fs.adl.AdlFileSystem
>   
>
>   
> fs.AbstractFileSystem.adl.impl
> org.apache.hadoop.fs.adl.Adl
>   
>
> I have the initial patch ready but hitting these live unit test failures:
>
> Failed tests:
>   
> TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.testListStatus:257
> expected:<1> but was:<10>
>
> Tests in error:
>   TestAdlFileContextMainOperationsLive>FileContextMainOperationsBaseTest.
> testMkdirsFailsForSubdirectoryOfExistingFile:254 » AccessControl
>   TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
> testMkdirsFailsForSubdirectoryOfExistingFile:190 » AccessControl
>
>
> Stay tuned...
>
> John Zhuge
> Software Engineer, Cloudera
>
> On Mon, Mar 20, 2017 at 10:02 AM, Junping Du > wrote:
>
> Thank you for reporting the issue, John! Does this issue only affect ADL
> (Azure Data Lake) which is a new feature for 2.8 rather than other existing
> FS? If so, I think we can leave the fix to 2.8.1 to fix given this is not a
> regression and just a new feature get broken.?
>
>
> Thanks,
>
>
> Junping
>
> 
> From: John Zhuge >
> Sent: Monday, March 20, 2017 9:07 AM
> To: Junping Du
> Cc: common-...@hadoop.apache.org;
> hdfs-...@hadoop.apache.org;
> yarn-dev@hadoop.apache.org;
> mapreduce-...@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
>
> Discovered https://issues.apache.org/jira/browse/HADOOP-14205 "No
> FileSystem for scheme: adl".
>
> The issue were caused by backporting HADOOP-13037 to branch-2 and earlier.
> HADOOP-12666 should not be backported, but some changes are needed:
> property fs.adl.impl in core-default.xml and hadoop-tools-dist/pom.xml.
>
> I am working on a patch.
>
>
> John Zhuge
> Software Engineer, Cloudera
>
> On Fri, Mar 17, 2017 at 2:18 AM, Junping Du  u...@hortonworks.com>> wrote:
> Hi all,
>  With fix of HDFS-11431 get in, I've created a new release candidate
> (RC3) for Apache Hadoop 2.8.0.
>
>  This is the next minor release to follow up 2.7.0 which has been
> released for more than 1 year. It comprises 2,900+ fixes, improvements, and
> new features. Most of these commits are released for the first time in
> branch-2.
>
>   More information about the 2.8.0 release plan can be found here:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>
>   New RC is available at: http://home.apache.org/~
> junping_du/hadoop-2.8.0-RC3
>
>   The RC tag in git is: release-2.8.0-RC3, and the latest commit id
> is: 91f2b7a13d1e97be65db92ddabc627cc29ac0009
>
>   The maven artifacts are available via repository.apache.org repository.apache.org> at: https://repository.apache.org/
> content/repositories/orgapachehadoop-1057
>
>   Please try the release and vote; the vote will run for the usual 5
> days, ending on 03/22/2017 PDT time.
>
> Thanks,
>
> Junping
>
>
>

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Yufei Gu]

Thank Junping for working on this.

I verified the following:
1. Verified the md5 of binary tar ball.
2. Deployed on a 4 node cluster with 3 node managers.
3. Configured fair scheduler
4. Ran Pi job and verified the results.
5. Ran SLS for fair scheduler and capacity scheduler.

All good except SLS, it doesn't work. I guess this is an known issue.

Best,

Yufei

On Tue, Mar 21, 2017 at 10:12 AM, Andrew Wang 
wrote:

> I poked around a bit. The 3.0.0-alpha2 binary tarball is only 246M and has
> more changes than 2.8.0.
>
> It looks like the 2.8.0 bin tarball has an extra 1.5GB of docs when
> extracted compared to 3.0.0-alpha2. I think it's from the extra src-html
> content:
>
> -> % find share/doc -name src-html | xargs du -sb | awk -e '{SUM+=$1} END
> {print SUM}'
> 1651891481
>
>
> On Tue, Mar 21, 2017 at 9:53 AM, Wei-Chiu Chuang 
> wrote:
>
> > Thanks Junping for taking on this huge effort!
> >
> > I found one tiny nit: the md5 files are not in conventional format
> > That is to say,
> > $ cat hadoop-2.8.0.tar.gz.md5
> >
> > $ /usr/bin/md5sum /build/source/target/artifacts/hadoop-2.8.0.tar.gz
> > c728a090b68d009070085367695ed507  /build/source/target/
> > artifacts/hadoop-2.8.0.tar.gz
> >
> > But a typical md5 file would have been:
> > c728a090b68d009070085367695ed507  hadoop-2.8.0.tar.gz
> >
> > I was pretty stunned finding hadoop-2.8.0.tar.gz is a whopping 410MB
> > binary, comparing to hadoop-2.7.3.tar.gz which is just 205 MB.
> > But later on I realized the source code hadoop-2.8.0-src.tar.gz is 33MB
> > comparing to hadoop-2.7.3-src.tar.gz which is 18MB. So probably it’s the
> > amount of changes made into Hadoop 2.8 makes such difference in size.
> >
> > Regards,
> > Wei-Chiu Chuang
> >
> > > On Mar 21, 2017, at 9:39 AM, Akira Ajisaka 
> wrote:
> > >
> > > Thanks Junping!
> > >
> > > +1 (binding)
> > >
> > > * Verified signatures and checksums
> > > * Built Hive 2.1.0 and Tez 0.8.5 with Hadoop 2.8.0 pom
> > > * Deployed a single node cluster and ran some Hive on Tez queries
> > successfully
> > > * The document looks good.
> > >
> > > I found a trivial issue in the doc. It does not block the release.
> > > https://issues.apache.org/jira/browse/HADOOP-14208
> > >
> > > Regards,
> > > Akira
> > >
> > >
> > > On 2017/03/17 18:18, Junping Du wrote:
> > >> Hi all,
> > >> With fix of HDFS-11431 get in, I've created a new release
> candidate
> > (RC3) for Apache Hadoop 2.8.0.
> > >>
> > >> This is the next minor release to follow up 2.7.0 which has been
> > released for more than 1 year. It comprises 2,900+ fixes, improvements,
> and
> > new features. Most of these commits are released for the first time in
> > branch-2.
> > >>
> > >>  More information about the 2.8.0 release plan can be found here:
> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
> > >>
> > >>  New RC is available at: http://home.apache.org/~
> > junping_du/hadoop-2.8.0-RC3
> > >>
> > >>  The RC tag in git is: release-2.8.0-RC3, and the latest commit id
> > is: 91f2b7a13d1e97be65db92ddabc627cc29ac0009
> > >>
> > >>  The maven artifacts are available via repository.apache.org at:
> > https://repository.apache.org/content/repositories/orgapachehadoop-1057
> > >>
> > >>  Please try the release and vote; the vote will run for the usual
> 5
> > days, ending on 03/22/2017 PDT time.
> > >>
> > >> Thanks,
> > >>
> > >> Junping
> > >>
> > >
> > > -
> > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> > >
> >
> >
>

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Andrew Wang]

I poked around a bit. The 3.0.0-alpha2 binary tarball is only 246M and has
more changes than 2.8.0.

It looks like the 2.8.0 bin tarball has an extra 1.5GB of docs when
extracted compared to 3.0.0-alpha2. I think it's from the extra src-html
content:

-> % find share/doc -name src-html | xargs du -sb | awk -e '{SUM+=$1} END
{print SUM}'
1651891481


On Tue, Mar 21, 2017 at 9:53 AM, Wei-Chiu Chuang 
wrote:

> Thanks Junping for taking on this huge effort!
>
> I found one tiny nit: the md5 files are not in conventional format
> That is to say,
> $ cat hadoop-2.8.0.tar.gz.md5
>
> $ /usr/bin/md5sum /build/source/target/artifacts/hadoop-2.8.0.tar.gz
> c728a090b68d009070085367695ed507  /build/source/target/
> artifacts/hadoop-2.8.0.tar.gz
>
> But a typical md5 file would have been:
> c728a090b68d009070085367695ed507  hadoop-2.8.0.tar.gz
>
> I was pretty stunned finding hadoop-2.8.0.tar.gz is a whopping 410MB
> binary, comparing to hadoop-2.7.3.tar.gz which is just 205 MB.
> But later on I realized the source code hadoop-2.8.0-src.tar.gz is 33MB
> comparing to hadoop-2.7.3-src.tar.gz which is 18MB. So probably it’s the
> amount of changes made into Hadoop 2.8 makes such difference in size.
>
> Regards,
> Wei-Chiu Chuang
>
> > On Mar 21, 2017, at 9:39 AM, Akira Ajisaka  wrote:
> >
> > Thanks Junping!
> >
> > +1 (binding)
> >
> > * Verified signatures and checksums
> > * Built Hive 2.1.0 and Tez 0.8.5 with Hadoop 2.8.0 pom
> > * Deployed a single node cluster and ran some Hive on Tez queries
> successfully
> > * The document looks good.
> >
> > I found a trivial issue in the doc. It does not block the release.
> > https://issues.apache.org/jira/browse/HADOOP-14208
> >
> > Regards,
> > Akira
> >
> >
> > On 2017/03/17 18:18, Junping Du wrote:
> >> Hi all,
> >> With fix of HDFS-11431 get in, I've created a new release candidate
> (RC3) for Apache Hadoop 2.8.0.
> >>
> >> This is the next minor release to follow up 2.7.0 which has been
> released for more than 1 year. It comprises 2,900+ fixes, improvements, and
> new features. Most of these commits are released for the first time in
> branch-2.
> >>
> >>  More information about the 2.8.0 release plan can be found here:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
> >>
> >>  New RC is available at: http://home.apache.org/~
> junping_du/hadoop-2.8.0-RC3
> >>
> >>  The RC tag in git is: release-2.8.0-RC3, and the latest commit id
> is: 91f2b7a13d1e97be65db92ddabc627cc29ac0009
> >>
> >>  The maven artifacts are available via repository.apache.org at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1057
> >>
> >>  Please try the release and vote; the vote will run for the usual 5
> days, ending on 03/22/2017 PDT time.
> >>
> >> Thanks,
> >>
> >> Junping
> >>
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
>
>

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Wei-Chiu Chuang]

Thanks Junping for taking on this huge effort!

I found one tiny nit: the md5 files are not in conventional format
That is to say, 
$ cat hadoop-2.8.0.tar.gz.md5 

$ /usr/bin/md5sum /build/source/target/artifacts/hadoop-2.8.0.tar.gz
c728a090b68d009070085367695ed507  
/build/source/target/artifacts/hadoop-2.8.0.tar.gz

But a typical md5 file would have been:
c728a090b68d009070085367695ed507  hadoop-2.8.0.tar.gz

I was pretty stunned finding hadoop-2.8.0.tar.gz is a whopping 410MB binary, 
comparing to hadoop-2.7.3.tar.gz which is just 205 MB.
But later on I realized the source code hadoop-2.8.0-src.tar.gz is 33MB 
comparing to hadoop-2.7.3-src.tar.gz which is 18MB. So probably it’s the amount 
of changes made into Hadoop 2.8 makes such difference in size.

Regards,
Wei-Chiu Chuang

> On Mar 21, 2017, at 9:39 AM, Akira Ajisaka  wrote:
> 
> Thanks Junping!
> 
> +1 (binding)
> 
> * Verified signatures and checksums
> * Built Hive 2.1.0 and Tez 0.8.5 with Hadoop 2.8.0 pom
> * Deployed a single node cluster and ran some Hive on Tez queries successfully
> * The document looks good.
> 
> I found a trivial issue in the doc. It does not block the release.
> https://issues.apache.org/jira/browse/HADOOP-14208
> 
> Regards,
> Akira
> 
> 
> On 2017/03/17 18:18, Junping Du wrote:
>> Hi all,
>> With fix of HDFS-11431 get in, I've created a new release candidate 
>> (RC3) for Apache Hadoop 2.8.0.
>> 
>> This is the next minor release to follow up 2.7.0 which has been 
>> released for more than 1 year. It comprises 2,900+ fixes, improvements, and 
>> new features. Most of these commits are released for the first time in 
>> branch-2.
>> 
>>  More information about the 2.8.0 release plan can be found here: 
>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>> 
>>  New RC is available at: 
>> http://home.apache.org/~junping_du/hadoop-2.8.0-RC3
>> 
>>  The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: 
>> 91f2b7a13d1e97be65db92ddabc627cc29ac0009
>> 
>>  The maven artifacts are available via repository.apache.org at: 
>> https://repository.apache.org/content/repositories/orgapachehadoop-1057
>> 
>>  Please try the release and vote; the vote will run for the usual 5 
>> days, ending on 03/22/2017 PDT time.
>> 
>> Thanks,
>> 
>> Junping
>> 
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Akira Ajisaka]

Thanks Junping!

+1 (binding)

* Verified signatures and checksums
* Built Hive 2.1.0 and Tez 0.8.5 with Hadoop 2.8.0 pom
* Deployed a single node cluster and ran some Hive on Tez queries 
successfully

* The document looks good.

I found a trivial issue in the doc. It does not block the release.
https://issues.apache.org/jira/browse/HADOOP-14208

Regards,
Akira


On 2017/03/17 18:18, Junping Du wrote:

Hi all,
 With fix of HDFS-11431 get in, I've created a new release candidate (RC3) 
for Apache Hadoop 2.8.0.

 This is the next minor release to follow up 2.7.0 which has been released 
for more than 1 year. It comprises 2,900+ fixes, improvements, and new 
features. Most of these commits are released for the first time in branch-2.

  More information about the 2.8.0 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.0-RC3

  The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: 
91f2b7a13d1e97be65db92ddabc627cc29ac0009

  The maven artifacts are available via repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1057

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 03/22/2017 PDT time.

Thanks,

Junping



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Masatake Iwasaki]

Thanks for putting this up, Junping.

+1(non-binding)

* verified signature and mds.
* built from source on centos 7 and openjdk 8 with native profile.
* built docs and skimmed the contents.
* deployed 3 nodes ha cluster and run some sample jobs with caller 
context enabled.


Masatake Iwasaki

On 3/17/17 18:18, Junping Du wrote:

Hi all,
  With fix of HDFS-11431 get in, I've created a new release candidate (RC3) 
for Apache Hadoop 2.8.0.

  This is the next minor release to follow up 2.7.0 which has been released 
for more than 1 year. It comprises 2,900+ fixes, improvements, and new 
features. Most of these commits are released for the first time in branch-2.

   More information about the 2.8.0 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

   New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.0-RC3

   The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: 
91f2b7a13d1e97be65db92ddabc627cc29ac0009

   The maven artifacts are available via repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1057

   Please try the release and vote; the vote will run for the usual 5 days, 
ending on 03/22/2017 PDT time.

Thanks,

Junping




-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

RE: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Brahma Reddy Battula]

Hi Junping,

Thank you for working on this.

+1 (Non-Binding)

verified the following:
1. Deployed HA cluster with three Nodes.
2. Verified the basic shell operations
3. Ran wordount and pi jobs

Regards
Brahma Reddy Battula

-Original Message-
From: Junping Du [mailto:j...@hortonworks.com] 
Sent: 17 March 2017 17:18
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

Hi all,
 With fix of HDFS-11431 get in, I've created a new release candidate (RC3) 
for Apache Hadoop 2.8.0.

 This is the next minor release to follow up 2.7.0 which has been released 
for more than 1 year. It comprises 2,900+ fixes, improvements, and new 
features. Most of these commits are released for the first time in branch-2.

  More information about the 2.8.0 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.0-RC3

  The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: 
91f2b7a13d1e97be65db92ddabc627cc29ac0009

  The maven artifacts are available via repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1057

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 03/22/2017 PDT time.

Thanks,

Junping

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org