[jira] [Created] (HADOOP-17763) DistCp job fails when AM is killed

[Bilwa S T (Jira)]

Bilwa S T created HADOOP-17763:
--

 Summary: DistCp job fails when AM is killed
 Key: HADOOP-17763
 URL: https://issues.apache.org/jira/browse/HADOOP-17763
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Bilwa S T
Assignee: Bilwa S T


Job fails as tasks fail with below exception
{code:java}
2021-06-11 18:48:47,047 | ERROR | IPC Server handler 0 on 27101 | Task: 
attempt_1623387358383_0006_m_00_1000 - exited : 
java.io.FileNotFoundException: File does not exist: 
hdfs://hacluster/staging-dir/dsperf/.staging/_distcp-646531269/fileList.seq
 at 
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1637)
 at 
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1630)
 at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
 at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1645)
 at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1863)
 at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1886)
 at 
org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.initialize(SequenceFileRecordReader.java:54)
 at 
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:560)
 at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:798)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
 at org.apache.hadoop.mapred.YarnChild$1.run(YarnChild.java:183)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1761)
 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:177)
 | TaskAttemptListenerImpl.java:304{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/539/

[Jun 14, 2021 5:48:22 PM] (noreply) HDFS-16055. Quota is not preserved in 
snapshot INode (#3078)
[Jun 14, 2021 8:33:04 PM] (Szilard Nemeth) YARN-10802. Change Capacity 
Scheduler minimum-user-limit-percent to accept decimal values. Contributed by 
Benjamin Teke
[Jun 14, 2021 9:23:50 PM] (noreply) HDFS-16016. BPServiceActor to provide new 
thread to handle IBR (#2998)
[Jun 15, 2021 7:50:09 AM] (noreply) HDFS-16068. WebHdfsFileSystem has a 
possible connection leak in connection with HttpFS (#3104)




-1 overall


The following subsystems voted -1:
blanks pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

Failed junit tests :

   hadoop.yarn.csi.client.TestCsiClient 
   hadoop.tools.dynamometer.TestDynamometerInfra 
   hadoop.tools.dynamometer.TestDynamometerInfra 
  

   cc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/539/artifact/out/results-compile-cc-root.txt
 [96K]

   javac:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/539/artifact/out/results-compile-javac-root.txt
 [376K]

   blanks:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/539/artifact/out/blanks-eol.txt
 [13M]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/539/artifact/out/blanks-tabs.txt
 [2.0M]

   checkstyle:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/539/artifact/out/results-checkstyle-root.txt
 [16M]

   pathlen:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/539/artifact/out/results-pathlen.txt
 [16K]

   pylint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/539/artifact/out/results-pylint.txt
 [20K]

   shellcheck:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/539/artifact/out/results-shellcheck.txt
 [28K]

   xml:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/539/artifact/out/xml.txt
 [24K]

   javadoc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/539/artifact/out/results-javadoc-javadoc-root.txt
 [408K]

   unit:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/539/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-csi.txt
 [24K]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/539/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-infra.txt
 [12K]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/539/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer.txt
 [24K]

Powered by Apache Yetus 0.14.0-SNAPSHOT   https://yetus.apache.org

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [ANNOUNCE] Apache Hadoop 3.3.1 release

[Xun Liu]

Thank you Wei-Chiu!

Best regards
Xun Liu

On Wed, Jun 16, 2021 at 11:03 AM Brahma Reddy Battula 
wrote:

> Hi Wei-Chiu,
>
> Thanks for driving this..
>
>
> Looks index.html is not updated and Changelog is broken.
>
>
> On Wed, Jun 16, 2021 at 7:59 AM Wei-Chiu Chuang 
> wrote:
>
> > Hi All,
> >
> > It gives me great pleasure to announce that the Apache Hadoop community
> has
> > voted to release Apache Hadoop 3.3.1.
> >
> > This is the first stable release of Apache Hadoop 3.3.x line. It contains
> > 697 bug fixes, improvements and enhancements since 3.3.0.
> >
> > Users are encouraged to read the overview of major changes
> >  since 3.3.0. For
> > details
> > of 697 bug fixes, improvements, and other enhancements since the previous
> > 3.3.0 release, please check release notes
> > <
> >
> http://hadoop.apache.org/docs/r3.3.1/hadoop-project-dist/hadoop-common/release/3.3.1/RELEASENOTES.3.3.1.html
> > >
> >  and changelog
> > <
> >
> http://hadoop.apache.org/docs/r3.3.1/hadoop-project-dist/hadoop-common/release/3.3.1/CHANGES.3.3.1.html
> > >
> > detail
> > the changes since 3.3.0.
> >
> > Many thanks to everyone who contributed to the release, and everyone in
> the
> > Apache Hadoop community! This release is a direct result of your great
> > contributions.
> >
> > Many thanks to everyone who helped in this release process!
> >
> > Many thanks to Sean Busbey, Chao Sun, Steve Loughran, Masatake Iwasaki,
> > Michael Stack, Viraj Jasani, Eric Payne, Ayush Saxena, Vinayakumar B,
> > Takanobu Asanuma, Xiaoqiao He and other folks who continued helps for
> this
> > release process.
> >
> > Best Regards,
> > Wei-Chiu Chuang
> >
>
>
> --
>
>
>
> --Brahma Reddy Battula
>

Re: [ANNOUNCE] Apache Hadoop 3.3.1 release

[Brahma Reddy Battula]

Hi Wei-Chiu,

Thanks for driving this..


Looks index.html is not updated and Changelog is broken.


On Wed, Jun 16, 2021 at 7:59 AM Wei-Chiu Chuang  wrote:

> Hi All,
>
> It gives me great pleasure to announce that the Apache Hadoop community has
> voted to release Apache Hadoop 3.3.1.
>
> This is the first stable release of Apache Hadoop 3.3.x line. It contains
> 697 bug fixes, improvements and enhancements since 3.3.0.
>
> Users are encouraged to read the overview of major changes
>  since 3.3.0. For
> details
> of 697 bug fixes, improvements, and other enhancements since the previous
> 3.3.0 release, please check release notes
> <
> http://hadoop.apache.org/docs/r3.3.1/hadoop-project-dist/hadoop-common/release/3.3.1/RELEASENOTES.3.3.1.html
> >
>  and changelog
> <
> http://hadoop.apache.org/docs/r3.3.1/hadoop-project-dist/hadoop-common/release/3.3.1/CHANGES.3.3.1.html
> >
> detail
> the changes since 3.3.0.
>
> Many thanks to everyone who contributed to the release, and everyone in the
> Apache Hadoop community! This release is a direct result of your great
> contributions.
>
> Many thanks to everyone who helped in this release process!
>
> Many thanks to Sean Busbey, Chao Sun, Steve Loughran, Masatake Iwasaki,
> Michael Stack, Viraj Jasani, Eric Payne, Ayush Saxena, Vinayakumar B,
> Takanobu Asanuma, Xiaoqiao He and other folks who continued helps for this
> release process.
>
> Best Regards,
> Wei-Chiu Chuang
>


-- 



--Brahma Reddy Battula

re??[ANNOUNCE] Apache Hadoop 3.3.1 release

[Qi Zhu]

Great work!
Thank you Wei-Chiu!


Qi Zhu


--  --
??: 
   "Wei-Chiu Chuang"

https://hadoop.apache.org/docs/r3.3.1/index.html>; since 3.3.0. For details
of 697 bug fixes, improvements, and other enhancements since the previous
3.3.0 release, please check release notes
;
 and changelog
;
detail
the changes since 3.3.0.

Many thanks to everyone who contributed to the release, and everyone in the
Apache Hadoop community! This release is a direct result of your great
contributions.

Many thanks to everyone who helped in this release process!

Many thanks to Sean Busbey, Chao Sun, Steve Loughran, Masatake Iwasaki,
Michael Stack, Viraj Jasani, Eric Payne, Ayush Saxena, Vinayakumar B,
Takanobu Asanuma, Xiaoqiao He and other folks who continued helps for this
release process.

Best Regards,
Wei-Chiu Chuang

Re: [ANNOUNCE] Apache Hadoop 3.3.1 release

[David]

Thank you Wei-Chiu!  Hopefully we can now move Hive forward!

On Tue, Jun 15, 2021, 10:30 PM Wei-Chiu Chuang  wrote:

> Hi All,
>
> It gives me great pleasure to announce that the Apache Hadoop community has
> voted to release Apache Hadoop 3.3.1.
>
> This is the first stable release of Apache Hadoop 3.3.x line. It contains
> 697 bug fixes, improvements and enhancements since 3.3.0.
>
> Users are encouraged to read the overview of major changes
>  since 3.3.0. For
> details
> of 697 bug fixes, improvements, and other enhancements since the previous
> 3.3.0 release, please check release notes
> <
> http://hadoop.apache.org/docs/r3.3.1/hadoop-project-dist/hadoop-common/release/3.3.1/RELEASENOTES.3.3.1.html
> >
>  and changelog
> <
> http://hadoop.apache.org/docs/r3.3.1/hadoop-project-dist/hadoop-common/release/3.3.1/CHANGES.3.3.1.html
> >
> detail
> the changes since 3.3.0.
>
> Many thanks to everyone who contributed to the release, and everyone in the
> Apache Hadoop community! This release is a direct result of your great
> contributions.
>
> Many thanks to everyone who helped in this release process!
>
> Many thanks to Sean Busbey, Chao Sun, Steve Loughran, Masatake Iwasaki,
> Michael Stack, Viraj Jasani, Eric Payne, Ayush Saxena, Vinayakumar B,
> Takanobu Asanuma, Xiaoqiao He and other folks who continued helps for this
> release process.
>
> Best Regards,
> Wei-Chiu Chuang
>

[ANNOUNCE] Apache Hadoop 3.3.1 release

[Wei-Chiu Chuang]

Hi All,

It gives me great pleasure to announce that the Apache Hadoop community has
voted to release Apache Hadoop 3.3.1.

This is the first stable release of Apache Hadoop 3.3.x line. It contains
697 bug fixes, improvements and enhancements since 3.3.0.

Users are encouraged to read the overview of major changes
 since 3.3.0. For details
of 697 bug fixes, improvements, and other enhancements since the previous
3.3.0 release, please check release notes

 and changelog

detail
the changes since 3.3.0.

Many thanks to everyone who contributed to the release, and everyone in the
Apache Hadoop community! This release is a direct result of your great
contributions.

Many thanks to everyone who helped in this release process!

Many thanks to Sean Busbey, Chao Sun, Steve Loughran, Masatake Iwasaki,
Michael Stack, Viraj Jasani, Eric Payne, Ayush Saxena, Vinayakumar B,
Takanobu Asanuma, Xiaoqiao He and other folks who continued helps for this
release process.

Best Regards,
Wei-Chiu Chuang

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/185/

[Jun 13, 2021 5:37:10 PM] (noreply) HADOOP-17645 Fix test failures in 
org.apache.hadoop.fs.azure.ITestOutputStreamSemantics. (#2926)
[Jun 13, 2021 6:22:29 PM] (noreply) HADOOP-17714 ABFS: 
testBlobBackCompatibility, testRandomRead & WasbAbfsCompatibility tests fail 
when triggered with default configs (#3035)
[Jun 14, 2021 1:17:39 AM] (noreply) HADOOP-17756. Increase precommit job 
timeout from 20 hours to 24 hours. (#3090)
[Jun 14, 2021 5:48:22 PM] (noreply) HDFS-16055. Quota is not preserved in 
snapshot INode (#3078)
[Jun 14, 2021 8:33:04 PM] (Szilard Nemeth) YARN-10802. Change Capacity 
Scheduler minimum-user-limit-percent to accept decimal values. Contributed by 
Benjamin Teke
[Jun 14, 2021 9:23:50 PM] (noreply) HDFS-16016. BPServiceActor to provide new 
thread to handle IBR (#2998)




-1 overall


The following subsystems voted -1:
blanks mvnsite pathlen spotbugs unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

spotbugs :

   module:hadoop-hdfs-project/hadoop-hdfs 
   Redundant nullcheck of oldLock, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory)
 Redundant null check at DataStorage.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory)
 Redundant null check at DataStorage.java:[line 695] 
   
org.apache.hadoop.hdfs.server.namenode.top.window.RollingWindowManager$UserCounts
 doesn't override java.util.ArrayList.equals(Object) At 
RollingWindowManager.java:At RollingWindowManager.java:[line 1] 

spotbugs :

   module:hadoop-yarn-project/hadoop-yarn 
   Redundant nullcheck of it, which is known to be non-null in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker,
 NMStateStoreService$LocalResourceTrackerState) Redundant null check at 
ResourceLocalizationService.java:is known to be non-null in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker,
 NMStateStoreService$LocalResourceTrackerState) Redundant null check at 
ResourceLocalizationService.java:[line 343] 
   Redundant nullcheck of it, which is known to be non-null in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker,
 NMStateStoreService$LocalResourceTrackerState) Redundant null check at 
ResourceLocalizationService.java:is known to be non-null in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker,
 NMStateStoreService$LocalResourceTrackerState) Redundant null check at 
ResourceLocalizationService.java:[line 356] 
   Boxed value is unboxed and then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:[line 333] 

spotbugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server 
   Redundant nullcheck of it, which is known to be non-null in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker,
 NMStateStoreService$LocalResourceTrackerState) Redundant null check at 
ResourceLocalizationService.java:is known to 

[jira] [Created] (HADOOP-17762) branch-2.10 daily build fails to pull latest changes

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17762:
--

 Summary: branch-2.10 daily build fails to pull latest changes
 Key: HADOOP-17762
 URL: https://issues.apache.org/jira/browse/HADOOP-17762
 Project: Hadoop Common
  Issue Type: Bug
  Components: build, yetus
Affects Versions: 2.10.1
Reporter: Ahmed Hussein


I noticed that the build for branch-2.10 failed to pull the latest changes for 
the last few days.

CC: [~aajisaka], [~tasanuma], [~Jim_Brennan]

https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/329/console

{code:bash}
Started by timer
Running as SYSTEM
Building remotely on H20 (Hadoop) in workspace 
/home/jenkins/jenkins-home/workspace/hadoop-qbt-branch-2.10-java7-linux-x86_64
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Using shallow clone with depth 10
Avoid fetching tags
Cloning repository https://github.com/apache/hadoop
ERROR: Failed to clean the workspace
jenkins.util.io.CompositeIOException: Unable to delete 
'/home/jenkins/jenkins-home/workspace/hadoop-qbt-branch-2.10-java7-linux-x86_64/sourcedir'.
 Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 
1 additional exceptions)
at 
jenkins.util.io.PathRemover.forceRemoveDirectoryContents(PathRemover.java:90)
at hudson.Util.deleteContentsRecursive(Util.java:262)
at hudson.Util.deleteContentsRecursive(Util.java:251)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:743)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:375)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: java.nio.file.AccessDeniedException: 
/home/jenkins/jenkins-home/workspace/hadoop-qbt-branch-2.10-java7-linux-x86_64/sourcedir/hadoop-hdfs-project/hadoop-hdfs/target/test/data/3/dfs/data/data1/current
at 
sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at 
sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
at java.nio.file.Files.newDirectoryStream(Files.java:457)
at 
jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:224)
at 
jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at 
jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at 
jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at 
jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at 
jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at 
jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at 
jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at 
jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at 
jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at 
jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at 
jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at 
jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at 
jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at 
jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at 
jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at 
jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at 
jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at 
jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at 
jenkins.util.io.PathRemo

Re: [VOTE] Hadoop 3.1.x EOL

[Wei-Chiu Chuang]

Dropped 3.1.4 from website.
Removed from https://dist.apache.org/repos/dist/release/hadoop/common/

On Thu, Jun 10, 2021 at 3:42 PM Akira Ajisaka  wrote:

> This vote has passed with 18 binding +1. I'll update the JIRA and the wiki.
>
> Thanks all for your participation.
>
> On Tue, Jun 8, 2021 at 3:03 AM Steve Loughran  wrote:
> >
> >
> >
> > On Thu, 3 Jun 2021 at 07:14, Akira Ajisaka  wrote:
> >>
> >> Dear Hadoop developers,
> >>
> >> Given the feedback from the discussion thread [1], I'd like to start
> >> an official vote
> >> thread for the community to vote and start the 3.1 EOL process.
> >>
> >> What this entails:
> >>
> >> (1) an official announcement that no further regular Hadoop 3.1.x
> releases
> >> will be made after 3.1.4.
> >> (2) resolve JIRAs that specifically target 3.1.5 as won't fix.
> >>
> >> This vote will run for 7 days and conclude by June 10th, 16:00 JST [2].
> >>
> >> Committers are eligible to cast binding votes. Non-committers are
> welcomed
> >> to cast non-binding votes.
> >>
> >> Here is my vote, +1
> >
> >
> >
> > +1 (binding)
> >>
> >>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

[Wei-Chiu Chuang]

Thanks all for the input!

I released the maven artifacts earlier today, so soon you should be able to
use the 3.3.1 dependencies in downstream applications.
The official website is refreshed to include 3.3.1:
https://hadoop.apache.org/
The tarball was uploaded and verified the links are good to go.

Will send out the announcement to the user mailing list later.

On Tue, Jun 15, 2021 at 9:56 PM Xiaoqiao He  wrote:

> +0.
>
> - Built successfully from tag release-3.3.1-RC3 on Ubuntu 20.04
> - Verified signature and checksum
> - Deployed pseudo-distributed cluster with 3 nodes
> - Ran basic HDFS shell commands and sample MR Jobs, it worked well.
> - Browsed NN/DN/RM/NM UI. *YARN UI2 DOES NOT work from my side.*
>
> NOTE: attach configuration for YARN. Please correct me if I missed
> something.
>
> Thanks Wei-Chiu for your great work!
>
> - He Xiaoqiao
>
>
> On Mon, Jun 14, 2021 at 12:28 PM Takanobu Asanuma 
> wrote:
>
>> +1.
>>  - Verified hashes
>>  - Confirmed native build on CentOS7
>>  - Started kerberized cluster (using docker)
>>  - Checked NN/RBF Web UI
>>  - Ran basic Erasure Coding shell commands
>>
>> Thanks for the great work, Wei-Chiu.
>>
>> - Takanobu
>>
>> 2021年6月13日(日) 3:25 Vinayakumar B :
>>
>> > +1 (Binding)
>> >
>> > 1. Built from Tag.
>> > 2. Successful Native Build on Ubuntu 20.04
>> > 3. Verified Checksums
>> > 4. Deployed the docker cluster with 3 nodes
>> > 5. Ran sample MR Jobs
>> >
>> > -Vinay
>> >
>> >
>> > On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena 
>> wrote:
>> >
>> > > +1,
>> > > Built from Source.
>> > > Successful Native Build on Ubuntu 20.04
>> > > Verified Checksums
>> > > Ran basic hdfs shell commands.
>> > > Ran simple MR jobs.
>> > > Browsed NN,DN,RM and NM UI.
>> > >
>> > > Thanx Wei-Chiu for driving the release.
>> > >
>> > > -Ayush
>> > >
>> > >
>> > > > On 12-Jun-2021, at 1:45 AM, epa...@apache.org wrote:
>> > > >
>> > > > +1 (binding)
>> > > > Eric
>> > > >
>> > > >
>> > > > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
>> > > weic...@apache.org> wrote:
>> > > >
>> > > > Hi community,
>> > > >
>> > > > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
>> > > blocker
>> > > > issues have been resolved [1] again.
>> > > >
>> > > > There are 2 additional issues resolved for RC3:
>> > > > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
>> > > > HADOOP-16878
>> > > > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
>> > > source
>> > > > and destination are the same
>> > > >
>> > > > There are 4 issues resolved for RC2:
>> > > > * HADOOP-17666. Update LICENSE for 3.3.1
>> > > > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
>> > > > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)"
>> (#3055)
>> > > > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>> > > >
>> > > > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two
>> > extra
>> > > > fixes compared to hadoop-thirdparty 1.1.0:
>> > > > * HADOOP-17707. Remove jaeger document from site index.
>> > > > * HADOOP-17730. Add back error_prone
>> > > >
>> > > > *RC tag is release-3.3.1-RC3
>> > > > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>> > > >
>> > > > *The RC3 artifacts are at*:
>> > > > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
>> > > > ARM artifacts:
>> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>> > > >
>> > > > *The maven artifacts are hosted here:*
>> > > >
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>> > > >
>> > > > *My public key is available here:*
>> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > > >
>> > > >
>> > > > Things I've verified:
>> > > > * all blocker issues targeting 3.3.1 have been resolved.
>> > > > * stable/evolving API changes between 3.3.0 and 3.3.1 are
>> compatible.
>> > > > * LICENSE and NOTICE files checked
>> > > > * RELEASENOTES and CHANGELOG
>> > > > * rat check passed.
>> > > > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit
>> tests.
>> > > > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
>> > > > * Extra: built 50 other open source projects on top of Hadoop 3.3.1
>> > RC2.
>> > > > Had to patch some of them due to commons-lang migration (Hadoop
>> 3.2.0)
>> > > and
>> > > > dependency divergence. Issues are being identified but so far
>> nothing
>> > > > blocker for Hadoop itself.
>> > > >
>> > > > Please try the release and vote. The vote will run for 5 days.
>> > > >
>> > > > My +1 to start,
>> > > >
>> > > > [1] https://issues.apache.org/jira/issues/?filter=12350491
>> > > > [2]
>> > > >
>> > >
>> >
>> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>> > >
>> > > -
>> > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>> > >
>> > >
>> >

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

[Xiaoqiao He]

+0.

- Built successfully from tag release-3.3.1-RC3 on Ubuntu 20.04
- Verified signature and checksum
- Deployed pseudo-distributed cluster with 3 nodes
- Ran basic HDFS shell commands and sample MR Jobs, it worked well.
- Browsed NN/DN/RM/NM UI. *YARN UI2 DOES NOT work from my side.*

NOTE: attach configuration for YARN. Please correct me if I missed
something.

Thanks Wei-Chiu for your great work!

- He Xiaoqiao


On Mon, Jun 14, 2021 at 12:28 PM Takanobu Asanuma 
wrote:

> +1.
>  - Verified hashes
>  - Confirmed native build on CentOS7
>  - Started kerberized cluster (using docker)
>  - Checked NN/RBF Web UI
>  - Ran basic Erasure Coding shell commands
>
> Thanks for the great work, Wei-Chiu.
>
> - Takanobu
>
> 2021年6月13日(日) 3:25 Vinayakumar B :
>
> > +1 (Binding)
> >
> > 1. Built from Tag.
> > 2. Successful Native Build on Ubuntu 20.04
> > 3. Verified Checksums
> > 4. Deployed the docker cluster with 3 nodes
> > 5. Ran sample MR Jobs
> >
> > -Vinay
> >
> >
> > On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena  wrote:
> >
> > > +1,
> > > Built from Source.
> > > Successful Native Build on Ubuntu 20.04
> > > Verified Checksums
> > > Ran basic hdfs shell commands.
> > > Ran simple MR jobs.
> > > Browsed NN,DN,RM and NM UI.
> > >
> > > Thanx Wei-Chiu for driving the release.
> > >
> > > -Ayush
> > >
> > >
> > > > On 12-Jun-2021, at 1:45 AM, epa...@apache.org wrote:
> > > >
> > > > +1 (binding)
> > > > Eric
> > > >
> > > >
> > > > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
> > > weic...@apache.org> wrote:
> > > >
> > > > Hi community,
> > > >
> > > > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> > > blocker
> > > > issues have been resolved [1] again.
> > > >
> > > > There are 2 additional issues resolved for RC3:
> > > > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > > > HADOOP-16878
> > > > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> > > source
> > > > and destination are the same
> > > >
> > > > There are 4 issues resolved for RC2:
> > > > * HADOOP-17666. Update LICENSE for 3.3.1
> > > > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > > > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)"
> (#3055)
> > > > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> > > >
> > > > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two
> > extra
> > > > fixes compared to hadoop-thirdparty 1.1.0:
> > > > * HADOOP-17707. Remove jaeger document from site index.
> > > > * HADOOP-17730. Add back error_prone
> > > >
> > > > *RC tag is release-3.3.1-RC3
> > > > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> > > >
> > > > *The RC3 artifacts are at*:
> > > > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > > > ARM artifacts:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> > > >
> > > > *The maven artifacts are hosted here:*
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> > > >
> > > > *My public key is available here:*
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > >
> > > > Things I've verified:
> > > > * all blocker issues targeting 3.3.1 have been resolved.
> > > > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > > > * LICENSE and NOTICE files checked
> > > > * RELEASENOTES and CHANGELOG
> > > > * rat check passed.
> > > > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit
> tests.
> > > > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > > > * Extra: built 50 other open source projects on top of Hadoop 3.3.1
> > RC2.
> > > > Had to patch some of them due to commons-lang migration (Hadoop
> 3.2.0)
> > > and
> > > > dependency divergence. Issues are being identified but so far nothing
> > > > blocker for Hadoop itself.
> > > >
> > > > Please try the release and vote. The vote will run for 5 days.
> > > >
> > > > My +1 to start,
> > > >
> > > > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > > > [2]
> > > >
> > >
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
> > >
> > > -
> > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> > >
> > >
> >
>








  fs.defaultFS
  hdfs://localhost
  The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.


  hadoop.http.filter.initializers
  org.apache.hadoop.security.HttpCrossOriginFilterInitializer

  Enable/disable the cross-origin (CORS) filter.
  hadoop.http.cross-origin.enabled
  true

  Comma separated list of origins tha

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/538/

[Jun 13, 2021 5:37:10 PM] (noreply) HADOOP-17645 Fix test failures in 
org.apache.hadoop.fs.azure.ITestOutputStreamSemantics. (#2926)
[Jun 13, 2021 6:22:29 PM] (noreply) HADOOP-17714 ABFS: 
testBlobBackCompatibility, testRandomRead & WasbAbfsCompatibility tests fail 
when triggered with default configs (#3035)
[Jun 14, 2021 1:17:39 AM] (noreply) HADOOP-17756. Increase precommit job 
timeout from 20 hours to 24 hours. (#3090)




-1 overall


The following subsystems voted -1:
blanks pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

Failed junit tests :

   hadoop.hdfs.TestRollingUpgrade 
   hadoop.yarn.csi.client.TestCsiClient 
   hadoop.tools.dynamometer.TestDynamometerInfra 
   hadoop.tools.dynamometer.TestDynamometerInfra 
  

   cc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/538/artifact/out/results-compile-cc-root.txt
 [96K]

   javac:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/538/artifact/out/results-compile-javac-root.txt
 [376K]

   blanks:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/538/artifact/out/blanks-eol.txt
 [13M]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/538/artifact/out/blanks-tabs.txt
 [2.0M]

   checkstyle:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/538/artifact/out/results-checkstyle-root.txt
 [16M]

   pathlen:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/538/artifact/out/results-pathlen.txt
 [16K]

   pylint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/538/artifact/out/results-pylint.txt
 [20K]

   shellcheck:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/538/artifact/out/results-shellcheck.txt
 [28K]

   xml:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/538/artifact/out/xml.txt
 [24K]

   javadoc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/538/artifact/out/results-javadoc-javadoc-root.txt
 [408K]

   unit:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/538/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 [556K]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/538/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-csi.txt
 [24K]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/538/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-infra.txt
 [8.0K]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/538/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer.txt
 [24K]

Powered by Apache Yetus 0.14.0-SNAPSHOT   https://yetus.apache.org

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org