RE: Apache Hadoop 3.1.1 release plan

2018-05-10 Thread Brahma Reddy Battula
Thanks Wangda Tan for driving the 3.1.1 release.Yes,This can be better addition 
to 3.1 line release for improving quality.

Looks only following two are pending which are in review state. Hope you are 
monitoring these two.

https://issues.apache.org/jira/browse/YARN-8265
https://issues.apache.org/jira/browse/YARN-8236



Note : https://issues.apache.org/jira/browse/YARN-8247==> committed branch-3.1


-Original Message-
From: Wangda Tan [mailto:wheele...@gmail.com] 
Sent: 19 April 2018 17:49
To: Hadoop Common ; 
mapreduce-...@hadoop.apache.org; Hdfs-dev ; 
yarn-dev@hadoop.apache.org
Subject: Apache Hadoop 3.1.1 release plan

Hi, All

We have released Apache Hadoop 3.1.0 on Apr 06. To further improve the quality 
of the release, we plan to release 3.1.1 at May 06. The focus of 3.1.1 will be 
fixing blockers / critical bugs and other enhancements. So far there are 100 
JIRAs [1] have fix version marked to 3.1.1.

We plan to cut branch-3.1.1 on May 01 and vote for RC on the same day.

Please feel free to share your insights.

Thanks,
Wangda Tan

[1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop Map/Reduce") AND 
fixVersion = 3.1.1


Re: Apache Hadoop 3.1.1 release plan

2018-05-10 Thread Sunil G
Thanks Brahma.
Yes, Billie is reviewing YARN-8265 and I am helping in YARN-8236.

- Sunil


On Thu, May 10, 2018 at 2:25 PM Brahma Reddy Battula <
brahmareddy.batt...@huawei.com> wrote:

> Thanks Wangda Tan for driving the 3.1.1 release.Yes,This can be better
> addition to 3.1 line release for improving quality.
>
> Looks only following two are pending which are in review state. Hope you
> are monitoring these two.
>
> https://issues.apache.org/jira/browse/YARN-8265
> https://issues.apache.org/jira/browse/YARN-8236
>
>
>
> Note : https://issues.apache.org/jira/browse/YARN-8247==> committed
> branch-3.1
>
>
> -Original Message-
> From: Wangda Tan [mailto:wheele...@gmail.com]
> Sent: 19 April 2018 17:49
> To: Hadoop Common ;
> mapreduce-...@hadoop.apache.org; Hdfs-dev ;
> yarn-dev@hadoop.apache.org
> Subject: Apache Hadoop 3.1.1 release plan
>
> Hi, All
>
> We have released Apache Hadoop 3.1.0 on Apr 06. To further improve the
> quality of the release, we plan to release 3.1.1 at May 06. The focus of
> 3.1.1 will be fixing blockers / critical bugs and other enhancements. So
> far there are 100 JIRAs [1] have fix version marked to 3.1.1.
>
> We plan to cut branch-3.1.1 on May 01 and vote for RC on the same day.
>
> Please feel free to share your insights.
>
> Thanks,
> Wangda Tan
>
> [1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop Map/Reduce")
> AND fixVersion = 3.1.1
>


Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2018-05-10 Thread Apache Jenkins Server
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/777/

[May 9, 2018 1:58:42 AM] (fabbri) HADOOP-15420 s3guard ITestS3GuardToolLocal 
failures in diff tests (Gabor
[May 9, 2018 4:51:41 AM] (msingh) HDDS-33. Ozone : Fix the test logic in
[May 9, 2018 6:30:43 AM] (rohithsharmaks) YARN-8239. [UI2] Clicking on Node 
Manager UI under AM container info /
[May 9, 2018 6:34:31 AM] (rohithsharmaks) YARN-8264. [UI2] GPU Info tab 
disappears if we click any sub link under
[May 9, 2018 6:47:04 AM] (rohithsharmaks) YARN-8260. [UI2] Per-application 
tracking URL is no longer available in
[May 9, 2018 1:36:07 PM] (msingh) HDDS-20. Ozone: Add support for rename key 
within a bucket for rpc
[May 9, 2018 2:50:21 PM] (msingh) HDDS-28. Ozone:Duplicate declaration in
[May 9, 2018 5:32:51 PM] (eyang) YARN-8261.  Fixed a bug in creation of 
localized container directory.   
[May 9, 2018 9:15:51 PM] (mackrorysd) HADOOP-15356. Make HTTP timeout 
configurable in ADLS connector.
[May 9, 2018 11:52:09 PM] (inigoiri) HDFS-13537. TestHdfsHelper does not 
generate jceks path properly for




-1 overall


The following subsystems voted -1:
asflicense findbugs unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-hdds/common 
   Found reliance on default encoding in 
org.apache.hadoop.utils.MetadataKeyFilters$KeyPrefixFilter.filterKey(byte[], 
byte[], byte[]):in 
org.apache.hadoop.utils.MetadataKeyFilters$KeyPrefixFilter.filterKey(byte[], 
byte[], byte[]): String.getBytes() At MetadataKeyFilters.java:[line 97] 

Failed junit tests :

   hadoop.util.TestReadWriteDiskValidator 
   hadoop.util.TestBasicDiskValidator 
   hadoop.fs.viewfs.TestViewFileSystemWithAuthorityLocalFileSystem 
   hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy 
   hadoop.hdfs.TestLeaseRecovery2 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy 
   
hadoop.yarn.server.nodemanager.containermanager.scheduler.TestContainerSchedulerQueuing
 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/777/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/777/artifact/out/diff-compile-javac-root.txt
  [328K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/777/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/777/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/777/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/777/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/777/artifact/out/whitespace-eol.txt
  [9.4M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/777/artifact/out/whitespace-tabs.txt
  [1.1M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/777/artifact/out/xml.txt
  [4.0K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/777/artifact/out/branch-findbugs-hadoop-hdds_client.txt
  [268K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/777/artifact/out/branch-findbugs-hadoop-hdds_common-warnings.html
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/777/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt
  [64K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/777/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt
  [48K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/777/artifact/out/branch-findbugs-hadoop-hdds_tools.txt
  [28K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/777/artifact/out/branch-findbugs-hadoop-ozone_client.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/777/artifact/out/branch-findbugs-hadoop-ozone_common.txt
  [24K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/777/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/777/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/777/artifact/out/branch-findbugs-hadoop-ozone_tools.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/777/artifact/out/branch-findbugs-hadoop-tools_hado

[jira] [Created] (YARN-8273) Log aggregation does not warn if HDFS quota in target directory is exceeded

2018-05-10 Thread Gergo Repas (JIRA)
Gergo Repas created YARN-8273:
-

 Summary: Log aggregation does not warn if HDFS quota in target 
directory is exceeded
 Key: YARN-8273
 URL: https://issues.apache.org/jira/browse/YARN-8273
 Project: Hadoop YARN
  Issue Type: Bug
  Components: log-aggregation
Affects Versions: 3.1.0
Reporter: Gergo Repas
Assignee: Gergo Repas


It appears that if an HDFS space quota is set on a target directory for log 
aggregation and the quota is already exceeded when log aggregation is 
attempted, zero-byte log files will be written to the HDFS directory, however 
NodeManager logs do not reflect a failure to write the files successfully (i.e. 
there are no ERROR or WARN messages to this effect).

An improvement may be worth investigating to alert users to this scenario, as 
otherwise logs for a YARN application may be missing both on HDFS and locally 
(after local log cleanup is done) and the user may not otherwise be informed.

Steps to reproduce:
* Set a small HDFS space quota on /tmp/logs/username/logs (e.g. 2MB)
* Write files to HDFS such that /tmp/logs/username/logs is almost 2MB full
* Run a Spark or MR job in the cluster
* Observe that zero byte files are written to HDFS after job completion
* Observe that YARN container logs are also not present on the NM hosts (or are 
deleted after yarn.nodemanager.delete.debug-delay-sec)
* Observe that no ERROR or WARN messages appear to be logged in the NM role log



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org



[jira] [Created] (YARN-8274) Docker command error during container relaunch

2018-05-10 Thread Billie Rinaldi (JIRA)
Billie Rinaldi created YARN-8274:


 Summary: Docker command error during container relaunch
 Key: YARN-8274
 URL: https://issues.apache.org/jira/browse/YARN-8274
 Project: Hadoop YARN
  Issue Type: Task
Reporter: Billie Rinaldi


I initiated container relaunch with a "sleep 60; exit 1" launch command and saw 
a "not a docker command" error on relaunch. Haven't figured out why this is 
happening, but it seems like it has been introduced recently to 
trunk/branch-3.1. cc [~shaneku...@gmail.com] [~ebadger]
{noformat}
org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException:
 Relaunch container failed
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.relaunchContainer(DockerLinuxContainerRuntime.java:954)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.relaunchContainer(DelegatingLinuxContainerRuntime.java:150)
at 
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.handleLaunchForLaunchType(LinuxContainerExecutor.java:562)
at 
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.relaunchContainer(LinuxContainerExecutor.java:486)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.relaunchContainer(ContainerLaunch.java:504)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch.call(ContainerRelaunch.java:111)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch.call(ContainerRelaunch.java:47)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-05-09 21:41:46,631 INFO 
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from 
container-launch.
2018-05-09 21:41:46,631 INFO 
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: 
container_1525897486447_0003_01_02
2018-05-09 21:41:46,631 INFO 
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 7
2018-05-09 21:41:46,631 INFO 
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception message: 
Relaunch container failed
2018-05-09 21:41:46,631 INFO 
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Shell error 
output: docker: 'container_1525897486447_0003_01_02' is not a docker 
command.
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org



[jira] [Created] (YARN-8275) Create a JNI interface to interact with Windows

2018-05-10 Thread Giovanni Matteo Fumarola (JIRA)
Giovanni Matteo Fumarola created YARN-8275:
--

 Summary: Create a JNI interface to interact with Windows
 Key: YARN-8275
 URL: https://issues.apache.org/jira/browse/YARN-8275
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: nodemanager
Reporter: Giovanni Matteo Fumarola
Assignee: Giovanni Matteo Fumarola


I did a quick investigation of the performance of WinUtils. In average NM calls 
4.76 times per second and 65.51 per container.

 
| |Requests|Requests/sec|Requests/min|Requests/container|
|*Sum [WinUtils]*|*135354*|*4.761*|*286.160*|*65.51*|
|[WinUtils] Execute -help|4148|0.145|8.769|2.007|
|[WinUtils] Execute -ls|2842|0.0999|6.008|1.37|
|[WinUtils] Execute -systeminfo|9153|0.321|19.35|4.43|
|[WinUtils] Execute -symlink|115096|4.048|243.33|57.37|
|[WinUtils] Execute -task isAlive|4115|0.144|8.699|2.05|

 Interval: 7 hours, 53 minutes and 48 seconds

Each execution of WinUtils does around *140 IO ops*, of which 130 are DDL ops.

This means *666.58* IO ops/second due to WinUtils.

We should start considering to remove WinUtils from Hadoop and creating a JNI 
interface.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org



Re: Apache Hadoop 3.1.1 release plan

2018-05-10 Thread Wangda Tan
Thanks Brahma/Sunil,

For YARN-8265, it is a too big change for 3.1.1, I just removed 3.1.1 from
target version.
For YARN-8236, it is a severe issue and I think it is close to finish.



On Thu, May 10, 2018 at 3:08 AM, Sunil G  wrote:

>
> Thanks Brahma.
> Yes, Billie is reviewing YARN-8265 and I am helping in YARN-8236.
>
> - Sunil
>
>
> On Thu, May 10, 2018 at 2:25 PM Brahma Reddy Battula <
> brahmareddy.batt...@huawei.com> wrote:
>
>> Thanks Wangda Tan for driving the 3.1.1 release.Yes,This can be better
>> addition to 3.1 line release for improving quality.
>>
>> Looks only following two are pending which are in review state. Hope you
>> are monitoring these two.
>>
>> https://issues.apache.org/jira/browse/YARN-8265
>> https://issues.apache.org/jira/browse/YARN-8236
>>
>>
>>
>> Note : https://issues.apache.org/jira/browse/YARN-8247==> committed
>> branch-3.1
>>
>>
>> -Original Message-
>> From: Wangda Tan [mailto:wheele...@gmail.com]
>> Sent: 19 April 2018 17:49
>> To: Hadoop Common ;
>> mapreduce-...@hadoop.apache.org; Hdfs-dev ;
>> yarn-dev@hadoop.apache.org
>> Subject: Apache Hadoop 3.1.1 release plan
>>
>> Hi, All
>>
>> We have released Apache Hadoop 3.1.0 on Apr 06. To further improve the
>> quality of the release, we plan to release 3.1.1 at May 06. The focus of
>> 3.1.1 will be fixing blockers / critical bugs and other enhancements. So
>> far there are 100 JIRAs [1] have fix version marked to 3.1.1.
>>
>> We plan to cut branch-3.1.1 on May 01 and vote for RC on the same day.
>>
>> Please feel free to share your insights.
>>
>> Thanks,
>> Wangda Tan
>>
>> [1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop
>> Map/Reduce") AND fixVersion = 3.1.1
>>
>


Re: Apache Hadoop 3.1.1 release plan

2018-05-10 Thread Wangda Tan
Hi all,

As we previously proposed RC time (May 1st), we want to release 3.1.1
sooner if possible. As of now, 3.1.1 has 187 fixes [1] on top of 3.1.0, and
there're 10 open blockers/criticals which target to 3.1.1 [2]. I just
posted comments to these open criticals/blockers ticket owners asking about
statuses.

If everybody agrees, I propose start code freeze of branch-3.1 from Sat PDT
time this week, only blockers/criticals can be committed to branch-3.1. To
avoid the burden of committers, I want to delay cutting branch-3.1.1 as
late as possible. If you have any major/minors (For severe issues please
update priorities) tickets want to go to 3.1.1, please reply this email
thread and we can look at them and make a call together.

Please feel free to share your comments and suggestions.

Thanks,
Wangda

[1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop Map/Reduce")
AND status = Resolved AND  fixVersion = 3.1.1
[2] project in (YARN, HADOOP, MAPREDUCE, "Hadoop Development Tools") AND
priority in (Blocker, Critical) AND resolution = Unresolved AND "Target
Version/s" = 3.1.1 ORDER BY priority DESC


On Thu, May 10, 2018 at 5:48 PM, Wangda Tan  wrote:

> Thanks Brahma/Sunil,
>
> For YARN-8265, it is a too big change for 3.1.1, I just removed 3.1.1 from
> target version.
> For YARN-8236, it is a severe issue and I think it is close to finish.
>
>
>
> On Thu, May 10, 2018 at 3:08 AM, Sunil G  wrote:
>
>>
>> Thanks Brahma.
>> Yes, Billie is reviewing YARN-8265 and I am helping in YARN-8236.
>>
>> - Sunil
>>
>>
>> On Thu, May 10, 2018 at 2:25 PM Brahma Reddy Battula <
>> brahmareddy.batt...@huawei.com> wrote:
>>
>>> Thanks Wangda Tan for driving the 3.1.1 release.Yes,This can be better
>>> addition to 3.1 line release for improving quality.
>>>
>>> Looks only following two are pending which are in review state. Hope you
>>> are monitoring these two.
>>>
>>> https://issues.apache.org/jira/browse/YARN-8265
>>> https://issues.apache.org/jira/browse/YARN-8236
>>>
>>>
>>>
>>> Note : https://issues.apache.org/jira/browse/YARN-8247==> committed
>>> branch-3.1
>>>
>>>
>>> -Original Message-
>>> From: Wangda Tan [mailto:wheele...@gmail.com]
>>> Sent: 19 April 2018 17:49
>>> To: Hadoop Common ;
>>> mapreduce-...@hadoop.apache.org; Hdfs-dev ;
>>> yarn-dev@hadoop.apache.org
>>> Subject: Apache Hadoop 3.1.1 release plan
>>>
>>> Hi, All
>>>
>>> We have released Apache Hadoop 3.1.0 on Apr 06. To further improve the
>>> quality of the release, we plan to release 3.1.1 at May 06. The focus of
>>> 3.1.1 will be fixing blockers / critical bugs and other enhancements. So
>>> far there are 100 JIRAs [1] have fix version marked to 3.1.1.
>>>
>>> We plan to cut branch-3.1.1 on May 01 and vote for RC on the same day.
>>>
>>> Please feel free to share your insights.
>>>
>>> Thanks,
>>> Wangda Tan
>>>
>>> [1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop
>>> Map/Reduce") AND fixVersion = 3.1.1
>>>
>>
>


Re: Apache Hadoop 3.1.1 release plan

2018-05-10 Thread Weiwei Yang
Hi Wangda

I would propose to have https://issues.apache.org/jira/browse/YARN-8015 
included in 3.1.1.

Once this is done, we get both intra and inter placement constraint covered so 
users could start to explore this feature. Otherwise the functionality is 
pretty limited. It has been Patch Available for a while, I just promoted it 
targeting to 3.1.1. Hope that makes sense.

Thanks!

--
Weiwei

On 11 May 2018, 9:02 AM +0800, Wangda Tan , wrote:
Hi all,

As we previously proposed RC time (May 1st), we want to release 3.1.1
sooner if possible. As of now, 3.1.1 has 187 fixes [1] on top of 3.1.0, and
there're 10 open blockers/criticals which target to 3.1.1 [2]. I just
posted comments to these open criticals/blockers ticket owners asking about
statuses.

If everybody agrees, I propose start code freeze of branch-3.1 from Sat PDT
time this week, only blockers/criticals can be committed to branch-3.1. To
avoid the burden of committers, I want to delay cutting branch-3.1.1 as
late as possible. If you have any major/minors (For severe issues please
update priorities) tickets want to go to 3.1.1, please reply this email
thread and we can look at them and make a call together.

Please feel free to share your comments and suggestions.

Thanks,
Wangda

[1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop Map/Reduce")
AND status = Resolved AND fixVersion = 3.1.1
[2] project in (YARN, HADOOP, MAPREDUCE, "Hadoop Development Tools") AND
priority in (Blocker, Critical) AND resolution = Unresolved AND "Target
Version/s" = 3.1.1 ORDER BY priority DESC


On Thu, May 10, 2018 at 5:48 PM, Wangda Tan  wrote:

Thanks Brahma/Sunil,

For YARN-8265, it is a too big change for 3.1.1, I just removed 3.1.1 from
target version.
For YARN-8236, it is a severe issue and I think it is close to finish.



On Thu, May 10, 2018 at 3:08 AM, Sunil G  wrote:


Thanks Brahma.
Yes, Billie is reviewing YARN-8265 and I am helping in YARN-8236.

- Sunil


On Thu, May 10, 2018 at 2:25 PM Brahma Reddy Battula <
brahmareddy.batt...@huawei.com> wrote:

Thanks Wangda Tan for driving the 3.1.1 release.Yes,This can be better
addition to 3.1 line release for improving quality.

Looks only following two are pending which are in review state. Hope you
are monitoring these two.

https://issues.apache.org/jira/browse/YARN-8265
https://issues.apache.org/jira/browse/YARN-8236



Note : https://issues.apache.org/jira/browse/YARN-8247==> committed
branch-3.1


-Original Message-
From: Wangda Tan [mailto:wheele...@gmail.com]
Sent: 19 April 2018 17:49
To: Hadoop Common ;
mapreduce-...@hadoop.apache.org; Hdfs-dev ;
yarn-dev@hadoop.apache.org
Subject: Apache Hadoop 3.1.1 release plan

Hi, All

We have released Apache Hadoop 3.1.0 on Apr 06. To further improve the
quality of the release, we plan to release 3.1.1 at May 06. The focus of
3.1.1 will be fixing blockers / critical bugs and other enhancements. So
far there are 100 JIRAs [1] have fix version marked to 3.1.1.

We plan to cut branch-3.1.1 on May 01 and vote for RC on the same day.

Please feel free to share your insights.

Thanks,
Wangda Tan

[1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop
Map/Reduce") AND fixVersion = 3.1.1





Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64

2018-05-10 Thread Apache Jenkins Server
For more details, see https://builds.apache.org/job/hadoop-trunk-win/463/

[May 9, 2018 1:36:07 PM] (msingh) HDDS-20. Ozone: Add support for rename key 
within a bucket for rpc
[May 9, 2018 2:50:21 PM] (msingh) HDDS-28. Ozone:Duplicate declaration in
[May 9, 2018 5:32:51 PM] (eyang) YARN-8261.  Fixed a bug in creation of 
localized container directory.   
[May 9, 2018 9:15:51 PM] (mackrorysd) HADOOP-15356. Make HTTP timeout 
configurable in ADLS connector.
[May 9, 2018 11:52:09 PM] (inigoiri) HDFS-13537. TestHdfsHelper does not 
generate jceks path properly for
[May 10, 2018 3:45:46 AM] (bibinchundatt) YARN-8201. Skip stacktrace of few 
exception from ClientRMService.
[May 10, 2018 5:17:48 AM] (vrushali) YARN-8247 Incorrect HTTP status code 
returned by ATSv2 for
[May 10, 2018 6:00:13 AM] (aengineer) HDDS-18. Ozone Shell should use 
RestClient and RpcClient. Contributed by
[May 10, 2018 9:38:08 AM] (aajisaka) HADOOP-15354. hadoop-aliyun & hadoop-azure 
modules to mark hadoop-common




-1 overall


The following subsystems voted -1:
compile mvninstall unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h 00m 00s)
unit


Specific tests:

Failed junit tests :

   hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec 
   hadoop.fs.contract.rawlocal.TestRawlocalContractAppend 
   hadoop.fs.TestFsShellCopy 
   hadoop.fs.TestFsShellList 
   hadoop.fs.TestLocalFileSystem 
   hadoop.fs.TestRawLocalFileSystemContract 
   hadoop.fs.TestSymlinkLocalFSFileContext 
   hadoop.fs.TestTrash 
   hadoop.http.TestHttpServer 
   hadoop.http.TestHttpServerLogs 
   hadoop.io.nativeio.TestNativeIO 
   hadoop.ipc.TestCallQueueManager 
   hadoop.ipc.TestProtoBufRpc 
   hadoop.ipc.TestSocketFactory 
   hadoop.metrics2.impl.TestStatsDMetrics 
   hadoop.metrics2.sink.TestRollingFileSystemSinkWithLocal 
   hadoop.security.TestSecurityUtil 
   hadoop.security.TestShellBasedUnixGroupsMapping 
   hadoop.security.token.TestDtUtilShell 
   hadoop.test.TestLambdaTestUtils 
   hadoop.util.TestNativeCodeLoader 
   hadoop.util.TestNodeHealthScriptRunner 
   hadoop.util.TestWinUtils 
   hadoop.fs.TestResolveHdfsSymlink 
   hadoop.hdfs.client.impl.TestBlockReaderLocal 
   hadoop.hdfs.crypto.TestHdfsCryptoStreams 
   hadoop.hdfs.qjournal.client.TestQuorumJournalManager 
   hadoop.hdfs.qjournal.server.TestJournalNode 
   hadoop.hdfs.qjournal.server.TestJournalNodeSync 
   hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages 
   hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks 
   
hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy 
   
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement 
   
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestProvidedImpl 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation 
   hadoop.hdfs.server.datanode.TestBlockPoolSliceStorage 
   hadoop.hdfs.server.datanode.TestBlockRecovery 
   hadoop.hdfs.server.datanode.TestBlockScanner 
   hadoop.hdfs.server.datanode.TestDataNodeFaultInjector 
   hadoop.hdfs.server.datanode.TestDataNodeMetrics 
   hadoop.hdfs.server.datanode.TestDataNodeUUID 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.hdfs.server.datanode.TestHSync 
   hadoop.hdfs.server.datanode.TestStorageReport 
   hadoop.hdfs.server.datanode.web.TestDatanodeHttpXFrame 
   hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand 
   hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC 
   hadoop.hdfs.server.mover.TestStorageMover 
   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA 
   hadoop.hdfs.server.namenode.ha.TestEditLogTailer 
   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA 
   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics 
   
hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot 
   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot 
   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots 
   hadoop.hdfs.server.namenode.snapshot.TestSnapRootDescendantDiff 
   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport 
   hadoop.hdfs.server.namenode.TestAddBlock