Re: About 2.7.4 Release

[Allen Wittenauer]

> On Mar 8, 2017, at 1:54 PM, Allen Wittenauer  
> wrote:
> 
>   This is already possible:
>   * don’t use —asfrelease
>   * use —sign, —native, and, if appropriate for your platform, 
> —docker and —dockercache


Oh yeah, I forgot about this:

https://effectivemachines.com/2016/08/16/building-your-own-apache-hadoop-distribution/



-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: About 2.7.4 Release

[Allen Wittenauer]

> On Mar 8, 2017, at 10:55 AM, Marton Elek  wrote:
> 
> I think the main point here is the testing of the release script, not the 
> creation of the official release.

… except the Hadoop PMC was doing exactly this from 2.3.0 up until 
recently. Which means we have a few years worth of releases that are 
effectively untrustworthy despite being signed.  One of the (many) reasons I 
rewrote the release process was to get Hadoop back in line with ASF policy.  
Given the massive turn over in committers, I don’t want us to repeat the same 
mistakes (like we usually do).

> I think there should be an option to configure the release tool to use a 
> forked github repo and/or a private playground nexus instead of official 
> apache repos. In this case it would be easy to test regularly the tool, even 
> by a non-committer (or even from Jenkins). But it would be just a smoketest 
> of the release script…

This is already possible:
* don’t use —asfrelease
* use —sign, —native, and, if appropriate for your platform, 
—docker and —dockercache


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/339/

[Mar 7, 2017 6:12:35 PM] (arp) HDFS-11477. Simplify file IO profiling 
configuration. Contributed by
[Mar 7, 2017 7:41:05 PM] (arp) HDFS-11508. Fix bind failure in SimpleTCPServer 
& Portmap where bind
[Mar 7, 2017 7:58:48 PM] (templedf) YARN-6287. 
RMCriticalThreadUncaughtExceptionHandler.rmContext should be
[Mar 7, 2017 9:34:46 PM] (rkanter) MAPREDUCE-6839. TestRecovery.testCrashed 
failed (pairg via rkanter)
[Mar 7, 2017 9:47:52 PM] (rkanter) YARN-6275. Fail to show real-time tracking 
charts in SLS (yufeigu via
[Mar 7, 2017 10:55:52 PM] (liuml07) HADOOP-14150. Implement getHomeDirectory() 
method in
[Mar 8, 2017 6:34:30 AM] (sunilg) YARN-6207. Move application across queues 
should handle delayed event




-1 overall


The following subsystems voted -1:
asflicense unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.server.balancer.TestBalancer 
   hadoop.yarn.server.timeline.webapp.TestTimelineWebServices 
   
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption 
   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.client.api.impl.TestAMRMClient 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/339/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/339/artifact/out/diff-compile-javac-root.txt
  [176K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/339/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/339/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/339/artifact/out/diff-patch-shellcheck.txt
  [24K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/339/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/339/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/339/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/339/artifact/out/diff-javadoc-javadoc-root.txt
  [2.2M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/339/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [268K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/339/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/339/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [60K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/339/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [324K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/339/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/339/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
  [8.0K]

   asflicense:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/339/artifact/out/patch-asflicense-problems.txt
  [4.0K]

Powered by Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org



-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-6856) TestRecovery.testSpeculative fails if testCrashed fails

[JIRA]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gergő Pásztor resolved MAPREDUCE-6856.
--
Resolution: Fixed

> TestRecovery.testSpeculative fails if testCrashed fails
> ---
>
> Key: MAPREDUCE-6856
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6856
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>
> The test {{testSpeculative}} in 
> {{org.apache.hadoop.mapreduce.v2.app.TestRecovery}} is unstable.
> Based on my findings, the test itself is not problematic. It only fails if 
> {{testCrashed}} in the same class fails before it.
> The reason is not completely clear to me, but I whenever I explicitly stop 
> the MRAppMaster in {{testCrashed}} in a finally block, then the issue 
> disappears. I think the reason is that both tests uses the same folder for 
> staging.
> Solution: wrap logic in {{testCrashed}} in a try-finally block and then stop 
> the MRAppMaster.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/251/

[Mar 7, 2017 6:12:35 PM] (arp) HDFS-11477. Simplify file IO profiling 
configuration. Contributed by
[Mar 7, 2017 7:41:05 PM] (arp) HDFS-11508. Fix bind failure in SimpleTCPServer 
& Portmap where bind
[Mar 7, 2017 7:58:48 PM] (templedf) YARN-6287. 
RMCriticalThreadUncaughtExceptionHandler.rmContext should be
[Mar 7, 2017 9:34:46 PM] (rkanter) MAPREDUCE-6839. TestRecovery.testCrashed 
failed (pairg via rkanter)
[Mar 7, 2017 9:47:52 PM] (rkanter) YARN-6275. Fail to show real-time tracking 
charts in SLS (yufeigu via
[Mar 7, 2017 10:55:52 PM] (liuml07) HADOOP-14150. Implement getHomeDirectory() 
method in
[Mar 8, 2017 6:34:30 AM] (sunilg) YARN-6207. Move application across queues 
should handle delayed event




-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-4694) Inconsistency in reduce input record counters between the stable and evolving APIs

[Harsh J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J resolved MAPREDUCE-4694.

Resolution: Won't Fix
  Assignee: (was: Harsh J)

This bug's existed for over 5 years now. I've not seen new complaints/interest 
in unifying the behaviour. Closing out as expected behaviour.

> Inconsistency in reduce input record counters between the stable and evolving 
> APIs
> --
>
> Key: MAPREDUCE-4694
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4694
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: client
>Affects Versions: 2.0.0-alpha
>Reporter: Harsh J
>  Labels: inconsistency, regression, test
> Attachments: MAPREDUCE-4694.patch, MAPREDUCE-4694.patch
>
>
> In the stable (mapred) API execution, if the values iterator is skipped by a 
> user, the records underneath it aren't counted in the "Reduce input records" 
> counter as the key progresses to the next unique one. In the evolving API 
> (mapreduce) API execution, if the values iterator is skipped by a user, the 
> records underneath it is still counted as the key progresses to the next 
> unique one.
> This behavior comes to me as a faulty one in the old API. A "Reduce input 
> records" counter must always define all the records that have been passed 
> into a reducer (cause they are read regardless of skipping), and both API's 
> record counting despite user applications must be consistent.
> I'll post a test case illustrating this shortly.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6861) Add metrics tags for ShuffleClientMetrics

[Akira Ajisaka (JIRA)]

Akira Ajisaka created MAPREDUCE-6861:


 Summary: Add metrics tags for ShuffleClientMetrics
 Key: MAPREDUCE-6861
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6861
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 3.0.0-alpha1
Reporter: Akira Ajisaka


Metrics tags were unintentionally removed by MAPREDUCE-6526. Let's add them 
back.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org