Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

[Junping Du]

Thanks Andrew for reporting the issue. This JIRA is out of my radar as it? 
didn't specify any target version before.


>From my understanding, this issue is related to our previous improvements with 
>separating client and server jars in HDFS-6200. If we use the new "client" jar 
>in NN HA deployment, then we will hit the issue reported.


I can see two options here:

- Without any change in 2.8.0, if user hit the issue when they deploy HA 
cluster by using new client jar, adding back hdfs jar just like how things work 
previously

- Make the change now in 2.8.0, either moving ConfiguredFailoverProxyProvider 
to client jar or adding dependency between client jar and server jar. There 
must be some arguments there on which way to fix is better especially 
ConfiguredFailoverProxyProvider still has some sever side dependencies.


I would prefer the first option, given:

- The issue fixing time is unpredictable as there are still discussion on how 
to fix this issue. Our 2.8.0 release shouldn't be an endless journey which has 
been deferred several times for more serious issue.

- We have workaround for this improvement, no regression happens due to this 
issue. People can still use hdfs jar in old way. The worst case is improvement 
for HDFS doesn't work in some cases - that shouldn't block the whole release.


I think we should let vote keep going unless someone have more concerns which I 
could miss.



Thanks,


Junping



From: Andrew Wang 
Sent: Tuesday, March 14, 2017 2:50 PM
To: Junping Du
Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

Hi Junping,

Noticed this possible blocker float by my inbox today. It had an affects but no 
target version set:

https://issues.apache.org/jira/browse/HDFS-11431

Thoughts? Seems like the hadoop-hdfs-client artifact doesn't work right now.

Best,
Andrew


On Tue, Mar 14, 2017 at 1:41 AM, Junping Du 
> wrote:
Hi all,
 With several important fixes get merged last week, I've created a new 
release candidate (RC2) for Apache Hadoop 2.8.0.

 This is the next minor release to follow up 2.7.0 which has been released 
for more than 1 year. It comprises 2,919 fixes, improvements, and new features. 
Most of these commits are released for the first time in branch-2.

  More information about the 2.8.0 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  Please note that RC0 and RC1 are not voted public because significant 
issues are found just after RC tag getting published.

  The RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.0-RC2

  The RC tag in git is: release-2.8.0-RC2

  The maven artifacts are available via 
repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1056

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 03/20/2017 PDT time.

Thanks,

Junping

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

[Andrew Wang]

Hi Junping,

Noticed this possible blocker float by my inbox today. It had an affects
but no target version set:

https://issues.apache.org/jira/browse/HDFS-11431

Thoughts? Seems like the hadoop-hdfs-client artifact doesn't work right now.

Best,
Andrew


On Tue, Mar 14, 2017 at 1:41 AM, Junping Du  wrote:

> Hi all,
>  With several important fixes get merged last week, I've created a new
> release candidate (RC2) for Apache Hadoop 2.8.0.
>
>  This is the next minor release to follow up 2.7.0 which has been
> released for more than 1 year. It comprises 2,919 fixes, improvements, and
> new features. Most of these commits are released for the first time in
> branch-2.
>
>   More information about the 2.8.0 release plan can be found here:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>
>   Please note that RC0 and RC1 are not voted public because
> significant issues are found just after RC tag getting published.
>
>   The RC is available at: http://home.apache.org/~
> junping_du/hadoop-2.8.0-RC2
>
>   The RC tag in git is: release-2.8.0-RC2
>
>   The maven artifacts are available via repository.apache.org at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1056
>
>   Please try the release and vote; the vote will run for the usual 5
> days, ending on 03/20/2017 PDT time.
>
> Thanks,
>
> Junping
>

Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/257/

[Mar 13, 2017 3:11:25 PM] (jzhuge) HADOOP-14173. Remove unused 
AdlConfKeys#ADL_EVENTS_TRACKING_SOURCE.
[Mar 13, 2017 9:15:02 PM] (liuml07) HADOOP-14170. FileSystemContractBaseTest is 
not cleaning up test
[Mar 13, 2017 9:24:51 PM] (jing9) HDFS-11395. 
RequestHedgingProxyProvider#RequestHedgingInvocationHandler
[Mar 14, 2017 12:30:10 AM] (arp) HDFS-11482. Add storage type demand to into
[Mar 14, 2017 6:17:11 AM] (sunilg) YARN-5496. Make Node Heatmap Chart 
categories clickable in new YARN UI.
[Mar 14, 2017 9:49:48 AM] (yqlin) HDFS-11526. Fix confusing block recovery 
message. Contributed by Yiqun
[Mar 14, 2017 9:58:07 AM] (junping_du) YARN-6314. Potential infinite 
redirection on YARN log redirection web




-1 overall


The following subsystems voted -1:
compile unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.TestFileChecksum 
   hadoop.hdfs.TestEncryptedTransfer 
   hadoop.hdfs.TestHDFSFileSystemContract 
   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer 
   hadoop.hdfs.TestMaintenanceState 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.web.TestWebHdfsFileSystemContract 
   hadoop.yarn.server.timeline.TestRollingLevelDB 
   hadoop.yarn.server.timeline.TestTimelineDataManager 
   hadoop.yarn.server.timeline.TestLeveldbTimelineStore 
   hadoop.yarn.server.timeline.webapp.TestTimelineWebServices 
   hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore 
   hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore 
   
hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer 
   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler 
   hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore 
   hadoop.yarn.server.resourcemanager.TestRMRestart 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.client.api.impl.TestAMRMClient 
   hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore 
   hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient 
   hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.mapred.TestShuffleHandler 
   hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService 
   hadoop.fs.azure.TestNativeAzureFileSystemContractMocked 

Timed out junit tests :

   org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache 
  

   compile:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/257/artifact/out/patch-compile-root.txt
  [136K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/257/artifact/out/patch-compile-root.txt
  [136K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/257/artifact/out/patch-compile-root.txt
  [136K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/257/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [336K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/257/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/257/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [56K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/257/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [72K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/257/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [324K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/257/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/257/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt
  [28K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/257/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/257/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-ui.txt
  [8.0K]
   

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/345/

[Mar 13, 2017 7:15:53 AM] (aajisaka) HADOOP-14156. Fix grammar error in 
ConfTest.java.
[Mar 13, 2017 10:22:30 AM] (yqlin) HDFS-11512. Increase timeout on
[Mar 13, 2017 3:11:25 PM] (jzhuge) HADOOP-14173. Remove unused 
AdlConfKeys#ADL_EVENTS_TRACKING_SOURCE.
[Mar 13, 2017 9:15:02 PM] (liuml07) HADOOP-14170. FileSystemContractBaseTest is 
not cleaning up test
[Mar 13, 2017 9:24:51 PM] (jing9) HDFS-11395. 
RequestHedgingProxyProvider#RequestHedgingInvocationHandler
[Mar 14, 2017 12:30:10 AM] (arp) HDFS-11482. Add storage type demand to into
[Mar 14, 2017 6:17:11 AM] (sunilg) YARN-5496. Make Node Heatmap Chart 
categories clickable in new YARN UI.




-1 overall


The following subsystems voted -1:
asflicense compile unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes 
   hadoop.hdfs.TestHDFSFileSystemContract 
   hadoop.hdfs.web.TestWebHdfsFileSystemContract 
   hadoop.hdfs.server.balancer.TestBalancer 
   hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker 
   hadoop.yarn.server.nodemanager.containermanager.TestContainerManager 
   hadoop.yarn.server.timeline.webapp.TestTimelineWebServices 
   hadoop.yarn.server.resourcemanager.TestResourceTrackerService 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.fs.azure.TestNativeAzureFileSystemContractMocked 
  

   compile:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/345/artifact/out/patch-compile-root.txt
  [204K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/345/artifact/out/patch-compile-root.txt
  [204K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/345/artifact/out/patch-compile-root.txt
  [204K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/345/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/345/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/345/artifact/out/diff-patch-shellcheck.txt
  [24K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/345/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/345/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/345/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/345/artifact/out/diff-javadoc-javadoc-root.txt
  [2.2M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/345/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [392K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/345/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [36K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/345/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/345/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [60K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/345/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [324K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/345/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-ui.txt
  [32K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/345/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt
  [16K]

   asflicense:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/345/artifact/out/patch-asflicense-problems.txt
  [4.0K]

Powered by Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: 

[jira] [Created] (HADOOP-14182) Support for pluggable security

[Yuliya Feldman (JIRA)]

Yuliya Feldman created HADOOP-14182:
---

 Summary: Support for pluggable security
 Key: HADOOP-14182
 URL: https://issues.apache.org/jira/browse/HADOOP-14182
 Project: Hadoop Common
  Issue Type: Improvement
  Components: common, security
Reporter: Yuliya Feldman


Today all SASL based authentications allow just Kerberos and Digest.
If we want to use yet another different SASL based authentication and adding 
one more "if/else" with some code specific just to that new way does not make 
much sense.
Proposal is to allow to plug custom SASL Authentication mechanism(s) without 
further changes in Hadoop code.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14181) Add validation of DynamoDB region

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14181:
--

 Summary: Add validation of DynamoDB region
 Key: HADOOP-14181
 URL: https://issues.apache.org/jira/browse/HADOOP-14181
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory
Assignee: Sean Mackrory


Currently you can configure a nonsensical AWS region for DynamoDB and it 
doesn't stop you - it'll just start using another region silently. I propose 
that we just check it's a valid region for the current SDK and throw an 
exception instead of proceeding.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14179) Namenode - changing config without restart

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HADOOP-14179.
-
Resolution: Invalid

https://wiki.apache.org/hadoop/InvalidJiraIssues

> Namenode - changing config without restart
> --
>
> Key: HADOOP-14179
> URL: https://issues.apache.org/jira/browse/HADOOP-14179
> Project: Hadoop Common
>  Issue Type: Wish
>  Components: conf
> Environment: SL5/6
>Reporter: Benedikt Maier
>   Original Estimate: 12h
>  Remaining Estimate: 12h
>
> Hi there,
> Is there a way to change the address of the secondary namenode in the 
> configuration files of the primary namenode without having to restart the 
> latter for the changes to have an effect? If one can avoid any downtime for 
> the namenode for such a change, that would be great. 
> Thanks



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[VOTE] Release Apache Hadoop 2.8.0 (RC2)

[Junping Du]

Hi all,
 With several important fixes get merged last week, I've created a new 
release candidate (RC2) for Apache Hadoop 2.8.0.

 This is the next minor release to follow up 2.7.0 which has been released 
for more than 1 year. It comprises 2,919 fixes, improvements, and new features. 
Most of these commits are released for the first time in branch-2.

  More information about the 2.8.0 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  Please note that RC0 and RC1 are not voted public because significant 
issues are found just after RC tag getting published.

  The RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.0-RC2

  The RC tag in git is: release-2.8.0-RC2

  The maven artifacts are available via repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1056

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 03/20/2017 PDT time.

Thanks,

Junping