[jira] [Created] (HADOOP-14285) Update minimum version of Maven from 3.0 to 3.3

[Akira Ajisaka (JIRA)]

Akira Ajisaka created HADOOP-14285:
--

 Summary: Update minimum version of Maven from 3.0 to 3.3
 Key: HADOOP-14285
 URL: https://issues.apache.org/jira/browse/HADOOP-14285
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Akira Ajisaka
Assignee: Akira Ajisaka


YARN-6421 requires Apache Maven 3.1+



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Upgrading minimum version of Maven to 3.1 from 3.0

[Akira Ajisaka]

Based on the positive feedback from Junping, filed HADOOP-14285 for the 
update.

Discussion can continue in the JIRA.

Thanks,
Akira

On 2017/04/05 12:56, Akira Ajisaka wrote:

Agreed to upgrade to 3.3+ and change the version only in trunk.

-Akira

On 2017/04/05 11:21, Sunil G wrote:

Thanks Akira and Junping.

@Junping. Yes, we plan to change this only in trunk and new YARN UI is
available there.
I also feel that there is no issue in moving to a latest version such as
3.3.

+ Sunil

On Tue, Apr 4, 2017 at 11:35 PM Junping Du mailto:j...@hortonworks.com>> wrote:

Thanks Sunil to bring it up. However, any reason not to go to latest
version of maven (3.3+)? Even version 3.1 is quite old - released 3+
years from now (please refer:
https://archive.apache.org/dist/maven/binaries/).
btw, I assume we only plan to change maven version on trunk given
new YARN UI only merge to trunk branch. Isn't it?

Thanks,

Junping

From: Akira Ajisaka mailto:aajis...@apache.org>>
Sent: Monday, April 03, 2017 10:02 PM
To: Sunil G; common-dev@hadoop.apache.org
; hdfs-...@hadoop.apache.org
; yarn-...@hadoop.apache.org
; mapreduce-...@hadoop.apache.org

Subject: Re: Upgrading minimum version of Maven to 3.1 from 3.0

As I mentioned in YARN-6421. I'm +1 for upgrading to 3.1+ because
the latest version of Maven 3.0.x is quite old. (4 years ago)

We need to update dev-support/docker/Dockerfile to enable Maven
3.1+ for
precommit Jenkins job.

Regards,
Akira

On 2017/04/03 18:49, Sunil G wrote:
> Hi Folks,
>
> Recently we were doing build framework up-gradation for Yarn Ui.
In order
> to compile yarn-ui on various architectures, we were using
> frontend-maven-plugin 0.0.22 version.
> However build is failing in *ppc64le.* If we could use latest
version of
> frontend-maven-plugin, we could resolve this error. (such as
using 1.1
> version). But this requires maven version 3.1 minimum. YARN-6421 is
> tracking this issue, and we thought we can propose to upgrade to
maven 3.1
>
> Kindly share your thoughts.
>
> Thanks
> + Sunil
>

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org

For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org




-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14286) S3A index title links in hadoop-aws don't work in docs

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-14286:
---

 Summary: S3A index title links in hadoop-aws don't work in docs
 Key: HADOOP-14286
 URL: https://issues.apache.org/jira/browse/HADOOP-14286
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation, fs/s3
Affects Versions: 2.8.0
Reporter: Steve Loughran
Priority: Minor


Some of the indexed title entries for S3A: "authentication", "other 
properties", don't resolve when clicked as there are other entries for S3 and 
S3N with the same title. They need to be given unique titles to the links work



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/368/

[Apr 5, 2017 8:33:07 AM] (aajisaka) YARN-5007. Remove deprecated constructors 
of MiniYARNCluster and
[Apr 5, 2017 5:30:45 PM] (jlowe) YARN-6403. Invalid local resource request can 
raise NPE and make NM
[Apr 5, 2017 6:28:30 PM] (wangda) YARN-6406. Remove SchedulerRequestKeys when 
no more pending
[Apr 5, 2017 7:50:41 PM] (varunsaxena) YARN-6437. 
TestSignalContainer#testSignalRequestDeliveryToNM fails
[Apr 5, 2017 10:12:03 PM] (wang) HADOOP-14280. Fix compilation of 
TestKafkaMetrics.
[Apr 5, 2017 10:33:08 PM] (arp) HDFS-11628. Clarify the behavior of HDFS Mover 
in documentation.
[Apr 5, 2017 10:43:14 PM] (kasha) YARN-6448. Continuous scheduling thread 
crashes while sorting nodes.
[Apr 5, 2017 11:04:09 PM] (wang) HFDS-11596. hadoop-hdfs-client jar is in the 
wrong directory in release
[Apr 5, 2017 11:07:24 PM] (templedf) YARN-6381. FSAppAttempt has several 
variables that should be final
[Apr 6, 2017 12:25:40 AM] (templedf) MAPREDUCE-6846. Fragments specified for 
libjar paths are not handled
[Apr 6, 2017 12:28:00 AM] (xyao) HDFS-11302. Improve Logging for 
SSLHostnameVerifier. Contributed by Chen
[Apr 6, 2017 12:54:30 AM] (arp) HDFS-11131. 
TestThrottledAsyncChecker#testCancellation is flaky.
[Apr 6, 2017 4:45:22 AM] (rohithsharmaks) YARN-6424. TimelineCollector is not 
stopped when an app finishes in RM.




-1 overall


The following subsystems voted -1:
asflicense unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.TestErasureCodeBenchmarkThroughput 
   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock 
   hadoop.mapreduce.TestMRJobClient 
   hadoop.tools.TestHadoopArchiveLogsRunner 
   hadoop.metrics2.impl.TestKafkaMetrics 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/368/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/368/artifact/out/diff-compile-javac-root.txt
  [184K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/368/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/368/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/368/artifact/out/diff-patch-shellcheck.txt
  [24K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/368/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/368/artifact/out/whitespace-eol.txt
  [12M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/368/artifact/out/whitespace-tabs.txt
  [1.2M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/368/artifact/out/diff-javadoc-javadoc-root.txt
  [2.2M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/368/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [160K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/368/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
  [88K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/368/artifact/out/patch-unit-hadoop-tools_hadoop-archive-logs.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/368/artifact/out/patch-unit-hadoop-tools_hadoop-kafka.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/368/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [324K]

   asflicense:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/368/artifact/out/patch-asflicense-problems.txt
  [4.0K]

Powered by Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/280/

[Apr 5, 2017 5:30:45 PM] (jlowe) YARN-6403. Invalid local resource request can 
raise NPE and make NM
[Apr 5, 2017 6:28:30 PM] (wangda) YARN-6406. Remove SchedulerRequestKeys when 
no more pending
[Apr 5, 2017 7:50:41 PM] (varunsaxena) YARN-6437. 
TestSignalContainer#testSignalRequestDeliveryToNM fails
[Apr 5, 2017 10:12:03 PM] (wang) HADOOP-14280. Fix compilation of 
TestKafkaMetrics.
[Apr 5, 2017 10:33:08 PM] (arp) HDFS-11628. Clarify the behavior of HDFS Mover 
in documentation.
[Apr 5, 2017 10:43:14 PM] (kasha) YARN-6448. Continuous scheduling thread 
crashes while sorting nodes.
[Apr 5, 2017 11:04:09 PM] (wang) HFDS-11596. hadoop-hdfs-client jar is in the 
wrong directory in release
[Apr 5, 2017 11:07:24 PM] (templedf) YARN-6381. FSAppAttempt has several 
variables that should be final
[Apr 6, 2017 12:25:40 AM] (templedf) MAPREDUCE-6846. Fragments specified for 
libjar paths are not handled
[Apr 6, 2017 12:28:00 AM] (xyao) HDFS-11302. Improve Logging for 
SSLHostnameVerifier. Contributed by Chen
[Apr 6, 2017 12:54:30 AM] (arp) HDFS-11131. 
TestThrottledAsyncChecker#testCancellation is flaky.
[Apr 6, 2017 4:45:22 AM] (rohithsharmaks) YARN-6424. TimelineCollector is not 
stopped when an app finishes in RM.




-1 overall


The following subsystems voted -1:
compile unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.TestWriteReadStripedFile 
   hadoop.hdfs.server.namenode.ha.TestHAMetrics 
   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.server.timeline.TestRollingLevelDB 
   hadoop.yarn.server.timeline.TestTimelineDataManager 
   hadoop.yarn.server.timeline.TestLeveldbTimelineStore 
   hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore 
   hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore 
   
hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer 
   hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore 
   hadoop.yarn.server.resourcemanager.TestRMRestart 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.client.api.impl.TestAMRMClient 
   hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore 
   hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient 
   hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.mapred.TestShuffleHandler 
   hadoop.mapreduce.v2.app.TestRuntimeEstimators 
   hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService 
   hadoop.tools.TestHadoopArchiveLogsRunner 
   hadoop.metrics2.impl.TestKafkaMetrics 

Timed out junit tests :

   org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache 
  

   compile:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/280/artifact/out/patch-compile-root.txt
  [136K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/280/artifact/out/patch-compile-root.txt
  [136K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/280/artifact/out/patch-compile-root.txt
  [136K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/280/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [336K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/280/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/280/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [52K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/280/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [72K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/280/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [324K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/280/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/280/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt
  [28K]
   
https://builds.apache.org/job/hadoop-

[jira] [Created] (HADOOP-14287) Compiling trunk with -DskipShade fails

[Arpit Agarwal (JIRA)]

Arpit Agarwal created HADOOP-14287:
--

 Summary: Compiling trunk with -DskipShade fails 
 Key: HADOOP-14287
 URL: https://issues.apache.org/jira/browse/HADOOP-14287
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Affects Versions: 3.0.0-alpha3
Reporter: Arpit Agarwal


Get the following errors when compiling trunk with -DskipShade. It succeeds 
with shading.

{code}
[ERROR] COMPILATION ERROR :
[ERROR] 
/hadoop/hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java:[41,30]
 cannot find symbol
  symbol:   class HdfsConfiguration
  location: package org.apache.hadoop.hdfs
[ERROR] 
/hadoop/hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java:[45,34]
 cannot find symbol
  symbol:   class WebHdfsConstants
  location: package org.apache.hadoop.hdfs.web
[ERROR] 
/hadoop/hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java:[71,36]
 cannot find symbol
  symbol:   class HdfsConfiguration
  location: class org.apache.hadoop.example.ITUseMiniCluster
[ERROR] 
/hadoop/hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java:[85,53]
 cannot access org.apache.hadoop.hdfs.DistributedFileSystem
  class file for org.apache.hadoop.hdfs.DistributedFileSystem not found
[ERROR] 
/hadoop/hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java:[109,38]
 cannot find symbol
  symbol:   variable WebHdfsConstants
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14243) Add S3A sensitive config keys to default list

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge resolved HADOOP-14243.
-
Resolution: Not A Problem

{{fs.s3a.secret.key}} already on the default list.
{{fs.s3a.access.key}} is not on the default list by design.

> Add S3A sensitive config keys to default list
> -
>
> Key: HADOOP-14243
> URL: https://issues.apache.org/jira/browse/HADOOP-14243
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/s3, security
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>
> S3A sensitive credential config keys should be added to the default list for 
> {{hadoop.security.sensitive-config-keys}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14270) HADOOP 2.8 Release tar ball size should be smaller

[Junping Du (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junping Du resolved HADOOP-14270.
-
Resolution: Duplicate

> HADOOP 2.8 Release tar ball size should be smaller
> --
>
> Key: HADOOP-14270
> URL: https://issues.apache.org/jira/browse/HADOOP-14270
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build
>Reporter: Junping Du
>Assignee: Junping Du
>Priority: Critical
>
> In voting stage of 2.8.0, [~jojochuang] were reporting that 2.8.0 tar ball 
> are 410 MB while previous release of 2.7.3 is just 205MB. [~andrew.wang] did 
> some advanced investigations that most of added size are from src-html get 
> included in tar.gz which is not necessary in release tar ball.
> I tried to manually build tar ball (using "mvn package -Pdist,native,docs,src 
> -DskipTests -Dtar"), and the dist tar ball is still ~200MB. The only 
> differences is we are using a new dev tool - create-release to create release 
> bits since 2.8 and we should fix from there.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14215) DynamoDB client should waitForActive on existing tables

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu resolved HADOOP-14215.

   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: HADOOP-13345

Committed to feature branch. Thanks for your contribution [~mackrorysd]. Thanks 
[~fabbri] for review and [~rajesh.balamohan] for offline discussion.

> DynamoDB client should waitForActive on existing tables
> ---
>
> Key: HADOOP-14215
> URL: https://issues.apache.org/jira/browse/HADOOP-14215
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
>Priority: Critical
> Fix For: HADOOP-13345
>
> Attachments: HADOOP-14215-HADOOP-13345.000.patch, 
> HADOOP-14215-HADOOP-13345.001.patch, HADOOP-14215-HADOOP-13345.002.patch
>
>
> I saw a case where 2 separate applications tried to use the same 
> non-pre-existing table with table.create = true at about the same time. One 
> failed with a ResourceInUse exception. If a table does not exist, we attempt 
> to create it and then wait for it to enter the active state. If another jumps 
> in in the middle of that, the table may exist, thus bypassing our call to 
> waitForActive(), and then try to use the table immediately.
> While we're at it, let's also make sure that the race condition where a table 
> might get created between checking if it exists and attempting to create it 
> is handled gracefully.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14288) TestDynamoDBMetadataStore is broken unless we can fail faster without a table version

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14288:
--

 Summary: TestDynamoDBMetadataStore is broken unless we can fail 
faster without a table version
 Key: HADOOP-14288
 URL: https://issues.apache.org/jira/browse/HADOOP-14288
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory
Priority: Critical


HADOOP-14215 causes TestDynamoDBMetadataStore to fail because not having a 
version meta-entry in the table no longer fails immediately, but must timeout 
all the retries.

Let's make those configurable and have the test just turn them way down low.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org