[jira] [Created] (YARN-9487) NodeManager shouldn't link against librt on macOS

2019-04-15 Thread Siyao Meng (JIRA) 
Siyao Meng created YARN-9487:


 Summary: NodeManager shouldn't link against librt on macOS
 Key: YARN-9487
 URL: https://issues.apache.org/jira/browse/YARN-9487
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 3.2.0, 3.3.0
Reporter: Siyao Meng


I'm compiling hadoop/trunk native on a Mac. Found that when compiling 
NodeManager it is trying to link against librt on Darwin as well - and it 
failed for sure. Diff for a quick fix:
{code}
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
index f0f005d53b5..9c3f0135aaf 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
@@ -190,5 +190,10 @@ add_executable(test-oom-listener
 main/native/oom-listener/impl/oom_listener.h
 main/native/oom-listener/test/oom_listener_test_main.cc
 )
-target_link_libraries(test-oom-listener gtest rt)
+if(CMAKE_SYSTEM_NAME MATCHES "Darwin")
+# macOS does not have rt
+target_link_libraries(test-oom-listener gtest)
+else()
+target_link_libraries(test-oom-listener gtest rt)
+endif()
 output_directory(test-oom-listener test)
{code}

This line of code is added in YARN-8437, so this affects 3.2.0+.

Will upload the patch after I'm added to YARN contributor list.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

VOTE: Hadoop Ozone 0.4.0-alpha RC1

2019-04-15 Thread Ajay Kumar 
Hi all,

We have created the second release candidate (RC1) for Apache Hadoop Ozone 
0.4.0-alpha.

This release contains security payload for Ozone. Below are some important 
features in it:

  *   Hadoop Delegation Tokens and Block Tokens supported for Ozone.
  *   Transparent Data Encryption (TDE) Support - Allows data blocks to be 
encrypted-at-rest.
  *   Kerberos support for Ozone.
  *   Certificate Infrastructure for Ozone  - Tokens use PKI instead of shared 
secrets.
  *   Datanode to Datanode communication secured via mutual TLS.
  *   Ability secure ozone cluster that works with Yarn, Hive, and Spark.
  *   Skaffold support to deploy Ozone clusters on K8s.
  *   Support S3 Authentication Mechanisms like - S3 v4 Authentication protocol.
  *   S3 Gateway supports Multipart upload.
  *   S3A file system is tested and supported.
  *   Support for Tracing and Profiling for all Ozone components.
  *   Audit Support - including Audit Parser tools.
  *   Apache Ranger Support in Ozone.
  *   Extensive failure testing for Ozone.

The RC artifacts are available at 
https://home.apache.org/~ajay/ozone-0.4.0-alpha-rc1

The RC tag in git is ozone-0.4.0-alpha-RC1 (git hash 
d673e16d14bb9377f27c9017e2ffc1bcb03eebfb)

Please try 
out,
 vote, or just give us feedback.

The vote will run for 5 days, ending on April 20, 2019, 19:00 UTC.

Thank you very much,

Ajay

[jira] [Created] (YARN-9486) Docker container exited with failure does not get clean up correctly

2019-04-15 Thread Eric Yang (JIRA) 
Eric Yang created YARN-9486:
---

 Summary: Docker container exited with failure does not get clean 
up correctly
 Key: YARN-9486
 URL: https://issues.apache.org/jira/browse/YARN-9486
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Eric Yang
Assignee: Eric Yang


When docker container encounters error and exit prematurely 
(EXITED_WITH_FAILURE), ContainerCleanup does not remove container, instead we 
get messages that look like this:

{code}
java.io.IOException: Could not find 
nmPrivate/application_1555111445937_0008/container_1555111445937_0008_01_07//container_1555111445937_0008_01_07.pid
 in any of the directories
2019-04-15 20:42:16,454 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
 Container container_1555111445937_0008_01_07 transitioned from RELAUNCHING 
to EXITED_WITH_FAILURE
2019-04-15 20:42:16,455 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerCleanup:
 Cleaning up container container_1555111445937_0008_01_07
2019-04-15 20:42:16,455 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerCleanup:
 Container container_1555111445937_0008_01_07 not launched. No cleanup 
needed to be done
2019-04-15 20:42:16,455 WARN 
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hbase
OPERATION=Container Finished - Failed   TARGET=ContainerImplRESULT=FAILURE  
DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE
APPID=application_1555111445937_0008
CONTAINERID=container_1555111445937_0008_01_07
2019-04-15 20:42:16,458 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
 Container container_1555111445937_0008_01_07 transitioned from 
EXITED_WITH_FAILURE to DONE
2019-04-15 20:42:16,458 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
 Removing container_1555111445937_0008_01_07 from application 
application_1555111445937_0008
2019-04-15 20:42:16,458 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
 Stopping resource-monitoring for container_1555111445937_0008_01_07
2019-04-15 20:42:16,458 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl:
 Considering container container_1555111445937_0008_01_07 for 
log-aggregation
2019-04-15 20:42:16,804 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
 Getting container-status for container_1555111445937_0008_01_07
2019-04-15 20:42:16,804 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
 Getting localization status for container_1555111445937_0008_01_07
2019-04-15 20:42:16,804 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
 Returning ContainerStatus: [ContainerId: 
container_1555111445937_0008_01_07, ExecutionType: GUARANTEED, State: 
COMPLETE, Capability: , Diagnostics: ..., ExitStatus: 
-1, IP: null, Host: null, ExposedPorts: , ContainerSubState: DONE]
2019-04-15 20:42:18,464 INFO 
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed 
completed containers from NM context: [container_1555111445937_0008_01_07]
2019-04-15 20:43:50,476 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
 Stopping container with container Id: container_1555111445937_0008_01_07
{code}

There is no docker rm command performed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-9485) Add Target Cache Size to Cache Cleaner Output

2019-04-15 Thread David Mollitor (JIRA) 
David Mollitor created YARN-9485:


 Summary: Add Target Cache Size to Cache Cleaner Output
 Key: YARN-9485
 URL: https://issues.apache.org/jira/browse/YARN-9485
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: nodemanager
Affects Versions: 2.9.2, 3.2.0
Reporter: David Mollitor
Assignee: David Mollitor






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-9484) Fix SLF4j logging API in AbstractManagedParentQueue#removeChildQueue

2019-04-15 Thread Akira Ajisaka (JIRA) 


 [ 
https://issues.apache.org/jira/browse/YARN-9484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka resolved YARN-9484.
-
Resolution: Duplicate

This issue was fixed by YARN-9349. Closing.

> Fix SLF4j logging API in AbstractManagedParentQueue#removeChildQueue
> 
>
> Key: YARN-9484
> URL: https://issues.apache.org/jira/browse/YARN-9484
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Affects Versions: 3.2.0
>Reporter: hunshenshi
>Priority: Major
>
> {code:java}
> if (LOG.isDebugEnabled()) {
>   LOG.debug("Removed child queue: {}" + cs.getQueueName());
> }
> {code}
> The '+' should be ','
> {code:java}
> if (LOG.isDebugEnabled()) {
>   LOG.debug("Removed child queue: {}", cs.getQueueName());
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2019-04-15 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1107/

[Apr 15, 2019 1:46:10 AM] (aajisaka) HADOOP-16227. Upgrade checkstyle to 8.19




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 
   Null passed for non-null parameter of 
com.google.common.util.concurrent.SettableFuture.set(Object) in 
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppTransition.transition(RMStateStore,
 RMStateStoreEvent) At RMStateStore.java:of 
com.google.common.util.concurrent.SettableFuture.set(Object) in 
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppTransition.transition(RMStateStore,
 RMStateStoreEvent) At RMStateStore.java:[line 291] 
   Null passed for non-null parameter of 
com.google.common.util.concurrent.SettableFuture.set(Object) in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.updateApplicationPriority(Priority,
 ApplicationId, SettableFuture, UserGroupInformation) At 
CapacityScheduler.java:of 
com.google.common.util.concurrent.SettableFuture.set(Object) in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.updateApplicationPriority(Priority,
 ApplicationId, SettableFuture, UserGroupInformation) At 
CapacityScheduler.java:[line 2650] 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-documentstore
 
   Unread field:TimelineEventSubDoc.java:[line 56] 
   Unread field:TimelineMetricSubDoc.java:[line 44] 

Failed junit tests :

   hadoop.hdfs.TestMaintenanceState 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.server.datanode.TestIncrementalBlockReports 
   
hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService 
   hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.mapreduce.v2.app.TestRuntimeEstimators 
   hadoop.yarn.sls.TestSLSStreamAMSynth 
   
hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1107/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1107/artifact/out/diff-compile-javac-root.txt
  [336K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1107/artifact/out/diff-checkstyle-root.txt
  [17M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1107/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1107/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1107/artifact/out/diff-patch-pylint.txt
  [84K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1107/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1107/artifact/out/diff-patch-shelldocs.txt
  [44K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1107/artifact/out/whitespace-eol.txt
  [9.6M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1107/artifact/out/whitespace-tabs.txt
  [1.1M]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1107/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1107/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-documentstore-warnings.html
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1107/artifact/out/branch-findbugs-hadoop-submarine_hadoop-submarine-yarnservice-runtime.txt
  [4.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1107/artifact/out/diff-javadoc-javadoc-root.txt
  [752K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1107/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [332K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1107/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

2019-04-15 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/292/

No changes




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml 
   hadoop-build-tools/src/main/resources/checkstyle/suppressions.xml 
   
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml
 
   hadoop-tools/hadoop-azure/src/config/checkstyle.xml 
   hadoop-tools/hadoop-resourceestimator/src/config/checkstyle.xml 
   hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml
 

FindBugs :

   module:hadoop-common-project/hadoop-common 
   Class org.apache.hadoop.fs.GlobalStorageStatistics defines non-transient 
non-serializable instance field map In GlobalStorageStatistics.java:instance 
field map In GlobalStorageStatistics.java 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client
 
   Boxed value is unboxed and then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:[line 335] 

Failed junit tests :

   hadoop.hdfs.tools.TestDFSZKFailoverController 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   hadoop.registry.secure.TestSecureLogins 
   hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 
   hadoop.yarn.client.api.impl.TestAMRMProxy 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/292/artifact/out/diff-compile-cc-root-jdk1.7.0_95.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/292/artifact/out/diff-compile-javac-root-jdk1.7.0_95.txt
  [328K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/292/artifact/out/diff-compile-cc-root-jdk1.8.0_191.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/292/artifact/out/diff-compile-javac-root-jdk1.8.0_191.txt
  [308K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/292/artifact/out/diff-checkstyle-root.txt
  [16M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/292/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/292/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/292/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/292/artifact/out/diff-patch-shellcheck.txt
  [72K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/292/artifact/out/diff-patch-shelldocs.txt
  [8.0K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/292/artifact/out/whitespace-eol.txt
  [12M]
   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/292/artifact/out/whitespace-tabs.txt
  [1.2M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/292/artifact/out/xml.txt
  [20K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/292/artifact/out/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/292/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/292/artifact/out/diff-javadoc-javadoc-root-jdk1.7.0_95.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/292/artifact/out/diff-javadoc-javadoc-root-jdk1.8.0_191.txt
  [1.1M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/292/artifact/out/patch-unit-hadoop-hdfs-project_hadoop

[jira] [Created] (YARN-9484) Fix SLF4j logging API in AbstractManagedParentQueue#removeChildQueue

2019-04-15 Thread hunshenshi (JIRA) 
hunshenshi created YARN-9484:


 Summary: Fix SLF4j logging API in 
AbstractManagedParentQueue#removeChildQueue
 Key: YARN-9484
 URL: https://issues.apache.org/jira/browse/YARN-9484
 Project: Hadoop YARN
  Issue Type: Bug
  Components: capacity scheduler
Affects Versions: 3.2.0
Reporter: hunshenshi


{code:java}
if (LOG.isDebugEnabled()) {
  LOG.debug("Removed child queue: {}" + cs.getQueueName());
}
{code}
The '+' should be ','
{code:java}
if (LOG.isDebugEnabled()) {
  LOG.debug("Removed child queue: {}", cs.getQueueName());
}
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-9483) DistributedShell does not release container when failed to localize at launch

2019-04-15 Thread Prabhu Joseph (JIRA) 
Prabhu Joseph created YARN-9483:
---

 Summary: DistributedShell does not release container when failed 
to localize at launch
 Key: YARN-9483
 URL: https://issues.apache.org/jira/browse/YARN-9483
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Prabhu Joseph
Assignee: Prabhu Joseph


DistributedShell does not release container when failed to localize at launch. 
The launch threads does not increment completed & failed containers when failed 
to localize. And the main thread waits for the containers to complete without 
failing the job.

{code}
yarn jar 
/HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
 -shell_command ls  -shell_args / -jar  
/HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
 -localize_files /tmp/prabhu
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-9482) DistributedShell job with localization fails in unsecure cluster

2019-04-15 Thread Prabhu Joseph (JIRA) 
Prabhu Joseph created YARN-9482:
---

 Summary: DistributedShell job with localization fails in unsecure 
cluster
 Key: YARN-9482
 URL: https://issues.apache.org/jira/browse/YARN-9482
 Project: Hadoop YARN
  Issue Type: Bug
  Components: distributed-shell
Affects Versions: 3.3.0
Reporter: Prabhu Joseph
Assignee: Prabhu Joseph


DistributedShell job with localization fails in unsecure cluster. The client 
localizes the input files to home directory (job user) whereas the AM runs as 
yarn user reads from it's home directory.

*Command:*

{code}
yarn jar 
/HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
 -shell_command ls  -shell_args / -jar  
/HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
 -localize_files /tmp/prabhu
{code}

{code}
Exception in thread "Thread-4" java.io.UncheckedIOException: Error during 
localization setup
at 
org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.lambda$run$0(ApplicationMaster.java:1495)
at 
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at 
java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at 
org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.run(ApplicationMaster.java:1481)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: File does not exist: 
hdfs://yarn-ats-1:8020/user/yarn/DistributedShell/application_1554817981283_0003/prabhu
at 
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1586)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1594)
at 
org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.lambda$run$0(ApplicationMaster.java:1487)
{code}  





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org