[jira] [Created] (MAPREDUCE-7207) Make job history done dir permissions configurable
Anthony Hsu created MAPREDUCE-7207: -- Summary: Make job history done dir permissions configurable Key: MAPREDUCE-7207 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7207 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Anthony Hsu MAPREDUCE-7010 made the permissions of the intermediate-user-done-dir configurable but not the intermediate-done-dir or done-dir. It would be useful to make the permissions of the intermediate-done-dir and done-dir configurable as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7206) ShuffleHandler cannot access file.out
Tarek Abouzeid created MAPREDUCE-7206:
-
Summary: ShuffleHandler cannot access file.out
Key: MAPREDUCE-7206
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7206
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.1.1
Environment: HDP 3.1 (3.1.0.0-78)
Reporter: Tarek Abouzeid
i am running HDP 3.1 (3.1.0.0-78) , i have 10 data nodes , Hive execution
engine is TEZ, when i run a query i get this error,
{code:java}
ERROR : FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex re-running, vertexName=Map
1, vertexId=vertex_1557754551780_1091_2_00Vertex re-running, vertexName=Map 1,
vertexId=vertex_1557754551780_1091_2_00Vertex re-running, vertexName=Map 1,
vertexId=vertex_1557754551780_1091_2_00Vertex failed, vertexName=Map 1,
vertexId=vertex_1557754551780_1091_2_00, diagnostics=[Vertex
vertex_1557754551780_1091_2_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE,
Vertex vertex_1557754551780_1091_2_00 [Map 1] failed as task
task_1557754551780_1091_2_00_01 failed after vertex succeeded.]DAG did not
succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
INFO : Completed executing
command(queryId=hive_20190516161715_09090e6d-e513-4fcc-9c96-0b48e9b43822); Time
taken: 17.935 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 2
from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex re-running,
vertexName=Map 1, vertexId=vertex_1557754551780_1091_2_00Vertex re-running,
vertexName=Map 1, vertexId=vertex_1557754551780_1091_2_00Vertex re-running,
vertexName=Map 1, vertexId=vertex_1557754551780_1091_2_00Vertex failed,
vertexName=Map 1, vertexId=vertex_1557754551780_1091_2_00, diagnostics=[Vertex
vertex_1557754551780_1091_2_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE,
Vertex vertex_1557754551780_1091_2_00 [Map 1] failed as task
task_1557754551780_1091_2_00_01 failed after vertex succeeded.]DAG did not
succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
(state=08S01,code=2)
{code}
when i traced the logs, for example the application id is
*application_1557754551780_1091*
checked the node manager logs
{code:java}
2019-05-16 16:19:05,801 INFO mapred.ShuffleHandler
(ShuffleHandler.java:sendMapOutput(1268)) -
/var/lib/hadoop/yarn/local/usercache/hive/appcache/application_1557754551780_1091/output/attempt_1557754551780_1091_2_00_00_0_10003/file.out
not found
2019-05-16 16:19:05,818 INFO mapred.ShuffleHandler
(ShuffleHandler.java:sendMapOutput(1268)) -
/var/lib/hadoop/yarn/local/usercache/hive/appcache/application_1557754551780_1091/output/attempt_1557754551780_1091_2_00_00_0_10003/file.out
not found
2019-05-16 16:19:05,821 INFO mapred.ShuffleHandler
(ShuffleHandler.java:sendMapOutput(1268)) -
/var/lib/hadoop/yarn/local/usercache/hive/appcache/application_1557754551780_1091/output/attempt_1557754551780_1091_2_00_00_0_10003/file.out
not found
2019-05-16 16:19:05,822 INFO mapred.ShuffleHandler
(ShuffleHandler.java:sendMapOutput(1268)) -
/var/lib/hadoop/yarn/local/usercache/hive/appcache/application_1557754551780_1091/output/attempt_1557754551780_1091_2_00_00_0_10003/file.out
not found
2019-05-16 16:19:05,824 INFO mapred.ShuffleHandler
(ShuffleHandler.java:sendMapOutput(1268)) -
/var/lib/hadoop/yarn/local/usercache/hive/appcache/application_1557754551780_1091/output/attempt_1557754551780_1091_2_00_00_0_10003/file.out
not found
2019-05-16 16:19:05,826 INFO mapred.ShuffleHandler
(ShuffleHandler.java:sendMapOutput(1268)) -
/var/lib/hadoop/yarn/local/usercache/hive/appcache/application_1557754551780_1091/output/attempt_1557754551780_1091_2_00_00_0_10003/file.out
not found
{code}
i checked the path where the output of the Map will be there in (
*/var/lib/hadoop/yarn/local/usercache/hive/appcache/application_1557754551780_1091/output/attempt_1557754551780_1091_2_00_00_0_10003*
)
{code:java}
drwx--x---. 3 hive hadoop 16 May 16 16:16 filecache
drwxr-s---. 3 hive hadoop 60 May 16 16:16 output
{code}
inside the output :
{code:java}
-rw---. 1 hive hadoop 28 May 16 16:17 file.out
-rw-r-. 1 hive hadoop 32 May 16 16:17 file.out.index
{code}
so the *file.out* is not readable by other users in same group (switched to
yarn user and tried to open this file and got permission denied)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1138/ [May 15, 2019 7:51:39 AM] (bibinchundatt) YARN-9547. ContainerStatusPBImpl default execution type is not returned. [May 15, 2019 8:00:09 AM] (bibinchundatt) YARN-9508. YarnConfiguration areNodeLabel enabled is costly in [May 15, 2019 5:26:01 PM] (arp7) HDDS-1511. Space tracking for Open Containers in HDDS Volumes. [May 15, 2019 6:50:46 PM] (gifuma) YARN-9552. FairScheduler: NODE_UPDATE can cause NoSuchElementException. [May 15, 2019 7:15:47 PM] (virajith) HDFS-14390. Provide kerberos support for AliasMap service used by [May 16, 2019 12:41:36 AM] (bharat) HADOOP-16247. NPE in FsUrlConnection. Contributed by Karthik Palanisamy. -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-documentstore Unread field:TimelineEventSubDoc.java:[line 56] Unread field:TimelineMetricSubDoc.java:[line 44] FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core Class org.apache.hadoop.applications.mawo.server.common.TaskStatus implements Cloneable but does not define or use clone method At TaskStatus.java:does not define or use clone method At TaskStatus.java:[lines 39-346] Equals method for org.apache.hadoop.applications.mawo.server.worker.WorkerId assumes the argument is of type WorkerId At WorkerId.java:the argument is of type WorkerId At WorkerId.java:[line 114] org.apache.hadoop.applications.mawo.server.worker.WorkerId.equals(Object) does not check for null argument At WorkerId.java:null argument At WorkerId.java:[lines 114-115] Failed junit tests : hadoop.hdfs.server.balancer.TestBalancer hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService hadoop.yarn.server.nodemanager.webapp.TestNMWebServices hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 hadoop.yarn.client.cli.TestLogsCLI hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapreduce.v2.app.TestRuntimeEstimators cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1138/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1138/artifact/out/diff-compile-javac-root.txt [332K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1138/artifact/out/diff-checkstyle-root.txt [17M] hadolint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1138/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1138/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1138/artifact/out/diff-patch-pylint.txt [96K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1138/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1138/artifact/out/diff-patch-shelldocs.txt [44K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1138/artifact/out/whitespace-eol.txt [9.6M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1138/artifact/out/whitespace-tabs.txt [1.1M] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1138/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-documentstore-warnings.html [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1138/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-mawo_hadoop-yarn-applications-mawo-core-warnings.html [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1138/artifact/out/branch-findbugs-hadoop-submarine_hadoop-submarine-tony-runtime.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1138/artifact/out/branch-findbugs-hadoop-submarine_hadoop-submarine-yarnservice-runtime.txt [4.0K] javadoc: https://bui
Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/323/ No changes -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml FindBugs : module:hadoop-common-project/hadoop-common Class org.apache.hadoop.fs.GlobalStorageStatistics defines non-transient non-serializable instance field map In GlobalStorageStatistics.java:instance field map In GlobalStorageStatistics.java FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:[line 335] Failed junit tests : hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.registry.secure.TestSecureLogins hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 hadoop.yarn.client.api.impl.TestAMRMProxy cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/323/artifact/out/diff-compile-cc-root-jdk1.7.0_95.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/323/artifact/out/diff-compile-javac-root-jdk1.7.0_95.txt [328K] cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/323/artifact/out/diff-compile-cc-root-jdk1.8.0_212.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/323/artifact/out/diff-compile-javac-root-jdk1.8.0_212.txt [308K] checkstyle: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/323/artifact/out/diff-checkstyle-root.txt [16M] hadolint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/323/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/323/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/323/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/323/artifact/out/diff-patch-shellcheck.txt [72K] shelldocs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/323/artifact/out/diff-patch-shelldocs.txt [8.0K] whitespace: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/323/artifact/out/whitespace-eol.txt [12M] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/323/artifact/out/whitespace-tabs.txt [1.2M] xml: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/323/artifact/out/xml.txt [12K] findbugs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/323/artifact/out/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html [8.0K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/323/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client-warnings.html [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/323/artifact/out/diff-javadoc-javadoc-root-jdk1.7.0_95.txt [16K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/323/artifact/out/diff-javadoc-javadoc-root-jdk1.8.0_212.txt [1.1M] unit: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/323/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [284K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/323/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-registry.txt [12K] https://builds.apache.
