kaleyroy opened a new issue #8202: URL: https://github.com/apache/dolphinscheduler/issues/8202
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues. ### What happened ## 配置MapReduce任务，DS生成的执行命令中 -D参数位置错误，导致hadoop jar 命令执行异常： Hadoop版本： **hadoop-3.2.1** 生成命令如下： ` hadoop jar WordCount.jar WordCount -Dmapreduce.job.name=WordCount -Dmapreduce.job.queuename=default /input /output ` 执行异常如下： ` [LOG-PATH]: /opt/apache-dolphinscheduler-2.0.2-bin/logs/4254049161440_2/18/25.log, [HOST]: 172.19.0.4 [INFO] 2022-01-25 04:41:30.723 [TaskLogInfo- - [taskAppId=TASK-4254049161440_2-18-25]] - mapreduce task params {"mainClass":"WordCount","mainJar":{"id":2,"res":"WordCount.jar","resourceName":"/WordCount.jar"},"resourceList":[],"localParams":[],"appName":"WordCount","mainArgs":"/input /output","others":"","programType":"JAVA","dependence":{},"conditionResult":{"successNode":[],"failedNode":[]},"waitStartTimeout":{},"switchResult":{}} [INFO] 2022-01-25 04:41:30.727 [TaskLogInfo- - [taskAppId=TASK-4254049161440_2-18-25]] - mapreduce task command: hadoop jar WordCount.jar WordCount -Dmapreduce.job.name=WordCount -Dmapreduce.job.queuename=default /input /output [INFO] 2022-01-25 04:41:30.728 [TaskLogInfo- - [taskAppId=TASK-4254049161440_2-18-25]] - tenantCode user:klrylab, task dir:18_25 [INFO] 2022-01-25 04:41:30.729 [TaskLogInfo- - [taskAppId=TASK-4254049161440_2-18-25]] - create command file:/tmp/dolphinscheduler/exec/process/4253999453664/4254049161440_2/18/25/18_25.command [INFO] 2022-01-25 04:41:30.729 [TaskLogInfo- - [taskAppId=TASK-4254049161440_2-18-25]] - command : #!/bin/sh BASEDIR=$(cd `dirname $0`; pwd) cd $BASEDIR export PYTHON_HOME=/usr/bin/python3 export HADOOP_HOME=/opt/soft/hadoop export HADOOP_CONF_DIR=/opt/soft/hadoop/etc/hadoop export JAVA_HOME=/usr/local/openjdk-8 export PATH=/usr/local/openjdk-8/bin:/opt/soft/hadoop/bin:/opt/soft/hadoop/sbin:$PATH echo $PATH hadoop jar WordCount.jar WordCount -Dmapreduce.job.name=WordCount -Dmapreduce.job.queuename=default /input /output [INFO] 2022-01-25 04:41:30.734 [TaskLogInfo- - [taskAppId=TASK-4254049161440_2-18-25]] - task run command: sudo -u klrylab sh /tmp/dolphinscheduler/exec/process/4253999453664/4254049161440_2/18/25/18_25.command [INFO] 2022-01-25 04:41:30.740 [TaskLogInfo- - [taskAppId=TASK-4254049161440_2-18-25]] - process start, process id is: 17381 [INFO] 2022-01-25 04:41:31.744 [TaskLogInfo- - [taskAppId=TASK-4254049161440_2-18-25]-getOutputLogService] - -> welcome to use bigdata scheduling system... /usr/local/openjdk-8/bin:/opt/soft/hadoop/bin:/opt/soft/hadoop/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin [INFO] 2022-01-25 04:41:36.760 [TaskLogInfo- - [taskAppId=TASK-4254049161440_2-18-25]-getOutputLogService] - -> 2022-01-25 04:41:35,759 INFO client.RMProxy: Connecting to ResourceManager at resourcemanager/172.19.0.8:8032 2022-01-25 04:41:36,568 INFO client.AHSProxy: Connecting to Application History server at historyserver/172.19.0.10:10200 [INFO] 2022-01-25 04:41:37.764 [TaskLogInfo- - [taskAppId=TASK-4254049161440_2-18-25]-getOutputLogService] - -> 2022-01-25 04:41:37,386 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2022-01-25 04:41:37,475 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/klrylab/.staging/job_1643019406017_0006 [INFO] 2022-01-25 04:41:38.766 [TaskLogInfo- - [taskAppId=TASK-4254049161440_2-18-25]-getOutputLogService] - -> 2022-01-25 04:41:37,845 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2022-01-25 04:41:38,332 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/klrylab/.staging/job_1643019406017_0006 Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://namenode:9000/user/klrylab/-Dmapreduce.job.name=WordCount at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:332) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:274) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:396) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:310) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:327) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:200) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1588) at WordCount.main(WordCount.java:59) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:323) at org.apache.hadoop.util.RunJar.main(RunJar.java:236) [INFO] 2022-01-25 04:41:38.802 [TaskLogInfo- - [taskAppId=TASK-4254049161440_2-18-25]] - process has exited, execute path:/tmp/dolphinscheduler/exec/process/4253999453664/4254049161440_2/18/25, processId:17381 ,exitStatusCode:1 ,processWaitForStatus:true ,processExitValue:1 ` ### What you expected to happen 正确命令如下，**-D 参数应该放在最后位置** ` hadoop jar /opt/soft/WordCount.jar WordCount /input /output/task2 -Dmapreduce.job.name=WordCount -Dmapreduce.job.queuename=default ` ### How to reproduce ### MR任务配置参考截图  ### Anything else _No response_ ### Version 2.0.2 ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
