[ https://issues.apache.org/jira/browse/YARN-6550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120554#comment-16120554 ]
Suma Shivaprasad commented on YARN-6550: ---------------------------------------- [~aw] Thanks for checking on the bash version portability. With the updated patch to use command groups, there is a UT failure that exposed some unexpected The UT - TestContainerLaunch.testInvalidSyntax checks for failures being propagated from a bunch of invalid commands through ShellCommandExecutor. Behaviour without the patch Below is the script that the UT executes -------- {noformat} #!/bin/bash export APPLICATION_WORKFLOW_CONTEXT="{"workflowId":"609f91c5cd83","workflowName":" insert table partition (cd_education_status) select cd_demo_sk, cd_gender, " exec /bin/bash -c "" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi {noformat} Expected error -------- */tmp/unit_test_pass.sh: line 5: insert: command not found* /tmp/unit_test_pass.sh: line 6: syntax error near unexpected token `cd_education_status' /tmp/unit_test_pass.sh: line 6: `partition (cd_education_status)' Behaviour with the patch Script that the UT executes with the patch (command groups) ---------------------------------------------------------------------------------- {noformat} { echo "Setting up env variables" export APPLICATION_WORKFLOW_CONTEXT=""workflowId":"609f91c5cd83","workflowName":" insert table partition (cd_education_status) select cd_demo_sk, cd_gender, " echo "Setting up job resources" echo "Launching container" } 1> >(tee -a "${STDOUT}" >&1) 2> >(tee -a "${STDERR}" >&2) # Note that redirection doesnt matter. Having a command group causes it. exec /bin/bash -c "" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi {noformat} Error ======== /tmp/unit_test_fail.sh: line 6: syntax error near unexpected token `cd_education_status' /tmp/unit_test_fail.sh: line 6: `partition (cd_education_status)' Please note that the "insert table" command being an invalid command is not even thrown. Given the above issues, I was exploring other ways of achieving redirection without doing it per line. Using exec with redirection seems like a more concise way to achieve this - http://tldp.org/LDP/abs/html/x17974.html I also tested the above UT script with exec and it works fine. If you dont have any objections, will update the patch to use exec with redirection instead. {noformat} export STDOUT="/tmp/1.out" export STDERR="/tmp/1.err" exec 1> >(tee -a "${STDOUT}" >&1) 2> >(tee -a "${STDERR}" >&2) export APPLICATION_WORKFLOW_CONTEXT="{"workflowId":"609f91c5cd83","workflowName":" insert table partition (cd_education_status) select cd_demo_sk, cd_gender, " exec /bin/bash -c "" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi {noformat} Error ====== */tmp/unit_test_exec.sh: line 9: insert: command not found* /tmp/unit_test_exec.sh: line 10: syntax error near unexpected token `cd_education_status' /tmp/unit_test_exec.sh: line 10: `partition (cd_education_status)' > Capture launch_container.sh logs > -------------------------------- > > Key: YARN-6550 > URL: https://issues.apache.org/jira/browse/YARN-6550 > Project: Hadoop YARN > Issue Type: Sub-task > Affects Versions: 3.0.0-beta1 > Reporter: Wangda Tan > Assignee: Suma Shivaprasad > Fix For: 3.0.0-beta1 > > Attachments: YARN-6550.002.patch, YARN-6550.003.patch, YARN-6550.patch > > > launch_container.sh which generated by NM will do a bunch of things (like > create link, etc.) while launch a process. No logs captured until {{exec}} is > called. We need capture all failures of launch_container.sh for easier > troubleshooting. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org