[jira] Subscription: Oozie Patch Available
Issue Subscription Filter: Oozie Patch Available (94 issues) Subscriber: ooziedaily Key Summary OOZIE-3409 Oozie Server : Possible memory leak in EL evaluation https://issues.apache.org/jira/browse/OOZIE-3409 OOZIE-3407 Cleanup TestPurgeXCommand https://issues.apache.org/jira/browse/OOZIE-3407 OOZIE-3404 The env variable of SPARK_HOME needs to be set when running pySpark https://issues.apache.org/jira/browse/OOZIE-3404 OOZIE-3401 TestPySpark failure https://issues.apache.org/jira/browse/OOZIE-3401 OOZIE-3375 Can't use empty in coordinator https://issues.apache.org/jira/browse/OOZIE-3375 OOZIE-3367 Using && in EL expressions in oozie bundle.xml files generates parse errors https://issues.apache.org/jira/browse/OOZIE-3367 OOZIE-3366 Update workflow status and subworkflow status on suspend command https://issues.apache.org/jira/browse/OOZIE-3366 OOZIE-3364 Rerunning Oozie bundle jobs starts the coordinators in indeterminate order https://issues.apache.org/jira/browse/OOZIE-3364 OOZIE-3362 When killed, SSH action should kill the spawned processes on target host https://issues.apache.org/jira/browse/OOZIE-3362 OOZIE-3326 Sqoop Action should support tez delegation tokens for hive-import https://issues.apache.org/jira/browse/OOZIE-3326 OOZIE-3320 Oozie ShellAction should support absolute bash file path https://issues.apache.org/jira/browse/OOZIE-3320 OOZIE-3319 Log SSH action callback error output https://issues.apache.org/jira/browse/OOZIE-3319 OOZIE-3301 Update NOTICE file https://issues.apache.org/jira/browse/OOZIE-3301 OOZIE-3274 Remove slf4j https://issues.apache.org/jira/browse/OOZIE-3274 OOZIE-3266 Coord action rerun support RERUN_SKIP_NODES option https://issues.apache.org/jira/browse/OOZIE-3266 OOZIE-3265 properties RERUN_FAIL_NODES and RERUN_SKIP_NODES should be able to appear together https://issues.apache.org/jira/browse/OOZIE-3265 OOZIE-3256 refactor OozieCLI class https://issues.apache.org/jira/browse/OOZIE-3256 OOZIE-3249 [tools] Instrumentation log parser https://issues.apache.org/jira/browse/OOZIE-3249 OOZIE-3199 Let system property restriction configurable https://issues.apache.org/jira/browse/OOZIE-3199 OOZIE-3196 Authorization: restrict world readability by user https://issues.apache.org/jira/browse/OOZIE-3196 OOZIE-3179 Adding a configurable config-default.xml location to a workflow https://issues.apache.org/jira/browse/OOZIE-3179 OOZIE-3170 Oozie Diagnostic Bundle tool fails with NPE due to missing service class https://issues.apache.org/jira/browse/OOZIE-3170 OOZIE-3137 Add support for log4j2 in HiveMain https://issues.apache.org/jira/browse/OOZIE-3137 OOZIE-3135 Configure log4j2 in SqoopMain https://issues.apache.org/jira/browse/OOZIE-3135 OOZIE-3091 Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper" https://issues.apache.org/jira/browse/OOZIE-3091 OOZIE-3071 Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0 https://issues.apache.org/jira/browse/OOZIE-3071 OOZIE-3063 Sanitizing variables that are part of openjpa.ConnectionProperties https://issues.apache.org/jira/browse/OOZIE-3063 OOZIE-3062 Set HADOOP_CONF_DIR for spark action https://issues.apache.org/jira/browse/OOZIE-3062 OOZIE-2952 Fix Findbugs warnings in oozie-sharelib-oozie https://issues.apache.org/jira/browse/OOZIE-2952 OOZIE-2949 Escape quotes whitespaces in Sqoop field https://issues.apache.org/jira/browse/OOZIE-2949 OOZIE-2927 Append new line character for Hive2 query using query tag https://issues.apache.org/jira/browse/OOZIE-2927 OOZIE-2834 ParameterVerifier logging non-useful warning for workflow definition https://issues.apache.org/jira/browse/OOZIE-2834 OOZIE-2833 when using uber mode the regex pattern used in the extractHeapSizeMB method does not allow heap sizes specified in bytes. https://issues.apache.org/jira/browse/OOZIE-2833 OOZIE-2812 SparkConfigurationService should support loading configurations from multiple Spark versions https://issues.apache.org/jira/browse/OOZIE-2812 OOZIE-2795 Create lib directory or symlink for Oozie CLI during packaging https://issues.apache.org/jira/browse/OOZIE-2795 OOZIE-2784 Include WEEK as a parameter in the Coordinator Expression Language Evaulator https://issues.apache.org/jira/browse/OOZIE-2784 OOZIE-2779 Mask Hive2 action Beeline JDBC password https://issues.apache.org/jira/browse/OOZIE-2779 OOZIE-2736 Reduce the number of threads during test execution
[jira] [Commented] (OOZIE-2949) Escape quotes whitespaces in Sqoop field
[ https://issues.apache.org/jira/browse/OOZIE-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734379#comment-16734379 ] Hadoop QA commented on OOZIE-2949: -- Testing JIRA OOZIE-2949 Cleaning local git workspace {color:green}+1 PATCH_APPLIES{color} {color:green}+1 CLEAN{color} {color:green}+1 RAW_PATCH_ANALYSIS{color} .{color:green}+1{color} the patch does not introduce any @author tags .{color:green}+1{color} the patch does not introduce any tabs .{color:green}+1{color} the patch does not introduce any trailing spaces .{color:green}+1{color} the patch does not introduce any star imports .{color:green}+1{color} the patch does not introduce any line longer than 132 .{color:green}+1{color} the patch adds/modifies 2 testcase(s) {color:green}+1 RAT{color} .{color:green}+1{color} the patch does not seem to introduce new RAT warnings {color:green}+1 JAVADOC{color} .{color:green}+1{color} Javadoc generation succeeded with the patch .{color:green}+1{color} the patch does not seem to introduce new Javadoc warning(s) .{color:orange}WARNING{color}: the current HEAD has 100 Javadoc warning(s) {color:green}+1 COMPILE{color} .{color:green}+1{color} HEAD compiles .{color:green}+1{color} patch compiles .{color:green}+1{color} the patch does not seem to introduce new javac warnings {color:green}+1{color} There are no new bugs found in total. . {color:green}+1{color} There are no new bugs found in [sharelib/hive2]. . {color:green}+1{color} There are no new bugs found in [sharelib/spark]. . {color:green}+1{color} There are no new bugs found in [sharelib/oozie]. . {color:green}+1{color} There are no new bugs found in [sharelib/pig]. . {color:green}+1{color} There are no new bugs found in [sharelib/streaming]. . {color:green}+1{color} There are no new bugs found in [sharelib/hive]. . {color:green}+1{color} There are no new bugs found in [sharelib/distcp]. . {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog]. . {color:green}+1{color} There are no new bugs found in [sharelib/sqoop]. . {color:green}+1{color} There are no new bugs found in [sharelib/git]. . {color:green}+1{color} There are no new bugs found in [client]. . {color:green}+1{color} There are no new bugs found in [docs]. . {color:green}+1{color} There are no new bugs found in [tools]. . {color:green}+1{color} There are no new bugs found in [fluent-job/fluent-job-api]. . {color:green}+1{color} There are no new bugs found in [server]. . {color:green}+1{color} There are no new bugs found in [webapp]. . {color:green}+1{color} There are no new bugs found in [examples]. . {color:green}+1{color} There are no new bugs found in [core]. {color:green}+1 BACKWARDS_COMPATIBILITY{color} .{color:green}+1{color} the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations .{color:green}+1{color} the patch does not modify JPA files {color:green}+1 TESTS{color} .Tests run: 3125 .{color:orange}Tests failed at first run:{color} TestPurgeXCommand#testPurgeBundleWithCoordChildWithWFChild3MoreThanLimit .For the complete list of flaky tests, see TEST-SUMMARY-FULL files. {color:green}+1 DISTRO{color} .{color:green}+1{color} distro tarball builds with the patch {color:green}*+1 Overall result, good!, no -1s*{color} {color:red}. There is at least one warning, please check{color} The full output of the test-patch run is available at . https://builds.apache.org/job/PreCommit-OOZIE-Build/962/ > Escape quotes whitespaces in Sqoop field > -- > > Key: OOZIE-2949 > URL: https://issues.apache.org/jira/browse/OOZIE-2949 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Peter Cseh >Assignee: Andras Salamon >Priority: Major > Fix For: 5.2.0 > > Attachments: OOZIE-2949-0.patch, OOZIE-2949-01.patch, > OOZIE-2949-02.patch > > > The current behavior of the Sqoop action is: > {noformat} > The Sqoop command can be specified either using the command element or > multiple arg elements. > When using the command element, Oozie will split the command on every space > into multiple arguments. > When using the arg elements, Oozie will pass each argument value as an > argument to Sqoop. > {noformat} > This prevents the user to simply copy-paste the command worked in the shell > into the workflow.xml. > We should split the field by taking quotes into account, similar to > what OOZIE-2391 > did for the Spark action's field. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Failed: OOZIE-2949 PreCommit Build #962
Jira: https://issues.apache.org/jira/browse/OOZIE-2949 Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/962/ ### ## LAST 100 LINES OF THE CONSOLE ### [...truncated 1.91 MB...] [DEBUG] There are no new bugs found in [tools]. [TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts [DEBUG] There are no new bugs found in [fluent-job/fluent-job-api]. [TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts [DEBUG] There are no new bugs found in [server]. [TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts [DEBUG] There are no new bugs found in [webapp]. [TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts [DEBUG] There are no new bugs found in [examples]. [TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts [DEBUG] There are no new bugs found in [core]. [INFO] There are no new bugs found totally]. [TRACE] FindBugs diffs checked and reports created [TRACE] Summary file size is 2560 bytes [TRACE] Full summary file size is 1471 bytes [TRACE] File [/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar] removed [TRACE] File [/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum] removed Running test-patch task BACKWARDS_COMPATIBILITY Running test-patch task TESTS Running test-patch task DISTRO Testing JIRA OOZIE-2949 Cleaning local git workspace +1 PATCH_APPLIES +1 CLEAN +1 RAW_PATCH_ANALYSIS +1 the patch does not introduce any @author tags +1 the patch does not introduce any tabs +1 the patch does not introduce any trailing spaces +1 the patch does not introduce any star imports +1 the patch does not introduce any line longer than 132 +1 the patch adds/modifies 2 testcase(s) +1 RAT +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC +1 Javadoc generation succeeded with the patch +1 the patch does not seem to introduce new Javadoc warning(s) WARNING: the current HEAD has 100 Javadoc warning(s) +1 COMPILE +1 HEAD compiles +1 patch compiles +1 the patch does not seem to introduce new javac warnings +1 There are no new bugs found in total. +1 There are no new bugs found in [sharelib/hive2]. +1 There are no new bugs found in [sharelib/spark]. +1 There are no new bugs found in [sharelib/oozie]. +1 There are no new bugs found in [sharelib/pig]. +1 There are no new bugs found in [sharelib/streaming]. +1 There are no new bugs found in [sharelib/hive]. +1 There are no new bugs found in [sharelib/distcp]. +1 There are no new bugs found in [sharelib/hcatalog]. +1 There are no new bugs found in [sharelib/sqoop]. +1 There are no new bugs found in [sharelib/git]. +1 There are no new bugs found in [client]. +1 There are no new bugs found in [docs]. +1 There are no new bugs found in [tools]. +1 There are no new bugs found in [fluent-job/fluent-job-api]. +1 There are no new bugs found in [server]. +1 There are no new bugs found in [webapp]. +1 There are no new bugs found in [examples]. +1 There are no new bugs found in [core]. +1 BACKWARDS_COMPATIBILITY +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations +1 the patch does not modify JPA files +1 TESTS Tests run: 3125 Tests failed at first run: TestPurgeXCommand#testPurgeBundleWithCoordChildWithWFChild3MoreThanLimit For the complete list of flaky tests, see TEST-SUMMARY-FULL files. +1 DISTRO +1 distro tarball builds with the patch +1 Overall result, good!, no -1s There is at least one warning, please check The full output of the test-patch run is available at https://builds.apache.org/job/PreCommit-OOZIE-Build/962/ Adding comment to JIRA % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0100 34650 0 100 3465 0 3626 --:--:-- --:--:-- --:--:-- 3624{"self":"https://issues.apache.org/jira/rest/api/2/issue/13080339/comment/16734379","id":"16734379","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop
[jira] [Commented] (OOZIE-2949) Escape quotes whitespaces in Sqoop field
[ https://issues.apache.org/jira/browse/OOZIE-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734270#comment-16734270 ] Hadoop QA commented on OOZIE-2949: -- PreCommit-OOZIE-Build started > Escape quotes whitespaces in Sqoop field > -- > > Key: OOZIE-2949 > URL: https://issues.apache.org/jira/browse/OOZIE-2949 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Peter Cseh >Assignee: Andras Salamon >Priority: Major > Fix For: 5.2.0 > > Attachments: OOZIE-2949-0.patch, OOZIE-2949-01.patch, > OOZIE-2949-02.patch > > > The current behavior of the Sqoop action is: > {noformat} > The Sqoop command can be specified either using the command element or > multiple arg elements. > When using the command element, Oozie will split the command on every space > into multiple arguments. > When using the arg elements, Oozie will pass each argument value as an > argument to Sqoop. > {noformat} > This prevents the user to simply copy-paste the command worked in the shell > into the workflow.xml. > We should split the field by taking quotes into account, similar to > what OOZIE-2391 > did for the Spark action's field. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-2949) Escape quotes whitespaces in Sqoop field
[ https://issues.apache.org/jira/browse/OOZIE-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734252#comment-16734252 ] Andras Salamon commented on OOZIE-2949: --- Thanks [~gezapeti] for the list to the gist, I've created a bash-like splitter using that code. I moved this code to a nested class and created a separate unit test class for this. I've also added a new unit test method to {{TestSqoopActionExecutor}} because the existing methods were not really testing this feature. > Escape quotes whitespaces in Sqoop field > -- > > Key: OOZIE-2949 > URL: https://issues.apache.org/jira/browse/OOZIE-2949 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Peter Cseh >Assignee: Andras Salamon >Priority: Major > Fix For: 5.2.0 > > Attachments: OOZIE-2949-0.patch, OOZIE-2949-01.patch, > OOZIE-2949-02.patch > > > The current behavior of the Sqoop action is: > {noformat} > The Sqoop command can be specified either using the command element or > multiple arg elements. > When using the command element, Oozie will split the command on every space > into multiple arguments. > When using the arg elements, Oozie will pass each argument value as an > argument to Sqoop. > {noformat} > This prevents the user to simply copy-paste the command worked in the shell > into the workflow.xml. > We should split the field by taking quotes into account, similar to > what OOZIE-2391 > did for the Spark action's field. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Review Request 69668: OOZIE 2949 - Escape quotes whitespaces in Sqoop field
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69668/ --- Review request for oozie, András Piros, Denes Bodo, Peter Cseh, and Kinga Marton. Repository: oozie-git Description --- OOZIE 2949 - Escape quotes whitespaces in Sqoop field Diffs - core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 556f2cfd1 core/src/test/java/org/apache/oozie/action/hadoop/TestShellSplitter.java PRE-CREATION sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopActionExecutor.java edfe0c739 Diff: https://reviews.apache.org/r/69668/diff/1/ Testing --- Unit tests locally Thanks, Andras Salamon
[jira] [Updated] (OOZIE-2949) Escape quotes whitespaces in Sqoop field
[ https://issues.apache.org/jira/browse/OOZIE-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-2949: -- Attachment: OOZIE-2949-02.patch > Escape quotes whitespaces in Sqoop field > -- > > Key: OOZIE-2949 > URL: https://issues.apache.org/jira/browse/OOZIE-2949 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Peter Cseh >Assignee: Andras Salamon >Priority: Major > Fix For: 5.2.0 > > Attachments: OOZIE-2949-0.patch, OOZIE-2949-01.patch, > OOZIE-2949-02.patch > > > The current behavior of the Sqoop action is: > {noformat} > The Sqoop command can be specified either using the command element or > multiple arg elements. > When using the command element, Oozie will split the command on every space > into multiple arguments. > When using the arg elements, Oozie will pass each argument value as an > argument to Sqoop. > {noformat} > This prevents the user to simply copy-paste the command worked in the shell > into the workflow.xml. > We should split the field by taking quotes into account, similar to > what OOZIE-2391 > did for the Spark action's field. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3218) Oozie Sqoop action with command splits the select clause into multiple parts due to delimiter being space
[ https://issues.apache.org/jira/browse/OOZIE-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734035#comment-16734035 ] Denes Bodo commented on OOZIE-3218: --- I prefer to handle the quotes. In my opinion using quotes in linux environment is a common when I need to write whitespaces. Also the implementation could be more maintainable due to we do not have to watch newly created options/token borders. > Oozie Sqoop action with command splits the select clause into multiple parts > due to delimiter being space > - > > Key: OOZIE-3218 > URL: https://issues.apache.org/jira/browse/OOZIE-3218 > Project: Oozie > Issue Type: Bug > Components: action, workflow >Affects Versions: 3.3.2, 4.1.0, 4.2.0, 4.3.0, 5.0.0 > Environment: Hortonworks Hadoop HDP-2.6.4.x release > oozie admin -version: Oozie server build version: 4.2.0.2.6.4.0-91 >Reporter: Mahesh Balakrishnan >Assignee: Mahesh Balakrishnan >Priority: Major > Attachments: OOZIE-3218-2.patch, OOZIE-3218-3.patch, OOZIE-3218.patch > > > When running a Oozie Sqoop action which has command with {{--query}} in place > the query is split into multiple parts causing {{"Unrecognized argument:"}} > and in-turn fails. > {code:xml} > > ${resourceManager} > ${nameNode} > import --verbose --connect jdbc:mysql://test.openstacklocal/db > --query select * from abc where $CONDITIONS --username test --password test > --driver com.mysql.jdbc.Driver -m 1 > > > {code} > > Oozie Launcher logs: > {noformat} > Sqoop command arguments : > import > --verbose > --connect > jdbc:mysql://test.openstacklocal/db > --query > "select > * > from > abc > where > $CONDITIONS" > --username > hive > --password > > --driver > com.mysql.jdbc.Driver > -m > 1 > Fetching child yarn jobs > tag id : oozie-a1bbe03a0983b9e822d12ae7bb269ee3 > 2791 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to > ResourceManager at hdp263-3.openstacklocal/172.26.105.248:8050 > Child yarn jobs are found - > = > >>> Invoking Sqoop command line now >>> > 3172 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not > been set in the environment. Cannot check for additional configuration. > 3172 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not > been set in the environment. Cannot check for additional configuration. > 3218 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: > 1.4.6.2.6.4.0-91 > 3218 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: > 1.4.6.2.6.4.0-91 > 3287 [main] DEBUG org.apache.sqoop.tool.BaseSqoopTool - Enabled debug > logging. > 3287 [main] DEBUG org.apache.sqoop.tool.BaseSqoopTool - Enabled debug > logging. > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing > arguments for import: > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing > arguments for import: > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: * > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: * > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: from > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: from > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: abc > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: abc > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: where > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: where > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: $CONDITIONS" > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: $CONDITIONS" > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: --username > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: --username > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: abc > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: abc > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: --password > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: --password > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: abc > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: abc > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: --driver
[jira] [Commented] (OOZIE-3218) Oozie Sqoop action with command splits the select clause into multiple parts due to delimiter being space
[ https://issues.apache.org/jira/browse/OOZIE-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734039#comment-16734039 ] Andras Salamon commented on OOZIE-3218: --- Thanks [~kmarton], [~dionusos]. Closing this issue as duplicate, and continue the development on OOZIE-2949 . > Oozie Sqoop action with command splits the select clause into multiple parts > due to delimiter being space > - > > Key: OOZIE-3218 > URL: https://issues.apache.org/jira/browse/OOZIE-3218 > Project: Oozie > Issue Type: Bug > Components: action, workflow >Affects Versions: 3.3.2, 4.1.0, 4.2.0, 4.3.0, 5.0.0 > Environment: Hortonworks Hadoop HDP-2.6.4.x release > oozie admin -version: Oozie server build version: 4.2.0.2.6.4.0-91 >Reporter: Mahesh Balakrishnan >Assignee: Mahesh Balakrishnan >Priority: Major > Attachments: OOZIE-3218-2.patch, OOZIE-3218-3.patch, OOZIE-3218.patch > > > When running a Oozie Sqoop action which has command with {{--query}} in place > the query is split into multiple parts causing {{"Unrecognized argument:"}} > and in-turn fails. > {code:xml} > > ${resourceManager} > ${nameNode} > import --verbose --connect jdbc:mysql://test.openstacklocal/db > --query select * from abc where $CONDITIONS --username test --password test > --driver com.mysql.jdbc.Driver -m 1 > > > {code} > > Oozie Launcher logs: > {noformat} > Sqoop command arguments : > import > --verbose > --connect > jdbc:mysql://test.openstacklocal/db > --query > "select > * > from > abc > where > $CONDITIONS" > --username > hive > --password > > --driver > com.mysql.jdbc.Driver > -m > 1 > Fetching child yarn jobs > tag id : oozie-a1bbe03a0983b9e822d12ae7bb269ee3 > 2791 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to > ResourceManager at hdp263-3.openstacklocal/172.26.105.248:8050 > Child yarn jobs are found - > = > >>> Invoking Sqoop command line now >>> > 3172 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not > been set in the environment. Cannot check for additional configuration. > 3172 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not > been set in the environment. Cannot check for additional configuration. > 3218 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: > 1.4.6.2.6.4.0-91 > 3218 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: > 1.4.6.2.6.4.0-91 > 3287 [main] DEBUG org.apache.sqoop.tool.BaseSqoopTool - Enabled debug > logging. > 3287 [main] DEBUG org.apache.sqoop.tool.BaseSqoopTool - Enabled debug > logging. > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing > arguments for import: > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing > arguments for import: > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: * > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: * > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: from > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: from > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: abc > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: abc > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: where > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: where > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: $CONDITIONS" > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: $CONDITIONS" > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: --username > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: --username > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: abc > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: abc > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: --password > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: --password > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: abc > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: abc > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: --driver > 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized > argument: --driver > 3288 [main] ERROR
[jira] [Commented] (OOZIE-3067) [core] Remove duplicate logic from ZKJobsConcurrencyService
[ https://issues.apache.org/jira/browse/OOZIE-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734013#comment-16734013 ] Andras Piros commented on OOZIE-3067: - Thanks for the contribution [~dionusos]! +1 > [core] Remove duplicate logic from ZKJobsConcurrencyService > --- > > Key: OOZIE-3067 > URL: https://issues.apache.org/jira/browse/OOZIE-3067 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 4.2.0 >Reporter: Denes Bodo >Assignee: Denes Bodo >Priority: Minor > Fix For: trunk > > Attachments: > 0001-OOZIE-3067-Remove-duplicated-logic-from-ZKJobsConcur.patch, > OOZIE-3067.002.patch > > > In ZKJobsConcurrencyService getServerUrls and getOtherServerUrls methods > basically the same. However, the logic is written inside both methods. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3067) [core] Remove duplicate logic from ZKJobsConcurrencyService
[ https://issues.apache.org/jira/browse/OOZIE-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Piros updated OOZIE-3067: Summary: [core] Remove duplicate logic from ZKJobsConcurrencyService (was: Remove duplicated logic from ZKJobsConcurrencyService) > [core] Remove duplicate logic from ZKJobsConcurrencyService > --- > > Key: OOZIE-3067 > URL: https://issues.apache.org/jira/browse/OOZIE-3067 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 4.2.0 >Reporter: Denes Bodo >Assignee: Denes Bodo >Priority: Minor > Fix For: trunk > > Attachments: > 0001-OOZIE-3067-Remove-duplicated-logic-from-ZKJobsConcur.patch, > OOZIE-3067.002.patch > > > In ZKJobsConcurrencyService getServerUrls and getOtherServerUrls methods > basically the same. However, the logic is written inside both methods. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3186) [core] Oozie is unable to use configuration linked using jceks://file/
[ https://issues.apache.org/jira/browse/OOZIE-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Piros updated OOZIE-3186: Summary: [core] Oozie is unable to use configuration linked using jceks://file/ (was: Oozie is unable to use configuration linked using jceks://file/...) > [core] Oozie is unable to use configuration linked using jceks://file/ > -- > > Key: OOZIE-3186 > URL: https://issues.apache.org/jira/browse/OOZIE-3186 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 5.0.0b1, 5.0.0, 4.3.1 >Reporter: Denes Bodo >Assignee: Denes Bodo >Priority: Critical > Labels: usability > Fix For: trunk > > Attachments: OOZIE-3186-001.patch, OOZIE-3186-002.patch, > OOZIE-3186-003.patch > > > When Oozie is used with Ambari, the next configuration makes Oozie fail to > start: > {noformat} > > hadoop.security.credential.provider.path > jceks://file/.../oozie-site.jceks > > {noformat} > Value should have *localjceks://* instead of *jceks://*. But Ambari does not > let change this value. I propose change the url when Oozie loads it. > > Stacktrace, when the issue occurs: > {code:java} > org.apache.oozie.service.ServiceException: E0103: Could not load service > classes, Could not load password for [oozie.service.JPAService.jdbc.password] > at org.apache.oozie.service.Services.loadServices(Services.java:309) at > org.apache.oozie.service.Services.init(Services.java:213) at > org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46) > at > org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4276) > at org.apache.catalina.core.StandardContext.start(StandardContext.java:4779) > at > org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:803) > at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:780) > at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583) at > org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:944) at > org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:779) at > org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:505) at > org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322) at > org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325) at > org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142) > at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1069) at > org.apache.catalina.core.StandardHost.start(StandardHost.java:822) at > org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061) at > org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463) at > org.apache.catalina.core.StandardService.start(StandardService.java:525) at > org.apache.catalina.core.StandardServer.start(StandardServer.java:761) at > org.apache.catalina.startup.Catalina.start(Catalina.java:595) at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) at > org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by: > java.lang.IllegalArgumentException: Could not load password for > [oozie.service.JPAService.jdbc.password] at > org.apache.oozie.service.ConfigurationService.getPassword(ConfigurationService.java:615) > at > org.apache.oozie.service.ConfigurationService.getPassword(ConfigurationService.java:602) > at org.apache.oozie.service.JPAService.init(JPAService.java:147) at > org.apache.oozie.service.Services.setServiceInternal(Services.java:386) at > org.apache.oozie.service.Services.setService(Services.java:372) at > org.apache.oozie.service.Services.loadServices(Services.java:305) ... 26 more > Caused by: java.lang.reflect.InvocationTargetException at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.apache.oozie.service.ConfigurationService.getPassword(ConfigurationService.java:608) > ... 31 more Caused by: java.lang.UnsupportedOperationException: Accessing > local file system is not allowed at > org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:48) > at org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSystem.java:47) >
[jira] [Commented] (OOZIE-3186) [core] Oozie is unable to use configuration linked using jceks://file/
[ https://issues.apache.org/jira/browse/OOZIE-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734008#comment-16734008 ] Andras Piros commented on OOZIE-3186: - Thanks for the contribution [~dionusos]! +1 > [core] Oozie is unable to use configuration linked using jceks://file/ > -- > > Key: OOZIE-3186 > URL: https://issues.apache.org/jira/browse/OOZIE-3186 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 5.0.0b1, 5.0.0, 4.3.1 >Reporter: Denes Bodo >Assignee: Denes Bodo >Priority: Critical > Labels: usability > Fix For: trunk > > Attachments: OOZIE-3186-001.patch, OOZIE-3186-002.patch, > OOZIE-3186-003.patch > > > When Oozie is used with Ambari, the next configuration makes Oozie fail to > start: > {noformat} > > hadoop.security.credential.provider.path > jceks://file/.../oozie-site.jceks > > {noformat} > Value should have *localjceks://* instead of *jceks://*. But Ambari does not > let change this value. I propose change the url when Oozie loads it. > > Stacktrace, when the issue occurs: > {code:java} > org.apache.oozie.service.ServiceException: E0103: Could not load service > classes, Could not load password for [oozie.service.JPAService.jdbc.password] > at org.apache.oozie.service.Services.loadServices(Services.java:309) at > org.apache.oozie.service.Services.init(Services.java:213) at > org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46) > at > org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4276) > at org.apache.catalina.core.StandardContext.start(StandardContext.java:4779) > at > org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:803) > at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:780) > at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583) at > org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:944) at > org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:779) at > org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:505) at > org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322) at > org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325) at > org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142) > at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1069) at > org.apache.catalina.core.StandardHost.start(StandardHost.java:822) at > org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061) at > org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463) at > org.apache.catalina.core.StandardService.start(StandardService.java:525) at > org.apache.catalina.core.StandardServer.start(StandardServer.java:761) at > org.apache.catalina.startup.Catalina.start(Catalina.java:595) at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) at > org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by: > java.lang.IllegalArgumentException: Could not load password for > [oozie.service.JPAService.jdbc.password] at > org.apache.oozie.service.ConfigurationService.getPassword(ConfigurationService.java:615) > at > org.apache.oozie.service.ConfigurationService.getPassword(ConfigurationService.java:602) > at org.apache.oozie.service.JPAService.init(JPAService.java:147) at > org.apache.oozie.service.Services.setServiceInternal(Services.java:386) at > org.apache.oozie.service.Services.setService(Services.java:372) at > org.apache.oozie.service.Services.loadServices(Services.java:305) ... 26 more > Caused by: java.lang.reflect.InvocationTargetException at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.apache.oozie.service.ConfigurationService.getPassword(ConfigurationService.java:608) > ... 31 more Caused by: java.lang.UnsupportedOperationException: Accessing > local file system is not allowed at > org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:48) > at org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSystem.java:47) > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2795) at
[jira] [Comment Edited] (OOZIE-2949) Escape quotes whitespaces in Sqoop field
[ https://issues.apache.org/jira/browse/OOZIE-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731989#comment-16731989 ] Andras Salamon edited comment on OOZIE-2949 at 1/4/19 9:00 AM: --- I've checked the current version of the patch in more detail. The plan was to handle commands like {{\-\-query 'SELECT * FROM employee'}} and use two tokens in this case ({{"\-\-query"}} and {{"SELECT * FROM employee"}}). The current patch works in this case. It fails if single and double quotes are mixed like ' ... " " ' because it thinks that the first double quote closes the single quote. It would be easy to fix the patch and recognise the whole command like one token. I'm not sure if this is a good enough solution. It will still fail if we want to use a single quote inside the query (e.g. search for O'Briens). I've also talked to sqoop experts. The confirmed that sqoop command parsing is a rather complex task. If we want to be able to block-copy from shell to oozie, the best approach is probably mimicking the shell parsing. We would still need real-life examples to create lots of test cases. was (Author: asalamon74): I've checked the current version of the patch in more detail. The plan was to handle commands like {{--query 'SELECT * FROM employee'}} and use two tokens in this case ({{"--query"}} and {{"SELECT * FROM employee"}}). The current patch works in this case. It fails if single and double quotes are mixed like ' ... " " ' because it thinks that the first double quote closes the single quote. It would be easy to fix the patch and recognise the whole command like one token. I'm not sure if this is a good enough solution. It will still fail if we want to use a single quote inside the query (e.g. search for O'Briens). I've also talked to sqoop experts. The confirmed that sqoop command parsing is a rather complex task. If we want to be able to block-copy from shell to oozie, the best approach is probably mimicking the shell parsing. We would still need real-life examples to create lots of test cases. > Escape quotes whitespaces in Sqoop field > -- > > Key: OOZIE-2949 > URL: https://issues.apache.org/jira/browse/OOZIE-2949 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Peter Cseh >Assignee: Andras Salamon >Priority: Major > Fix For: 5.2.0 > > Attachments: OOZIE-2949-0.patch, OOZIE-2949-01.patch > > > The current behavior of the Sqoop action is: > {noformat} > The Sqoop command can be specified either using the command element or > multiple arg elements. > When using the command element, Oozie will split the command on every space > into multiple arguments. > When using the arg elements, Oozie will pass each argument value as an > argument to Sqoop. > {noformat} > This prevents the user to simply copy-paste the command worked in the shell > into the workflow.xml. > We should split the field by taking quotes into account, similar to > what OOZIE-2391 > did for the Spark action's field. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3405) SSH action shows empty error Message and Error code
[ https://issues.apache.org/jira/browse/OOZIE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733936#comment-16733936 ] Andras Salamon commented on OOZIE-3405: --- [~orova] I suggest creating one new error code with a message template like 'Script failed with exit value XX'. If you store the exit value in the {{.error}} file, you could use that information here. If someone needs more information, they can check the log files. > SSH action shows empty error Message and Error code > --- > > Key: OOZIE-3405 > URL: https://issues.apache.org/jira/browse/OOZIE-3405 > Project: Oozie > Issue Type: Improvement >Reporter: Peter Orova >Assignee: Peter Orova >Priority: Minor > > Currently, when an SSH action fails the only message that is returned is the > Status. Neither the {{error Message}} nor {{Error code}} fields are filled. > This makes reporting on the causes of SSH Action failures via Oozie highly > impractical: the only meaningful bit of information there is on a failed SSH > Action is the Status. > The Status is filled based on the presence (or lack of) the {{.error file}} > that is produced in case the user submitted script returns with any other > value than 0. > {noformat} > SshActionExecutor#getActionStatus > ... > String outFile = getRemoteFileName(context, action, "error", false, true); > String checkErrorCmd = SSH_COMMAND_BASE + action.getTrackerUri() + " ls " + > outFile; > int retVal = getReturnValue(checkErrorCmd); > ... > {noformat} > > User requirement is to provide some more detailed information on the > success/failure of the user-submitted script. That could be at a minimum the > return value, optionally the last ~1K of the stderr that is drained. This > information could then be communicated via {{errorMessage}} and {{ErrorCode}} -- This message was sent by Atlassian JIRA (v7.6.3#76005)