[jira] [Commented] (OOZIE-2301) JA017: Could not lookup launched hadoop Job ID

2018-05-25 Thread Mobin Ranjbar (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491511#comment-16491511
 ] 

Mobin Ranjbar commented on OOZIE-2301:
--

[~dbist13] Sure, Thanks. I had no idea whether it is a bug or error because 
there were many people who had this problem and no solution found. Anyway, 
Thanks again.

> JA017: Could not lookup launched hadoop Job ID 
> ---
>
> Key: OOZIE-2301
> URL: https://issues.apache.org/jira/browse/OOZIE-2301
> Project: Oozie
>  Issue Type: Bug
>Reporter: Mobin Ranjbar
>Priority: Major
>
> How can I solve this problem when I submit a mapreduce job in Oozie Editor in 
> Hue? :
> JA017: Could not lookup launched hadoop Job ID [job_local152843681_0009] 
> which was associated with action 
> [009-150711083342968-oozie-root-W@mapreduce-f660]. Failing this action!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OOZIE-3260) Remove item after first try from in-memory SLA map

2018-05-25 Thread Andras Piros (JIRA) 
Andras Piros created OOZIE-3260:
---

 Summary: Remove item after first try from in-memory SLA map
 Key: OOZIE-3260
 URL: https://issues.apache.org/jira/browse/OOZIE-3260
 Project: Oozie
  Issue Type: Bug
  Components: coordinator, core, workflow
Affects Versions: 5.0.0
Reporter: Andras Piros
Assignee: Andras Piros


Despite having implemented OOZIE-3134, there are still cases where 
{{SLACalculatorMemory#slaMap}} and database contents still get out of sync. 
E.g. database contents of {{SLA_SUMMARY}} table have been purged manually from 
DB, or no corresponding {{WF_JOBS}} or {{COORD_JOBS}} entries exist anymore in 
DB.

In those rare cases, we see {{JPAExecutorException}} instances like:
{noformat}
2017-10-09 17:00:18,185 DEBUG openjpa.jdbc.SQL: SERVER[HOST]  [0 ms] spent
2017-10-09 17:00:18,185 ERROR org.apache.oozie.sla.SLACalculatorMemory: 
SERVER[tplhc01c001.iuser.iroot.adidom.com] USER[-] GROUP[-] TOKEN[-] APP[-] 
JOB[438-170916014916144-oozie-oozi-C@556] ACTION[-] Exception in SLA 
processing for job [438-170916014916144-oozie-oozi-C@556]
org.apache.oozie.executor.jpa.JPAExecutorException: E0604: Job does not exist 
[select w.eventProcessed from SLASummaryBean w where w.jobId = :id]
at 
org.apache.oozie.executor.jpa.SLASummaryQueryExecutor.getSingleValue(SLASummaryQueryExecutor.java:161)
at 
org.apache.oozie.sla.SLACalculatorMemory.updateJobSla(SLACalculatorMemory.java:480)
at 
org.apache.oozie.sla.SLACalculatorMemory.updateAllSlaStatus(SLACalculatorMemory.java:601)
{noformat}
or
{noformat}
2017-10-09 17:00:53,085 WARN 
org.apache.oozie.service.CallableQueueService$CompositeCallable: SERVER[HOST] 
USER[-] GROUP[-] TOKEN[-] APP[-] JOB[011-170813033731256-oozie-oozi-W] 
ACTION[011-170813033731256-oozie-oozi-W@sqoop_full_tbl_unload] exception 
callable [action.check], E0604: Job does not exist [select w.statusStr from 
WorkflowJobBean w where w.id = :id]
org.apache.oozie.command.CommandException: E0604: Job does not exist [select 
w.statusStr from WorkflowJobBean w where w.id = :id]
at 
org.apache.oozie.command.wf.ActionCheckXCommand.eagerLoadState(ActionCheckXCommand.java:97)
at org.apache.oozie.command.XCommand.call(XCommand.java:256)
at 
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:332)
at 
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:261)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.oozie.executor.jpa.JPAExecutorException: E0604: Job does 
not exist [select w.statusStr from WorkflowJobBean w where w.id = :id]
at 
org.apache.oozie.executor.jpa.WorkflowJobQueryExecutor.get(WorkflowJobQueryExecutor.java:345)
at 
org.apache.oozie.executor.jpa.WorkflowJobQueryExecutor.get(WorkflowJobQueryExecutor.java:38)
at 
org.apache.oozie.command.wf.ActionCheckXCommand.eagerLoadState(ActionCheckXCommand.java:90)
{noformat}

Solution here is to remove any {{SLACalculatorMemory#slaMap}} entries that are 
causing those {{JPAExecutorException}} instances after the first unsuccessful 
run, to not cause huge logfiles. The items to be logged don't exist anymore, 
anyways.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3258) Surefire should not trim stack traces

2018-05-25 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490738#comment-16490738
 ] 

Hadoop QA commented on OOZIE-3258:
--


Testing JIRA OOZIE-3258

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
error(s)
.{color:red}ERROR{color}: the current HEAD has 2 Javadoc error(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2132
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/580/



> Surefire should not trim stack traces
> -
>
> Key: OOZIE-3258
> URL: https://issues.apache.org/jira/browse/OOZIE-3258
> Project: Oozie
>  Issue Type: Bug
>  Components: core, tests
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: OOZIE-3258-001.patch
>
>
> The current setting of surefire trims stack traces if it reaches JUnit. This 
> makes debugging harder, because it's not obvious what code path is affected:
> {noformat}
> java.io.EOFException: End of File Exception between local host is: 
> "asf918.gq1.ygridcore.net/67.195.81.138"; destination host is: 
> "localhost":35969; : java.io.EOFException; For more details see:  
> http://wiki.apache.org/hadoop/EOFException
>   at 
> org.apache.oozie.util.TestZKUtilsWithSecurity.setUp(TestZKUtilsWithSecurity.java:36)
> Caused by: java.io.EOFException
> {noformat}
> We have to see the full trace in order to address problems efficiently.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Failed: OOZIE-3258 PreCommit Build #580

2018-05-25 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-3258
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/580/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.71 MB...]
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/streaming].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hive].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hcatalog].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/sqoop].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/oozie].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/distcp].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/spark].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [client].
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2366 bytes
[TRACE] Full summary file size is 1314 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build@2/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build@2/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO


Testing JIRA OOZIE-3258

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
-1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any line longer than 132
-1 the patch does not add/modify any testcase
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warning(s)
+1 the patch does not seem to introduce new Javadoc error(s)
ERROR: the current HEAD has 2 Javadoc error(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [webapp].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [client].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2132
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)


The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/580/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0{"self":"https://issues.apache.org/jira/rest/api/2/issue/13162055/comment/16490738","id":"16490738","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","emailAddress":"blackhole
 at hadoop dot apache dot

[jira] [Commented] (OOZIE-1178) Workflow Application Master in YARN

2018-05-25 Thread Peter Cseh (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490736#comment-16490736
 ] 

Peter Cseh commented on OOZIE-1178:
---

Yeah, this is for run and manage the workflow execution from a Yarn AM instead 
of the Oozie server to make Oozie more scalable.
There are some issues with this though:
# how to not DDOS the database? If every WFAM communicates with the Oozie 
server to talk to the database, would it help the scalability at all?
# As [~andras.piros] mentioned there are some issues with synchronous actions?
## how to run ssh action? - will it be even supported?
## Email and FS action are looking more managable
# How we handle getting and injecting delegation tokens to the WFAM for every 
action? We certainly don't want to distribute the Oozie keytab within the Yarn 
cluster

There are some crazy upsides in this though: it would open up the possibility 
to execute way more dynamic workflows (e.g. workflow defined by code) as user 
code would run more contained.


> Workflow Application Master in YARN
> ---
>
> Key: OOZIE-1178
> URL: https://issues.apache.org/jira/browse/OOZIE-1178
> Project: Oozie
>  Issue Type: New Feature
>Reporter: Bo Wang
>Priority: Major
> Attachments: MAPREDUCE-4495-v1.1.patch, MAPREDUCE-4495-v1.patch, 
> MapReduceWorkflowAM.pdf, yapp_proposal.txt
>
>
> It is useful to have a workflow application master, which will be capable of 
> running a DAG of jobs. The workflow client submits a DAG request to the AM 
> and then the AM will manage the life cycle of this application in terms of 
> requesting the needed resources from the RM, and starting, monitoring and 
> retrying the application's individual tasks.
> Compared to running Oozie with the current MapReduce Application Master, 
> these are some of the advantages:
>  - Less number of consumed resources, since only one application master will 
> be spawned for the whole workflow.
>  - Reuse of resources, since the same resources can be used by multiple 
> consecutive jobs in the workflow (no need to request/wait for resources for 
> every individual job from the central RM).
>  - More optimization opportunities in terms of collective resource requests.
>  - Optimization opportunities in terms of rewriting and composing jobs in the 
> workflow (e.g. pushing down Mappers).
>  - This Application Master can be reused/extended by higher systems like Pig 
> and hive to provide an optimized way of running their workflows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3257) Testcase TestHiveActionExecutor#testHiveAction still fails

2018-05-25 Thread Andras Piros (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-3257:

Summary: Testcase TestHiveActionExecutor#testHiveAction still fails  (was: 
Testcase TestHiveActionExecutor#testHiveAction still fails after OOZIE-3236)

> Testcase TestHiveActionExecutor#testHiveAction still fails
> --
>
> Key: OOZIE-3257
> URL: https://issues.apache.org/jira/browse/OOZIE-3257
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: OOZIE-3257-001.patch
>
>
> OOZIE-3236 did not fully fix {{testHiveAction}}. The reason is that two Hive 
> action is started from this test and the second one still uses the default 
> timeout of 60 seconds.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3257) TestHiveActionExecutor#testHiveAction still fails

2018-05-25 Thread Andras Piros (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-3257:

Summary: TestHiveActionExecutor#testHiveAction still fails  (was: Testcase 
TestHiveActionExecutor#testHiveAction still fails)

> TestHiveActionExecutor#testHiveAction still fails
> -
>
> Key: OOZIE-3257
> URL: https://issues.apache.org/jira/browse/OOZIE-3257
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: OOZIE-3257-001.patch
>
>
> OOZIE-3236 did not fully fix {{testHiveAction}}. The reason is that two Hive 
> action is started from this test and the second one still uses the default 
> timeout of 60 seconds.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3257) Testcase TestHiveActionExecutor#testHiveAction still fails after OOZIE-3236

2018-05-25 Thread Andras Piros (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490725#comment-16490725
 ] 

Andras Piros commented on OOZIE-3257:
-

Thanks for the contribution [~pbacsko]! +1

> Testcase TestHiveActionExecutor#testHiveAction still fails after OOZIE-3236
> ---
>
> Key: OOZIE-3257
> URL: https://issues.apache.org/jira/browse/OOZIE-3257
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: OOZIE-3257-001.patch
>
>
> OOZIE-3236 did not fully fix {{testHiveAction}}. The reason is that two Hive 
> action is started from this test and the second one still uses the default 
> timeout of 60 seconds.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-1178) Workflow Application Master in YARN

2018-05-25 Thread Andras Piros (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490722#comment-16490722
 ] 

Andras Piros commented on OOZIE-1178:
-

[~dbist13] not exactly. This JIRA is about having the whole workflow (all 
applications) run on YARN in a single {{WorkflowAM}} ApplicationMaster, whereas 
[*Oozie On YARN*|https://issues.apache.org/jira/browse/OOZIE-1770] was about 
having one workflow application's launcher run on YARN as a {{LauncherAM}} 
ApplicationMaster. I wouldn't close for that reason.

Another question can be if we really want to support something like that; in 
the meanwhile we have workflow actions that are meant to run on one of the 
Oozie servers (synchronous actions) that cannot be run directly on a YARN 
NodeManager container in any case. [~gezapeti] what are your two cents?

> Workflow Application Master in YARN
> ---
>
> Key: OOZIE-1178
> URL: https://issues.apache.org/jira/browse/OOZIE-1178
> Project: Oozie
>  Issue Type: New Feature
>Reporter: Bo Wang
>Priority: Major
> Attachments: MAPREDUCE-4495-v1.1.patch, MAPREDUCE-4495-v1.patch, 
> MapReduceWorkflowAM.pdf, yapp_proposal.txt
>
>
> It is useful to have a workflow application master, which will be capable of 
> running a DAG of jobs. The workflow client submits a DAG request to the AM 
> and then the AM will manage the life cycle of this application in terms of 
> requesting the needed resources from the RM, and starting, monitoring and 
> retrying the application's individual tasks.
> Compared to running Oozie with the current MapReduce Application Master, 
> these are some of the advantages:
>  - Less number of consumed resources, since only one application master will 
> be spawned for the whole workflow.
>  - Reuse of resources, since the same resources can be used by multiple 
> consecutive jobs in the workflow (no need to request/wait for resources for 
> every individual job from the central RM).
>  - More optimization opportunities in terms of collective resource requests.
>  - Optimization opportunities in terms of rewriting and composing jobs in the 
> workflow (e.g. pushing down Mappers).
>  - This Application Master can be reused/extended by higher systems like Pig 
> and hive to provide an optimized way of running their workflows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3218) Oozie Sqoop action with command splits the select clause into multiple parts due to delimiter being space

2018-05-25 Thread Andras Piros (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490699#comment-16490699
 ] 

Andras Piros commented on OOZIE-3218:
-

[~dionusos] reviewed. Till the glorious moment :) when we'll begin to use 
GitHub pull requests both for reviews and pre-commit hook, can you please 
upload both to ReviewBoard (review) and JIRA issue (pre-commit hook)? Thanks!

> Oozie Sqoop action with command splits the select clause into multiple parts 
> due to delimiter being space
> -
>
> Key: OOZIE-3218
> URL: https://issues.apache.org/jira/browse/OOZIE-3218
> Project: Oozie
>  Issue Type: Bug
>  Components: action, workflow
>Affects Versions: 3.3.2, 4.1.0, 4.2.0, 4.3.0, 5.0.0
> Environment: Hortonworks Hadoop HDP-2.6.4.x release 
>  oozie admin -version: Oozie server build version: 4.2.0.2.6.4.0-91
>Reporter: Mahesh Balakrishnan
>Assignee: Mahesh Balakrishnan
>Priority: Major
> Attachments: OOZIE-3218-2.patch, OOZIE-3218.patch
>
>
> When running a Oozie Sqoop action which has command with {{--query}} in place 
> the query is split into multiple parts causing {{"Unrecognized argument:"}} 
> and in-turn fails.
> {code:xml}
> 
> ${resourceManager}
> ${nameNode}
> import --verbose --connect jdbc:mysql://test.openstacklocal/db 
> --query select * from abc where $CONDITIONS --username test --password test 
> --driver com.mysql.jdbc.Driver -m 1 
> 
> 
> {code}
>  
> Oozie Launcher logs:
> {noformat}
>  Sqoop command arguments :
>  import
>  --verbose
>  --connect
>  jdbc:mysql://test.openstacklocal/db
>  --query
>  "select
>  *
>  from
>  abc
>  where
>  $CONDITIONS"
>  --username
>  hive
>  --password
>  
>  --driver
>  com.mysql.jdbc.Driver
>  -m
>  1
>  Fetching child yarn jobs
>  tag id : oozie-a1bbe03a0983b9e822d12ae7bb269ee3
>  2791 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to 
> ResourceManager at hdp263-3.openstacklocal/172.26.105.248:8050
>  Child yarn jobs are found - 
>  =
> >>> Invoking Sqoop command line now >>>
> 3172 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not 
> been set in the environment. Cannot check for additional configuration.
>  3172 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not 
> been set in the environment. Cannot check for additional configuration.
>  3218 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 
> 1.4.6.2.6.4.0-91
>  3218 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 
> 1.4.6.2.6.4.0-91
>  3287 [main] DEBUG org.apache.sqoop.tool.BaseSqoopTool - Enabled debug 
> logging.
>  3287 [main] DEBUG org.apache.sqoop.tool.BaseSqoopTool - Enabled debug 
> logging.
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing 
> arguments for import:
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing 
> arguments for import:
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: *
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: *
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: from
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: from
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: where
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: where
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: $CONDITIONS"
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: $CONDITIONS"
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --username
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --username
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --password
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --password
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --driver
>  3288 [main] ERROR

Re: Review Request 67314: Oozie Sqoop action with command splits the select clause into multiple parts due to delimiter being space

2018-05-25 Thread András Piros via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67314/#review203872
---




core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java
Lines 97 (patched)


I'd at most log at `TRACE` level here. But I'd rather leave that and log at 
`DEBUG` level the resulting `argList` that's gonna be forwarded to 
`setSqoopCommand()`.



core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java
Lines 100 (patched)


I'd at most log at `TRACE` level here. But I'd rather leave that and log at 
`DEBUG` level the resulting `argList` that's gonna be forwarded to 
`setSqoopCommand()`.



core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java
Lines 105 (patched)


Don't we want to replace all `\n\n`s?



core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java
Lines 108 (patched)


`arrayLength` isn't a nice name for a variable that's incremented in a loop.



sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopActionExecutor.java
Line 46 (original), 46 (patched)


Please cover old functionality that uses `"\n"` as word separator inside 
the query, and new functionality that user `" "` as word separator inside the 
query, as well.

Please also test a mixed case that uses both separators inside the same 
query.



sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopActionExecutor.java
Lines 50 (patched)


`SQOOP_QUERY_COMMAND`


- András Piros


On May 25, 2018, 7:37 a.m., Denes Bodo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67314/
> ---
> 
> (Updated May 25, 2018, 7:37 a.m.)
> 
> 
> Review request for oozie and András Piros.
> 
> 
> Bugs: OOZIE-3218
> https://issues.apache.org/jira/browse/OOZIE-3218
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> When running a Oozie Sqoop action which has command with --query in place the 
> query is split into multiple parts causing "Unrecognized argument:" and 
> in-turn fails.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 69d5e7e 
>   
> sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopActionExecutor.java
>  edfe0c7 
> 
> 
> Diff: https://reviews.apache.org/r/67314/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Denes Bodo
> 
>

[jira] [Commented] (OOZIE-1178) Workflow Application Master in YARN

2018-05-25 Thread Artem Ervits (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490688#comment-16490688
 ] 

Artem Ervits commented on OOZIE-1178:
-

[~andras.piros] with OYA this can be closed, no?

> Workflow Application Master in YARN
> ---
>
> Key: OOZIE-1178
> URL: https://issues.apache.org/jira/browse/OOZIE-1178
> Project: Oozie
>  Issue Type: New Feature
>Reporter: Bo Wang
>Priority: Major
> Attachments: MAPREDUCE-4495-v1.1.patch, MAPREDUCE-4495-v1.patch, 
> MapReduceWorkflowAM.pdf, yapp_proposal.txt
>
>
> It is useful to have a workflow application master, which will be capable of 
> running a DAG of jobs. The workflow client submits a DAG request to the AM 
> and then the AM will manage the life cycle of this application in terms of 
> requesting the needed resources from the RM, and starting, monitoring and 
> retrying the application's individual tasks.
> Compared to running Oozie with the current MapReduce Application Master, 
> these are some of the advantages:
>  - Less number of consumed resources, since only one application master will 
> be spawned for the whole workflow.
>  - Reuse of resources, since the same resources can be used by multiple 
> consecutive jobs in the workflow (no need to request/wait for resources for 
> every individual job from the central RM).
>  - More optimization opportunities in terms of collective resource requests.
>  - Optimization opportunities in terms of rewriting and composing jobs in the 
> workflow (e.g. pushing down Mappers).
>  - This Application Master can be reused/extended by higher systems like Pig 
> and hive to provide an optimized way of running their workflows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (OOZIE-2545) Output data exceeds its limit [2048]

2018-05-25 Thread Artem Ervits (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits resolved OOZIE-2545.
-
Resolution: Not A Problem

[~jingtao] please see response from [~satishsaley] to address the issue.

> Output data exceeds its limit [2048]
> 
>
> Key: OOZIE-2545
> URL: https://issues.apache.org/jira/browse/OOZIE-2545
> Project: Oozie
>  Issue Type: Bug
>Reporter: jingtao
>Priority: Major
>
> I am trying to run a simple oozie workflow executing a hive script. 
> This hive script just calls joining(tables is very large);
> Once the hive script execution ends I was expecting to see the workflow 
> status changing from RUNNING to SUCCESFUL, but this is not happening. 
> This is the content of the workflow log:
> 2016-05-31 15:52:34,591 WARN 
> org.apache.oozie.action.hadoop.HiveActionExecutor: SERVER[hadoop02] 
> USER[scapp] GROUP[-] TOKEN[] APP[wf-sqoop-hive-agreement] 
> JOB[001-160531143657136-oozie-oozi-W] 
> ACTION[001-160531143657136-oozie-oozi-W@hive-query-agreement] Launcher 
> exception: Output data exceeds its limit [2048]
> org.apache.oozie.action.hadoop.LauncherException: Output data exceeds its 
> limit [2048]
>   at 
> org.apache.oozie.action.hadoop.LauncherMapper.getLocalFileContentStr(LauncherMapper.java:415)
>   at 
> org.apache.oozie.action.hadoop.LauncherMapper.handleActionData(LauncherMapper.java:391)
>   at 
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:275)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
>   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3257) Testcase TestHiveActionExecutor#testHiveAction still fails after OOZIE-3236

2018-05-25 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490655#comment-16490655
 ] 

Hadoop QA commented on OOZIE-3257:
--


Testing JIRA OOZIE-3257

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
error(s)
.{color:red}ERROR{color}: the current HEAD has 2 Javadoc error(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2134
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/579/



> Testcase TestHiveActionExecutor#testHiveAction still fails after OOZIE-3236
> ---
>
> Key: OOZIE-3257
> URL: https://issues.apache.org/jira/browse/OOZIE-3257
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: OOZIE-3257-001.patch
>
>
> OOZIE-3236 did not fully fix {{testHiveAction}}. The reason is that two Hive 
> action is started from this test and the second one still uses the default 
> timeout of 60 seconds.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Success: OOZIE-3257 PreCommit Build #579

2018-05-25 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-3257
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/579/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.71 MB...]
[DEBUG] There are no new bugs found in [sharelib/pig].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/streaming].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hive].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hcatalog].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/sqoop].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/oozie].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/distcp].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/spark].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [client].
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2365 bytes
[TRACE] Full summary file size is 1314 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO


Testing JIRA OOZIE-3257

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 1 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warning(s)
+1 the patch does not seem to introduce new Javadoc error(s)
ERROR: the current HEAD has 2 Javadoc error(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [webapp].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [client].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2134
+1 DISTRO
+1 distro tarball builds with the patch 


+1 Overall result, good!, no -1s


The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/579/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100  30020 0  100  3002  0   3372 --:--:-- --:--:-- --:--:--  
3373{"self":"https://issues.apache.org/jira/rest/api/2/issue/13162048/comment/16490655","id":"16490655","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","emailAddress":"blackhole
 at hadoop dot apache dot

[jira] [Reopened] (OOZIE-3228) [Spark action] Can't load properties from spark-defaults.conf

2018-05-25 Thread Andras Piros (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros reopened OOZIE-3228:
-

Reopening as the test 
{{TestSparkArgsExtractor.testPropertiesArePrependedToSparkArgs}} fails 
sometimes:
{noformat}
java.lang.AssertionError: Spark args mismatch expected:<[--master, yarn, 
--deploy-mode, client, --name, Spark Copy File, --class, 
org.apache.oozie.example.SparkFileCopy, --conf, 
spark.executor.extraClassPath=/etc/hbase/conf:/etc/hive/conf:$PWD/*, --conf, 
spark.driver.extraClassPath=/etc/hbase/conf:/etc/hive/conf:$PWD/*, --conf, 
spark.yarn.security.tokens.hadoopfs.enabled=false, --conf, 
spark.yarn.security.tokens.hive.enabled=false, --conf, 
spark.yarn.security.tokens.hbase.enabled=false, --conf, 
spark.yarn.security.credentials.hadoopfs.enabled=false, --conf, 
spark.yarn.security.credentials.hive.enabled=false, --conf, 
spark.yarn.security.credentials.hbase.enabled=false, --conf, 
spark.executor.extraJavaOptions=-XX:+UseG1GC -XX:+PrintGC 
-XX:+UnlockExperimentalVMOptions -Dlog4j.configuration=spark-log4j.properties, 
--conf, spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:+PrintGC 
-XX:+UnlockExperimentalVMOptions -Dlog4j.configuration=spark-log4j.properties, 
--properties-file, spark-defaults-oozie-generated.properties, --files, 
spark-log4j.properties,hive-site.xml, --conf, spark.yarn.jar=null, --verbose, 
/lib/test.jar]> but was:<[--master, yarn, --deploy-mode, client, --name, Spark 
Copy File, --class, org.apache.oozie.example.SparkFileCopy, --conf, 
spark.executor.extraClassPath=/etc/hbase/conf:/etc/hive/conf:$PWD/*, --conf, 
spark.driver.extraClassPath=/etc/hbase/conf:/etc/hive/conf:$PWD/*, --conf, 
spark.yarn.security.tokens.hive.enabled=false, --conf, 
spark.yarn.security.tokens.hbase.enabled=false, --conf, 
spark.executor.extraJavaOptions=-XX:+UseG1GC -XX:+PrintGC 
-XX:+UnlockExperimentalVMOptions -Dlog4j.configuration=spark-log4j.properties, 
--conf, spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:+PrintGC 
-XX:+UnlockExperimentalVMOptions -Dlog4j.configuration=spark-log4j.properties, 
--properties-file, spark-defaults-oozie-generated.properties, --files, 
spark-log4j.properties, --conf, spark.yarn.jar=null, --verbose, /lib/test.jar]>
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at 
org.apache.oozie.action.hadoop.TestSparkArgsExtractor.testPropertiesArePrependedToSparkArgs(TestSparkArgsExtractor.java:304)
{noformat}

> [Spark action] Can't load properties from spark-defaults.conf
> -
>
> Key: OOZIE-3228
> URL: https://issues.apache.org/jira/browse/OOZIE-3228
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 5.0.0, 4.3.1
>Reporter: Tang Yan
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3228.001.patch, OOZIE-3228.002.patch, 
> OOZIE-3228.003.patch, OOZIE-3228.004.patch, OOZIE-3228.005.patch
>
>
> When I create a Oozie workflow to launch a spark action, the spark job can't 
> load the configured properties in {{spark-defaults.conf}}. I've configured 
> each NodeManager as the Spark gateway role, so {{spark-defaults.conf}} is 
> generated in {{/etc/spark/conf/}} on each worker node.
> In {{spark-defaults.conf}} some configuration I've set into.
> {noformat}
> spark.executor.extraClassPath=/etc/hbase/conf:/etc/hive/conf
> spark.driver.extraClassPath=/etc/hbase/conf:/etc/hive/conf
> {noformat}
> But in the Oozie spark job, they're not loaded automatically.
> {noformat}
> --conf spark.executor.extraClassPath=$PWD/*
> --conf spark.driver.extraClassPath=$PWD/*
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3258) Surefire should not trim stack traces

2018-05-25 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490626#comment-16490626
 ] 

Hadoop QA commented on OOZIE-3258:
--

PreCommit-OOZIE-Build started


> Surefire should not trim stack traces
> -
>
> Key: OOZIE-3258
> URL: https://issues.apache.org/jira/browse/OOZIE-3258
> Project: Oozie
>  Issue Type: Bug
>  Components: core, tests
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: OOZIE-3258-001.patch
>
>
> The current setting of surefire trims stack traces if it reaches JUnit. This 
> makes debugging harder, because it's not obvious what code path is affected:
> {noformat}
> java.io.EOFException: End of File Exception between local host is: 
> "asf918.gq1.ygridcore.net/67.195.81.138"; destination host is: 
> "localhost":35969; : java.io.EOFException; For more details see:  
> http://wiki.apache.org/hadoop/EOFException
>   at 
> org.apache.oozie.util.TestZKUtilsWithSecurity.setUp(TestZKUtilsWithSecurity.java:36)
> Caused by: java.io.EOFException
> {noformat}
> We have to see the full trace in order to address problems efficiently.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3259) Flaky test TestZKUtilsWithSecurity#testNewUsingACLs

2018-05-25 Thread Peter Bacsko (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490625#comment-16490625
 ] 

Peter Bacsko commented on OOZIE-3259:
-

Todo: add full stack trace after OOZIE-3258 is committed.

> Flaky test TestZKUtilsWithSecurity#testNewUsingACLs
> ---
>
> Key: OOZIE-3259
> URL: https://issues.apache.org/jira/browse/OOZIE-3259
> Project: Oozie
>  Issue Type: Sub-task
>  Components: security, tests
>Reporter: Peter Bacsko
>Priority: Major
>
> The test case TestZKUtilsWithSecurity#testNewUsingACLs failed with the 
> following error:
> {noformat}
> java.io.EOFException: End of File Exception between local host is: 
> "asf918.gq1.ygridcore.net/67.195.81.138"; destination host is: 
> "localhost":35969; : java.io.EOFException; For more details see:  
> http://wiki.apache.org/hadoop/EOFException
>   at 
> org.apache.oozie.util.TestZKUtilsWithSecurity.setUp(TestZKUtilsWithSecurity.java:36)
> Caused by: java.io.EOFException
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OOZIE-3259) Flaky test TestZKUtilsWithSecurity#testNewUsingACLs

2018-05-25 Thread Peter Bacsko (JIRA) 
Peter Bacsko created OOZIE-3259:
---

 Summary: Flaky test TestZKUtilsWithSecurity#testNewUsingACLs
 Key: OOZIE-3259
 URL: https://issues.apache.org/jira/browse/OOZIE-3259
 Project: Oozie
  Issue Type: Sub-task
  Components: security, tests
Reporter: Peter Bacsko


The test case TestZKUtilsWithSecurity#testNewUsingACLs failed with the 
following error:

{noformat}
java.io.EOFException: End of File Exception between local host is: 
"asf918.gq1.ygridcore.net/67.195.81.138"; destination host is: 
"localhost":35969; : java.io.EOFException; For more details see:  
http://wiki.apache.org/hadoop/EOFException
at 
org.apache.oozie.util.TestZKUtilsWithSecurity.setUp(TestZKUtilsWithSecurity.java:36)
Caused by: java.io.EOFException
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3258) Surefire should not trim stack traces

2018-05-25 Thread Peter Cseh (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490621#comment-16490621
 ] 

Peter Cseh commented on OOZIE-3258:
---

This will make debugging easier for sure.
+1 (pending jenkins)

> Surefire should not trim stack traces
> -
>
> Key: OOZIE-3258
> URL: https://issues.apache.org/jira/browse/OOZIE-3258
> Project: Oozie
>  Issue Type: Bug
>  Components: core, tests
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: OOZIE-3258-001.patch
>
>
> The current setting of surefire trims stack traces if it reaches JUnit. This 
> makes debugging harder, because it's not obvious what code path is affected:
> {noformat}
> java.io.EOFException: End of File Exception between local host is: 
> "asf918.gq1.ygridcore.net/67.195.81.138"; destination host is: 
> "localhost":35969; : java.io.EOFException; For more details see:  
> http://wiki.apache.org/hadoop/EOFException
>   at 
> org.apache.oozie.util.TestZKUtilsWithSecurity.setUp(TestZKUtilsWithSecurity.java:36)
> Caused by: java.io.EOFException
> {noformat}
> We have to see the full trace in order to address problems efficiently.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3258) Surefire should not trim stack traces

2018-05-25 Thread Peter Bacsko (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490617#comment-16490617
 ] 

Peter Bacsko commented on OOZIE-3258:
-

[~gezapeti] please check.

> Surefire should not trim stack traces
> -
>
> Key: OOZIE-3258
> URL: https://issues.apache.org/jira/browse/OOZIE-3258
> Project: Oozie
>  Issue Type: Bug
>  Components: core, tests
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: OOZIE-3258-001.patch
>
>
> The current setting of surefire trims stack traces if it reaches JUnit. This 
> makes debugging harder, because it's not obvious what code path is affected:
> {noformat}
> java.io.EOFException: End of File Exception between local host is: 
> "asf918.gq1.ygridcore.net/67.195.81.138"; destination host is: 
> "localhost":35969; : java.io.EOFException; For more details see:  
> http://wiki.apache.org/hadoop/EOFException
>   at 
> org.apache.oozie.util.TestZKUtilsWithSecurity.setUp(TestZKUtilsWithSecurity.java:36)
> Caused by: java.io.EOFException
> {noformat}
> We have to see the full trace in order to address problems efficiently.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3258) Surefire should not trim stack traces

2018-05-25 Thread Peter Bacsko (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-3258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Bacsko updated OOZIE-3258:

Attachment: OOZIE-3258-001.patch

> Surefire should not trim stack traces
> -
>
> Key: OOZIE-3258
> URL: https://issues.apache.org/jira/browse/OOZIE-3258
> Project: Oozie
>  Issue Type: Bug
>  Components: core, tests
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: OOZIE-3258-001.patch
>
>
> The current setting of surefire trims stack traces if it reaches JUnit. This 
> makes debugging harder, because it's not obvious what code path is affected:
> {noformat}
> java.io.EOFException: End of File Exception between local host is: 
> "asf918.gq1.ygridcore.net/67.195.81.138"; destination host is: 
> "localhost":35969; : java.io.EOFException; For more details see:  
> http://wiki.apache.org/hadoop/EOFException
>   at 
> org.apache.oozie.util.TestZKUtilsWithSecurity.setUp(TestZKUtilsWithSecurity.java:36)
> Caused by: java.io.EOFException
> {noformat}
> We have to see the full trace in order to address problems efficiently.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-2352) Unportable shebang in shell scripts

2018-05-25 Thread Andras Piros (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2352:

Description: 
Some shell scripts in oozie distribution use unportable shebangs:
{noformat}
#!/bin/bash
{noformat}

bash shell interpreter may be installed in different locations (e.g 
/usr/local/bin/bash on FreeBSD).

The proposed solution is to use more portable
{noformat}
#!/usr/bin/env bash
{noformat}
instead.

Affected files:
{noformat}
bin/mkdistro.sh
client/src/main/conf/oozie-client-env.sh
client/target/oozie-client-5.0.0-client/oozie-client-5.0.0/conf/oozie-client-env.sh
core/src/main/conf/oozie-env.sh
core/src/main/resources/ssh-wrapper.sh
distro/src/main/bin/addtowar.sh
distro/src/main/bin/oozie-jetty-server.sh
distro/src/main/bin/oozie-run.sh
distro/src/main/bin/oozie-setup.sh
distro/src/main/bin/oozie-start.sh
distro/src/main/bin/oozie-stop.sh
distro/src/main/bin/oozie-sys.sh
distro/src/main/bin/oozied.sh
tools/src/main/bin/oozie-diag-bundle-collector.sh
tools/src/main/bin/ooziedb.sh
{noformat}

  was:
Some shell scripts in oozie distribution use unportable shebangs:
#!/bin/bash

bash shell interpreter may be installed in different locations (e.g 
/usr/local/bin/bash on FreeBSD).

The proposed solution is to use more portable
#!/usr/bin/env bash
instead.

Affected files:
client/src/main/bin/oozie
distro/src/main/bin/addtowar.sh
distro/src/main/bin/oozied.sh
distro/src/main/bin/oozie-setup.sh
distro/src/main/bin/oozie-sys.sh
tools/src/main/bin/ooziedb.sh
core/src/main/conf/oozie-env.sh


> Unportable shebang in shell scripts
> ---
>
> Key: OOZIE-2352
> URL: https://issues.apache.org/jira/browse/OOZIE-2352
> Project: Oozie
>  Issue Type: Bug
>  Components: scripts
>Affects Versions: 4.2.0
>Reporter: Dmitry Sivachenko
>Assignee: Artem Ervits
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2352-00.patch
>
>
> Some shell scripts in oozie distribution use unportable shebangs:
> {noformat}
> #!/bin/bash
> {noformat}
> bash shell interpreter may be installed in different locations (e.g 
> /usr/local/bin/bash on FreeBSD).
> The proposed solution is to use more portable
> {noformat}
> #!/usr/bin/env bash
> {noformat}
> instead.
> Affected files:
> {noformat}
> bin/mkdistro.sh
> client/src/main/conf/oozie-client-env.sh
> client/target/oozie-client-5.0.0-client/oozie-client-5.0.0/conf/oozie-client-env.sh
> core/src/main/conf/oozie-env.sh
> core/src/main/resources/ssh-wrapper.sh
> distro/src/main/bin/addtowar.sh
> distro/src/main/bin/oozie-jetty-server.sh
> distro/src/main/bin/oozie-run.sh
> distro/src/main/bin/oozie-setup.sh
> distro/src/main/bin/oozie-start.sh
> distro/src/main/bin/oozie-stop.sh
> distro/src/main/bin/oozie-sys.sh
> distro/src/main/bin/oozied.sh
> tools/src/main/bin/oozie-diag-bundle-collector.sh
> tools/src/main/bin/ooziedb.sh
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-2352) Unportable shebang in shell scripts

2018-05-25 Thread Andras Piros (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2352:

Component/s: scripts

> Unportable shebang in shell scripts
> ---
>
> Key: OOZIE-2352
> URL: https://issues.apache.org/jira/browse/OOZIE-2352
> Project: Oozie
>  Issue Type: Bug
>  Components: scripts
>Affects Versions: 4.2.0
>Reporter: Dmitry Sivachenko
>Assignee: Artem Ervits
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2352-00.patch
>
>
> Some shell scripts in oozie distribution use unportable shebangs:
> #!/bin/bash
> bash shell interpreter may be installed in different locations (e.g 
> /usr/local/bin/bash on FreeBSD).
> The proposed solution is to use more portable
> #!/usr/bin/env bash
> instead.
> Affected files:
> client/src/main/bin/oozie
> distro/src/main/bin/addtowar.sh
> distro/src/main/bin/oozied.sh
> distro/src/main/bin/oozie-setup.sh
> distro/src/main/bin/oozie-sys.sh
> tools/src/main/bin/ooziedb.sh
> core/src/main/conf/oozie-env.sh



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2211) Remove OozieCLI#validateCommandV41

2018-05-25 Thread Andras Piros (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490595#comment-16490595
 ] 

Andras Piros commented on OOZIE-2211:
-

[~dbist13] having looked at a wrong patch. +1 for patch 002.

> Remove OozieCLI#validateCommandV41
> --
>
> Key: OOZIE-2211
> URL: https://issues.apache.org/jira/browse/OOZIE-2211
> Project: Oozie
>  Issue Type: Task
>Reporter: Azrael Seoeun
>Assignee: Artem Ervits
>Priority: Trivial
> Fix For: 5.1.0
>
> Attachments: OOZIE-2211-00.patch, OOZIE-2211.01.patch, 
> OOZIE-2211.02.patch
>
>
> XML validation API is move to server side (OOZIE-2159)
> OozieCLI#validateCommandV41 is for the backward compatibility.
> TestValidation is for the xml validation on client side.
> Those should be removed on next version. (might be 4.2.0 higher version)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3257) Testcase TestHiveActionExecutor#testHiveAction still fails after OOZIE-3236

2018-05-25 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490571#comment-16490571
 ] 

Hadoop QA commented on OOZIE-3257:
--

PreCommit-OOZIE-Build started


> Testcase TestHiveActionExecutor#testHiveAction still fails after OOZIE-3236
> ---
>
> Key: OOZIE-3257
> URL: https://issues.apache.org/jira/browse/OOZIE-3257
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: OOZIE-3257-001.patch
>
>
> OOZIE-3236 did not fully fix {{testHiveAction}}. The reason is that two Hive 
> action is started from this test and the second one still uses the default 
> timeout of 60 seconds.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OOZIE-3257) Testcase TestHiveActionExecutor#testHiveAction still fails after OOZIE-3236

2018-05-25 Thread Peter Bacsko (JIRA) 
Peter Bacsko created OOZIE-3257:
---

 Summary: Testcase TestHiveActionExecutor#testHiveAction still 
fails after OOZIE-3236
 Key: OOZIE-3257
 URL: https://issues.apache.org/jira/browse/OOZIE-3257
 Project: Oozie
  Issue Type: Sub-task
Reporter: Peter Bacsko
Assignee: Peter Bacsko


OOZIE-3236 did not fully fix {{testHiveAction}}. The reason is that two Hive 
action is started from this test and the second one still uses the default 
timeout of 60 seconds.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3160) PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting

2018-05-25 Thread Andras Piros (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490535#comment-16490535
 ] 

Andras Piros commented on OOZIE-3160:
-

Thanks for your work on this [~pbacsko]! {{PriorityDelayQueue}} plays an 
important part of Oozie. So a bit more thinking while working on the design doc 
is a good approach.

> PriorityDelayQueue put()/take() can cause significant CPU load due to busy 
> waiting
> --
>
> Key: OOZIE-3160
> URL: https://issues.apache.org/jira/browse/OOZIE-3160
> Project: Oozie
>  Issue Type: Bug
>  Components: core
> Environment: all platforms
>Reporter: jj
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: 11.png, 22.png, 
> OOZIE-3160-POC01.patch
>
>
> oozie process always  consume  high cpu. in my mechine,around 10%. 
> I check the source code，find take() method in PriorityDelayQueue class。
> code:
> {code:java}
> public QueueElement take() throws InterruptedException {
> QueueElement e = poll();
> while (e == null) {
> Thread.sleep(10);
> e = poll();
> }
> return e;
> }
> {code}
> i think it's the reason of this problem. it's keep while, not await.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3160) PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting

2018-05-25 Thread Peter Bacsko (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490526#comment-16490526
 ] 

Peter Bacsko commented on OOZIE-3160:
-

Just talked to the guys here, they suggested that we create a design document 
about this. The change required here is non-trivial and it's a critical part in 
Oozie, so if we mess it up, huge problems can occur. So let's be a bit 
conservative about this ticket.

> PriorityDelayQueue put()/take() can cause significant CPU load due to busy 
> waiting
> --
>
> Key: OOZIE-3160
> URL: https://issues.apache.org/jira/browse/OOZIE-3160
> Project: Oozie
>  Issue Type: Bug
>  Components: core
> Environment: all platforms
>Reporter: jj
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: 11.png, 22.png, 
> OOZIE-3160-POC01.patch
>
>
> oozie process always  consume  high cpu. in my mechine,around 10%. 
> I check the source code，find take() method in PriorityDelayQueue class。
> code:
> {code:java}
> public QueueElement take() throws InterruptedException {
> QueueElement e = poll();
> while (e == null) {
> Thread.sleep(10);
> e = poll();
> }
> return e;
> }
> {code}
> i think it's the reason of this problem. it's keep while, not await.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3218) Oozie Sqoop action with command splits the select clause into multiple parts due to delimiter being space

2018-05-25 Thread Denes Bodo (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490381#comment-16490381
 ] 

Denes Bodo commented on OOZIE-3218:
---

[~mbalakrishnan], [~andras.piros] Patch002 is on review board 
https://reviews.apache.org/r/67314/ and has been modified to cleanly apply on 
master branch.

> Oozie Sqoop action with command splits the select clause into multiple parts 
> due to delimiter being space
> -
>
> Key: OOZIE-3218
> URL: https://issues.apache.org/jira/browse/OOZIE-3218
> Project: Oozie
>  Issue Type: Bug
>  Components: action, workflow
>Affects Versions: 3.3.2, 4.1.0, 4.2.0, 4.3.0, 5.0.0
> Environment: Hortonworks Hadoop HDP-2.6.4.x release 
>  oozie admin -version: Oozie server build version: 4.2.0.2.6.4.0-91
>Reporter: Mahesh Balakrishnan
>Assignee: Mahesh Balakrishnan
>Priority: Major
> Attachments: OOZIE-3218-2.patch, OOZIE-3218.patch
>
>
> When running a Oozie Sqoop action which has command with {{--query}} in place 
> the query is split into multiple parts causing {{"Unrecognized argument:"}} 
> and in-turn fails.
> {code:xml}
> 
> ${resourceManager}
> ${nameNode}
> import --verbose --connect jdbc:mysql://test.openstacklocal/db 
> --query select * from abc where $CONDITIONS --username test --password test 
> --driver com.mysql.jdbc.Driver -m 1 
> 
> 
> {code}
>  
> Oozie Launcher logs:
> {noformat}
>  Sqoop command arguments :
>  import
>  --verbose
>  --connect
>  jdbc:mysql://test.openstacklocal/db
>  --query
>  "select
>  *
>  from
>  abc
>  where
>  $CONDITIONS"
>  --username
>  hive
>  --password
>  
>  --driver
>  com.mysql.jdbc.Driver
>  -m
>  1
>  Fetching child yarn jobs
>  tag id : oozie-a1bbe03a0983b9e822d12ae7bb269ee3
>  2791 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to 
> ResourceManager at hdp263-3.openstacklocal/172.26.105.248:8050
>  Child yarn jobs are found - 
>  =
> >>> Invoking Sqoop command line now >>>
> 3172 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not 
> been set in the environment. Cannot check for additional configuration.
>  3172 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not 
> been set in the environment. Cannot check for additional configuration.
>  3218 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 
> 1.4.6.2.6.4.0-91
>  3218 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 
> 1.4.6.2.6.4.0-91
>  3287 [main] DEBUG org.apache.sqoop.tool.BaseSqoopTool - Enabled debug 
> logging.
>  3287 [main] DEBUG org.apache.sqoop.tool.BaseSqoopTool - Enabled debug 
> logging.
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing 
> arguments for import:
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing 
> arguments for import:
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: *
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: *
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: from
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: from
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: where
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: where
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: $CONDITIONS"
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: $CONDITIONS"
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --username
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --username
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --password
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --password
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --driver
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --driver
>  3288 [main]

Review Request 67314: Oozie Sqoop action with command splits the select clause into multiple parts due to delimiter being space

2018-05-25 Thread Denes Bodo 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67314/
---

Review request for oozie and András Piros.


Bugs: OOZIE-3218
https://issues.apache.org/jira/browse/OOZIE-3218


Repository: oozie-git


Description
---

When running a Oozie Sqoop action which has command with --query in place the 
query is split into multiple parts causing "Unrecognized argument:" and in-turn 
fails.


Diffs
-

  core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
69d5e7e 
  
sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopActionExecutor.java
 edfe0c7 


Diff: https://reviews.apache.org/r/67314/diff/1/


Testing
---


Thanks,

Denes Bodo

[jira] Subscription: Oozie Patch Available

2018-05-25 Thread jira 
Issue Subscription
Filter: Oozie Patch Available (99 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3256  refactor OozieCLI class
https://issues.apache.org/jira/browse/OOZIE-3256
OOZIE-3249  [tools] Instrumentation log parser
https://issues.apache.org/jira/browse/OOZIE-3249
OOZIE-3218  Oozie Sqoop action with command splits the select clause into 
multiple parts due to delimiter being space
https://issues.apache.org/jira/browse/OOZIE-3218
OOZIE-3199  Let system property restriction configurable
https://issues.apache.org/jira/browse/OOZIE-3199
OOZIE-3196  Authorization: restrict world readability by user
https://issues.apache.org/jira/browse/OOZIE-3196
OOZIE-3194  Oozie should set proper permissions to sharelib after upload
https://issues.apache.org/jira/browse/OOZIE-3194
OOZIE-3193  Applications are not killed when submitted via subworkflow
https://issues.apache.org/jira/browse/OOZIE-3193
OOZIE-3186  Oozie is unable to use configuration linked using jceks://file/...
https://issues.apache.org/jira/browse/OOZIE-3186
OOZIE-3179  Adding a configurable config-default.xml location to a workflow
https://issues.apache.org/jira/browse/OOZIE-3179
OOZIE-3170  Oozie Diagnostic Bundle tool fails with NPE due to missing service 
class
https://issues.apache.org/jira/browse/OOZIE-3170
OOZIE-3160  PriorityDelayQueue put()/take() can cause significant CPU load due 
to busy waiting
https://issues.apache.org/jira/browse/OOZIE-3160
OOZIE-3135  Configure log4j2 in SqoopMain
https://issues.apache.org/jira/browse/OOZIE-3135
OOZIE-3109  Escape log-streaming's HTML-specific characters
https://issues.apache.org/jira/browse/OOZIE-3109
OOZIE-3091  Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: 
org/apache/avro/mapred/AvroWrapper"
https://issues.apache.org/jira/browse/OOZIE-3091
OOZIE-3071  Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 
than Spark 2.2.0
https://issues.apache.org/jira/browse/OOZIE-3071
OOZIE-3063  Sanitizing variables that are part of openjpa.ConnectionProperties
https://issues.apache.org/jira/browse/OOZIE-3063
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues.apache.org/jira/browse/OOZIE-3062
OOZIE-3061  Kill only those child jobs which are not already killed
https://issues.apache.org/jira/browse/OOZIE-3061
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  [examples] Fix Findbugs warnings
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues.apache.org/jira/browse/OOZIE-2829
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2795  Create lib directory or symlink for Oozie CLI during packaging
https://issues.apache.org/jira/browse/OOZIE-2795
OOZIE-2791  ShareLib installation may fail on busy Hadoop clusters
https://issues.apache.org/jira/browse/OOZIE-2791
OOZIE-2784  Include WEEK as a parameter in the Coordinator Expression Language 
Evaulator
https://issues.apache.org/jira/browse/OOZIE-2784
OOZIE-2779  Mask Hive2 action Beeline JDBC password
https://issues.apache.org/jira/browse/OOZIE-2779
OOZIE-2736  Reduce the number of threads during test execution
https://issues.apache.org/jira/browse/OOZIE-2736
OOZIE-2718  Improve -dryrun for bundles
https://issues.apache.org/jira/browse/OOZIE-2718