date:20180810

[jira] [Commented] (OOZIE-3326) Sqoop Action should support tez delegation tokens for hive-import

2018-08-10 Thread Brian Goerlitz (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576909#comment-16576909
 ] 

Brian Goerlitz commented on OOZIE-3326:
---

[~dionusos] - Please assign this to me for patch contribution. Thanks.

> Sqoop Action should support tez delegation tokens for hive-import
> -
>
> Key: OOZIE-3326
> URL: https://issues.apache.org/jira/browse/OOZIE-3326
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Brian Goerlitz
>Priority: Major
>
> SqoopMain needs to support tez delegation tokens for hive-imports. 
> Implementation is similar to that of HiveMain and Hive2Main.
> At present, hive-import will fail to start a tez session in secure 
> environment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3160) PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting

2018-08-10 Thread Andras Piros (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576376#comment-16576376
 ] 

Andras Piros commented on OOZIE-3160:
-

Thanks for the contribution [~pbacsko]! I like that the code is pretty much 
aligned with the specification.

I think most functionality is covered and error-free, the code is quite 
readable - I wouldn't call it a proof of concept anymore ;) I only made a 
couple of comments on ReviewBoard. Nice job!

> PriorityDelayQueue put()/take() can cause significant CPU load due to busy 
> waiting
> --
>
> Key: OOZIE-3160
> URL: https://issues.apache.org/jira/browse/OOZIE-3160
> Project: Oozie
>  Issue Type: Bug
>  Components: core
> Environment: all platforms
>Reporter: jj
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: 11.png, 22.png, 
> OOZIE-3160-POC01.patch, OOZIE-3160-POC02.patch, OOZIE-3160-POC02.patch, 
> OOZIE-3160-POC03.patch, OOZIE-3160-POC04.patch, OOZIE-3160-POC05.patch, 
> PriorityDelayQueue improvement - OOZIE-3160.pdf
>
>
> oozie process always  consume  high cpu. in my mechine,around 10%. 
> I check the source code，find take() method in PriorityDelayQueue class。
> code:
> {code:java}
> public QueueElement take() throws InterruptedException {
> QueueElement e = poll();
> while (e == null) {
> Thread.sleep(10);
> e = poll();
> }
> return e;
> }
> {code}
> i think it's the reason of this problem. it's keep while, not await.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67885: POC: OOZIE-3160 PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting

2018-08-10 Thread András Piros via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67885/#review207076
---




core/src/main/java/org/apache/oozie/service/AsyncXCommandExecutor.java
Lines 143 (patched)


What about using a `CopyOnWriteArrayList` or a `ConcurrentHashSet` instead?



core/src/main/java/org/apache/oozie/service/AsyncXCommandExecutor.java
Lines 154 (patched)


Shouldn't we remove only when `executor.execute()` happened before?



core/src/main/java/org/apache/oozie/service/AsyncXCommandExecutor.java
Lines 163 (patched)


Here we rely on the fact that `commandFinished()` is called only once per 
task. If called multiple times, we're in a problem.



core/src/main/java/org/apache/oozie/service/AsyncXCommandExecutor.java
Lines 167 (patched)


Sure we need to give reference to `ThreadPoolExecutor` in a `public` way?



core/src/main/java/org/apache/oozie/service/AsyncXCommandExecutor.java
Lines 195 (patched)


`Integer#compareTo()` makes it more readable.



core/src/main/java/org/apache/oozie/service/AsyncXCommandExecutor.java
Lines 270 (patched)


`CopyOnWriteArrayList` or `ConcurrentHashSet` instead?



core/src/main/java/org/apache/oozie/service/CallableQueueService.java
Lines 122-126 (patched)


Dead code.



core/src/main/java/org/apache/oozie/service/CallableQueueService.java
Lines 165 (patched)


Unsure why we need an `AtomicInteger` here.



core/src/main/java/org/apache/oozie/service/CallableQueueService.java
Lines 182 (patched)


Unsure why we need an `AtomicInteger` here.



core/src/main/java/org/apache/oozie/service/CallableQueueService.java
Lines 559 (patched)


`LOG.debug()` on which implementation is running would be nice.



core/src/main/java/org/apache/oozie/util/PriorityDelayQueue.java
Line 67 (original), 67-68 (patched)


A javadoc about the difference between the semantics of `delay` and 
`initialDelay` would be nice.



core/src/test/java/org/apache/oozie/service/TestAsyncXCommandExecutor.java
Lines 103-111 (patched)


Duplicate.



core/src/test/java/org/apache/oozie/service/TestAsyncXCommandExecutor.java
Lines 155-163 (patched)


Duplicate.



core/src/test/java/org/apache/oozie/service/TestAsyncXCommandExecutor.java
Lines 207 (patched)


Are there 4 priority levels, from 0 to 3?



core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java
Lines 412-415 (original), 427-440 (patched)


Dead code.



core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java
Line 425 (original), 455 (patched)


Dead code.



core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java
Lines 1075 (patched)


Better use `taskCount / 3`.


- András Piros


On Aug. 6, 2018, 3:28 p.m., Peter Bacsko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67885/
> ---
> 
> (Updated Aug. 6, 2018, 3:28 p.m.)
> 
> 
> Review request for oozie, András Piros, Peter Cseh, Kinga Marton, and Robert 
> Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Still just a POC.
> 
> Tests are almost completely missing.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/service/AsyncXCommandExecutor.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/service/CallableAccess.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/service/CallableQueueService.java 
> ef8d58da5 
>   core/src/main/java/org/apache/oozie/util/PriorityDelayQueue.java 75c20698c 
>   core/src/test/java/org/apache/oozie/service/TestAsyncXCommandExecutor.java 
> PRE-CREATION 
>   core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 
> 9c2a11d6f 
> 
> 
> Diff: https://reviews.apache.org/r/67885/diff/5/
> 
> 
> Testing
> ---
> 
> Executed TestCallableQueueService which passed completely.
> 
> 
> Thanks,
> 
> Peter Bacsko
> 
>

[jira] [Created] (OOZIE-3326) Sqoop Action should support tez delegation tokens for hive-import

2018-08-10 Thread Brian Goerlitz (JIRA)

Brian Goerlitz created OOZIE-3326:
-

 Summary: Sqoop Action should support tez delegation tokens for 
hive-import
 Key: OOZIE-3326
 URL: https://issues.apache.org/jira/browse/OOZIE-3326
 Project: Oozie
  Issue Type: Bug
  Components: action
Reporter: Brian Goerlitz


SqoopMain needs to support tez delegation tokens for hive-imports. 
Implementation is similar to that of HiveMain and Hive2Main.

At present, hive-import will fail to start a tez session in secure environment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Failed: OOZIE-3317 PreCommit Build #745

2018-08-10 Thread Apache Jenkins Server

Jira: https://issues.apache.org/jira/browse/OOZIE-3317
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/745/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 221.98 KB...]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 45.176 s
[INFO] Finished at: 2018-08-10T13:24:47Z
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) 
on project oozie-core: Compilation failure
[ERROR] 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/core/src/main/java/org/apache/oozie/service/ShareLibService.java:[128,55]
 variable dt is already defined in class 
org.apache.oozie.service.ShareLibService
[ERROR] 
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :oozie-core

Reports
  Running test-patch task CLEAN
  Running test-patch task RAW_PATCH_ANALYSIS
  Running test-patch task RAT
  Running test-patch task JAVADOC
  Running test-patch task COMPILE
  Running test-patch task FINDBUGS_DIFF
[TRACE] Downloading FindBugs diff JAR from 
https://repo1.maven.org/maven2/me/andrz/findbugs/findbugs-diff/0.1.0/findbugs-diff-0.1.0-all.jar
[TRACE] FindBugs diff JAR downloaded
grep: 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/bin/findbugs-diff-0.1.0-all.jar.md5sum:
 No such file or directory
[TRACE] FindBugs diff JAR checked, is safe to use
[TRACE] Performing FindBugs diffs
[TRACE] FindBugs diffs performed
[TRACE] Checking FindBugs diffs and creating reports
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 1239 bytes
[TRACE] Full summary file size is 184 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO


Testing JIRA OOZIE-3317

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 1 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warning(s)
+1 the patch does not seem to introduce new Javadoc error(s)
ERROR: the current HEAD has 2 Javadoc error(s)
-1 COMPILE
+1 HEAD compiles
-1 patch does not compile
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS - patch does not compile, cannot run testcases
-1 DISTRO
-1 distro tarball fails with the patch


-1 Overall result, please check the reported -1(s)


The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/745/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100  18930 0  100  1893  0   2714 --:--:-- --:--:-- --:--:--  
2712100  51620  3269  100  1893   4686   2713 --:--:-- --:--:-- --:--:--  
4683

[jira] [Commented] (OOZIE-3317) Fix false positive precommit reports

2018-08-10 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576284#comment-16576284
 ] 

Hadoop QA commented on OOZIE-3317:
--


Testing JIRA OOZIE-3317

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
error(s)
.{color:red}ERROR{color}: the current HEAD has 2 Javadoc error(s)
{color:red}-1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:red}-1{color} patch does not compile
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:red}-1 DISTRO{color}
.{color:red}-1{color} distro tarball fails with the patch


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/745/



> Fix false positive precommit reports
> 
>
> Key: OOZIE-3317
> URL: https://issues.apache.org/jira/browse/OOZIE-3317
> Project: Oozie
>  Issue Type: Sub-task
>  Components: scripts
>Affects Versions: 5.0.0
>Reporter: Julia Kinga Marton
>Assignee: Julia Kinga Marton
>Priority: Critical
> Fix For: 5.1.0
>
> Attachments: OOZIE-3317-TESTING.patch, consoleFull.txt
>
>
> Some steps of the current precommit are failing but we get a +1 anyways ( see 
> OOZIE-3304 for a recent example)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3317) Fix false positive precommit reports

2018-08-10 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576278#comment-16576278
 ] 

Hadoop QA commented on OOZIE-3317:
--

PreCommit-OOZIE-Build started


> Fix false positive precommit reports
> 
>
> Key: OOZIE-3317
> URL: https://issues.apache.org/jira/browse/OOZIE-3317
> Project: Oozie
>  Issue Type: Sub-task
>  Components: scripts
>Affects Versions: 5.0.0
>Reporter: Julia Kinga Marton
>Assignee: Julia Kinga Marton
>Priority: Critical
> Fix For: 5.1.0
>
> Attachments: OOZIE-3317-TESTING.patch, consoleFull.txt
>
>
> Some steps of the current precommit are failing but we get a +1 anyways ( see 
> OOZIE-3304 for a recent example)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3317) Fix false positive precommit reports

2018-08-10 Thread Julia Kinga Marton (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576274#comment-16576274
 ] 

Julia Kinga Marton commented on OOZIE-3317:
---

Please don`t check the patch yet. I have uploaded the 
[OOZIE-3317-TESTING.patch|https://issues.apache.org/jira/secure/attachment/12935135/OOZIE-3317-TESTING.patch]
 with testing purpose. I have introduced intentionally a bug, to check if in 
case of a timeout, the failure will be reported or not. 

> Fix false positive precommit reports
> 
>
> Key: OOZIE-3317
> URL: https://issues.apache.org/jira/browse/OOZIE-3317
> Project: Oozie
>  Issue Type: Sub-task
>  Components: scripts
>Affects Versions: 5.0.0
>Reporter: Julia Kinga Marton
>Assignee: Julia Kinga Marton
>Priority: Critical
> Fix For: 5.1.0
>
> Attachments: OOZIE-3317-TESTING.patch, consoleFull.txt
>
>
> Some steps of the current precommit are failing but we get a +1 anyways ( see 
> OOZIE-3304 for a recent example)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3317) Fix false positive precommit reports

2018-08-10 Thread Julia Kinga Marton (JIRA)



 [ 
https://issues.apache.org/jira/browse/OOZIE-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julia Kinga Marton updated OOZIE-3317:
--
Attachment: OOZIE-3317-TESTING.patch

> Fix false positive precommit reports
> 
>
> Key: OOZIE-3317
> URL: https://issues.apache.org/jira/browse/OOZIE-3317
> Project: Oozie
>  Issue Type: Sub-task
>  Components: scripts
>Affects Versions: 5.0.0
>Reporter: Julia Kinga Marton
>Assignee: Julia Kinga Marton
>Priority: Critical
> Fix For: 5.1.0
>
> Attachments: OOZIE-3317-TESTING.patch, consoleFull.txt
>
>
> Some steps of the current precommit are failing but we get a +1 anyways ( see 
> OOZIE-3304 for a recent example)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3324) Cannot compile with findbugs check

2018-08-10 Thread Andras Salamon (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576241#comment-16576241
 ] 

Andras Salamon commented on OOZIE-3324:
---

Seems to me Jenkins comment is missing again, here is the build info: 
[https://builds.apache.org/job/PreCommit-OOZIE-Build/743/]

 

> Cannot compile with findbugs check
> --
>
> Key: OOZIE-3324
> URL: https://issues.apache.org/jira/browse/OOZIE-3324
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Critical
> Attachments: OOZIE-3324-1.patch
>
>
> Latest snapshot compilation fails because of missing findbugs-filter.xml file:
> {noformat}
> $ mvn clean install -DskipTests
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 11.852 s
> [INFO] Finished at: 2018-08-09T09:11:41+02:00
> [INFO] 
> 
> [ERROR] Could not find resource 
> '/Users/andrassalamon/src/oozie/fluent-job/fluent-job-api/findbugs-filter.xml'.
>  -> [Help 1]{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3204) Oozie cannot run HBase code in Java action

2018-08-10 Thread Andras Piros (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-3204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576232#comment-16576232
 ] 

Andras Piros commented on OOZIE-3204:
-

[~dionusos] can you please provide a Java action example using HBase to show 
that it's been already fixed? You can provide a patch and the code in 
{{examples/}}.

> Oozie cannot run HBase code in Java action 
> ---
>
> Key: OOZIE-3204
> URL: https://issues.apache.org/jira/browse/OOZIE-3204
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.3.1
>Reporter: Denes Bodo
>Assignee: Peter Cseh
>Priority: Critical
>
> After having custom raw file system implementation, HBase (java) action 
> cannot run:
> {noformat}
> 018-03-28 06:55:46,372  WARN HbaseCredentials:523 - 
> SERVER[ctr-e138-1518143905142-137559-01-03.hwx.site] USER[hrt_qa] 
> GROUP[-] TOKEN[] APP[tpch_query1] JOB[002-180328065157516-oozie-oozi-W] 
> ACTION[002-180328065157516-oozie-oozi-W@tpch_query1] Exception in 
> receiving hbase credentials
> java.io.IOException: java.lang.reflect.InvocationTargetException
>   at 
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
>   at 
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
>   at 
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
>   at 
> org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:68)
>   at 
> org.apache.oozie.action.hadoop.HbaseCredentials$1.run(HbaseCredentials.java:86)
>   at 
> org.apache.oozie.action.hadoop.HbaseCredentials$1.run(HbaseCredentials.java:84)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
>   at 
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:313)
>   at 
> org.apache.oozie.action.hadoop.HbaseCredentials.obtainToken(HbaseCredentials.java:83)
>   at 
> org.apache.oozie.action.hadoop.HbaseCredentials.addtoJobConf(HbaseCredentials.java:56)
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:1338)
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1178)
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1424)
>   at 
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
>   at 
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
>   at org.apache.oozie.command.XCommand.call(XCommand.java:287)
>   at 
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:331)
>   at 
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:260)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
>   ... 24 more
> Caused by: java.lang.ExceptionInInitializerError
>   at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64)
>   at 
> org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:75)
>   at 
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:105)
>   at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:903)
>   at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:648)
>   ... 29 more
> Caused by: java.lang.UnsupportedOperationException: Accessing local file 
> system is not allowed
>   at 
>

[jira] [Commented] (OOZIE-3204) Oozie cannot run HBase code in Java action

2018-08-10 Thread Denes Bodo (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-3204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576231#comment-16576231
 ] 

Denes Bodo commented on OOZIE-3204:
---

Shall we close this as it seems already fixed in Oozie-5.x?

> Oozie cannot run HBase code in Java action 
> ---
>
> Key: OOZIE-3204
> URL: https://issues.apache.org/jira/browse/OOZIE-3204
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.3.1
>Reporter: Denes Bodo
>Assignee: Peter Cseh
>Priority: Critical
>
> After having custom raw file system implementation, HBase (java) action 
> cannot run:
> {noformat}
> 018-03-28 06:55:46,372  WARN HbaseCredentials:523 - 
> SERVER[ctr-e138-1518143905142-137559-01-03.hwx.site] USER[hrt_qa] 
> GROUP[-] TOKEN[] APP[tpch_query1] JOB[002-180328065157516-oozie-oozi-W] 
> ACTION[002-180328065157516-oozie-oozi-W@tpch_query1] Exception in 
> receiving hbase credentials
> java.io.IOException: java.lang.reflect.InvocationTargetException
>   at 
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
>   at 
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
>   at 
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
>   at 
> org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:68)
>   at 
> org.apache.oozie.action.hadoop.HbaseCredentials$1.run(HbaseCredentials.java:86)
>   at 
> org.apache.oozie.action.hadoop.HbaseCredentials$1.run(HbaseCredentials.java:84)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
>   at 
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:313)
>   at 
> org.apache.oozie.action.hadoop.HbaseCredentials.obtainToken(HbaseCredentials.java:83)
>   at 
> org.apache.oozie.action.hadoop.HbaseCredentials.addtoJobConf(HbaseCredentials.java:56)
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:1338)
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1178)
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1424)
>   at 
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
>   at 
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
>   at org.apache.oozie.command.XCommand.call(XCommand.java:287)
>   at 
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:331)
>   at 
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:260)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
>   ... 24 more
> Caused by: java.lang.ExceptionInInitializerError
>   at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64)
>   at 
> org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:75)
>   at 
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:105)
>   at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:903)
>   at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:648)
>   ... 29 more
> Caused by: java.lang.UnsupportedOperationException: Accessing local file 
> system is not allowed
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:48)
>   at 
> org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSystem.java:47)
>

[jira] [Commented] (OOZIE-3298) OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED

2018-08-10 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576206#comment-16576206
 ] 

Hadoop QA commented on OOZIE-3298:
--


Testing JIRA OOZIE-3298

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 2 line(s) longer than 132 
characters
.{color:green}+1{color} the patch adds/modifies 4 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:red}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
error(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2919
.Tests failed: 2
.Tests errors: 0

.The patch failed the following testcases:

testRerun(org.apache.oozie.action.hadoop.TestRerun)
testFailingMapReduceJobCausesOozieLauncherAMToFail(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)

.Tests failing with errors:


{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/744/



> OYA: external ID is not filled properly and failing MR job is treated as 
> SUCCEEDED
> --
>
> Key: OOZIE-3298
> URL: https://issues.apache.org/jira/browse/OOZIE-3298
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Peter Bacsko
>Assignee: Andras Piros
>Priority: Blocker
> Fix For: 5.1.0
>
> Attachments: OOZIE-3298.001.patch, OOZIE-3298.002.patch, 
> OOZIE-3298.003.patch, OOZIE-3298.004.patch
>
>
> When a MapReduce action is launched from Oozie (OYA), we don't properly fill 
> the external ID field. It gets populated with the YARN id of the LauncherAM, 
> not with the id of the actual MR job. If the MR job is succesfully submitted 
> then fails, it will be treated as a successfully executed action, which is 
> very misleading and can potentially confuse Oozie users.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Failed: OOZIE-3298 PreCommit Build #744

2018-08-10 Thread Apache Jenkins Server

Jira: https://issues.apache.org/jira/browse/OOZIE-3298
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/744/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 466.82 KB...]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 03:36 min
[INFO] Finished at: 2018-08-10T12:28:14Z
[INFO] 

Reports
  Running test-patch task CLEAN
  Running test-patch task RAW_PATCH_ANALYSIS
  Running test-patch task RAT
  Running test-patch task JAVADOC
  Running test-patch task COMPILE
  Running test-patch task FINDBUGS_DIFF
[TRACE] Downloading FindBugs diff JAR from 
https://repo1.maven.org/maven2/me/andrz/findbugs/findbugs-diff/0.1.0/findbugs-diff-0.1.0-all.jar
[TRACE] FindBugs diff JAR downloaded
grep: 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/bin/findbugs-diff-0.1.0-all.jar.md5sum:
 No such file or directory
[TRACE] FindBugs diff JAR checked, is safe to use
[TRACE] Performing FindBugs diffs
[TRACE] FindBugs diffs performed
[TRACE] Checking FindBugs diffs and creating reports
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 1239 bytes
[TRACE] Full summary file size is 184 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO


Testing JIRA OOZIE-3298

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
-1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
-1 the patch contains 2 line(s) longer than 132 characters
+1 the patch adds/modifies 4 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warning(s)
WARNING: the current HEAD has 100 Javadoc warning(s)
+1 the patch does not seem to introduce new Javadoc error(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS
Tests run: 2919
Tests failed: 2
Tests errors: 0

The patch failed the following testcases:

testRerun(org.apache.oozie.action.hadoop.TestRerun)
testFailingMapReduceJobCausesOozieLauncherAMToFail(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)

Tests failing with errors:


+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/744/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100  22390 0  100  2239  0   2536 --:--:-- --:--:-- --:--:--  
2535100  58540  3615  100  2239   4093   2535 --:--:-- --:--:-- --:--:--  
4093
{"self":"https://issues.apache.org/jira/rest/api/2/issue/13168377/comment/16576206","id":"16576206","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop
 QA","active":true,"timeZone":"Etc/UTC"},"body":"\nTesting JIRA 
OOZIE-3298\n\nCleaning local git 
workspace\n\n\n\n{color:green}+1 
PATCH_APPLIES{color}\n{color:green}+1 CLEAN{color}\n{color:red}-1 
RAW_PATCH_ANALYSIS{color}\n.{color:green}+1{color} the patch does not 
introduce any @author tags\n.{color:green}+1{color} the patch does not 
introduce any tabs\n.{color:green}+1{color} the

Failed: OOZIE-3324 PreCommit Build #743

2018-08-10 Thread Apache Jenkins Server

Jira: https://issues.apache.org/jira/browse/OOZIE-3324
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/743/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.11 MB...]
 0 There are [2] new bugs found in [fluent-job/fluent-job-api] that would be 
nice to have fixed.
 You can find the FindBugs diff here: 
fluent-job/fluent-job-api/findbugs-new.html
 -1 There are [4] new bugs found below threshold in [sharelib/hive2] that must 
be fixed.
 You can find the FindBugs diff here (look for the red and orange ones): 
sharelib/hive2/findbugs-new.html
 The most important FindBugs errors are:
 At Hive2Main.java:[line 271]: 
org.apache.oozie.action.hadoop.Hive2Main.runBeeline(String[], String) invokes 
System.exit(...), which shuts down the entire virtual machine
 At Hive2Main.java:[line 279]: 
java/io/FileInputStream.init(Ljava/lang/String;)V reads a file whose 
location might be specified by user input
 At Hive2Main.java:[line 165]: At Hive2Main.java:[line 146]
 At Hive2Main.java:[line 149]: java/io/File.init(Ljava/lang/String;)V 
reads a file whose location might be specified by user input
 java/io/FileOutputStream.init(Ljava/lang/String;)V writes to a file 
whose location might be specified by user input: At Hive2Main.java:[line 146]
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/sqoop].
 -1 There are [26] new bugs found below threshold in [sharelib/oozie] that must 
be fixed, listing only the first [5] ones.
 You can find the FindBugs diff here (look for the red and orange ones): 
sharelib/oozie/findbugs-new.html
 The top [5] most important FindBugs errors are:
 At ActionStats.java:[line 37]: Unwritten public or protected field: 
org.apache.oozie.action.hadoop.ActionStats.currentActionType
 At LauncherAMUtils.java:[line 64]: Found reliance on default encoding in 
org.apache.oozie.action.hadoop.LauncherAMUtils.getLocalFileContentStr(File, 
String, int): new java.io.FileReader(File)
 Obligation to clean up resource created at LauncherAMUtils.java:[line 64] is 
not discharged: 
org.apache.oozie.action.hadoop.LauncherAMUtils.getLocalFileContentStr(File, 
String, int) may fail to clean up java.io.Reader on checked exception
 Path continues at LauncherAMUtils.java:[line 67]: Path continues at 
LauncherAMUtils.java:[line 65]
 Found reliance on default encoding in 
org.apache.oozie.action.hadoop.LauncherMain.getHadoopJobIds(String, Pattern[]): 
new java.io.FileReader(String): Path continues at LauncherAMUtils.java:[line 68]
 -1 There are [3] new bugs found below threshold in [sharelib/pig] that must be 
fixed.
 You can find the FindBugs diff here (look for the red and orange ones): 
sharelib/pig/findbugs-new.html
 The most important FindBugs errors are:
 At PigMain.java:[line 258]: java/io/File.init(Ljava/lang/String;)V 
reads a file whose location might be specified by user input
 At PigMain.java:[line 141]: At PigMain.java:[line 131]
 At PigMain.java:[line 245]: At PigMain.java:[line 199]
 At PigMain.java:[line 137]: java/io/File.init(Ljava/lang/String;)V 
reads a file whose location might be specified by user input
 java/io/File.init(Ljava/lang/String;)V reads a file whose location 
might be specified by user input: At PigMain.java:[line 131]
 -1 There are [2] new bugs found below threshold in [sharelib/hive] that must 
be fixed.
 You can find the FindBugs diff here (look for the red and orange ones): 
sharelib/hive/findbugs-new.html
 The most important FindBugs errors are:
 At HiveMain.java:[line 317]: 
java/io/FileInputStream.init(Ljava/lang/String;)V reads a file whose 
location might be specified by user input
 At HiveMain.java:[line 226]: At HiveMain.java:[line 207]
 At HiveMain.java:[line 210]: java/io/File.init(Ljava/lang/String;)V 
reads a file whose location might be specified by user input
 At HiveMain.java:[line 207]
 -1 There are [5] new bugs found below threshold in [sharelib/spark] that must 
be fixed.
 You can find the FindBugs diff here (look for the red and orange ones): 
sharelib/spark/findbugs-new.html
 The most important FindBugs errors are:
 At SparkMain.java:[line 133]: 
java/io/File.init(Ljava/io/File;Ljava/lang/String;)V reads a file whose 
location might be specified by user input
 java/io/File.init(Ljava/lang/String;)V reads a file whose location 
might be specified by user input: At SparkMain.java:[line 133]
 At SparkMain.java:[line 166]: At SparkMain.java:[line 173]
 The regular expression 
"^spark-assembly((?:(-|_|(\\d+\\.))\\d+(?:\\.\\d+)*))*\\.jar$" is vulnerable to 
a denial of service attack (ReDOS): At SparkMain.java:[line 172]
 The regular expression 
"^spark-yarn((?:(-|_|(\\d+\\.))\\d+(?:\\.\\d+)*))*\\.jar$" is vulnerable to a 
denial of service attack (ReDOS): At

[jira] [Commented] (OOZIE-3298) OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED

2018-08-10 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576104#comment-16576104
 ] 

Hadoop QA commented on OOZIE-3298:
--

PreCommit-OOZIE-Build started


> OYA: external ID is not filled properly and failing MR job is treated as 
> SUCCEEDED
> --
>
> Key: OOZIE-3298
> URL: https://issues.apache.org/jira/browse/OOZIE-3298
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Peter Bacsko
>Assignee: Andras Piros
>Priority: Blocker
> Fix For: 5.1.0
>
> Attachments: OOZIE-3298.001.patch, OOZIE-3298.002.patch, 
> OOZIE-3298.003.patch, OOZIE-3298.004.patch
>
>
> When a MapReduce action is launched from Oozie (OYA), we don't properly fill 
> the external ID field. It gets populated with the YARN id of the LauncherAM, 
> not with the id of the actual MR job. If the MR job is succesfully submitted 
> then fails, it will be treated as a successfully executed action, which is 
> very misleading and can potentially confuse Oozie users.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3298) OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED

2018-08-10 Thread Andras Piros (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576099#comment-16576099
 ] 

Andras Piros commented on OOZIE-3298:
-

Changes in patch 004:
* removed new test inside {{TestJavaActionExecutor}}
* {{JavaActionExecutor#findEffectiveYarnApplicationId()}} now falls back to 
looking for an available YARN child application that isn't the launcher via 
{{YarnClient#getApplications()}} in case there is still no {{action-data.seq}} 
written by {{LauncherMain}} at the end of the MapReduce action
* {{TestMapReduceActionExecutor#startWorkflowAndFailChildMRJob()}} now waits 
until a real YARN child that isn't the Oozie Launcher appears

The test 
{{TestMapReduceActionExecutor#testFailingMapReduceJobCausesOozieLauncherAMToFail()}}
 fails locally. [~pbacsko] can you please look at the code and advise what do I 
have to change?

> OYA: external ID is not filled properly and failing MR job is treated as 
> SUCCEEDED
> --
>
> Key: OOZIE-3298
> URL: https://issues.apache.org/jira/browse/OOZIE-3298
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Peter Bacsko
>Assignee: Andras Piros
>Priority: Blocker
> Fix For: 5.1.0
>
> Attachments: OOZIE-3298.001.patch, OOZIE-3298.002.patch, 
> OOZIE-3298.003.patch, OOZIE-3298.004.patch
>
>
> When a MapReduce action is launched from Oozie (OYA), we don't properly fill 
> the external ID field. It gets populated with the YARN id of the LauncherAM, 
> not with the id of the actual MR job. If the MR job is succesfully submitted 
> then fails, it will be treated as a successfully executed action, which is 
> very misleading and can potentially confuse Oozie users.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3298) OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED

2018-08-10 Thread Andras Piros (JIRA)



 [ 
https://issues.apache.org/jira/browse/OOZIE-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-3298:

Attachment: OOZIE-3298.004.patch

> OYA: external ID is not filled properly and failing MR job is treated as 
> SUCCEEDED
> --
>
> Key: OOZIE-3298
> URL: https://issues.apache.org/jira/browse/OOZIE-3298
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Peter Bacsko
>Assignee: Andras Piros
>Priority: Blocker
> Fix For: 5.1.0
>
> Attachments: OOZIE-3298.001.patch, OOZIE-3298.002.patch, 
> OOZIE-3298.003.patch, OOZIE-3298.004.patch
>
>
> When a MapReduce action is launched from Oozie (OYA), we don't properly fill 
> the external ID field. It gets populated with the YARN id of the LauncherAM, 
> not with the id of the actual MR job. If the MR job is succesfully submitted 
> then fails, it will be treated as a successfully executed action, which is 
> very misleading and can potentially confuse Oozie users.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68237: OOZIE-3298 OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED

2018-08-10 Thread András Piros via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68237/
---

(Updated Aug. 10, 2018, 10:52 a.m.)


Review request for oozie, Attila Sasvari and Peter Bacsko.


Repository: oozie-git


Description
---

OOZIE-3298 OYA: external ID is not filled properly and failing MR job is 
treated as SUCCEEDED


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
8f0f244013932476d8ae454d224f235948529b34 
  
core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 
83a23f5220aa72ba15edc8b98ef80a74213fcee8 
  core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 
f39bba2c691435354dac6da7794e5142b511d937 
  core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
a31079a41d30677d35a253a4a69505c21aa585f6 
  
core/src/test/java/org/apache/oozie/action/oozie/TestSubWorkflowActionExecutor.java
 893405e38ad71d22460768b76ed03ac4d9a0b95d 
  
sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java
 f460b6bd11f60dfb397c6bba82be1427c2d1b570 


Diff: https://reviews.apache.org/r/68237/diff/2/

Changes: https://reviews.apache.org/r/68237/diff/1-2/


Testing
---

DO NOT MERGE YET.

Tested on a real cluster, plus added test cases to `TestJavaActionExecutor` and 
`TestMapReduceActionExecutor`.

Please note that I could find no proper way of actually getting MapReduce to 
start a new job while using `OozieClient#submit()` - apparently no YARN child 
application is created. Please advise what's the best way to advance, maybe 
call `JobClient#submitJob()` directly from 
`ActionExecutorTestCase#startWorkflowAndFailChildMRJob()`.


Thanks,

András Piros

[jira] [Commented] (OOZIE-3324) Cannot compile with findbugs check

2018-08-10 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576029#comment-16576029
 ] 

Hadoop QA commented on OOZIE-3324:
--

PreCommit-OOZIE-Build started


> Cannot compile with findbugs check
> --
>
> Key: OOZIE-3324
> URL: https://issues.apache.org/jira/browse/OOZIE-3324
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Critical
> Attachments: OOZIE-3324-1.patch
>
>
> Latest snapshot compilation fails because of missing findbugs-filter.xml file:
> {noformat}
> $ mvn clean install -DskipTests
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 11.852 s
> [INFO] Finished at: 2018-08-09T09:11:41+02:00
> [INFO] 
> 
> [ERROR] Could not find resource 
> '/Users/andrassalamon/src/oozie/fluent-job/fluent-job-api/findbugs-filter.xml'.
>  -> [Help 1]{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3324) Cannot compile with findbugs check

2018-08-10 Thread Andras Piros (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576031#comment-16576031
 ] 

Andras Piros commented on OOZIE-3324:
-

Retriggering last Jenkins run to get an up-to-date pre-commit output.

> Cannot compile with findbugs check
> --
>
> Key: OOZIE-3324
> URL: https://issues.apache.org/jira/browse/OOZIE-3324
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Critical
> Attachments: OOZIE-3324-1.patch
>
>
> Latest snapshot compilation fails because of missing findbugs-filter.xml file:
> {noformat}
> $ mvn clean install -DskipTests
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 11.852 s
> [INFO] Finished at: 2018-08-09T09:11:41+02:00
> [INFO] 
> 
> [ERROR] Could not find resource 
> '/Users/andrassalamon/src/oozie/fluent-job/fluent-job-api/findbugs-filter.xml'.
>  -> [Help 1]{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3317) Fix false positive precommit reports

2018-08-10 Thread Julia Kinga Marton (JIRA)



 [ 
https://issues.apache.org/jira/browse/OOZIE-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julia Kinga Marton updated OOZIE-3317:
--
Attachment: consoleFull.txt

> Fix false positive precommit reports
> 
>
> Key: OOZIE-3317
> URL: https://issues.apache.org/jira/browse/OOZIE-3317
> Project: Oozie
>  Issue Type: Sub-task
>  Components: scripts
>Affects Versions: 5.0.0
>Reporter: Julia Kinga Marton
>Assignee: Julia Kinga Marton
>Priority: Critical
> Fix For: 5.1.0
>
> Attachments: consoleFull.txt
>
>
> Some steps of the current precommit are failing but we get a +1 anyways ( see 
> OOZIE-3304 for a recent example)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3317) Fix false positive precommit reports

2018-08-10 Thread Julia Kinga Marton (JIRA)



 [ 
https://issues.apache.org/jira/browse/OOZIE-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julia Kinga Marton updated OOZIE-3317:
--
Attachment: (was: consoleFull)

> Fix false positive precommit reports
> 
>
> Key: OOZIE-3317
> URL: https://issues.apache.org/jira/browse/OOZIE-3317
> Project: Oozie
>  Issue Type: Sub-task
>  Components: scripts
>Affects Versions: 5.0.0
>Reporter: Julia Kinga Marton
>Assignee: Julia Kinga Marton
>Priority: Critical
> Fix For: 5.1.0
>
> Attachments: consoleFull.txt
>
>
> Some steps of the current precommit are failing but we get a +1 anyways ( see 
> OOZIE-3304 for a recent example)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3317) Fix false positive precommit reports

2018-08-10 Thread Julia Kinga Marton (JIRA)



 [ 
https://issues.apache.org/jira/browse/OOZIE-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julia Kinga Marton updated OOZIE-3317:
--
Attachment: consoleFull

> Fix false positive precommit reports
> 
>
> Key: OOZIE-3317
> URL: https://issues.apache.org/jira/browse/OOZIE-3317
> Project: Oozie
>  Issue Type: Sub-task
>  Components: scripts
>Affects Versions: 5.0.0
>Reporter: Julia Kinga Marton
>Assignee: Julia Kinga Marton
>Priority: Critical
> Fix For: 5.1.0
>
> Attachments: consoleFull
>
>
> Some steps of the current precommit are failing but we get a +1 anyways ( see 
> OOZIE-3304 for a recent example)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] Subscription: Oozie Patch Available

2018-08-10 Thread jira

Issue Subscription
Filter: Oozie Patch Available (95 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3324  Cannot compile with findbugs check
https://issues.apache.org/jira/browse/OOZIE-3324
OOZIE-3320  Oozie ShellAction should support absolute bash file path
https://issues.apache.org/jira/browse/OOZIE-3320
OOZIE-3319  Log SSH action callback error output
https://issues.apache.org/jira/browse/OOZIE-3319
OOZIE-3318  Fix Javadoc check in the pre-commit
https://issues.apache.org/jira/browse/OOZIE-3318
OOZIE-3301  Update NOTICE file
https://issues.apache.org/jira/browse/OOZIE-3301
OOZIE-3298  OYA: external ID is not filled properly and failing MR job is 
treated as SUCCEEDED
https://issues.apache.org/jira/browse/OOZIE-3298
OOZIE-3274  Remove slf4j
https://issues.apache.org/jira/browse/OOZIE-3274
OOZIE-3266  Coord action rerun support RERUN_SKIP_NODES option
https://issues.apache.org/jira/browse/OOZIE-3266
OOZIE-3265  properties RERUN_FAIL_NODES and RERUN_SKIP_NODES should be able to 
appear together
https://issues.apache.org/jira/browse/OOZIE-3265
OOZIE-3256  refactor OozieCLI class
https://issues.apache.org/jira/browse/OOZIE-3256
OOZIE-3249  [tools] Instrumentation log parser
https://issues.apache.org/jira/browse/OOZIE-3249
OOZIE-3229  Improved filtering options in V2SLAServlet
https://issues.apache.org/jira/browse/OOZIE-3229
OOZIE-3218  Oozie Sqoop action with command splits the select clause into 
multiple parts due to delimiter being space
https://issues.apache.org/jira/browse/OOZIE-3218
OOZIE-3199  Let system property restriction configurable
https://issues.apache.org/jira/browse/OOZIE-3199
OOZIE-3196  Authorization: restrict world readability by user
https://issues.apache.org/jira/browse/OOZIE-3196
OOZIE-3194  Oozie should set proper permissions to sharelib after upload
https://issues.apache.org/jira/browse/OOZIE-3194
OOZIE-3186  Oozie is unable to use configuration linked using jceks://file/...
https://issues.apache.org/jira/browse/OOZIE-3186
OOZIE-3179  Adding a configurable config-default.xml location to a workflow
https://issues.apache.org/jira/browse/OOZIE-3179
OOZIE-3170  Oozie Diagnostic Bundle tool fails with NPE due to missing service 
class
https://issues.apache.org/jira/browse/OOZIE-3170
OOZIE-3160  PriorityDelayQueue put()/take() can cause significant CPU load due 
to busy waiting
https://issues.apache.org/jira/browse/OOZIE-3160
OOZIE-3135  Configure log4j2 in SqoopMain
https://issues.apache.org/jira/browse/OOZIE-3135
OOZIE-3091  Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: 
org/apache/avro/mapred/AvroWrapper"
https://issues.apache.org/jira/browse/OOZIE-3091
OOZIE-3071  Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 
than Spark 2.2.0
https://issues.apache.org/jira/browse/OOZIE-3071
OOZIE-3063  Sanitizing variables that are part of openjpa.ConnectionProperties
https://issues.apache.org/jira/browse/OOZIE-3063
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues.apache.org/jira/browse/OOZIE-3062
OOZIE-3061  Kill only those child jobs which are not already killed
https://issues.apache.org/jira/browse/OOZIE-3061
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2795  Create lib directory or symlink for Oozie CLI during packaging
https://issues.apache.org/jira/browse/OOZIE-2795
OOZIE-2784  Include WEEK as a parameter in the Coordinator Expression Language 
Evaulator
https://issues.apache.org/jira/browse/OOZIE-2784
OOZIE-2779  Mask Hive2 action Beeline JDBC password
https://issues.apache.org/jira/browse/OOZIE-2779
OOZIE-2736  Reduce the number of threads during test execution
https://issues.apache.org/jira/browse/OOZIE-2736
OOZIE-2734  Switch docs from twiki to markdown

[jira] [Commented] (OOZIE-3326) Sqoop Action should support tez delegation tokens for hive-import

[jira] [Commented] (OOZIE-3160) PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting

Re: Review Request 67885: POC: OOZIE-3160 PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting

[jira] [Created] (OOZIE-3326) Sqoop Action should support tez delegation tokens for hive-import

Failed: OOZIE-3317 PreCommit Build #745

[jira] [Commented] (OOZIE-3317) Fix false positive precommit reports

[jira] [Commented] (OOZIE-3317) Fix false positive precommit reports

[jira] [Commented] (OOZIE-3317) Fix false positive precommit reports

[jira] [Updated] (OOZIE-3317) Fix false positive precommit reports

[jira] [Commented] (OOZIE-3324) Cannot compile with findbugs check

[jira] [Commented] (OOZIE-3204) Oozie cannot run HBase code in Java action

[jira] [Commented] (OOZIE-3204) Oozie cannot run HBase code in Java action

[jira] [Commented] (OOZIE-3298) OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED

Failed: OOZIE-3298 PreCommit Build #744

Failed: OOZIE-3324 PreCommit Build #743

[jira] [Commented] (OOZIE-3298) OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED

[jira] [Commented] (OOZIE-3298) OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED

[jira] [Updated] (OOZIE-3298) OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED

Re: Review Request 68237: OOZIE-3298 OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED

[jira] [Commented] (OOZIE-3324) Cannot compile with findbugs check

[jira] [Commented] (OOZIE-3324) Cannot compile with findbugs check

[jira] [Updated] (OOZIE-3317) Fix false positive precommit reports

[jira] [Updated] (OOZIE-3317) Fix false positive precommit reports

[jira] [Updated] (OOZIE-3317) Fix false positive precommit reports

[jira] Subscription: Oozie Patch Available

25 matches

Site Navigation

Mail list logo

Footer information