[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (108 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3071  Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 
than Spark 2.2.0
https://issues.apache.org/jira/browse/OOZIE-3071
OOZIE-3063  Sanitizing variables that are part of openjpa.ConnectionProperties
https://issues.apache.org/jira/browse/OOZIE-3063
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues.apache.org/jira/browse/OOZIE-3062
OOZIE-3031  Coord job with only unresolved dependencies doesn't timeout
https://issues.apache.org/jira/browse/OOZIE-3031
OOZIE-3022  fix for warning has no file and won't be listed in dependency files 
details
https://issues.apache.org/jira/browse/OOZIE-3022
OOZIE-3002  address findbugs errors in client lib
https://issues.apache.org/jira/browse/OOZIE-3002
OOZIE-3001  core library has many instances of warnings with trailing spaces 
and lines longer than 132 chars
https://issues.apache.org/jira/browse/OOZIE-3001
OOZIE-2997  files contain trailing white spaces in client lib
https://issues.apache.org/jira/browse/OOZIE-2997
OOZIE-2996  add option for -UseGCOverheadLimit to maven opts as sometimes local 
testing fails
https://issues.apache.org/jira/browse/OOZIE-2996
OOZIE-2978  Remove code that handles Pig versions before 0.8 
https://issues.apache.org/jira/browse/OOZIE-2978
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues.apache.org/jira/browse/OOZIE-2975
OOZIE-2969  Drop support for Java 1.7
https://issues.apache.org/jira/browse/OOZIE-2969
OOZIE-2964  Add -Xdoclint:none to javadoc opts to avoid warnings
https://issues.apache.org/jira/browse/OOZIE-2964
OOZIE-2963  getting error in build ArtifactNotFoundException: Could not find 
artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde
https://issues.apache.org/jira/browse/OOZIE-2963
OOZIE-2962  bump maven-javadoc-plugin to 2.10.4
https://issues.apache.org/jira/browse/OOZIE-2962
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues.apache.org/jira/browse/OOZIE-2934
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2914  Consolidate Trim 
https://issues.apache.org/jira/browse/OOZIE-2914
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues.apache.org/jira/browse/OOZIE-2867
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues.apache.org/jira/browse/OOZIE-2829
OOZIE-2826  Falcon feed fails to aws s3; Oozie joda time version does not meet 
required jar version 2.2 or later
https://issues.apache.org/jira/browse/OOZIE-2826
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

[Robert Kanter]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186754
---




docs/src/site/twiki/DG_CommandLineTool.twiki
Lines 1948 (patched)


It would be good to add something about the children aspect of the tool 
(e.g. that if you get a coordinator, it will also get some number of child 
workflows)



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 97-98 (patched)


Something to consider for all output of the tool, not just here: we're 
outputting most of the info in a human-readable format.  Should we think about 
using a machine-readable format?  Or maybe having the option for one?  Or doing 
both?  The idea being that someone would then be able to write their own tool 
that could analyze stuff.  We already have some code somewhere that converts a 
WorkflowJob into JSON, so it shouldn't be a lot of work to add this either.  
That might also be a good idea from a compatibility perspective - i.e. what's 
the compatibility story on this out?  If there's a new field, what do we do?



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 99-119 (patched)


I'm not sure that's necessary here - we're not concatinating strings, we're 
writing to a Writer.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 151 (patched)


Should we skip control types?  That's probably fine for the start action, 
but what about a decision action?  That has some useful info.  Maybe we should 
still print these out, but with fewer fields or something (e.g. no need for a 
console url).



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 163 (patched)


I think the JHS may also be required, in the cases where the RM has 
forgotten about the job.

And what about HDFS?  That's required too.

I'm thinking we might be best off not doing these checks.  It's too 
complicated (CM spent a lot of effort on this) and we can't check for 
everything (e.g. what if log aggregation is turned off?).  Besides, we're 
already handling exceptions below when trying to get the logs - if the RM, JHS, 
HDFS, etc isn't working, the call will fail anyway.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 185 (patched)


Please create a followup JIRA to change this in the future to use 
OOZIE-2983 ("Stream the Launcher AM Logs") once it's done.  This will also be 
nice in that we can get rid of the RM up check.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 191 (patched)


Is there not a cleaner way to do this than using a CLI like this?



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 221 (patched)


This won't work right if using RM HA...

I'd recommend using a YarnClient and passing it the 
hadoopConfig so it can figure out the RM address for you.  There must 
be a benign simple YarnClient command you can run to verify 
connectivity.



tools/src/main/java/org/apache/oozie/tools/diag/BundleCollectorDriver.java
Lines 37 (patched)


I'm not sure I like the name "BundleXYZ" for these classes.  It's ambiguous 
with a Bundle Job.  Perhaps 
"DiagBundleXYZ" instead?



tools/src/main/java/org/apache/oozie/tools/diag/BundleCompressor.java
Lines 32 (patched)


Can use the Java 7 try-with-resources.



tools/src/main/java/org/apache/oozie/tools/diag/Client.java
Lines 32 (patched)


I think you have a typo here: "... so we get that for free"



tools/src/main/java/org/apache/oozie/tools/diag/Client.java
Lines 38 (patched)


I would add a comment here explaining why we're doing this.  (i.e. that 
Oozie has a jsp page with a thread dump, and the Oozie client normally doesn't 
have a way of getting it, so we're doing that here; reusing all the fancy HTTP 
handling code in AuthOozieClient)



tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java
Lines 47-49 (patched)


Strange lining here



tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java
Lines 45-47 (patched)


Strange lin

[jira] [Created] (OOZIE-3072) oozie.service.HadoopAccessorService.action.configurations should overwrite define default values set in Hadoop's configuration files

[Peter Cseh (JIRA)]

Peter Cseh created OOZIE-3072:
-

 Summary: oozie.service.HadoopAccessorService.action.configurations 
 should overwrite define default values set in Hadoop's configuration files
 Key: OOZIE-3072
 URL: https://issues.apache.org/jira/browse/OOZIE-3072
 Project: Oozie
  Issue Type: Bug
Reporter: Peter Cseh


When Oozie process the files defined in 
oozie.service.HadoopAccessorService.action.configurations   it uses 
Configuration.injectDefauts(). This prevents overwriting things defined inside 
any of the configuration files Oozie is providing default values from (like 
yarn-site.xml, mapred-site.xml)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2714) Detect conflicting resources during class loading

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186058#comment-16186058
 ] 

Hadoop QA commented on OOZIE-2714:
--

Testing JIRA OOZIE-2714

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:red}-1{color} There are [2] new bugs found below threshold in total that 
must be fixed.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:red}-1{color} There are [2] new bugs found below threshold in 
[sharelib/oozie] that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones): 
sharelib/oozie/findbugs-new.html
. The most important FindBugs errors are:
. At ConflictDetectingClassLoader.java:[line 106]: File(...) reads a file whose 
location might be specified by user input
. Exception is caught when Exception is not thrown in 
org.apache.oozie.action.hadoop.ConflictDetectingClassLoader.loadClass(String): 
At ConflictDetectingClassLoader.java:[line 106]
. At ConflictDetectingClassLoader.java:[line 114]
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/58/

> Detect conflicting resources during class loading
> -
>
> Key: OOZIE-2714
> URL: https://issues.apache.org/jira/browse/OOZIE-2714
> Project: Oozie
>  Issue Type: New Feature
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: ClassLoaderTest.java, OOZIE-2714-POC01.patch
>
>
> There are a bunch of issues in Oozie which are related to class loading. 
> The main problem is that the classpath is constructed in a way which is very 
> specific to Oozie:
> - Hadoop lib jars
> - Sharelib jars
> - User-defined jars
> Sometimes there is a conflict between sharelib and hadoop lib version. Also, 
> users can add their own jars which sometimes contain a different version of 
> popular libraries such as Guava, Apache commons, etc.
> We should be able to detect these conflicts and print exact error message so 
> that Oozie users can take appropriate actions to resolve the problem.
> A possible approach is the following:
> * start the execution of an action on a different thread
> * replace the thread's context classloader with a classloader which can 
> detect conflicts
> * when the JVM invokes the {{loadClass()}} method of the classloader, i

Failed: OOZIE-2714 PreCommit Build #58

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-2714
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/58/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.54 MB...]
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 77 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
-1 There are [2] new bugs found below threshold in total that must be fixed.
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [sharelib/distcp].
 -1 There are [2] new bugs found below threshold in [sharelib/oozie] that must 
be fixed.
 You can find the FindBugs diff here (look for the red and orange ones): 
sharelib/oozie/findbugs-new.html
 The most important FindBugs errors are:
 At ConflictDetectingClassLoader.java:[line 106]: File(...) reads a file whose 
location might be specified by user input
 Exception is caught when Exception is not thrown in 
org.apache.oozie.action.hadoop.ConflictDetectingClassLoader.loadClass(String): 
At ConflictDetectingClassLoader.java:[line 106]
 At ConflictDetectingClassLoader.java:[line 114]
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/pig].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS - patch does not compile, cannot run testcases
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/58/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 3706k0 101360 0  19037  0  0:03:19 --:--:--  0:03:19 
19037100 3706k  100 3706k0 0  4906k  0 --:--:-- --:--:-- --:--:-- 
16.2M
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-2714
Archiving artifacts
[Fast Archiver] Compressed 1.80 MB of artifacts by 48.5% relative to #54
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Updated] (OOZIE-1401) PurgeCommand should purge the workflow jobs w/o end_time

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-1401:

Description: 
Currently, {{PurgeXCommand}} logic is not working with those workflow jobs with 
{{end_time=null}}. This command needs to take care of those jobs as well. This 
happens in the case of long stuck jobs after Hadoop restarts or DB failures. It 
could be done by checking {{last_modified_time}} instead, if {{end_time}} is 
not available.

The current query:
{code:sql}
select w from WorkflowJobBean w where w.endTimestamp < :endTime
{code}

There is also an issue when:
* there is a parent workflow that has its {{end_time}} set
* is otherwise eligible for {{PurgeXCommand}}: {{end_time}} is older than 
configured number of days, and has {{status}} either {{KILLED}}, or {{FAILED}}, 
or {{SUCCEEDED}}
* has a child workflow that has the {{parent_id}} set to the {{id}} of the 
parent workflow
* child workflow has its {{end_time = NULL}}

In this case, 
[*{{PurgeXCommand#fetchTerminatedWorkflow()}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/command/PurgeXCommand.java#L249]
 throws a {{NullPointerException}} like this:
{noformat}
2017-09-29 07:59:46,365 DEBUG org.apache.oozie.command.PurgeXCommand: 
SERVER[host-10-17-101-90.coe.cloudera.com] USER[-] GROUP[-] TOKEN[-] APP[-] 
JOB[-] ACTION[-] Purging workflows of long running coordinators is turned on
2017-09-29 07:59:46,371 DEBUG org.apache.oozie.command.PurgeXCommand: 
SERVER[host-10-17-101-90.coe.cloudera.com] USER[-] GROUP[-] TOKEN[-] APP[-] 
JOB[-] ACTION[-] Execute command [purge] key [null]
2017-09-29 07:59:46,371 INFO org.apache.oozie.command.PurgeXCommand: 
SERVER[host-10-17-101-90.coe.cloudera.com] USER[-] GROUP[-] TOKEN[-] APP[-] 
JOB[-] ACTION[-] STARTED Purge to purge Workflow Jobs older than [1] days, 
Coordinator Jobs older than [1] days, and Bundlejobs older than [1] days.
2017-09-29 07:59:46,375 ERROR org.apache.oozie.command.PurgeXCommand: 
SERVER[host-10-17-101-90.coe.cloudera.com] USER[-] GROUP[-] TOKEN[-] APP[-] 
JOB[-] ACTION[-] Exception, 
java.lang.NullPointerException
at 
org.apache.oozie.command.PurgeXCommand.fetchTerminatedWorkflow(PurgeXCommand.java:249)
at 
org.apache.oozie.command.PurgeXCommand.processWorkflowsHelper(PurgeXCommand.java:227)
at 
org.apache.oozie.command.PurgeXCommand.processWorkflows(PurgeXCommand.java:199)
at 
org.apache.oozie.command.PurgeXCommand.execute(PurgeXCommand.java:150)
at org.apache.oozie.command.PurgeXCommand.execute(PurgeXCommand.java:53)
at org.apache.oozie.command.XCommand.call(XCommand.java:286)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}

  was:
Currently, Purge logic is not working with those workflow jobs with 
end_time=null. This command needs to take care of those jobs as well. This 
happens in the case of long stuck jobs after Hadoop restarts or DB failures. It 
could be done by checking created_time if end_time is not available.

The current query:
select w from WorkflowJobBean w where w.endTimestamp < :endTime



> PurgeCommand should purge the workflow jobs w/o end_time
> 
>
> Key: OOZIE-1401
> URL: https://issues.apache.org/jira/browse/OOZIE-1401
> Project: Oozie
>  Issue Type: Sub-task
>  Components: bundle, coordinator, workflow
>Affects Versions: trunk
>Reporter: Mona Chitnis
>Assignee: Andras Piros
>
> Currently, {{PurgeXCommand}} logic is not working with those workflow jobs 
> with {{end_time=null}}. This command needs to take care of those jobs as 
> well. This happens in the case of long stuck jobs after Hadoop restarts or DB 
> failures. It could be done by checking {{last_modified_time}} instead, if 
> {{end_time}} is not available.
> The current query:
> {code:sql}
> select w from WorkflowJobBean w where w.endTimestamp < :endTime
> {code}
> There is also an issue when:
> * there is a parent workflow that has its {{end_time}} set
> * is otherwise eligible for {{PurgeXCommand}}: {{end_time}} is older than 
> configured number of days, and has {{status}} either {{KILLED}}, or 
> {{FAILED}}, or {{SUCCEEDED}}
> * has a child workflow that has the {{parent_id}} set to the {{id}} of the 
> parent workflow
> * child workflow has its {{end_time = NULL}}
> In this case, 
> [*{{PurgeXCommand#fetchTerminatedWorkflow()}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozi

[jira] [Assigned] (OOZIE-1401) PurgeCommand should purge the workflow jobs w/o end_time

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros reassigned OOZIE-1401:
---

Assignee: Andras Piros  (was: Jaydeep Vishwakarma)

> PurgeCommand should purge the workflow jobs w/o end_time
> 
>
> Key: OOZIE-1401
> URL: https://issues.apache.org/jira/browse/OOZIE-1401
> Project: Oozie
>  Issue Type: Sub-task
>  Components: bundle, coordinator, workflow
>Affects Versions: trunk
>Reporter: Mona Chitnis
>Assignee: Andras Piros
>
> Currently, Purge logic is not working with those workflow jobs with 
> end_time=null. This command needs to take care of those jobs as well. This 
> happens in the case of long stuck jobs after Hadoop restarts or DB failures. 
> It could be done by checking created_time if end_time is not available.
> The current query:
> select w from WorkflowJobBean w where w.endTimestamp < :endTime



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-1401) PurgeCommand should purge the workflow jobs w/o end_time

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185946#comment-16185946
 ] 

Andras Piros commented on OOZIE-1401:
-

[~jaydeepvishwakarma] taking this issue over, I have to fix that.

> PurgeCommand should purge the workflow jobs w/o end_time
> 
>
> Key: OOZIE-1401
> URL: https://issues.apache.org/jira/browse/OOZIE-1401
> Project: Oozie
>  Issue Type: Sub-task
>  Components: bundle, coordinator, workflow
>Affects Versions: trunk
>Reporter: Mona Chitnis
>Assignee: Jaydeep Vishwakarma
>
> Currently, Purge logic is not working with those workflow jobs with 
> end_time=null. This command needs to take care of those jobs as well. This 
> happens in the case of long stuck jobs after Hadoop restarts or DB failures. 
> It could be done by checking created_time if end_time is not available.
> The current query:
> select w from WorkflowJobBean w where w.endTimestamp < :endTime



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2714) Detect conflicting resources during class loading

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185916#comment-16185916
 ] 

Peter Bacsko commented on OOZIE-2714:
-

By the way - there were some concerns regarding the possible overhead of the 
duplication checking. The good news: it doesn't have any. At least it's so fast 
that's unnoticeable.

> Detect conflicting resources during class loading
> -
>
> Key: OOZIE-2714
> URL: https://issues.apache.org/jira/browse/OOZIE-2714
> Project: Oozie
>  Issue Type: New Feature
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: ClassLoaderTest.java, OOZIE-2714-POC01.patch
>
>
> There are a bunch of issues in Oozie which are related to class loading. 
> The main problem is that the classpath is constructed in a way which is very 
> specific to Oozie:
> - Hadoop lib jars
> - Sharelib jars
> - User-defined jars
> Sometimes there is a conflict between sharelib and hadoop lib version. Also, 
> users can add their own jars which sometimes contain a different version of 
> popular libraries such as Guava, Apache commons, etc.
> We should be able to detect these conflicts and print exact error message so 
> that Oozie users can take appropriate actions to resolve the problem.
> A possible approach is the following:
> * start the execution of an action on a different thread
> * replace the thread's context classloader with a classloader which can 
> detect conflicts
> * when the JVM invokes the {{loadClass()}} method of the classloader, it  
> scans through the jars (which are available as {{URLClassPath}} objects). If 
> it finds the given resource in at least two jars, it can do different things 
> depending on the setup:
> ** throws an error immediately, mentioning the conflicting jars (this is 
> probably too strict - but still an option)
> ** loads the two resource into a byte array and compares them - it only 
> throws an error if there is difference
> ** compares the jars but only emits an error message if there is a conflict
> ** something else (user defined action?)
> Implementing such a classloader is not difficult and would greatly enhance 
> the supportability of Oozie. It could work in multiple modes depending on the 
> setup - perhaps being able to control it from a workflow config is desirable. 
> If there's any problem, we should be able to turn it off completely, too.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (OOZIE-2714) Detect conflicting resources during class loading

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185895#comment-16185895
 ] 

Peter Bacsko edited comment on OOZIE-2714 at 9/29/17 2:23 PM:
--

Ping [~andras.piros], [~gezapeti], [~asasvari], [~rkanter] - what do you guys 
think about this approach?


was (Author: pbacsko):
Ping [~andras.piros], [~gezapeti], [~rkanter] - what do you guys think about 
this approach?

> Detect conflicting resources during class loading
> -
>
> Key: OOZIE-2714
> URL: https://issues.apache.org/jira/browse/OOZIE-2714
> Project: Oozie
>  Issue Type: New Feature
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: ClassLoaderTest.java, OOZIE-2714-POC01.patch
>
>
> There are a bunch of issues in Oozie which are related to class loading. 
> The main problem is that the classpath is constructed in a way which is very 
> specific to Oozie:
> - Hadoop lib jars
> - Sharelib jars
> - User-defined jars
> Sometimes there is a conflict between sharelib and hadoop lib version. Also, 
> users can add their own jars which sometimes contain a different version of 
> popular libraries such as Guava, Apache commons, etc.
> We should be able to detect these conflicts and print exact error message so 
> that Oozie users can take appropriate actions to resolve the problem.
> A possible approach is the following:
> * start the execution of an action on a different thread
> * replace the thread's context classloader with a classloader which can 
> detect conflicts
> * when the JVM invokes the {{loadClass()}} method of the classloader, it  
> scans through the jars (which are available as {{URLClassPath}} objects). If 
> it finds the given resource in at least two jars, it can do different things 
> depending on the setup:
> ** throws an error immediately, mentioning the conflicting jars (this is 
> probably too strict - but still an option)
> ** loads the two resource into a byte array and compares them - it only 
> throws an error if there is difference
> ** compares the jars but only emits an error message if there is a conflict
> ** something else (user defined action?)
> Implementing such a classloader is not difficult and would greatly enhance 
> the supportability of Oozie. It could work in multiple modes depending on the 
> setup - perhaps being able to control it from a workflow config is desirable. 
> If there's any problem, we should be able to turn it off completely, too.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2714) Detect conflicting resources during class loading

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185895#comment-16185895
 ] 

Peter Bacsko commented on OOZIE-2714:
-

Ping [~andras.piros], [~gezapeti], [~rkanter] - what do you guys think about 
this approach?

> Detect conflicting resources during class loading
> -
>
> Key: OOZIE-2714
> URL: https://issues.apache.org/jira/browse/OOZIE-2714
> Project: Oozie
>  Issue Type: New Feature
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: ClassLoaderTest.java, OOZIE-2714-POC01.patch
>
>
> There are a bunch of issues in Oozie which are related to class loading. 
> The main problem is that the classpath is constructed in a way which is very 
> specific to Oozie:
> - Hadoop lib jars
> - Sharelib jars
> - User-defined jars
> Sometimes there is a conflict between sharelib and hadoop lib version. Also, 
> users can add their own jars which sometimes contain a different version of 
> popular libraries such as Guava, Apache commons, etc.
> We should be able to detect these conflicts and print exact error message so 
> that Oozie users can take appropriate actions to resolve the problem.
> A possible approach is the following:
> * start the execution of an action on a different thread
> * replace the thread's context classloader with a classloader which can 
> detect conflicts
> * when the JVM invokes the {{loadClass()}} method of the classloader, it  
> scans through the jars (which are available as {{URLClassPath}} objects). If 
> it finds the given resource in at least two jars, it can do different things 
> depending on the setup:
> ** throws an error immediately, mentioning the conflicting jars (this is 
> probably too strict - but still an option)
> ** loads the two resource into a byte array and compares them - it only 
> throws an error if there is difference
> ** compares the jars but only emits an error message if there is a conflict
> ** something else (user defined action?)
> Implementing such a classloader is not difficult and would greatly enhance 
> the supportability of Oozie. It could work in multiple modes depending on the 
> setup - perhaps being able to control it from a workflow config is desirable. 
> If there's any problem, we should be able to turn it off completely, too.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2714) Detect conflicting resources during class loading

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185889#comment-16185889
 ] 

Peter Bacsko commented on OOZIE-2714:
-

I've worked on this a little bit recently.

I ran into a bunch of tricky problems. But this POC seems to do the job. It did 
find the conflicts when I inserted a near-duplication of a Guava class into the 
sharelib.

I don't know if it's possible to do this simpler. I tried to mess around with 
overriding only {{findClass()}} but that didn't work. Looks like it's necessary 
to invoke {{defineClass()}} which enforces that all dependencies of the 
initially loaded class (in our case, {{LauncherAM}}) will subsequently be 
loaded by the custom classloader, not the default AppClassLoader. Anyway, it 
was fun :) BTW it was also necessary to add a new main class which loads 
{{LauncherAM}} with reflection.

There is a second approach - use {{-Djava.system.class.loader}} in the command 
line but that seems to be more problematic. It's because we have to check 
{{java.class.path}} and construct the {{URL[]}} array by hand. To do this, we 
either re-use code from {{sun.misc.*}} packages or copy-paste some code from 
there to do it the same way. Neither seems to be a convincing solution, so I'd 
vote for the reflection-based approach.

> Detect conflicting resources during class loading
> -
>
> Key: OOZIE-2714
> URL: https://issues.apache.org/jira/browse/OOZIE-2714
> Project: Oozie
>  Issue Type: New Feature
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: ClassLoaderTest.java, OOZIE-2714-POC01.patch
>
>
> There are a bunch of issues in Oozie which are related to class loading. 
> The main problem is that the classpath is constructed in a way which is very 
> specific to Oozie:
> - Hadoop lib jars
> - Sharelib jars
> - User-defined jars
> Sometimes there is a conflict between sharelib and hadoop lib version. Also, 
> users can add their own jars which sometimes contain a different version of 
> popular libraries such as Guava, Apache commons, etc.
> We should be able to detect these conflicts and print exact error message so 
> that Oozie users can take appropriate actions to resolve the problem.
> A possible approach is the following:
> * start the execution of an action on a different thread
> * replace the thread's context classloader with a classloader which can 
> detect conflicts
> * when the JVM invokes the {{loadClass()}} method of the classloader, it  
> scans through the jars (which are available as {{URLClassPath}} objects). If 
> it finds the given resource in at least two jars, it can do different things 
> depending on the setup:
> ** throws an error immediately, mentioning the conflicting jars (this is 
> probably too strict - but still an option)
> ** loads the two resource into a byte array and compares them - it only 
> throws an error if there is difference
> ** compares the jars but only emits an error message if there is a conflict
> ** something else (user defined action?)
> Implementing such a classloader is not difficult and would greatly enhance 
> the supportability of Oozie. It could work in multiple modes depending on the 
> setup - perhaps being able to control it from a workflow config is desirable. 
> If there's any problem, we should be able to turn it off completely, too.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2714) Detect conflicting resources during class loading

[Peter Bacsko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Bacsko updated OOZIE-2714:

Attachment: OOZIE-2714-POC01.patch

> Detect conflicting resources during class loading
> -
>
> Key: OOZIE-2714
> URL: https://issues.apache.org/jira/browse/OOZIE-2714
> Project: Oozie
>  Issue Type: New Feature
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: ClassLoaderTest.java, OOZIE-2714-POC01.patch
>
>
> There are a bunch of issues in Oozie which are related to class loading. 
> The main problem is that the classpath is constructed in a way which is very 
> specific to Oozie:
> - Hadoop lib jars
> - Sharelib jars
> - User-defined jars
> Sometimes there is a conflict between sharelib and hadoop lib version. Also, 
> users can add their own jars which sometimes contain a different version of 
> popular libraries such as Guava, Apache commons, etc.
> We should be able to detect these conflicts and print exact error message so 
> that Oozie users can take appropriate actions to resolve the problem.
> A possible approach is the following:
> * start the execution of an action on a different thread
> * replace the thread's context classloader with a classloader which can 
> detect conflicts
> * when the JVM invokes the {{loadClass()}} method of the classloader, it  
> scans through the jars (which are available as {{URLClassPath}} objects). If 
> it finds the given resource in at least two jars, it can do different things 
> depending on the setup:
> ** throws an error immediately, mentioning the conflicting jars (this is 
> probably too strict - but still an option)
> ** loads the two resource into a byte array and compares them - it only 
> throws an error if there is difference
> ** compares the jars but only emits an error message if there is a conflict
> ** something else (user defined action?)
> Implementing such a classloader is not difficult and would greatly enhance 
> the supportability of Oozie. It could work in multiple modes depending on the 
> setup - perhaps being able to control it from a workflow config is desirable. 
> If there's any problem, we should be able to turn it off completely, too.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (OOZIE-2714) Detect conflicting resources during class loading

[Peter Bacsko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Bacsko reassigned OOZIE-2714:
---

Assignee: Peter Bacsko

> Detect conflicting resources during class loading
> -
>
> Key: OOZIE-2714
> URL: https://issues.apache.org/jira/browse/OOZIE-2714
> Project: Oozie
>  Issue Type: New Feature
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: ClassLoaderTest.java, OOZIE-2714-POC01.patch
>
>
> There are a bunch of issues in Oozie which are related to class loading. 
> The main problem is that the classpath is constructed in a way which is very 
> specific to Oozie:
> - Hadoop lib jars
> - Sharelib jars
> - User-defined jars
> Sometimes there is a conflict between sharelib and hadoop lib version. Also, 
> users can add their own jars which sometimes contain a different version of 
> popular libraries such as Guava, Apache commons, etc.
> We should be able to detect these conflicts and print exact error message so 
> that Oozie users can take appropriate actions to resolve the problem.
> A possible approach is the following:
> * start the execution of an action on a different thread
> * replace the thread's context classloader with a classloader which can 
> detect conflicts
> * when the JVM invokes the {{loadClass()}} method of the classloader, it  
> scans through the jars (which are available as {{URLClassPath}} objects). If 
> it finds the given resource in at least two jars, it can do different things 
> depending on the setup:
> ** throws an error immediately, mentioning the conflicting jars (this is 
> probably too strict - but still an option)
> ** loads the two resource into a byte array and compares them - it only 
> throws an error if there is difference
> ** compares the jars but only emits an error message if there is a conflict
> ** something else (user defined action?)
> Implementing such a classloader is not difficult and would greatly enhance 
> the supportability of Oozie. It could work in multiple modes depending on the 
> setup - perhaps being able to control it from a workflow config is desirable. 
> If there's any problem, we should be able to turn it off completely, too.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3071) Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0

[Artem Ervits (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185847#comment-16185847
 ] 

Artem Ervits commented on OOZIE-3071:
-

[~rangu] not sure why you're getting failures, I bumped the commons-lang3 to 
3.6 in pom.xml only and compiled with 
{code}
bin/mkdistro.sh -DjavaVersion=1.8 -fae -Dmaven.test.failure.ignore=true 
-Dspark.version=2.2.0 -DskipTests=true
{code}
I didn't get any failures. Perhaps you'd want to try that? Btw, I'm using the 
master branch.

My only change is
{noformat}
diff --git a/pom.xml b/pom.xml
index efccc346..492e7f62 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1308,7 +1308,7 @@
 
 org.apache.commons
 commons-lang3
-3.3.2
+3.6
 

 
{noformat}

> Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than 
> Spark 2.2.0
> 
>
> Key: OOZIE-3071
> URL: https://issues.apache.org/jira/browse/OOZIE-3071
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.3.0
>Reporter: Ran Gu
>  Labels: patch
> Attachments: OOZIE-3071.patch
>
>
> Currently Oozie 4.3.0 uses commons-lang3 version 3.3.2 in Spark sharelib.
> Spark 2.2.0 uses commons-lang3 version 3.5. 
> This causes Oozie(/Spark) job failures on EMR-5.8.0 clusters.
> Error message:
> 17/08/22 00:22:43 ERROR ApplicationMaster: User class threw exception: 
> java.lang.IllegalArgumentException: Illegal pattern component: XXX
> java.lang.IllegalArgumentException: Illegal pattern component: XXX
> at 
> org.apache.commons.lang3.time.FastDatePrinter.parsePattern(FastDatePrinter.java:282)
> at 
> org.apache.commons.lang3.time.FastDatePrinter.init(FastDatePrinter.java:149)
> at 
> org.apache.commons.lang3.time.FastDatePrinter.(FastDatePrinter.java:142)
> at 
> org.apache.commons.lang3.time.FastDateFormat.(FastDateFormat.java:384)
> at 
> org.apache.commons.lang3.time.FastDateFormat.(FastDateFormat.java:369)
> at 
> org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:91)
> at 
> org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:88)
> at org.apache.commons.lang3.time.FormatCache.getInstance(FormatCache.java:82) 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

[Peter Cseh]

> On Sept. 29, 2017, 8:17 a.m., András Piros wrote:
> > tools/src/main/bin/oozie-diag-bundle-collector.sh
> > Lines 55-58 (patched)
> > 
> >
> > Extract function.

I don't like the idea of extracting all these functions in a shell script. None 
of the extracted functions would deduplicate code and none of the will be used 
in other shell scripts either. I find it way easier just to read the shell 
script as it is rather that following function calls in it.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186568
---


On Sept. 27, 2017, 2:12 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Sept. 27, 2017, 2:12 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml efccc346932514ada578a3462eb3c3cfe519a323 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/BundleCollectorDriver.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/BundleCompressor.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/BundleEntryWriter.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/Client.java PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/7/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

[jira] [Commented] (OOZIE-3070) Remove references to org.mortbay.jetty

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185531#comment-16185531
 ] 

Peter Bacsko commented on OOZIE-3070:
-

Thanks everyone, committed to master

> Remove references to org.mortbay.jetty
> --
>
> Key: OOZIE-3070
> URL: https://issues.apache.org/jira/browse/OOZIE-3070
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Fix For: 5.0.0
>
> Attachments: OOZIE-3070-001.patch, OOZIE-3070-002.patch, 
> OOZIE-3070-003.patch
>
>
> A couple of months ago we migrated from Tomcat to Jetty.
> However, Oozie as a production code uses {{org.eclipse.jetty}}, whereas 
> {{EmbeddedServletContainer}} still uses code from {{org.mortbay.jetty}}. We 
> don't need two Jettys, so all remaining code should use the newest Jetty.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2296) Add an Oozie diagnostic bundle tool

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185490#comment-16185490
 ] 

Andras Piros commented on OOZIE-2296:
-

[~asasvari] left some comments on RB.

> Add an Oozie diagnostic bundle tool
> ---
>
> Key: OOZIE-2296
> URL: https://issues.apache.org/jira/browse/OOZIE-2296
> Project: Oozie
>  Issue Type: New Feature
>  Components: tools
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Attachments: OOZIE-2296-002.patch, OOZIE-2296-003.patch, 
> OOZIE-2296-004.patch, OOZIE-2296-005.patch, OOZIE-2296-006.patch, 
> OOZIE-2296-009.patch, OOZIE-2296-010.patch
>
>
> To help with our support cases, I've built a tool that collects a bunch of 
> job and other information from Oozie that throws it all in a tarball.  The 
> idea is that the user can just click a button in Cloudera Manager, and it 
> will run this tool.  
> This tool could be useful for others as an easy way to get information out of 
> Oozie, so I thought I'd contribute it here.  It's built as a "tool" (so it 
> sits next to the sharelib and database tools), and simply uses the Oozie 
> client for getting pretty much everything, so it doesn't require anything 
> special.
> Here's the information that it can get:
> # Sharelib: {{ooze admin -shareliblist}} and {{oozie admin -shareliblist 
> }}
> # Oozie Server's resolved loaded configuration (from admin endpoint)
> # Other admin commands output (queue dump, env vars, etc)
> # Thread dump (HOST:11000/oozie/admin/jvminfo.jsp)
> # Details from last n jobs and/or specific list of jobs
> #- job.properties contents
> #- XML definition
> #- verbose status for each job and each action etc
> #- Oozie logs
> #- Unfortunately, we can't get the launcher jobs' logs from Hadoop
> # Metrics/Instrumentation



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

[András Piros]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186568
---




tools/src/main/bin/oozie-diag-bundle-collector.sh
Lines 23-31 (patched)


Extract function.



tools/src/main/bin/oozie-diag-bundle-collector.sh
Lines 33-34 (patched)


Extract function.



tools/src/main/bin/oozie-diag-bundle-collector.sh
Lines 36 (patched)


`OOZIECP` or `OOZIECLASSPATH` would be a better name.



tools/src/main/bin/oozie-diag-bundle-collector.sh
Lines 37-45 (patched)


Extract function.



tools/src/main/bin/oozie-diag-bundle-collector.sh
Lines 48-53 (patched)


Extract function.



tools/src/main/bin/oozie-diag-bundle-collector.sh
Lines 55-58 (patched)


Extract function.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 51 (patched)


`MAX_TIMEOUT_SECONDS` would be a better name.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 60 (patched)


`storeWorkflowJobDetails()` might be a better name.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 67-73 (patched)


This is happening three times in code. Extract to one and reuse.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 69 (patched)


Returning here and having no `else` path seems cleaner to me.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 71-77 (patched)


Extract method.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 76 (patched)


`"Could not create resolved actions directory:"`



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 81-83 (patched)


Nice :)



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 88 (patched)


`"Got details for ... successfully"`, maybe?



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 93 (patched)


Maybe printing only the `Exception` message is a better idea.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 97-98 (patched)


Put `resolvedActionsDir` and `configEntryWriter` to fields, and you got two 
parameters less.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 99-119 (patched)


Using a `StringBuilder`, maybe?



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 118 (patched)


`ACTIONS`



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 120 (patched)


Put this just before `while`.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 124-127 (patched)


Substitute w/ a normal `while` loop might be more readable.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 128-149 (patched)


Using a `StringBuilder`, maybe?



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 158-160 (patched)


`persistWorkflowAction()`



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 170 (patched)


It could implement `Runnable`, in which case you don't need to `return 
null`.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 182 (patched)


`StandardCharsets.UTF_8.toString()` seems cleaner to me.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 189 (patched)


`IllegalStateException`



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 194 (patched)