[jira] [Updated] (OOZIE-3077) SLA documentation fixes

[Satish Subhashrao Saley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-3077:
---
Attachment: OOZIE-3077-1.patch

> SLA documentation fixes
> ---
>
> Key: OOZIE-3077
> URL: https://issues.apache.org/jira/browse/OOZIE-3077
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Minor
> Attachments: OOZIE-3077-1.patch
>
>
> Following properties can be specified in job.xml -> Following properties can 
> be specified in job.xml, job.properties
> Default is 24 hours -> Default is 48 hours
> oozie.sla.disable.alerts -> oozie.sla.enable.alerts for property name in the 
> example configuration having 2009-01-01T01:00Z::2009-05-31T23:59Z as value



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3062) Set HADOOP_CONF_DIR for spark action

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190569#comment-16190569
 ] 

Hadoop QA commented on OOZIE-3062:
--

Testing JIRA OOZIE-3062

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2034
.Tests rerun: 80
.Tests failed at first run: 
org.apache.oozie.command.coord.TestCoordActionsKillXCommand,org.apache.oozie.command.coord.TestCoordActionInputCheckXCommandNonUTC,org.apache.oozie.command.TestXCommand,org.apache.oozie.action.hadoop.TestJavaActionExecutor,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/74/

> Set HADOOP_CONF_DIR for spark action
> 
>
> Key: OOZIE-3062
> URL: https://issues.apache.org/jira/browse/OOZIE-3062
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3062-1.patch, OOZIE-3062-2.patch, 
> OOZIE-3062-3.patch, OOZIE-3062-4.patch
>
>
> OOZIE-2569 created core-site.xml, hdfs-site.xml, mapred-site.xml, 
> yarn-site.xml for spark action. But for spark to consider these files ([spark 
> code 
> ref|https://github.com/apache/spark/blob/83fe3b5e10f8dc62245ea37143abb96be0f39805/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala#L284]),
>  we need to set HADOOP_CONF_DIR environmental variable, otherwise it will 
> point to default HADOOP_CONF_DIR configured on the node. Therefore, Oozie 
> should set HADOOP_CONF_DIR.
> Due to a bug YARN-4727, yarn was not considering user specified value for 
> whiltelisted environment variables including HADOOP_CONF_DIR. So, this will 
> work on specified hadoop fix versions in YARN-4727.
> This fix in Oozie will be no-op on Hadoop without YARN-4727 fix and Oozie 
> will behave as it is.
> To verify this on one node machine. 
> 1. In hadoop mapred-site.xml, set 
> {code}
> 
>mapreduce.fileoutputcommitter.marksuccessfuljobs
>false
> 
> {code}
> 2. Run any spark file copy example from Oozie. Check output directory, 
> _SUCCESS flag is missing. It should be there because we explicitly set 
> conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "true"); in 
> 

Success: OOZIE-3062 PreCommit Build #74

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-3062
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/74/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.59 MB...]
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 77 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/pig].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2034
Tests rerun: 80
Tests failed at first run: 
org.apache.oozie.command.coord.TestCoordActionsKillXCommand,org.apache.oozie.command.coord.TestCoordActionInputCheckXCommandNonUTC,org.apache.oozie.command.TestXCommand,org.apache.oozie.action.hadoop.TestJavaActionExecutor,
+1 DISTRO
+1 distro tarball builds with the patch 


+1 Overall result, good!, no -1s

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/74/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
 67 3706k   67 2512k0 0  3232k  0  0:00:01 --:--:--  0:00:01 
3232k100 3706k  100 3706k0 0  4645k  0 --:--:-- --:--:-- --:--:-- 
58.2M
Adding comment to JIRA
Comment added.

test-patch exit code: 0

[description-setter] Description set: OOZIE-3062
Archiving artifacts
[Fast Archiver] Compressed 1.79 MB of artifacts by 67.9% relative to #71
Recording test results
Email was triggered for: Success
Sending email for trigger: Success



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190525#comment-16190525
 ] 

Hadoop QA commented on OOZIE-2406:
--

Testing JIRA OOZIE-2406

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 1 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 6 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:orange}0{color} There are [1] new bugs found in total that would be nice 
to have fixed.
. {color:orange}0{color} There are [1] new bugs found in [core] that would be 
nice to have fixed.
. You can find the FindBugs diff here: core/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [examples].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2039
.Tests failed: 0
.Tests errors: 1

.The patch failed the following testcases:

.  

.Tests failing with errors:
.  testJMXInstrumentation(org.apache.oozie.util.TestMetricsInstrumentation)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/73/

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png, 
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png, 
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch, 
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch, 
> OOZIE-2406.007.patch, OOZIE-2406.008.patch, OOZIE-2406.009.patch, 
> OOZIE-2406.010.patch, Screen Shot 2017-10-02 at 1.55.58 PM.png, Screen Shot 
> 2017-10-02 at 1.56.33 PM.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which 

Failed: OOZIE-2406 PreCommit Build #73

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-2406
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/73/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.99 MB...]
 0 There are [1] new bugs found in [core] that would be nice to have fixed.
 You can find the FindBugs diff here: core/findbugs-new.html
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [examples].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS
Tests run: 2039
Tests failed: 0
Tests errors: 1

The patch failed the following testcases:

  

Tests failing with errors:
  testJMXInstrumentation(org.apache.oozie.util.TestMetricsInstrumentation)

+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/73/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  0 
00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
100 3706k  100 3706k0 0  3676k  0  0:00:01  0:00:01 --:--:-- 3676k
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-2406
Archiving artifacts
[Fast Archiver] Compressed 2.19 MB of artifacts by 48.5% relative to #71
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
1 tests failed.
FAILED:  org.apache.oozie.util.TestMetricsInstrumentation.testJMXInstrumentation

Error Message:
Unable to open socket file: target process not responding or HotSpot VM not 
loaded

Stack Trace:
com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: 
target process not responding or HotSpot VM not loaded
at 
sun.tools.attach.LinuxVirtualMachine.(LinuxVirtualMachine.java:106)
at 
sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:78)
at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:250)
at 
org.apache.oozie.util.TestMetricsInstrumentation.testJMXInstrumentation(TestMetricsInstrumentation.java:235)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at 
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 

[jira] [Commented] (OOZIE-2896) Ensure compatibility for existing LauncherMapper settings

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190510#comment-16190510
 ] 

Robert Kanter commented on OOZIE-2896:
--

+1 with Rohini's suggestions

I assume that when there's an equivalent XML field in the new 1.0 schema set, 
that has the highest priority over any properties, right?  

We should also make sure to document this somewhere, including the list of 
properties.  Like what you have in your comment.

> Ensure compatibility for existing LauncherMapper settings
> -
>
> Key: OOZIE-2896
> URL: https://issues.apache.org/jira/browse/OOZIE-2896
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Bacsko
>Assignee: Andras Piros
>Priority: Blocker
>
> There are a couple of LauncherMapper settings that we should support for 
> backward compatibility reasons.
> For example, {{oozie.launcher.mapreduce.map.memory.mb}}, 
> {{mapred.child.env}}, {{mapred.map.env}}, etc.
> We also have to discuss the list of properties we're going to support, since 
> there are many of them.
> List of hadoop properties under this task:
> mapreduce.job.queuename
> mapreduce.map.memory.mb
> mapreduce.map.java.opts
> mapreduce.map.env
> mapreduce.job.priority
> yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms
> yarn.app.mapreduce.am.resource.mb
> yarn.app.mapreduce.am.command-opts
> yarn.app.mapreduce.am.env
> ACL related properties should go under OOZIE-2897:
> mapreduce.job.acl-view-job
> mapreduce.job.acl-modfy-job



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

[Robert Kanter]

> On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 221 (patched)
> > 
> >
> > This won't work right if using RM HA...
> > 
> > I'd recommend using a YarnClient and passing it the 
> > hadoopConfig so it can figure out the RM address for you.  There 
> > must be a benign simple YarnClient command you can run to verify 
> > connectivity.
> 
> Attila Sasvari wrote:
> I could not find such command, but please let me know which one do you 
> think of. Methods I tried retried to connect to RM multiple times for more 
> minutes in case of a connection error.

Ya, apparently there isn't really one.  Hadoop 3 has a CLI command that would 
tell you the state of each RM 
(https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAAdmin.java#L481)
  Maybe you can steal something from it?  I'm not really sure of a good 
solution here.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186754
---


On Oct. 2, 2017, 9:39 a.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Oct. 2, 2017, 9:39 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
>   
> tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCollectorDriver.java
>  PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCompressor.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleEntryWriter.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagOozieClient.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/9/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 62470: OOZIE-3062 Set HADOOP_CONF_DIR for spark action

[Rohini Palaniswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62470/#review187014
---




core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java
Lines 149-151 (original)


I missed this earlier in the review. If we have this, then we should set 
the env here (will require method signature change) and remove the earlier code 
for setting SPARK_HOME that was done in setupLauncherConf as well.


- Rohini Palaniswamy


On Oct. 3, 2017, 9:52 p.m., Satish Saley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62470/
> ---
> 
> (Updated Oct. 3, 2017, 9:52 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-3062
> https://issues.apache.org/jira/browse/OOZIE-3062
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-3062 Set HADOOP_CONF_DIR for spark action
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 76d0daa4 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 80d64ec8 
>   
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java
>  d97f1f06 
> 
> 
> Diff: https://reviews.apache.org/r/62470/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Satish Saley
> 
>

[jira] [Commented] (OOZIE-3071) Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190470#comment-16190470
 ] 

Hadoop QA commented on OOZIE-3071:
--

Testing JIRA OOZIE-3071

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/72/

> Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than 
> Spark 2.2.0
> 
>
> Key: OOZIE-3071
> URL: https://issues.apache.org/jira/browse/OOZIE-3071
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.3.0
>Reporter: Ran Gu
>  Labels: patch
> Attachments: OOZIE-3071-1.patch, OOZIE-3071.patch
>
>
> Currently Oozie 4.3.0 uses commons-lang3 version 3.3.2 in Spark sharelib.
> Spark 2.2.0 uses commons-lang3 version 3.5. 
> This causes Oozie(/Spark) job failures on EMR-5.8.0 clusters.
> Error message:
> 17/08/22 00:22:43 ERROR ApplicationMaster: User class threw exception: 
> java.lang.IllegalArgumentException: Illegal pattern component: XXX
> java.lang.IllegalArgumentException: Illegal pattern component: XXX
> at 
> org.apache.commons.lang3.time.FastDatePrinter.parsePattern(FastDatePrinter.java:282)
> at 
> org.apache.commons.lang3.time.FastDatePrinter.init(FastDatePrinter.java:149)
> at 
> org.apache.commons.lang3.time.FastDatePrinter.(FastDatePrinter.java:142)
> at 
> org.apache.commons.lang3.time.FastDateFormat.(FastDateFormat.java:384)
> at 
> org.apache.commons.lang3.time.FastDateFormat.(FastDateFormat.java:369)
> at 
> org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:91)
> at 
> org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:88)
> at org.apache.commons.lang3.time.FormatCache.getInstance(FormatCache.java:82) 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-3071 PreCommit Build #72

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-3071
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/72/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.53 MB...]
+1 the patch does not introduce any line longer than 132
-1 the patch does not add/modify any testcase
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 77 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/spark].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS - patch does not compile, cannot run testcases
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/72/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
 43 3706k   43 1604k0 0  2117k  0  0:00:01 --:--:--  0:00:01 
2117k100 3706k  100 3706k0 0  4479k  0 --:--:-- --:--:-- --:--:-- 
29.7M
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-3071
Archiving artifacts
[Fast Archiver] Compressed 1.80 MB of artifacts by 64.4% relative to #71
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Created] (OOZIE-3077) SLA documentation fixes

[Satish Subhashrao Saley (JIRA)]

Satish Subhashrao Saley created OOZIE-3077:
--

 Summary: SLA documentation fixes
 Key: OOZIE-3077
 URL: https://issues.apache.org/jira/browse/OOZIE-3077
 Project: Oozie
  Issue Type: Bug
Reporter: Satish Subhashrao Saley
Assignee: Satish Subhashrao Saley
Priority: Minor



Following properties can be specified in job.xml -> Following properties can be 
specified in job.xml, job.properties
Default is 24 hours -> Default is 48 hours
oozie.sla.disable.alerts -> oozie.sla.enable.alerts for property name in the 
example configuration having 2009-01-01T01:00Z::2009-05-31T23:59Z as value




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3062) Set HADOOP_CONF_DIR for spark action

[Satish Subhashrao Saley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-3062:
---
Attachment: OOZIE-3062-4.patch

> Set HADOOP_CONF_DIR for spark action
> 
>
> Key: OOZIE-3062
> URL: https://issues.apache.org/jira/browse/OOZIE-3062
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3062-1.patch, OOZIE-3062-2.patch, 
> OOZIE-3062-3.patch, OOZIE-3062-4.patch
>
>
> OOZIE-2569 created core-site.xml, hdfs-site.xml, mapred-site.xml, 
> yarn-site.xml for spark action. But for spark to consider these files ([spark 
> code 
> ref|https://github.com/apache/spark/blob/83fe3b5e10f8dc62245ea37143abb96be0f39805/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala#L284]),
>  we need to set HADOOP_CONF_DIR environmental variable, otherwise it will 
> point to default HADOOP_CONF_DIR configured on the node. Therefore, Oozie 
> should set HADOOP_CONF_DIR.
> Due to a bug YARN-4727, yarn was not considering user specified value for 
> whiltelisted environment variables including HADOOP_CONF_DIR. So, this will 
> work on specified hadoop fix versions in YARN-4727.
> This fix in Oozie will be no-op on Hadoop without YARN-4727 fix and Oozie 
> will behave as it is.
> To verify this on one node machine. 
> 1. In hadoop mapred-site.xml, set 
> {code}
> 
>mapreduce.fileoutputcommitter.marksuccessfuljobs
>false
> 
> {code}
> 2. Run any spark file copy example from Oozie. Check output directory, 
> _SUCCESS flag is missing. It should be there because we explicitly set 
> conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "true"); in 
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L293



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62470: OOZIE-3062 Set HADOOP_CONF_DIR for spark action

[Satish Saley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62470/
---

(Updated Oct. 3, 2017, 2:52 p.m.)


Review request for oozie.


Bugs: OOZIE-3062
https://issues.apache.org/jira/browse/OOZIE-3062


Repository: oozie-git


Description
---

OOZIE-3062 Set HADOOP_CONF_DIR for spark action


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
76d0daa4 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
80d64ec8 
  
sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java
 d97f1f06 


Diff: https://reviews.apache.org/r/62470/diff/4/

Changes: https://reviews.apache.org/r/62470/diff/3-4/


Testing
---


Thanks,

Satish Saley

[jira] [Commented] (OOZIE-3062) Set HADOOP_CONF_DIR for spark action

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190408#comment-16190408
 ] 

Hadoop QA commented on OOZIE-3062:
--

Testing JIRA OOZIE-3062

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2034
.Tests rerun: 71
.Tests failed at first run: 
org.apache.oozie.command.coord.TestCoordActionInputCheckXCommandNonUTC,org.apache.oozie.action.hadoop.TestJavaActionExecutor,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/71/

> Set HADOOP_CONF_DIR for spark action
> 
>
> Key: OOZIE-3062
> URL: https://issues.apache.org/jira/browse/OOZIE-3062
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3062-1.patch, OOZIE-3062-2.patch, 
> OOZIE-3062-3.patch
>
>
> OOZIE-2569 created core-site.xml, hdfs-site.xml, mapred-site.xml, 
> yarn-site.xml for spark action. But for spark to consider these files ([spark 
> code 
> ref|https://github.com/apache/spark/blob/83fe3b5e10f8dc62245ea37143abb96be0f39805/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala#L284]),
>  we need to set HADOOP_CONF_DIR environmental variable, otherwise it will 
> point to default HADOOP_CONF_DIR configured on the node. Therefore, Oozie 
> should set HADOOP_CONF_DIR.
> Due to a bug YARN-4727, yarn was not considering user specified value for 
> whiltelisted environment variables including HADOOP_CONF_DIR. So, this will 
> work on specified hadoop fix versions in YARN-4727.
> This fix in Oozie will be no-op on Hadoop without YARN-4727 fix and Oozie 
> will behave as it is.
> To verify this on one node machine. 
> 1. In hadoop mapred-site.xml, set 
> {code}
> 
>mapreduce.fileoutputcommitter.marksuccessfuljobs
>false
> 
> {code}
> 2. Run any spark file copy example from Oozie. Check output directory, 
> _SUCCESS flag is missing. It should be there because we explicitly set 
> conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "true"); in 
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L293



--

Success: OOZIE-3062 PreCommit Build #71

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-3062
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/71/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.59 MB...]
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 77 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/pig].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2034
Tests rerun: 71
Tests failed at first run: 
org.apache.oozie.command.coord.TestCoordActionInputCheckXCommandNonUTC,org.apache.oozie.action.hadoop.TestJavaActionExecutor,
+1 DISTRO
+1 distro tarball builds with the patch 


+1 Overall result, good!, no -1s

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/71/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100 3706k  100 3706k0 0  4458k  0 --:--:-- --:--:-- --:--:-- 7324k
Adding comment to JIRA
Comment added.

test-patch exit code: 0

[description-setter] Description set: OOZIE-3062
Archiving artifacts
[Fast Archiver] Compressed 1.79 MB of artifacts by 62.7% relative to #69
Recording test results
Email was triggered for: Success
Sending email for trigger: Success



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Updated] (OOZIE-2406) Completely rewrite GraphGenerator code

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2406:

Attachment: OOZIE-2406.010.patch

Addressed another comment round.

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png, 
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png, 
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch, 
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch, 
> OOZIE-2406.007.patch, OOZIE-2406.008.patch, OOZIE-2406.009.patch, 
> OOZIE-2406.010.patch, Screen Shot 2017-10-02 at 1.55.58 PM.png, Screen Shot 
> 2017-10-02 at 1.56.33 PM.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62352: OOZIE-2406 Completely rewrite GraphGenerator code

[András Piros]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62352/
---

(Updated Oct. 3, 2017, 9:05 p.m.)


Review request for oozie and Robert Kanter.


Changes
---

Addressed another comment round.


Repository: oozie-git


Description
---

OOZIE-2406 Completely rewrite GraphGenerator code


Diffs (updated)
-

  client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
74843dc9a3cbb8cef38e97f81e34731f191d9aab 
  core/pom.xml 6f9adb66af9344ac7d2212cdc31aa203ec06c286 
  core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 
059d3cf6dc251b49940af29d82cbdd817043a176 
  core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 
10812c675ebc0cc4aefca9f4a85ef2fc26d143e1 
  core/src/main/java/org/apache/oozie/util/GraphGenerator.java 
6ded2c6dc15c9e8453ff800407ff0324be185f41 
  core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/GraphRenderer.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/OutputFormat.java PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/WorkflowActionNode.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/servlet/TestV1JobServlet.java 
ee9ab556c9b6930c406ca5dcd54065cbbebb86c9 
  core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 
002e925b57cd830ea6d83a87cea4383165116b80 
  core/src/test/java/org/apache/oozie/util/graph/TestGraphGenerator.java 
PRE-CREATION 
  core/src/test/resources/graphWF.xml 6a7b0427a9951835a7533a04b66258ded369d5bf 
  core/src/test/resources/graphWF_26_actions.xml 
a091be0f3559ede195ccc3339adee4478a8da8c0 
  core/src/test/resources/graphWF_50_actions.xml PRE-CREATION 
  core/src/test/resources/graphWF_decision_fork_join.xml PRE-CREATION 
  docs/src/site/twiki/WebServicesAPI.twiki 
ef3e60242512decd48beb3d8c9ac747b7d128eda 
  examples/src/main/apps/java-main/workflow.xml 
98e01ca92187e1c567686f6c2b4f689bb2a5ef6a 
  pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
  webapp/src/main/webapp/oozie-console.js 
72c8a198a4ffe60f74a9f700831f65efcb3066c4 


Diff: https://reviews.apache.org/r/62352/diff/6/

Changes: https://reviews.apache.org/r/62352/diff/5-6/


Testing
---

`TestGraphGenerator`, `TestV1JobServlet`


Thanks,

András Piros

Re: Review Request 62352: OOZIE-2406 Completely rewrite GraphGenerator code

[András Piros]

> On Oct. 2, 2017, 8:11 p.m., Peter Cseh wrote:
> > core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java
> > Lines 365 (patched)
> > 
> >
> > UTF-8

This is a `Locale` instance, not an `Encoding`, or a `String`, or a `Charset` 
one.


> On Oct. 2, 2017, 8:11 p.m., Peter Cseh wrote:
> > core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java
> > Lines 366 (patched)
> > 
> >
> > UTF-8

This is a `Locale` instance, not an `Encoding`, or a `String`, or a `Charset` 
one.


> On Oct. 2, 2017, 8:11 p.m., Peter Cseh wrote:
> > core/src/test/java/org/apache/oozie/util/graph/TestGraphGenerator.java
> > Lines 148 (patched)
> > 
> >
> > How is this test testing performance?

5 graphs, 50+ nodes each... to render PNG images takes a bit of a time.


- András


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62352/#review186874
---


On Oct. 2, 2017, 9:04 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62352/
> ---
> 
> (Updated Oct. 2, 2017, 9:04 p.m.)
> 
> 
> Review request for oozie and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2406 Completely rewrite GraphGenerator code
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 74843dc9a3cbb8cef38e97f81e34731f191d9aab 
>   core/pom.xml 6f9adb66af9344ac7d2212cdc31aa203ec06c286 
>   core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 
> 059d3cf6dc251b49940af29d82cbdd817043a176 
>   core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 
> 10812c675ebc0cc4aefca9f4a85ef2fc26d143e1 
>   core/src/main/java/org/apache/oozie/util/GraphGenerator.java 
> 6ded2c6dc15c9e8453ff800407ff0324be185f41 
>   core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/GraphRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/OutputFormat.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/WorkflowActionNode.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java 
> PRE-CREATION 
>   core/src/test/java/org/apache/oozie/servlet/TestV1JobServlet.java 
> ee9ab556c9b6930c406ca5dcd54065cbbebb86c9 
>   core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 
> 002e925b57cd830ea6d83a87cea4383165116b80 
>   core/src/test/java/org/apache/oozie/util/graph/TestGraphGenerator.java 
> PRE-CREATION 
>   core/src/test/resources/graphWF.xml 
> 6a7b0427a9951835a7533a04b66258ded369d5bf 
>   core/src/test/resources/graphWF_26_actions.xml 
> a091be0f3559ede195ccc3339adee4478a8da8c0 
>   core/src/test/resources/graphWF_50_actions.xml PRE-CREATION 
>   core/src/test/resources/graphWF_decision_fork_join.xml PRE-CREATION 
>   docs/src/site/twiki/WebServicesAPI.twiki 
> ef3e60242512decd48beb3d8c9ac747b7d128eda 
>   examples/src/main/apps/java-main/workflow.xml 
> 98e01ca92187e1c567686f6c2b4f689bb2a5ef6a 
>   pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
>   webapp/src/main/webapp/oozie-console.js 
> 72c8a198a4ffe60f74a9f700831f65efcb3066c4 
> 
> 
> Diff: https://reviews.apache.org/r/62352/diff/5/
> 
> 
> Testing
> ---
> 
> `TestGraphGenerator`, `TestV1JobServlet`
> 
> 
> Thanks,
> 
> András Piros
> 
>

Re: Review Request 62352: OOZIE-2406 Completely rewrite GraphGenerator code

[András Piros]

> On Oct. 3, 2017, 11:12 a.m., Peter Bacsko wrote:
> > core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java
> > Lines 170-173 (patched)
> > 
> >
> > To me this looks weird. We have a single threaded executor and we wait 
> > without any kind of timeout. What does this method accomplish by rendering 
> > the PNG like that?

Added a comment to the field `GraphvizRenderer#EXECUTOR_SERVICE`.


> On Oct. 3, 2017, 11:12 a.m., Peter Bacsko wrote:
> > core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java
> > Lines 161 (patched)
> > 
> >
> > Why do we have to explicitly null this out? Is this class ever reused? 
> > Can't see a setter for "out".
> > 
> > I'm also concerned about closing the output stream here. This should be 
> > the caller's responsibility.

Left from old code. Nevermind, removed, and the code still works.


> On Oct. 3, 2017, 11:12 a.m., Peter Bacsko wrote:
> > core/src/test/java/org/apache/oozie/util/graph/TestGraphGenerator.java
> > Lines 83 (patched)
> > 
> >
> > Please also print the stack trace to improve debuggability

I'm not into having stack trace, making the log message clearer instead.


> On Oct. 3, 2017, 11:12 a.m., Peter Bacsko wrote:
> > core/src/test/java/org/apache/oozie/util/graph/TestGraphGenerator.java
> > Lines 96 (patched)
> > 
> >
> > Please also print the stack trace to improve debuggability

I'm not into having stack trace, making the log message clearer instead.


> On Oct. 3, 2017, 11:12 a.m., Peter Bacsko wrote:
> > core/src/test/java/org/apache/oozie/util/graph/TestGraphGenerator.java
> > Lines 111 (patched)
> > 
> >
> > Please also print the stack trace to improve debuggability

I'm not into having stack trace, making the log message clearer instead.


- András


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62352/#review186944
---


On Oct. 2, 2017, 9:04 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62352/
> ---
> 
> (Updated Oct. 2, 2017, 9:04 p.m.)
> 
> 
> Review request for oozie and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2406 Completely rewrite GraphGenerator code
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 74843dc9a3cbb8cef38e97f81e34731f191d9aab 
>   core/pom.xml 6f9adb66af9344ac7d2212cdc31aa203ec06c286 
>   core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 
> 059d3cf6dc251b49940af29d82cbdd817043a176 
>   core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 
> 10812c675ebc0cc4aefca9f4a85ef2fc26d143e1 
>   core/src/main/java/org/apache/oozie/util/GraphGenerator.java 
> 6ded2c6dc15c9e8453ff800407ff0324be185f41 
>   core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/GraphRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/OutputFormat.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/WorkflowActionNode.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java 
> PRE-CREATION 
>   core/src/test/java/org/apache/oozie/servlet/TestV1JobServlet.java 
> ee9ab556c9b6930c406ca5dcd54065cbbebb86c9 
>   core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 
> 002e925b57cd830ea6d83a87cea4383165116b80 
>   core/src/test/java/org/apache/oozie/util/graph/TestGraphGenerator.java 
> PRE-CREATION 
>   core/src/test/resources/graphWF.xml 
> 6a7b0427a9951835a7533a04b66258ded369d5bf 
>   core/src/test/resources/graphWF_26_actions.xml 
> a091be0f3559ede195ccc3339adee4478a8da8c0 
>   core/src/test/resources/graphWF_50_actions.xml PRE-CREATION 
>   core/src/test/resources/graphWF_decision_fork_join.xml PRE-CREATION 
>   docs/src/site/twiki/WebServicesAPI.twiki 
> ef3e60242512decd48beb3d8c9ac747b7d128eda 
>   examples/src/main/apps/java-main/workflow.xml 
> 98e01ca92187e1c567686f6c2b4f689bb2a5ef6a 
>   pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
>   webapp/src/main/webapp/oozie-console.js 
> 72c8a198a4ffe60f74a9f700831f65efcb3066c4 
> 
> 
> Diff: https://reviews.apache.org/r/62352/diff/5/
> 
> 
> 

Re: Review Request 62470: OOZIE-3062 Set HADOOP_CONF_DIR for spark action

[Rohini Palaniswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62470/#review186997
---




core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Line 331 (original), 331 (patched)


Would like to disagree with András Piros comment on creating yet another 
method in JavaActionExecutor. It does not make it clean. We should not be 
renaming method to include every step done inside the method. Adding default 
child env is still part of setting up the launcher config and it does not 
require special mention in the method name and a new method. I would suggest 
reverting back to old method name and overriding it as in the initial patch.



sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java
Lines 136 (patched)


You can get rid of this and just use ArrayUtils.contains method from 
commons-lang


- Rohini Palaniswamy


On Oct. 3, 2017, 7:35 p.m., Satish Saley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62470/
> ---
> 
> (Updated Oct. 3, 2017, 7:35 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-3062
> https://issues.apache.org/jira/browse/OOZIE-3062
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-3062 Set HADOOP_CONF_DIR for spark action
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> be056038 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 80d64ec8 
>   
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java
>  d97f1f06 
> 
> 
> Diff: https://reviews.apache.org/r/62470/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Satish Saley
> 
>

[jira] [Commented] (OOZIE-3071) Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190240#comment-16190240
 ] 

Peter Cseh commented on OOZIE-3071:
---

While this version bump solves the issue with Spark 2.2, won't it break 
Spark1.6 runs from Oozie? 
We should find a way to separate out server dependencies from the Sharelib 
dependencies in general so we can avoid problems like this.

> Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than 
> Spark 2.2.0
> 
>
> Key: OOZIE-3071
> URL: https://issues.apache.org/jira/browse/OOZIE-3071
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.3.0
>Reporter: Ran Gu
>  Labels: patch
> Attachments: OOZIE-3071-1.patch, OOZIE-3071.patch
>
>
> Currently Oozie 4.3.0 uses commons-lang3 version 3.3.2 in Spark sharelib.
> Spark 2.2.0 uses commons-lang3 version 3.5. 
> This causes Oozie(/Spark) job failures on EMR-5.8.0 clusters.
> Error message:
> 17/08/22 00:22:43 ERROR ApplicationMaster: User class threw exception: 
> java.lang.IllegalArgumentException: Illegal pattern component: XXX
> java.lang.IllegalArgumentException: Illegal pattern component: XXX
> at 
> org.apache.commons.lang3.time.FastDatePrinter.parsePattern(FastDatePrinter.java:282)
> at 
> org.apache.commons.lang3.time.FastDatePrinter.init(FastDatePrinter.java:149)
> at 
> org.apache.commons.lang3.time.FastDatePrinter.(FastDatePrinter.java:142)
> at 
> org.apache.commons.lang3.time.FastDateFormat.(FastDateFormat.java:384)
> at 
> org.apache.commons.lang3.time.FastDateFormat.(FastDateFormat.java:369)
> at 
> org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:91)
> at 
> org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:88)
> at org.apache.commons.lang3.time.FormatCache.getInstance(FormatCache.java:82) 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (OOZIE-3071) Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0

[Artem Ervits (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190224#comment-16190224
 ] 

Artem Ervits edited comment on OOZIE-3071 at 10/3/17 7:48 PM:
--

[~rangu] I attached my patch with 3.6 version for your convenience. Since I'm 
not a committer, a member of the PMC or a committer has to vote on the patch 
and get it merged into master. If that happens before Oozie 5.0 (next version) 
ships, then you will see the patch in the next release. Thank you for reporting 
the problem. [~gezapeti] [~pbacsko] [~rkanter][~rohini] [~abhishekbafna] please 
review.


was (Author: dbist13):
[~rangu] I attached my patch with 3.6 version for your convenience. Since I'm 
not a committer, a member of the PMC or a committer has to vote on the patch 
and get it merged into master. If that happens before Oozie 5.0 (next version) 
ships, then you will see the patch in the next release. Thank you for reporting 
the problem.

> Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than 
> Spark 2.2.0
> 
>
> Key: OOZIE-3071
> URL: https://issues.apache.org/jira/browse/OOZIE-3071
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.3.0
>Reporter: Ran Gu
>  Labels: patch
> Attachments: OOZIE-3071-1.patch, OOZIE-3071.patch
>
>
> Currently Oozie 4.3.0 uses commons-lang3 version 3.3.2 in Spark sharelib.
> Spark 2.2.0 uses commons-lang3 version 3.5. 
> This causes Oozie(/Spark) job failures on EMR-5.8.0 clusters.
> Error message:
> 17/08/22 00:22:43 ERROR ApplicationMaster: User class threw exception: 
> java.lang.IllegalArgumentException: Illegal pattern component: XXX
> java.lang.IllegalArgumentException: Illegal pattern component: XXX
> at 
> org.apache.commons.lang3.time.FastDatePrinter.parsePattern(FastDatePrinter.java:282)
> at 
> org.apache.commons.lang3.time.FastDatePrinter.init(FastDatePrinter.java:149)
> at 
> org.apache.commons.lang3.time.FastDatePrinter.(FastDatePrinter.java:142)
> at 
> org.apache.commons.lang3.time.FastDateFormat.(FastDateFormat.java:384)
> at 
> org.apache.commons.lang3.time.FastDateFormat.(FastDateFormat.java:369)
> at 
> org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:91)
> at 
> org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:88)
> at org.apache.commons.lang3.time.FormatCache.getInstance(FormatCache.java:82) 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3071) Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0

[Artem Ervits (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits updated OOZIE-3071:

Attachment: OOZIE-3071-1.patch

[~rangu] I attached my patch with 3.6 version for your convenience. Since I'm 
not a committer, a member of the PMC or a committer has to vote on the patch 
and get it merged into master. If that happens before Oozie 5.0 (next version) 
ships, then you will see the patch in the next release. Thank you for reporting 
the problem.

> Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than 
> Spark 2.2.0
> 
>
> Key: OOZIE-3071
> URL: https://issues.apache.org/jira/browse/OOZIE-3071
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.3.0
>Reporter: Ran Gu
>  Labels: patch
> Attachments: OOZIE-3071-1.patch, OOZIE-3071.patch
>
>
> Currently Oozie 4.3.0 uses commons-lang3 version 3.3.2 in Spark sharelib.
> Spark 2.2.0 uses commons-lang3 version 3.5. 
> This causes Oozie(/Spark) job failures on EMR-5.8.0 clusters.
> Error message:
> 17/08/22 00:22:43 ERROR ApplicationMaster: User class threw exception: 
> java.lang.IllegalArgumentException: Illegal pattern component: XXX
> java.lang.IllegalArgumentException: Illegal pattern component: XXX
> at 
> org.apache.commons.lang3.time.FastDatePrinter.parsePattern(FastDatePrinter.java:282)
> at 
> org.apache.commons.lang3.time.FastDatePrinter.init(FastDatePrinter.java:149)
> at 
> org.apache.commons.lang3.time.FastDatePrinter.(FastDatePrinter.java:142)
> at 
> org.apache.commons.lang3.time.FastDateFormat.(FastDateFormat.java:384)
> at 
> org.apache.commons.lang3.time.FastDateFormat.(FastDateFormat.java:369)
> at 
> org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:91)
> at 
> org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:88)
> at org.apache.commons.lang3.time.FormatCache.getInstance(FormatCache.java:82) 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3062) Set HADOOP_CONF_DIR for spark action

[Satish Subhashrao Saley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-3062:
---
Attachment: OOZIE-3062-3.patch

> Set HADOOP_CONF_DIR for spark action
> 
>
> Key: OOZIE-3062
> URL: https://issues.apache.org/jira/browse/OOZIE-3062
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3062-1.patch, OOZIE-3062-2.patch, 
> OOZIE-3062-3.patch
>
>
> OOZIE-2569 created core-site.xml, hdfs-site.xml, mapred-site.xml, 
> yarn-site.xml for spark action. But for spark to consider these files ([spark 
> code 
> ref|https://github.com/apache/spark/blob/83fe3b5e10f8dc62245ea37143abb96be0f39805/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala#L284]),
>  we need to set HADOOP_CONF_DIR environmental variable, otherwise it will 
> point to default HADOOP_CONF_DIR configured on the node. Therefore, Oozie 
> should set HADOOP_CONF_DIR.
> Due to a bug YARN-4727, yarn was not considering user specified value for 
> whiltelisted environment variables including HADOOP_CONF_DIR. So, this will 
> work on specified hadoop fix versions in YARN-4727.
> This fix in Oozie will be no-op on Hadoop without YARN-4727 fix and Oozie 
> will behave as it is.
> To verify this on one node machine. 
> 1. In hadoop mapred-site.xml, set 
> {code}
> 
>mapreduce.fileoutputcommitter.marksuccessfuljobs
>false
> 
> {code}
> 2. Run any spark file copy example from Oozie. Check output directory, 
> _SUCCESS flag is missing. It should be there because we explicitly set 
> conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "true"); in 
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L293



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62470: OOZIE-3062 Set HADOOP_CONF_DIR for spark action

[Satish Saley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62470/
---

(Updated Oct. 3, 2017, 12:35 p.m.)


Review request for oozie.


Bugs: OOZIE-3062
https://issues.apache.org/jira/browse/OOZIE-3062


Repository: oozie-git


Description
---

OOZIE-3062 Set HADOOP_CONF_DIR for spark action


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
be056038 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
80d64ec8 
  
sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java
 d97f1f06 


Diff: https://reviews.apache.org/r/62470/diff/3/

Changes: https://reviews.apache.org/r/62470/diff/2-3/


Testing
---


Thanks,

Satish Saley

[jira] [Commented] (OOZIE-3071) Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0

[Ran Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190159#comment-16190159
 ] 

Ran Gu commented on OOZIE-3071:
---

Sorry for the late response. I have verified that updating commons-lang3 
version to 3.6 in top level pom.xml worked properly.
Are you going to make that change? And will that be included in the next Oozie 
release?

> Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than 
> Spark 2.2.0
> 
>
> Key: OOZIE-3071
> URL: https://issues.apache.org/jira/browse/OOZIE-3071
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.3.0
>Reporter: Ran Gu
>  Labels: patch
> Attachments: OOZIE-3071.patch
>
>
> Currently Oozie 4.3.0 uses commons-lang3 version 3.3.2 in Spark sharelib.
> Spark 2.2.0 uses commons-lang3 version 3.5. 
> This causes Oozie(/Spark) job failures on EMR-5.8.0 clusters.
> Error message:
> 17/08/22 00:22:43 ERROR ApplicationMaster: User class threw exception: 
> java.lang.IllegalArgumentException: Illegal pattern component: XXX
> java.lang.IllegalArgumentException: Illegal pattern component: XXX
> at 
> org.apache.commons.lang3.time.FastDatePrinter.parsePattern(FastDatePrinter.java:282)
> at 
> org.apache.commons.lang3.time.FastDatePrinter.init(FastDatePrinter.java:149)
> at 
> org.apache.commons.lang3.time.FastDatePrinter.(FastDatePrinter.java:142)
> at 
> org.apache.commons.lang3.time.FastDateFormat.(FastDateFormat.java:384)
> at 
> org.apache.commons.lang3.time.FastDateFormat.(FastDateFormat.java:369)
> at 
> org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:91)
> at 
> org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:88)
> at org.apache.commons.lang3.time.FormatCache.getInstance(FormatCache.java:82) 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (OOZIE-3076) Allow logging OpenJPA SQL queries for debugging

[Attila Sasvari (JIRA)]

Attila Sasvari created OOZIE-3076:
-

 Summary: Allow logging OpenJPA SQL queries for debugging
 Key: OOZIE-3076
 URL: https://issues.apache.org/jira/browse/OOZIE-3076
 Project: Oozie
  Issue Type: Improvement
Reporter: Attila Sasvari


If you are curious about the exact SQL that OpenJPA issues to the datastore, 
you can set the following in {{persistence.xml}} in the {{oozie-core jar}}
{code}


{code}
( more info 
https://openjpa.apache.org/builds/2.4.0/apache-openjpa/docs/ref_guide_logging.html
 )

To have effect on OpenJPA logging, you need to update oozie core jar and 
restart oozie.   

Under the hood, {{org.apache.openjpa.conf.OpenJPAConfigurationImpl}} handles 
[this 
part|https://github.com/apache/openjpa/blob/49c1478cf3f107d9f7069bf2102d8cf0b6ca2b12/openjpa-kernel/src/main/java/org/apache/openjpa/conf/OpenJPAConfigurationImpl.java#L393].
 From Oozie, it is called from the {{JPAService}}'s {{init()}} method when [the 
factory 
|https://github.com/apache/oozie/blob/7bda357849e1006f2cc6df58320f8905da2bd3b8/core/src/main/java/org/apache/oozie/service/JPAService.java#L230]
 is created:
{code}
factory = Persistence.createEntityManagerFactory(persistentUnit, props);
{code} 

It shall be possible to control OpenJPA logging programatically via. A 
hard-coded example:
{code}
((EntityManagerFactoryImpl) 
factory).getBrokerFactory().getConfiguration().setLog("SQL=TRACE")
((EntityManagerFactoryImpl) 
factory).getBrokerFactory().getConfiguration().setConnectionFactoryProperties("PrintParameters=true")
{code}
After this openjpa related information will appear in the Oozie logs:
{code}
132816  oozie-derby  TRACE  [Scheduler-8] openjpa.jdbc.SQL -  executing prepstmnt 1558515073 SELECT t0.id, t0.job_id, t0.status, 
t0.external_id, t0.push_missing_dependencies FROM COORD_ACTIONS t0 WHERE 
((t0.status = ? OR t0.status = ?) AND t0.last_modified_time <= ?) 
[params=(String) WAITING, (String) SUBMITTED, (Timestamp) 2017-10-03 
17:02:16.986]
132816  oozie-derby  TRACE  [Scheduler-8] openjpa.jdbc.SQL -  [0 ms] spent
{code}
- To make it easily configurable, a new element should be added to 
{{oozie-default.xml}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3075) Follow-up on OOZIE-3054: create the lib directory if it doesn't exist

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189657#comment-16189657
 ] 

Peter Bacsko commented on OOZIE-3075:
-

Thanks, committed to master.

> Follow-up on OOZIE-3054: create the lib directory if it doesn't exist
> -
>
> Key: OOZIE-3075
> URL: https://issues.apache.org/jira/browse/OOZIE-3075
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: OOZIE-3075-001.patch
>
>
> If the folder {{/user/oozie/sharelib/lib}} doesn't exist, then disabling the 
> Erasure Coding on that directory is not possible because an exception will be 
> thrown.
> Solution is to create the path if it doesn't exist.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3075) Follow-up on OOZIE-3054: create the lib directory if it doesn't exist

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189613#comment-16189613
 ] 

Hadoop QA commented on OOZIE-3075:
--

Testing JIRA OOZIE-3075

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2032
.Tests rerun: 74
.Tests failed at first run: 
org.apache.oozie.command.coord.TestCoordActionsKillXCommand,org.apache.oozie.TestDagEngine,org.apache.oozie.command.coord.TestCoordActionInputCheckXCommandNonUTC,org.apache.oozie.action.hadoop.TestJavaActionExecutor,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/70/

> Follow-up on OOZIE-3054: create the lib directory if it doesn't exist
> -
>
> Key: OOZIE-3075
> URL: https://issues.apache.org/jira/browse/OOZIE-3075
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: OOZIE-3075-001.patch
>
>
> If the folder {{/user/oozie/sharelib/lib}} doesn't exist, then disabling the 
> Erasure Coding on that directory is not possible because an exception will be 
> thrown.
> Solution is to create the path if it doesn't exist.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-3075 PreCommit Build #70

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-3075
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/70/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.59 MB...]
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 77 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/pig].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2032
Tests rerun: 74
Tests failed at first run: 
org.apache.oozie.command.coord.TestCoordActionsKillXCommand,org.apache.oozie.TestDagEngine,org.apache.oozie.command.coord.TestCoordActionInputCheckXCommandNonUTC,org.apache.oozie.action.hadoop.TestJavaActionExecutor,
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/70/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  0 
00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
100 3706k  100 3706k0 0  4854k  0 --:--:-- --:--:-- --:--:-- 4854k
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-3075
Archiving artifacts
[Fast Archiver] Compressed 1.79 MB of artifacts by 74.9% relative to #69
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
All tests passed

Re: Review Request 62352: OOZIE-2406 Completely rewrite GraphGenerator code

[Peter Bacsko]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62352/#review186944
---




core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java
Lines 30 (patched)


Import



core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java
Lines 60 (patched)


I would rather call this getShape(). The word "calculate" is telling me 
that we do some math as well.



core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java
Lines 90 (patched)


getColor()



core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java
Lines 125 (patched)


What do we "ensure" here? To me this feels like another get() method with 
some extra logic.



core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java
Lines 170-173 (patched)


To me this looks weird. We have a single threaded executor and we wait 
without any kind of timeout. What does this method accomplish by rendering the 
PNG like that?



core/src/main/java/org/apache/oozie/util/graph/WorkflowActionNode.java
Lines 27 (patched)


We store this class in a hashmap and there is no hashCode() and equals() 
defined for this class. Could you generate them?



core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java
Lines 61 (patched)


Could you just add a short comment like "// nop" if the method body is 
intentionally empty?



core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java
Lines 92-100 (patched)


Can we live without "continue"? If we can, just remove it and put this 
block under an "if (child != null)" branch.



core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java
Lines 161 (patched)


Why do we have to explicitly null this out? Is this class ever reused? 
Can't see a setter for "out".

I'm also concerned about closing the output stream here. This should be the 
caller's responsibility.



core/src/test/java/org/apache/oozie/util/graph/TestGraphGenerator.java
Lines 83 (patched)


Please also print the stack trace to improve debuggability



core/src/test/java/org/apache/oozie/util/graph/TestGraphGenerator.java
Lines 96 (patched)


Please also print the stack trace to improve debuggability



core/src/test/java/org/apache/oozie/util/graph/TestGraphGenerator.java
Lines 111 (patched)


Please also print the stack trace to improve debuggability


- Peter Bacsko


On okt. 2, 2017, 9:04 du, András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62352/
> ---
> 
> (Updated okt. 2, 2017, 9:04 du)
> 
> 
> Review request for oozie and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2406 Completely rewrite GraphGenerator code
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 74843dc9a3cbb8cef38e97f81e34731f191d9aab 
>   core/pom.xml 6f9adb66af9344ac7d2212cdc31aa203ec06c286 
>   core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 
> 059d3cf6dc251b49940af29d82cbdd817043a176 
>   core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 
> 10812c675ebc0cc4aefca9f4a85ef2fc26d143e1 
>   core/src/main/java/org/apache/oozie/util/GraphGenerator.java 
> 6ded2c6dc15c9e8453ff800407ff0324be185f41 
>   core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/GraphRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/OutputFormat.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/WorkflowActionNode.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java 
> PRE-CREATION 
>   core/src/test/java/org/apache/oozie/servlet/TestV1JobServlet.java 
> ee9ab556c9b6930c406ca5dcd54065cbbebb86c9 
>   core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 
> 002e925b57cd830ea6d83a87cea4383165116b80 
>   

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

[Peter Bacsko]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186940
---




tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 77 (patched)


I think this should go to stdout instead



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 113 (patched)


I see writeString("\n") at least a dozen of times (actually 18).

I strongly suggest adding a method to the writer and call it writeNewLine().



tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCollectorDriver.java
Lines 41 (patched)


We might as we call this DATE_FORMAT. Also, if it's referenced from other 
classes (and I see it is), just make it public.



tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCollectorDriver.java
Lines 47 (patched)


I know that this is returned by the parser, but can we switch to 
List here if it's not a big deal? It just feels safer.



tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCompressor.java
Lines 31 (patched)


Nit: I know that zip() is expressive in itself, but still, the name 
compress() sounds better to me.



tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleEntryWriter.java
Lines 48 (patched)


This is usually thrown by the JVM when a static initializer block fails. 
Could we change this to something else? An ordinary RuntimeException looks good 
enough to me.



tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java
Lines 46 (patched)


Nit: space :D



tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java
Lines 52 (patched)


Do we need to call flush() here?



tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java
Lines 56 (patched)


Could this whole example metrics be stored in a file and not inline in the 
code?



tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java
Lines 122 (patched)


Simpler:

import static java.nio.file.Files.readAllBytes;
import static java.nio.file.Paths.get;

String s = new String(readAllBytes(get("test.txt";

This part should be extracted somewhere to avoid repeated coding to load a 
file



tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java
Lines 127 (patched)


The string "UTF-8" is repeated a couple of times. Make it const or use 
StandardCharsets.UTF_8.



tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java
Lines 134 (patched)


Same here, I would store this in an external file



tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java
Lines 215 (patched)


Simplify file loading



tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java
Lines 231 (patched)


Handle this catch properly



tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java
Lines 120 (patched)


Simplify file loading


- Peter Bacsko


On okt. 2, 2017, 9:39 de, Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated okt. 2, 2017, 9:39 de)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
>   
> tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCollectorDriver.java
>  PRE-CREATION 
>   

[jira] [Commented] (OOZIE-3072) oozie.service.HadoopAccessorService.action.configurations should overwrite default values set in Hadoop's configuration files

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189491#comment-16189491
 ] 

Peter Bacsko commented on OOZIE-3072:
-

+1

> oozie.service.HadoopAccessorService.action.configurations  should 
> overwrite default values set in Hadoop's configuration files
> --
>
> Key: OOZIE-3072
> URL: https://issues.apache.org/jira/browse/OOZIE-3072
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-3072.001.patch, OOZIE-3072.002.patch, 
> OOZIE-3072.003.patch, OOZIE-3072.004.patch
>
>
> When Oozie process the files defined in 
> oozie.service.HadoopAccessorService.action.configurations it uses 
> Configuration.injectDefauts(). This prevents overwriting things defined 
> inside any of the configuration files Oozie is providing default values from 
> (like yarn-site.xml, mapred-site.xml)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

[Attila Sasvari]

> On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 191 (patched)
> > 
> >
> > Is there not a cleaner way to do this than using a CLI like this?
> 
> Attila Sasvari wrote:
> I could not find one, please let me know if you have something in mind.

OK, I am going to inline/borrow code from dumpAllContainersLogs in LogsCLI. 
What we do is basically just fetching logs from HDFS. If there is an exception, 
we log it and move on.


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186754
---


On Oct. 2, 2017, 9:39 a.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Oct. 2, 2017, 9:39 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
>   
> tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCollectorDriver.java
>  PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCompressor.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleEntryWriter.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagOozieClient.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/9/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

[jira] [Commented] (OOZIE-3075) Follow-up on OOZIE-3054: create the lib directory if it doesn't exist

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189482#comment-16189482
 ] 

Peter Cseh commented on OOZIE-3075:
---

+1 (pending jenkins)
We have overlooked this in the previous issue

> Follow-up on OOZIE-3054: create the lib directory if it doesn't exist
> -
>
> Key: OOZIE-3075
> URL: https://issues.apache.org/jira/browse/OOZIE-3075
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: OOZIE-3075-001.patch
>
>
> If the folder {{/user/oozie/sharelib/lib}} doesn't exist, then disabling the 
> Erasure Coding on that directory is not possible because an exception will be 
> thrown.
> Solution is to create the path if it doesn't exist.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3075) Follow-up on OOZIE-3054: create the lib directory if it doesn't exist

[Peter Bacsko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Bacsko updated OOZIE-3075:

Attachment: OOZIE-3075-001.patch

> Follow-up on OOZIE-3054: create the lib directory if it doesn't exist
> -
>
> Key: OOZIE-3075
> URL: https://issues.apache.org/jira/browse/OOZIE-3075
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: OOZIE-3075-001.patch
>
>
> If the folder {{/user/oozie/sharelib/lib}} doesn't exist, then disabling the 
> Erasure Coding on that directory is not possible because an exception will be 
> thrown.
> Solution is to create the path if it doesn't exist.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (OOZIE-3075) Follow-up on OOZIE-3054: create the lib directory if it doesn't exist

[Peter Bacsko (JIRA)]

Peter Bacsko created OOZIE-3075:
---

 Summary: Follow-up on OOZIE-3054: create the lib directory if it 
doesn't exist
 Key: OOZIE-3075
 URL: https://issues.apache.org/jira/browse/OOZIE-3075
 Project: Oozie
  Issue Type: Bug
  Components: core
Reporter: Peter Bacsko
Assignee: Peter Bacsko


If the folder {{/user/oozie/sharelib/lib}} doesn't exist, then disabling the 
Erasure Coding on that directory is not possible because an exception will be 
thrown.

Solution is to create the path if it doesn't exist.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (OOZIE-2734) Switch docs from twiki to markdown

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros reassigned OOZIE-2734:
---

Assignee: Andras Piros

> Switch docs from twiki to markdown
> --
>
> Key: OOZIE-2734
> URL: https://issues.apache.org/jira/browse/OOZIE-2734
> Project: Oozie
>  Issue Type: Improvement
>  Components: docs
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Andras Piros
>
> This won't be a fun JIRA, but it would be great if we could switch to 
> something easier to work with and better supported like markdown, and get rid 
> of the super old twiki we're using for docs.
> I found two tools that I haven't tried out that claim to be able to convert 
> this for us:
> # http://pandoc.org/
> # https://github.com/jcodagnone/twiki2markdown
> The former supports a lot of different formats, and according to [this blog 
> post|http://goer.org/Journal/2013/02/converting_twiki_foswiki_to_markdown_restructuredtext.html]
>  it's better to convert from the rendered html instead of the twiki.  But who 
> knows which is better, or if the second tool is better than either.
> I think once we switch to a better format, it will be easier to keep the docs 
> in better shape, as they could really use some reorganizing.  Though we can 
> keep that for a separate task.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (108 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3072  oozie.service.HadoopAccessorService.action.configurationsshould 
overwrite default values set in Hadoop's configuration files
https://issues.apache.org/jira/browse/OOZIE-3072
OOZIE-3071  Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 
than Spark 2.2.0
https://issues.apache.org/jira/browse/OOZIE-3071
OOZIE-3063  Sanitizing variables that are part of openjpa.ConnectionProperties
https://issues.apache.org/jira/browse/OOZIE-3063
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues.apache.org/jira/browse/OOZIE-3062
OOZIE-3031  Coord job with only unresolved dependencies doesn't timeout
https://issues.apache.org/jira/browse/OOZIE-3031
OOZIE-3022  fix for warning has no file and won't be listed in dependency files 
details
https://issues.apache.org/jira/browse/OOZIE-3022
OOZIE-3002  address findbugs errors in client lib
https://issues.apache.org/jira/browse/OOZIE-3002
OOZIE-3001  core library has many instances of warnings with trailing spaces 
and lines longer than 132 chars
https://issues.apache.org/jira/browse/OOZIE-3001
OOZIE-2997  files contain trailing white spaces in client lib
https://issues.apache.org/jira/browse/OOZIE-2997
OOZIE-2996  add option for -UseGCOverheadLimit to maven opts as sometimes local 
testing fails
https://issues.apache.org/jira/browse/OOZIE-2996
OOZIE-2978  Remove code that handles Pig versions before 0.8 
https://issues.apache.org/jira/browse/OOZIE-2978
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues.apache.org/jira/browse/OOZIE-2975
OOZIE-2969  Drop support for Java 1.7
https://issues.apache.org/jira/browse/OOZIE-2969
OOZIE-2964  Add -Xdoclint:none to javadoc opts to avoid warnings
https://issues.apache.org/jira/browse/OOZIE-2964
OOZIE-2962  bump maven-javadoc-plugin to 2.10.4
https://issues.apache.org/jira/browse/OOZIE-2962
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues.apache.org/jira/browse/OOZIE-2934
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2914  Consolidate Trim 
https://issues.apache.org/jira/browse/OOZIE-2914
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues.apache.org/jira/browse/OOZIE-2867
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues.apache.org/jira/browse/OOZIE-2829
OOZIE-2826  Falcon feed fails to aws s3; Oozie joda time version does not meet 
required jar version 2.2 or later
https://issues.apache.org/jira/browse/OOZIE-2826
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions