[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189035#comment-16189035
 ] 

Rohini Palaniswamy commented on OOZIE-2406:
---

bq. I also considered the Oozie UI to have GraphGenerator render in svg format, 
it should take less effort on Oozie server side, as for SVGs we don't need 
BufferedImage instances
  Big +1 for switching to svg. We had to turn off graph rendering for large 
dags as the huge image objects caused the Oozie servers to go out of memory.

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png, 
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png, 
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch, 
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch, 
> OOZIE-2406.007.patch, OOZIE-2406.008.patch, OOZIE-2406.009.patch, Screen Shot 
> 2017-10-02 at 1.55.58 PM.png, Screen Shot 2017-10-02 at 1.56.33 PM.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189034#comment-16189034
 ] 

Hadoop QA commented on OOZIE-2406:
--

Testing JIRA OOZIE-2406

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 6 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:orange}0{color} There are [1] new bugs found in total that would be nice 
to have fixed.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:orange}0{color} There are [1] new bugs found in [core] that would be 
nice to have fixed.
. You can find the FindBugs diff here: core/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2037
.Tests rerun: 69
.Tests failed at first run: 
org.apache.oozie.command.coord.TestCoordActionInputCheckXCommandNonUTC,org.apache.oozie.action.hadoop.TestJavaActionExecutor,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/69/

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png, 
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png, 
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch, 
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch, 
> OOZIE-2406.007.patch, OOZIE-2406.008.patch, OOZIE-2406.009.patch, Screen Shot 
> 2017-10-02 at 1.55.58 PM.png, Screen Shot 2017-10-02 at 1.56.33 PM.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we 

Success: OOZIE-2406 PreCommit Build #69

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-2406
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/69/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.60 MB...]
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 77 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
0 There are [1] new bugs found in total that would be nice to have fixed.
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [tools].
 0 There are [1] new bugs found in [core] that would be nice to have fixed.
 You can find the FindBugs diff here: core/findbugs-new.html
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/pig].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2037
Tests rerun: 69
Tests failed at first run: 
org.apache.oozie.command.coord.TestCoordActionInputCheckXCommandNonUTC,org.apache.oozie.action.hadoop.TestJavaActionExecutor,
+1 DISTRO
+1 distro tarball builds with the patch 


+1 Overall result, good!, no -1s

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/69/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  0 
00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
100 3706k  100 3706k0 0  5020k  0 --:--:-- --:--:-- --:--:-- 5020k
Adding comment to JIRA
Comment added.

test-patch exit code: 0

[description-setter] Description set: OOZIE-2406
Archiving artifacts
[Fast Archiver] Compressed 1.80 MB of artifacts by 69.5% relative to #60
Recording test results
Email was triggered for: Success
Sending email for trigger: Success



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Commented] (OOZIE-2896) Ensure compatibility for existing LauncherMapper settings

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189031#comment-16189031
 ] 

Rohini Palaniswamy commented on OOZIE-2896:
---

bq. for properties within the same group, where the user has defined at least 
two different properties, emit an ERROR log stating that the user has inserted 
contradictory parameter values, and die. 
   Should not error out for these cases. Just need to keep current behavior as 
is which does not fail for any of the scenarios mentioned.
-  Map/Reduce specific settings have priority over generic child ones. For 
eg: mapred.map.child.env has precedence over mapred.child.env.
- If both new and deprecated setting is specified last one takes precedence
- admin settings are prepended to regular settings. For eg:  
yarn.app.mapreduce.am.admin-command-opts and yarn.app.mapreduce.am.command-opts 
both are applied when starting jvm. If there is a conflict, 
yarn.app.mapreduce.am.command-opts will take precedence because it comes last.

> Ensure compatibility for existing LauncherMapper settings
> -
>
> Key: OOZIE-2896
> URL: https://issues.apache.org/jira/browse/OOZIE-2896
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Bacsko
>Assignee: Andras Piros
>Priority: Blocker
>
> There are a couple of LauncherMapper settings that we should support for 
> backward compatibility reasons.
> For example, {{oozie.launcher.mapreduce.map.memory.mb}}, 
> {{mapred.child.env}}, {{mapred.map.env}}, etc.
> We also have to discuss the list of properties we're going to support, since 
> there are many of them.
> List of hadoop properties under this task:
> mapreduce.job.queuename
> mapreduce.map.memory.mb
> mapreduce.map.java.opts
> mapreduce.map.env
> mapreduce.job.priority
> yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms
> yarn.app.mapreduce.am.resource.mb
> yarn.app.mapreduce.am.command-opts
> yarn.app.mapreduce.am.env
> ACL related properties should go under OOZIE-2897:
> mapreduce.job.acl-view-job
> mapreduce.job.acl-modfy-job



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3068) Environmental variables should be separated by comma (,)

[Rohini Palaniswamy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohini Palaniswamy updated OOZIE-3068:
--
Issue Type: Sub-task  (was: Bug)
Parent: OOZIE-2889

> Environmental variables should be separated by comma (,)
> 
>
> Key: OOZIE-3068
> URL: https://issues.apache.org/jira/browse/OOZIE-3068
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Satish Subhashrao Saley
>
> Hadoop expects comma separated list of env variables.
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java#L289
> It will be confusing for users to deal with different conventions. 
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L1310-L1311
> Being a workflow scheduler system to manage Apache Hadoop jobs, Oozie should 
> avoid this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3068) Environmental variables should be separated by comma (,)

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189016#comment-16189016
 ] 

Rohini Palaniswamy commented on OOZIE-3068:
---

Another reason , is used as separator is that the path and file separators vary 
between Windows and Linux. So in general , is used in configuration and during 
the actual env variable construction appropriate separator is chosen.

> Environmental variables should be separated by comma (,)
> 
>
> Key: OOZIE-3068
> URL: https://issues.apache.org/jira/browse/OOZIE-3068
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>
> Hadoop expects comma separated list of env variables.
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java#L289
> It will be confusing for users to deal with different conventions. 
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L1310-L1311
> Being a workflow scheduler system to manage Apache Hadoop jobs, Oozie should 
> avoid this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (OOZIE-2783) TestHive2ActionExecutor.testHive2Action is flaky

[Satish Subhashrao Saley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley reassigned OOZIE-2783:
--

Assignee: (was: Satish Subhashrao Saley)

> TestHive2ActionExecutor.testHive2Action is flaky
> 
>
> Key: OOZIE-2783
> URL: https://issues.apache.org/jira/browse/OOZIE-2783
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>
> {code}
> junit.framework.ComparisonFailure: expected:<[SUCCEED]ED> but 
> was:<[FAILED/KILL]ED>
>   at junit.framework.Assert.assertEquals(Assert.java:85)
>   at junit.framework.Assert.assertEquals(Assert.java:91)
>   at 
> org.apache.oozie.action.hadoop.TestHive2ActionExecutor.testHive2Action(TestHive2ActionExecutor.java:225)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> {code}
> Test case stack trace. It is unable to find default database
> {code}
> 22:12:20,119 ERROR RetryingHMSHandler:135 - 
> NoSuchObjectException(message:There is no database named default)
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.getMDatabase(ObjectStore.java:487)
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.getDatabase(ObjectStore.java:498)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:483)
>   at 
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
>   at com.sun.proxy.$Proxy95.getDatabase(Unknown Source)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database(HiveMetaStore.java:796)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:483)
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
>   at com.sun.proxy.$Proxy97.get_database(Unknown Source)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:949)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:483)
>   at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
>   at com.sun.proxy.$Proxy98.getDatabase(Unknown Source)
>   at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1165)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:1296)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:1291)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:9944)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9180)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:422)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:322)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:975)
>   at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:968)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:99)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.run(SQLOperation.java:172)
>   at 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:231)
>   at 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:218)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:483)
>   at 
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79)
>   at 
> 

[jira] [Updated] (OOZIE-2406) Completely rewrite GraphGenerator code

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2406:

Attachment: OOZIE-2406.009.patch

Addressing Geza's review comments. Tests are passing locally.

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png, 
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png, 
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch, 
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch, 
> OOZIE-2406.007.patch, OOZIE-2406.008.patch, OOZIE-2406.009.patch, Screen Shot 
> 2017-10-02 at 1.55.58 PM.png, Screen Shot 2017-10-02 at 1.56.33 PM.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62352: OOZIE-2406 Completely rewrite GraphGenerator code

[András Piros]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62352/
---

(Updated Oct. 2, 2017, 9:04 p.m.)


Review request for oozie and Robert Kanter.


Changes
---

Addressing Geza's review comments.


Repository: oozie-git


Description
---

OOZIE-2406 Completely rewrite GraphGenerator code


Diffs (updated)
-

  client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
74843dc9a3cbb8cef38e97f81e34731f191d9aab 
  core/pom.xml 6f9adb66af9344ac7d2212cdc31aa203ec06c286 
  core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 
059d3cf6dc251b49940af29d82cbdd817043a176 
  core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 
10812c675ebc0cc4aefca9f4a85ef2fc26d143e1 
  core/src/main/java/org/apache/oozie/util/GraphGenerator.java 
6ded2c6dc15c9e8453ff800407ff0324be185f41 
  core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/GraphRenderer.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/OutputFormat.java PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/WorkflowActionNode.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/servlet/TestV1JobServlet.java 
ee9ab556c9b6930c406ca5dcd54065cbbebb86c9 
  core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 
002e925b57cd830ea6d83a87cea4383165116b80 
  core/src/test/java/org/apache/oozie/util/graph/TestGraphGenerator.java 
PRE-CREATION 
  core/src/test/resources/graphWF.xml 6a7b0427a9951835a7533a04b66258ded369d5bf 
  core/src/test/resources/graphWF_26_actions.xml 
a091be0f3559ede195ccc3339adee4478a8da8c0 
  core/src/test/resources/graphWF_50_actions.xml PRE-CREATION 
  core/src/test/resources/graphWF_decision_fork_join.xml PRE-CREATION 
  docs/src/site/twiki/WebServicesAPI.twiki 
ef3e60242512decd48beb3d8c9ac747b7d128eda 
  examples/src/main/apps/java-main/workflow.xml 
98e01ca92187e1c567686f6c2b4f689bb2a5ef6a 
  pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
  webapp/src/main/webapp/oozie-console.js 
72c8a198a4ffe60f74a9f700831f65efcb3066c4 


Diff: https://reviews.apache.org/r/62352/diff/5/

Changes: https://reviews.apache.org/r/62352/diff/4-5/


Testing
---

`TestGraphGenerator`, `TestV1JobServlet`


Thanks,

András Piros

[jira] [Commented] (OOZIE-3072) oozie.service.HadoopAccessorService.action.configurations should overwrite default values set in Hadoop's configuration files

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188790#comment-16188790
 ] 

Peter Cseh commented on OOZIE-3072:
---

Test passes locally.

> oozie.service.HadoopAccessorService.action.configurations  should 
> overwrite default values set in Hadoop's configuration files
> --
>
> Key: OOZIE-3072
> URL: https://issues.apache.org/jira/browse/OOZIE-3072
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-3072.001.patch, OOZIE-3072.002.patch, 
> OOZIE-3072.003.patch, OOZIE-3072.004.patch
>
>
> When Oozie process the files defined in 
> oozie.service.HadoopAccessorService.action.configurations it uses 
> Configuration.injectDefauts(). This prevents overwriting things defined 
> inside any of the configuration files Oozie is providing default values from 
> (like yarn-site.xml, mapred-site.xml)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62352: OOZIE-2406 Completely rewrite GraphGenerator code

[András Piros]

> On Oct. 2, 2017, 6:43 p.m., Peter Cseh wrote:
> > core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java
> > Lines 202 (patched)
> > 
> >
> > UTF-8

This is a `Locale` instance, not an `Encoding`, or a `String`, or a `Charset` 
one.


> On Oct. 2, 2017, 6:43 p.m., Peter Cseh wrote:
> > core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java
> > Lines 211 (patched)
> > 
> >
> > UTF-8

This is a `Locale` instance, not an `Encoding`, or a `String`, or a `Charset` 
one.


> On Oct. 2, 2017, 6:43 p.m., Peter Cseh wrote:
> > core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java
> > Lines 219 (patched)
> > 
> >
> > We should stick to UTF-8 as a locale.

This is a `Locale` instance, not an `Encoding`, or a `String`, or a `Charset` 
one.


> On Oct. 2, 2017, 6:43 p.m., Peter Cseh wrote:
> > core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java
> > Lines 226 (patched)
> > 
> >
> > UTF-8

This is a `Locale` instance, not an `Encoding`, or a `String`, or a `Charset` 
one.


> On Oct. 2, 2017, 6:43 p.m., Peter Cseh wrote:
> > core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java
> > Lines 233 (patched)
> > 
> >
> > UTF-8

This is a `Locale` instance, not an `Encoding`, or a `String`, or a `Charset` 
one.


> On Oct. 2, 2017, 6:43 p.m., Peter Cseh wrote:
> > core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java
> > Lines 242 (patched)
> > 
> >
> > UTF-8

This is a `Locale` instance, not an `Encoding`, or a `String`, or a `Charset` 
one.


- András


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62352/#review186863
---


On Oct. 2, 2017, 4:41 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62352/
> ---
> 
> (Updated Oct. 2, 2017, 4:41 p.m.)
> 
> 
> Review request for oozie and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2406 Completely rewrite GraphGenerator code
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 74843dc9a3cbb8cef38e97f81e34731f191d9aab 
>   core/pom.xml 6f9adb66af9344ac7d2212cdc31aa203ec06c286 
>   core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 
> 059d3cf6dc251b49940af29d82cbdd817043a176 
>   core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 
> 10812c675ebc0cc4aefca9f4a85ef2fc26d143e1 
>   core/src/main/java/org/apache/oozie/util/GraphGenerator.java 
> 6ded2c6dc15c9e8453ff800407ff0324be185f41 
>   core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/GraphRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/OutputFormat.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/WorkflowActionNode.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java 
> PRE-CREATION 
>   core/src/test/java/org/apache/oozie/servlet/TestV1JobServlet.java 
> ee9ab556c9b6930c406ca5dcd54065cbbebb86c9 
>   core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 
> 002e925b57cd830ea6d83a87cea4383165116b80 
>   core/src/test/java/org/apache/oozie/util/graph/TestGraphGenerator.java 
> PRE-CREATION 
>   core/src/test/resources/graphWF.xml 
> 6a7b0427a9951835a7533a04b66258ded369d5bf 
>   core/src/test/resources/graphWF_26_actions.xml 
> a091be0f3559ede195ccc3339adee4478a8da8c0 
>   core/src/test/resources/graphWF_50_actions.xml PRE-CREATION 
>   docs/src/site/twiki/WebServicesAPI.twiki 
> ef3e60242512decd48beb3d8c9ac747b7d128eda 
>   examples/src/main/apps/java-main/workflow.xml 
> 98e01ca92187e1c567686f6c2b4f689bb2a5ef6a 
>   pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
>   webapp/src/main/webapp/oozie-console.js 
> 72c8a198a4ffe60f74a9f700831f65efcb3066c4 
> 
> 
> Diff: https://reviews.apache.org/r/62352/diff/4/
> 
> 
> Testing
> ---
> 
> `TestGraphGenerator`, `TestV1JobServlet`
> 
> 
> Thanks,
> 
> András Piros
> 
>

[jira] [Commented] (OOZIE-3072) oozie.service.HadoopAccessorService.action.configurations should overwrite default values set in Hadoop's configuration files

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188780#comment-16188780
 ] 

Hadoop QA commented on OOZIE-3072:
--

Testing JIRA OOZIE-3072

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 3 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [tools].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2034
.Tests failed: 0
.Tests errors: 1

.The patch failed the following testcases:

.  

.Tests failing with errors:
.  testJMXInstrumentation(org.apache.oozie.util.TestMetricsInstrumentation)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/68/

> oozie.service.HadoopAccessorService.action.configurations  should 
> overwrite default values set in Hadoop's configuration files
> --
>
> Key: OOZIE-3072
> URL: https://issues.apache.org/jira/browse/OOZIE-3072
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-3072.001.patch, OOZIE-3072.002.patch, 
> OOZIE-3072.003.patch, OOZIE-3072.004.patch
>
>
> When Oozie process the files defined in 
> oozie.service.HadoopAccessorService.action.configurations it uses 
> Configuration.injectDefauts(). This prevents overwriting things defined 
> inside any of the configuration files Oozie is providing default values from 
> (like yarn-site.xml, mapred-site.xml)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-3072 PreCommit Build #68

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-3072
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/68/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.59 MB...]
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [tools].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS
Tests run: 2034
Tests failed: 0
Tests errors: 1

The patch failed the following testcases:

  

Tests failing with errors:
  testJMXInstrumentation(org.apache.oozie.util.TestMetricsInstrumentation)

+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/68/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100 3706k  100 3706k0 0  2910k  0  0:00:01  0:00:01 --:--:-- 5355k
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-3072
Archiving artifacts
[Fast Archiver] Compressed 1.79 MB of artifacts by 64.5% relative to #60
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
1 tests failed.
FAILED:  org.apache.oozie.util.TestMetricsInstrumentation.testJMXInstrumentation

Error Message:
Unable to open socket file: target process not responding or HotSpot VM not 
loaded

Stack Trace:
com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: 
target process not responding or HotSpot VM not loaded
at 
sun.tools.attach.LinuxVirtualMachine.(LinuxVirtualMachine.java:106)
at 
sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:78)
at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:250)
at 
org.apache.oozie.util.TestMetricsInstrumentation.testJMXInstrumentation(TestMetricsInstrumentation.java:235)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at 
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at 

Re: Review Request 62352: OOZIE-2406 Completely rewrite GraphGenerator code

[Peter Cseh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62352/#review186874
---




core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java
Lines 335 (patched)


Shouldn't an exception be raised here?



core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java
Lines 365 (patched)


UTF-8



core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java
Lines 366 (patched)


UTF-8



core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java
Lines 86-88 (patched)


Constants



core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java
Lines 202-205 (patched)


extract something like createRenderer



core/src/test/java/org/apache/oozie/util/graph/TestGraphGenerator.java
Lines 148 (patched)


How is this test testing performance?


- Peter Cseh


On Oct. 2, 2017, 4:41 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62352/
> ---
> 
> (Updated Oct. 2, 2017, 4:41 p.m.)
> 
> 
> Review request for oozie and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2406 Completely rewrite GraphGenerator code
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 74843dc9a3cbb8cef38e97f81e34731f191d9aab 
>   core/pom.xml 6f9adb66af9344ac7d2212cdc31aa203ec06c286 
>   core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 
> 059d3cf6dc251b49940af29d82cbdd817043a176 
>   core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 
> 10812c675ebc0cc4aefca9f4a85ef2fc26d143e1 
>   core/src/main/java/org/apache/oozie/util/GraphGenerator.java 
> 6ded2c6dc15c9e8453ff800407ff0324be185f41 
>   core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/GraphRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/OutputFormat.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/WorkflowActionNode.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java 
> PRE-CREATION 
>   core/src/test/java/org/apache/oozie/servlet/TestV1JobServlet.java 
> ee9ab556c9b6930c406ca5dcd54065cbbebb86c9 
>   core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 
> 002e925b57cd830ea6d83a87cea4383165116b80 
>   core/src/test/java/org/apache/oozie/util/graph/TestGraphGenerator.java 
> PRE-CREATION 
>   core/src/test/resources/graphWF.xml 
> 6a7b0427a9951835a7533a04b66258ded369d5bf 
>   core/src/test/resources/graphWF_26_actions.xml 
> a091be0f3559ede195ccc3339adee4478a8da8c0 
>   core/src/test/resources/graphWF_50_actions.xml PRE-CREATION 
>   docs/src/site/twiki/WebServicesAPI.twiki 
> ef3e60242512decd48beb3d8c9ac747b7d128eda 
>   examples/src/main/apps/java-main/workflow.xml 
> 98e01ca92187e1c567686f6c2b4f689bb2a5ef6a 
>   pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
>   webapp/src/main/webapp/oozie-console.js 
> 72c8a198a4ffe60f74a9f700831f65efcb3066c4 
> 
> 
> Diff: https://reviews.apache.org/r/62352/diff/4/
> 
> 
> Testing
> ---
> 
> `TestGraphGenerator`, `TestV1JobServlet`
> 
> 
> Thanks,
> 
> András Piros
> 
>

[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188627#comment-16188627
 ] 

Hadoop QA commented on OOZIE-2406:
--

Testing JIRA OOZIE-2406

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 5 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:orange}0{color} There are [2] new bugs found in total that would be nice 
to have fixed.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:orange}0{color} There are [2] new bugs found in [core] that would be 
nice to have fixed.
. You can find the FindBugs diff here: core/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2036
.Tests failed: 8
.Tests errors: 2

.The patch failed the following testcases:

.  testImportTablesOverflowBatchSize(org.apache.oozie.tools.TestDBLoadDump)
.  
testImportInvalidDataLeavesTablesEmpty(org.apache.oozie.tools.TestDBLoadDump)
.  
testImportToNonEmptyTablesCausesPrematureExit(org.apache.oozie.tools.TestDBLoadDump)
.  testImportedDBIsExportedCorrectly(org.apache.oozie.tools.TestDBLoadDump)
.  testMain(org.apache.oozie.action.hadoop.TestHiveMain)
.  testPig_withNullExternalID(org.apache.oozie.action.hadoop.TestPigMain)
.  testEmbeddedPigWithinPython(org.apache.oozie.action.hadoop.TestPigMain)
.  testPigScript(org.apache.oozie.action.hadoop.TestPigMain)

.Tests failing with errors:
.  testAddXIncludeFromReader(org.apache.oozie.util.TestXConfiguration)
.  testAddXIncludeFromStream(org.apache.oozie.util.TestXConfiguration)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/67/

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png, 
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png, 
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch, 
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch, 
> OOZIE-2406.007.patch, OOZIE-2406.008.patch, Screen Shot 2017-10-02 at 1.55.58 
> PM.png, Screen Shot 2017-10-02 at 1.56.33 PM.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user 

Failed: OOZIE-2406 PreCommit Build #67

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-2406
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/67/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.99 MB...]
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/pig].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS
Tests run: 2036
Tests failed: 8
Tests errors: 2

The patch failed the following testcases:

  testImportTablesOverflowBatchSize(org.apache.oozie.tools.TestDBLoadDump)
  
testImportInvalidDataLeavesTablesEmpty(org.apache.oozie.tools.TestDBLoadDump)
  
testImportToNonEmptyTablesCausesPrematureExit(org.apache.oozie.tools.TestDBLoadDump)
  testImportedDBIsExportedCorrectly(org.apache.oozie.tools.TestDBLoadDump)
  testMain(org.apache.oozie.action.hadoop.TestHiveMain)
  testPig_withNullExternalID(org.apache.oozie.action.hadoop.TestPigMain)
  testEmbeddedPigWithinPython(org.apache.oozie.action.hadoop.TestPigMain)
  testPigScript(org.apache.oozie.action.hadoop.TestPigMain)

Tests failing with errors:
  testAddXIncludeFromReader(org.apache.oozie.util.TestXConfiguration)
  testAddXIncludeFromStream(org.apache.oozie.util.TestXConfiguration)

+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/67/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  0 
00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
100 3706k  100 3706k0 0  3845k  0 --:--:-- --:--:-- --:--:-- 3845k
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-2406
Archiving artifacts
[Fast Archiver] Compressed 2.19 MB of artifacts by 45.7% relative to #60
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
10 tests failed.
FAILED:  org.apache.oozie.util.TestXConfiguration.testAddXIncludeFromReader

Error Message:
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build%402/core/target/test-classes/test-oozie-default.xml
 (No such file or directory)

Stack Trace:
java.io.FileNotFoundException: 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build%402/core/target/test-classes/test-oozie-default.xml
 (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.(FileInputStream.java:138)
at java.io.FileInputStream.(FileInputStream.java:93)
at java.io.FileReader.(FileReader.java:58)
at 
org.apache.oozie.util.TestXConfiguration.prepareXmlWithInclude(TestXConfiguration.java:94)
at 
org.apache.oozie.util.TestXConfiguration.testAddXIncludeFromReader(TestXConfiguration.java:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at 
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at 

Re: Review Request 62352: OOZIE-2406 Completely rewrite GraphGenerator code

[Peter Cseh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62352/#review186863
---




core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java
Lines 311-313 (patched)


Extract function.



core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java
Line 303 (original), 315-337 (patched)


setContentType()



core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java
Lines 121-168 (patched)


These three methods look very similar. Can something be extracted from it?



core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java
Lines 202 (patched)


UTF-8



core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java
Lines 211 (patched)


UTF-8



core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java
Lines 219 (patched)


We should stick to UTF-8 as a locale.



core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java
Lines 226 (patched)


UTF-8



core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java
Lines 233 (patched)


UTF-8



core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java
Lines 242 (patched)


UTF-8



core/src/test/resources/graphWF_26_actions.xml
Lines 1-7 (original), 1-7 (patched)


Can you add workflows with forks and decision nodes in them to the test?


- Peter Cseh


On Oct. 2, 2017, 4:41 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62352/
> ---
> 
> (Updated Oct. 2, 2017, 4:41 p.m.)
> 
> 
> Review request for oozie and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2406 Completely rewrite GraphGenerator code
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 74843dc9a3cbb8cef38e97f81e34731f191d9aab 
>   core/pom.xml 6f9adb66af9344ac7d2212cdc31aa203ec06c286 
>   core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 
> 059d3cf6dc251b49940af29d82cbdd817043a176 
>   core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 
> 10812c675ebc0cc4aefca9f4a85ef2fc26d143e1 
>   core/src/main/java/org/apache/oozie/util/GraphGenerator.java 
> 6ded2c6dc15c9e8453ff800407ff0324be185f41 
>   core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/GraphRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/OutputFormat.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/WorkflowActionNode.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java 
> PRE-CREATION 
>   core/src/test/java/org/apache/oozie/servlet/TestV1JobServlet.java 
> ee9ab556c9b6930c406ca5dcd54065cbbebb86c9 
>   core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 
> 002e925b57cd830ea6d83a87cea4383165116b80 
>   core/src/test/java/org/apache/oozie/util/graph/TestGraphGenerator.java 
> PRE-CREATION 
>   core/src/test/resources/graphWF.xml 
> 6a7b0427a9951835a7533a04b66258ded369d5bf 
>   core/src/test/resources/graphWF_26_actions.xml 
> a091be0f3559ede195ccc3339adee4478a8da8c0 
>   core/src/test/resources/graphWF_50_actions.xml PRE-CREATION 
>   docs/src/site/twiki/WebServicesAPI.twiki 
> ef3e60242512decd48beb3d8c9ac747b7d128eda 
>   examples/src/main/apps/java-main/workflow.xml 
> 98e01ca92187e1c567686f6c2b4f689bb2a5ef6a 
>   pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
>   webapp/src/main/webapp/oozie-console.js 
> 72c8a198a4ffe60f74a9f700831f65efcb3066c4 
> 
> 
> Diff: https://reviews.apache.org/r/62352/diff/4/
> 
> 
> Testing
> ---
> 
> `TestGraphGenerator`, `TestV1JobServlet`
> 
> 
> Thanks,
> 
> András Piros
> 
>

[jira] [Commented] (OOZIE-3072) oozie.service.HadoopAccessorService.action.configurations should overwrite default values set in Hadoop's configuration files

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188436#comment-16188436
 ] 

Hadoop QA commented on OOZIE-3072:
--

Testing JIRA OOZIE-3072

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 3 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/66/

> oozie.service.HadoopAccessorService.action.configurations  should 
> overwrite default values set in Hadoop's configuration files
> --
>
> Key: OOZIE-3072
> URL: https://issues.apache.org/jira/browse/OOZIE-3072
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-3072.001.patch, OOZIE-3072.002.patch, 
> OOZIE-3072.003.patch, OOZIE-3072.004.patch
>
>
> When Oozie process the files defined in 
> oozie.service.HadoopAccessorService.action.configurations it uses 
> Configuration.injectDefauts(). This prevents overwriting things defined 
> inside any of the configuration files Oozie is providing default values from 
> (like yarn-site.xml, mapred-site.xml)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-3072 PreCommit Build #66

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-3072
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/66/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.51 MB...]
+1 the patch does not introduce any line longer than 132
+1 the patch does adds/modifies 3 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 77 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/pig].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS - patch does not compile, cannot run testcases
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/66/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  0 
00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
100 3706k  100 3706k0 0  3564k  0  0:00:01  0:00:01 --:--:-- 3564k
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-3072
Archiving artifacts
[Fast Archiver] Compressed 1.79 MB of artifacts by 80.2% relative to #60
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
All tests passed

Re: Review Request 62352: OOZIE-2406 Completely rewrite GraphGenerator code

[András Piros]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62352/
---

(Updated Oct. 2, 2017, 4:41 p.m.)


Review request for oozie and Robert Kanter.


Changes
---

Addressing FindBugs error.


Repository: oozie-git


Description
---

OOZIE-2406 Completely rewrite GraphGenerator code


Diffs (updated)
-

  client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
74843dc9a3cbb8cef38e97f81e34731f191d9aab 
  core/pom.xml 6f9adb66af9344ac7d2212cdc31aa203ec06c286 
  core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 
059d3cf6dc251b49940af29d82cbdd817043a176 
  core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 
10812c675ebc0cc4aefca9f4a85ef2fc26d143e1 
  core/src/main/java/org/apache/oozie/util/GraphGenerator.java 
6ded2c6dc15c9e8453ff800407ff0324be185f41 
  core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/GraphRenderer.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/OutputFormat.java PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/WorkflowActionNode.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/servlet/TestV1JobServlet.java 
ee9ab556c9b6930c406ca5dcd54065cbbebb86c9 
  core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 
002e925b57cd830ea6d83a87cea4383165116b80 
  core/src/test/java/org/apache/oozie/util/graph/TestGraphGenerator.java 
PRE-CREATION 
  core/src/test/resources/graphWF.xml 6a7b0427a9951835a7533a04b66258ded369d5bf 
  core/src/test/resources/graphWF_26_actions.xml 
a091be0f3559ede195ccc3339adee4478a8da8c0 
  core/src/test/resources/graphWF_50_actions.xml PRE-CREATION 
  docs/src/site/twiki/WebServicesAPI.twiki 
ef3e60242512decd48beb3d8c9ac747b7d128eda 
  examples/src/main/apps/java-main/workflow.xml 
98e01ca92187e1c567686f6c2b4f689bb2a5ef6a 
  pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
  webapp/src/main/webapp/oozie-console.js 
72c8a198a4ffe60f74a9f700831f65efcb3066c4 


Diff: https://reviews.apache.org/r/62352/diff/4/

Changes: https://reviews.apache.org/r/62352/diff/3-4/


Testing
---

`TestGraphGenerator`, `TestV1JobServlet`


Thanks,

András Piros

[jira] [Updated] (OOZIE-2406) Completely rewrite GraphGenerator code

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2406:

Attachment: OOZIE-2406.008.patch

Addressing FindBugs error.

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png, 
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png, 
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch, 
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch, 
> OOZIE-2406.007.patch, OOZIE-2406.008.patch, Screen Shot 2017-10-02 at 1.55.58 
> PM.png, Screen Shot 2017-10-02 at 1.56.33 PM.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-3072 PreCommit Build #64

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-3072
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/64/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.53 MB...]
+1 the patch does not introduce any line longer than 132
+1 the patch does adds/modifies 2 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 77 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [tools].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS - patch does not compile, cannot run testcases
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/64/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
 19 3706k   19  705k0 0   966k  0  0:00:03 --:--:--  0:00:03  
966k100 3706k  100 3706k0 0  4690k  0 --:--:-- --:--:-- --:--:-- 
48.8M
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-3072
Archiving artifacts
[Fast Archiver] Compressed 1.79 MB of artifacts by 64.4% relative to #60
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Commented] (OOZIE-2296) Add an Oozie diagnostic bundle tool

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188264#comment-16188264
 ] 

Hadoop QA commented on OOZIE-2296:
--

Testing JIRA OOZIE-2296

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 4 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [examples].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/65/

> Add an Oozie diagnostic bundle tool
> ---
>
> Key: OOZIE-2296
> URL: https://issues.apache.org/jira/browse/OOZIE-2296
> Project: Oozie
>  Issue Type: New Feature
>  Components: tools
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Attachments: OOZIE-2296-002.patch, OOZIE-2296-003.patch, 
> OOZIE-2296-004.patch, OOZIE-2296-005.patch, OOZIE-2296-006.patch, 
> OOZIE-2296-009.patch, OOZIE-2296-010.patch, OOZIE-2296-011.patch
>
>
> To help with our support cases, I've built a tool that collects a bunch of 
> job and other information from Oozie that throws it all in a tarball.  The 
> idea is that the user can just click a button in Cloudera Manager, and it 
> will run this tool.  
> This tool could be useful for others as an easy way to get information out of 
> Oozie, so I thought I'd contribute it here.  It's built as a "tool" (so it 
> sits next to the sharelib and database tools), and simply uses the Oozie 
> client for getting pretty much everything, so it doesn't require anything 
> special.
> Here's the information that it can get:
> # Sharelib: {{ooze admin -shareliblist}} and {{oozie admin -shareliblist 
> }}
> # Oozie Server's resolved loaded configuration (from admin endpoint)
> # Other admin commands output (queue dump, env vars, etc)
> # Thread dump (HOST:11000/oozie/admin/jvminfo.jsp)
> # Details from last n jobs and/or specific list of jobs
> #- job.properties contents
> #- XML definition
> #- verbose status for each job and each action etc
> #- Oozie logs
> #- Unfortunately, we can't get the launcher jobs' logs from Hadoop
> # Metrics/Instrumentation



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-2296 PreCommit Build #65

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-2296
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/65/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.53 MB...]
+1 the patch does not introduce any line longer than 132
+1 the patch does adds/modifies 4 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 77 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [examples].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS - patch does not compile, cannot run testcases
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/65/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  0 
00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
 24 3706k   24  895k0 0   753k  0  0:00:04  0:00:01  0:00:03  
753k100 3706k  100 3706k0 0  2938k  0  0:00:01  0:00:01 --:--:-- 
38.1M
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-2296
Archiving artifacts
[Fast Archiver] Compressed 1.80 MB of artifacts by 62.7% relative to #60
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
All tests passed

Failed: OOZIE-3072 PreCommit Build #63

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-3072
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/63/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.60 MB...]
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/pig].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS
Tests run: 2033
Tests failed: 8
Tests errors: 2

The patch failed the following testcases:

  testImportTablesOverflowBatchSize(org.apache.oozie.tools.TestDBLoadDump)
  
testImportInvalidDataLeavesTablesEmpty(org.apache.oozie.tools.TestDBLoadDump)
  
testImportToNonEmptyTablesCausesPrematureExit(org.apache.oozie.tools.TestDBLoadDump)
  testImportedDBIsExportedCorrectly(org.apache.oozie.tools.TestDBLoadDump)
  testMain(org.apache.oozie.action.hadoop.TestHiveMain)
  testPig_withNullExternalID(org.apache.oozie.action.hadoop.TestPigMain)
  testEmbeddedPigWithinPython(org.apache.oozie.action.hadoop.TestPigMain)
  testPigScript(org.apache.oozie.action.hadoop.TestPigMain)

Tests failing with errors:
  testAddXIncludeFromReader(org.apache.oozie.util.TestXConfiguration)
  testAddXIncludeFromStream(org.apache.oozie.util.TestXConfiguration)

+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/63/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 3706k0 362000 0  48878  0  0:01:17 --:--:--  0:01:17 
48878100 3706k  100 3706k0 0  4160k  0 --:--:-- --:--:-- --:--:-- 
23.8M
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-3072
Archiving artifacts
[Fast Archiver] Compressed 1.80 MB of artifacts by 62.6% relative to #60
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
10 tests failed.
FAILED:  org.apache.oozie.util.TestXConfiguration.testAddXIncludeFromReader

Error Message:
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build%402/core/target/test-classes/test-oozie-default.xml
 (No such file or directory)

Stack Trace:
java.io.FileNotFoundException: 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build%402/core/target/test-classes/test-oozie-default.xml
 (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.(FileInputStream.java:138)
at java.io.FileInputStream.(FileInputStream.java:93)
at java.io.FileReader.(FileReader.java:58)
at 
org.apache.oozie.util.TestXConfiguration.prepareXmlWithInclude(TestXConfiguration.java:94)
at 
org.apache.oozie.util.TestXConfiguration.testAddXIncludeFromReader(TestXConfiguration.java:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at 
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at 

[jira] [Commented] (OOZIE-3072) oozie.service.HadoopAccessorService.action.configurations should overwrite default values set in Hadoop's configuration files

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188227#comment-16188227
 ] 

Hadoop QA commented on OOZIE-3072:
--

Testing JIRA OOZIE-3072

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2033
.Tests failed: 8
.Tests errors: 2

.The patch failed the following testcases:

.  testImportTablesOverflowBatchSize(org.apache.oozie.tools.TestDBLoadDump)
.  
testImportInvalidDataLeavesTablesEmpty(org.apache.oozie.tools.TestDBLoadDump)
.  
testImportToNonEmptyTablesCausesPrematureExit(org.apache.oozie.tools.TestDBLoadDump)
.  testImportedDBIsExportedCorrectly(org.apache.oozie.tools.TestDBLoadDump)
.  testMain(org.apache.oozie.action.hadoop.TestHiveMain)
.  testPig_withNullExternalID(org.apache.oozie.action.hadoop.TestPigMain)
.  testEmbeddedPigWithinPython(org.apache.oozie.action.hadoop.TestPigMain)
.  testPigScript(org.apache.oozie.action.hadoop.TestPigMain)

.Tests failing with errors:
.  testAddXIncludeFromReader(org.apache.oozie.util.TestXConfiguration)
.  testAddXIncludeFromStream(org.apache.oozie.util.TestXConfiguration)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/63/

> oozie.service.HadoopAccessorService.action.configurations  should 
> overwrite default values set in Hadoop's configuration files
> --
>
> Key: OOZIE-3072
> URL: https://issues.apache.org/jira/browse/OOZIE-3072
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-3072.001.patch, OOZIE-3072.002.patch, 
> OOZIE-3072.003.patch, OOZIE-3072.004.patch
>
>
> When Oozie process the files defined in 
> oozie.service.HadoopAccessorService.action.configurations it uses 
> Configuration.injectDefauts(). This prevents overwriting things defined 
> inside any of the configuration files Oozie is providing default values from 
> (like yarn-site.xml, mapred-site.xml)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2896) Ensure compatibility for existing LauncherMapper settings

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188226#comment-16188226
 ] 

Peter Cseh commented on OOZIE-2896:
---

The approach looks good to me.
Please make it somewhat configurable, at least give it the ability to turn the 
whole thing off.

> Ensure compatibility for existing LauncherMapper settings
> -
>
> Key: OOZIE-2896
> URL: https://issues.apache.org/jira/browse/OOZIE-2896
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Bacsko
>Assignee: Andras Piros
>Priority: Blocker
>
> There are a couple of LauncherMapper settings that we should support for 
> backward compatibility reasons.
> For example, {{oozie.launcher.mapreduce.map.memory.mb}}, 
> {{mapred.child.env}}, {{mapred.map.env}}, etc.
> We also have to discuss the list of properties we're going to support, since 
> there are many of them.
> List of hadoop properties under this task:
> mapreduce.job.queuename
> mapreduce.map.memory.mb
> mapreduce.map.java.opts
> mapreduce.map.env
> mapreduce.job.priority
> yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms
> yarn.app.mapreduce.am.resource.mb
> yarn.app.mapreduce.am.command-opts
> yarn.app.mapreduce.am.env
> ACL related properties should go under OOZIE-2897:
> mapreduce.job.acl-view-job
> mapreduce.job.acl-modfy-job



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3072) oozie.service.HadoopAccessorService.action.configurations should overwrite default values set in Hadoop's configuration files

[Peter Cseh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh updated OOZIE-3072:
--
Attachment: OOZIE-3072.004.patch

Thanks for the review and the test case suggestion!

# Adding new test case with  section in the workflow to test it does 
not get overwritten.
# Removing a function that is not used.
# Moving setName around so it won't be overwritten by action-defaults.

> oozie.service.HadoopAccessorService.action.configurations  should 
> overwrite default values set in Hadoop's configuration files
> --
>
> Key: OOZIE-3072
> URL: https://issues.apache.org/jira/browse/OOZIE-3072
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-3072.001.patch, OOZIE-3072.002.patch, 
> OOZIE-3072.003.patch, OOZIE-3072.004.patch
>
>
> When Oozie process the files defined in 
> oozie.service.HadoopAccessorService.action.configurations it uses 
> Configuration.injectDefauts(). This prevents overwriting things defined 
> inside any of the configuration files Oozie is providing default values from 
> (like yarn-site.xml, mapred-site.xml)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188178#comment-16188178
 ] 

Hadoop QA commented on OOZIE-2406:
--

Testing JIRA OOZIE-2406

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 5 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:red}-1{color} There are [1] new bugs found below threshold in total that 
must be fixed.
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:red}-1{color} There are [1] new bugs found below threshold in [core] 
that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones): 
core/findbugs-new.html
. The most important FindBugs errors are:
. At GraphGenerator.java:[line 81]: The use of XMLReader.parse(...) is 
vulnerable to XML External Entity attacks
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2036
.Tests rerun: 51
.Tests failed at first run: 
org.apache.oozie.action.hadoop.TestJavaActionExecutor,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/62/

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png, 
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png, 
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch, 
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch, 
> OOZIE-2406.007.patch, Screen Shot 2017-10-02 at 1.55.58 PM.png, Screen Shot 
> 2017-10-02 at 1.56.33 PM.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> 

Failed: OOZIE-2406 PreCommit Build #62

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-2406
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/62/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.98 MB...]
WARNING: the current HEAD has 77 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
-1 There are [1] new bugs found below threshold in total that must be fixed.
 +1 There are no new bugs found in [examples].
 -1 There are [1] new bugs found below threshold in [core] that must be fixed.
 You can find the FindBugs diff here (look for the red and orange ones): 
core/findbugs-new.html
 The most important FindBugs errors are:
 At GraphGenerator.java:[line 81]: The use of XMLReader.parse(...) is 
vulnerable to XML External Entity attacks
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2036
Tests rerun: 51
Tests failed at first run: 
org.apache.oozie.action.hadoop.TestJavaActionExecutor,
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/62/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100 3706k  100 3706k0 0  4577k  0 --:--:-- --:--:-- --:--:-- 5763k
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-2406
Archiving artifacts
[Fast Archiver] Compressed 2.18 MB of artifacts by 47.2% relative to #60
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188091#comment-16188091
 ] 

Hadoop QA commented on OOZIE-2406:
--

Testing JIRA OOZIE-2406

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 5 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:red}-1{color} There are [1] new bugs found below threshold in total that 
must be fixed.
. {color:red}-1{color} There are [1] new bugs found below threshold in [core] 
that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones): 
core/findbugs-new.html
. The most important FindBugs errors are:
. At GraphGenerator.java:[line 81]: The use of XMLReader.parse(...) is 
vulnerable to XML External Entity attacks
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [tools].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2036
.Tests failed: 0
.Tests errors: 1

.The patch failed the following testcases:

.  

.Tests failing with errors:
.  testJMXInstrumentation(org.apache.oozie.util.TestMetricsInstrumentation)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/61/

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png, 
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png, 
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch, 
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch, 
> OOZIE-2406.007.patch, Screen Shot 2017-10-02 at 1.55.58 PM.png, Screen Shot 
> 2017-10-02 at 1.56.33 PM.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork 

Failed: OOZIE-2406 PreCommit Build #61

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-2406
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/61/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.67 MB...]
 The most important FindBugs errors are:
 At GraphGenerator.java:[line 81]: The use of XMLReader.parse(...) is 
vulnerable to XML External Entity attacks
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [tools].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS
Tests run: 2036
Tests failed: 0
Tests errors: 1

The patch failed the following testcases:

  

Tests failing with errors:
  testJMXInstrumentation(org.apache.oozie.util.TestMetricsInstrumentation)

+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/61/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100 3706k  100 3706k0 0  3088k  0  0:00:01  0:00:01 --:--:-- 5161k
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-2406
Archiving artifacts
[Fast Archiver] Compressed 1.87 MB of artifacts by 55.0% relative to #60
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
1 tests failed.
FAILED:  org.apache.oozie.util.TestMetricsInstrumentation.testJMXInstrumentation

Error Message:
Unable to open socket file: target process not responding or HotSpot VM not 
loaded

Stack Trace:
com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: 
target process not responding or HotSpot VM not loaded
at 
sun.tools.attach.LinuxVirtualMachine.(LinuxVirtualMachine.java:106)
at 
sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:78)
at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:250)
at 
org.apache.oozie.util.TestMetricsInstrumentation.testJMXInstrumentation(TestMetricsInstrumentation.java:235)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at 
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 

[jira] [Commented] (OOZIE-3072) oozie.service.HadoopAccessorService.action.configurations should overwrite default values set in Hadoop's configuration files

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188075#comment-16188075
 ] 

Peter Bacsko commented on OOZIE-3072:
-

+1

> oozie.service.HadoopAccessorService.action.configurations  should 
> overwrite default values set in Hadoop's configuration files
> --
>
> Key: OOZIE-3072
> URL: https://issues.apache.org/jira/browse/OOZIE-3072
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-3072.001.patch, OOZIE-3072.002.patch, 
> OOZIE-3072.003.patch
>
>
> When Oozie process the files defined in 
> oozie.service.HadoopAccessorService.action.configurations it uses 
> Configuration.injectDefauts(). This prevents overwriting things defined 
> inside any of the configuration files Oozie is providing default values from 
> (like yarn-site.xml, mapred-site.xml)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3072) oozie.service.HadoopAccessorService.action.configurations should overwrite default values set in Hadoop's configuration files

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188074#comment-16188074
 ] 

Hadoop QA commented on OOZIE-3072:
--

Testing JIRA OOZIE-3072

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2033
.Tests rerun: 72
.Tests failed at first run: 
org.apache.oozie.command.coord.TestCoordActionsKillXCommand,org.apache.oozie.command.coord.TestCoordActionInputCheckXCommandNonUTC,org.apache.oozie.action.hadoop.TestJavaActionExecutor,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/60/

> oozie.service.HadoopAccessorService.action.configurations  should 
> overwrite default values set in Hadoop's configuration files
> --
>
> Key: OOZIE-3072
> URL: https://issues.apache.org/jira/browse/OOZIE-3072
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-3072.001.patch, OOZIE-3072.002.patch, 
> OOZIE-3072.003.patch
>
>
> When Oozie process the files defined in 
> oozie.service.HadoopAccessorService.action.configurations it uses 
> Configuration.injectDefauts(). This prevents overwriting things defined 
> inside any of the configuration files Oozie is providing default values from 
> (like yarn-site.xml, mapred-site.xml)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

[Attila Sasvari]

> On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 97-98 (patched)
> > 
> >
> > Something to consider for all output of the tool, not just here: we're 
> > outputting most of the info in a human-readable format.  Should we think 
> > about using a machine-readable format?  Or maybe having the option for one? 
> >  Or doing both?  The idea being that someone would then be able to write 
> > their own tool that could analyze stuff.  We already have some code 
> > somewhere that converts a WorkflowJob into JSON, so it shouldn't be a lot 
> > of work to add this either.  That might also be a good idea from a 
> > compatibility perspective - i.e. what's the compatibility story on this 
> > out?  If there's a new field, what do we do?

It is a good idea, and I would create a separate JIRA to discuss/design and 
implement it. OOZIE-3074


> On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 163 (patched)
> > 
> >
> > I think the JHS may also be required, in the cases where the RM has 
> > forgotten about the job.
> > 
> > And what about HDFS?  That's required too.
> > 
> > I'm thinking we might be best off not doing these checks.  It's too 
> > complicated (CM spent a lot of effort on this) and we can't check for 
> > everything (e.g. what if log aggregation is turned off?).  Besides, we're 
> > already handling exceptions below when trying to get the logs - if the RM, 
> > JHS, HDFS, etc isn't working, the call will fail anyway.

I agree with it, but failing fast would be a better experience than a 30 
seconds timeout. The default retry policy of YarnClient resulted in a lot of 
retries and I could not find the proper parameter to control it (I want to 
restrict retries to a few seconds instead of minutes).


> On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 185 (patched)
> > 
> >
> > Please create a followup JIRA to change this in the future to use 
> > OOZIE-2983 ("Stream the Launcher AM Logs") once it's done.  This will also 
> > be nice in that we can get rid of the RM up check.

I will do so.


> On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 191 (patched)
> > 
> >
> > Is there not a cleaner way to do this than using a CLI like this?

I could not find one, please let me know if you have something in mind.


> On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 221 (patched)
> > 
> >
> > This won't work right if using RM HA...
> > 
> > I'd recommend using a YarnClient and passing it the 
> > hadoopConfig so it can figure out the RM address for you.  There 
> > must be a benign simple YarnClient command you can run to verify 
> > connectivity.

I could not find such command, but please let me know which one do you think 
of. Methods I tried retried to connect to RM multiple times for more minutes in 
case of a connection error.


> On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/BundleCollectorDriver.java
> > Lines 37 (patched)
> > 
> >
> > I'm not sure I like the name "BundleXYZ" for these classes.  It's 
> > ambiguous with a Bundle Job.  Perhaps 
> > "DiagBundleXYZ" instead?

These classes are in the ``org.apache.oozie.tools.diag`` package that why I 
thought names like Client, BundleXYZ are not ambigous. Will fix it.


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186754
---


On Oct. 2, 2017, 9:39 a.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Oct. 2, 2017, 9:39 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   

[jira] [Created] (OOZIE-3074) Machine-readable format for Oozie diagnostic bundles

[Attila Sasvari (JIRA)]

Attila Sasvari created OOZIE-3074:
-

 Summary: Machine-readable format for Oozie diagnostic bundles
 Key: OOZIE-3074
 URL: https://issues.apache.org/jira/browse/OOZIE-3074
 Project: Oozie
  Issue Type: New Feature
Affects Versions: 5.0.0
Reporter: Attila Sasvari


Idea it add a machine-readable format for Oozie diagnostic bundles introduced 
by OOZIE-2296. It would allow users to write their own tool that could analyze 
stuff. 

We already have some code somewhere that converts a WorkflowJob into JSON, so 
it shouldn't be a lot of work to add this either. That might also be a good 
idea from a compatibility perspective - i.e. what's the compatibility story on 
this out?  If there's a new field, what do we do?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2296) Add an Oozie diagnostic bundle tool

[Attila Sasvari (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2296:
--
Attachment: OOZIE-2296-011.patch

> Add an Oozie diagnostic bundle tool
> ---
>
> Key: OOZIE-2296
> URL: https://issues.apache.org/jira/browse/OOZIE-2296
> Project: Oozie
>  Issue Type: New Feature
>  Components: tools
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Attachments: OOZIE-2296-002.patch, OOZIE-2296-003.patch, 
> OOZIE-2296-004.patch, OOZIE-2296-005.patch, OOZIE-2296-006.patch, 
> OOZIE-2296-009.patch, OOZIE-2296-010.patch, OOZIE-2296-011.patch
>
>
> To help with our support cases, I've built a tool that collects a bunch of 
> job and other information from Oozie that throws it all in a tarball.  The 
> idea is that the user can just click a button in Cloudera Manager, and it 
> will run this tool.  
> This tool could be useful for others as an easy way to get information out of 
> Oozie, so I thought I'd contribute it here.  It's built as a "tool" (so it 
> sits next to the sharelib and database tools), and simply uses the Oozie 
> client for getting pretty much everything, so it doesn't require anything 
> special.
> Here's the information that it can get:
> # Sharelib: {{ooze admin -shareliblist}} and {{oozie admin -shareliblist 
> }}
> # Oozie Server's resolved loaded configuration (from admin endpoint)
> # Other admin commands output (queue dump, env vars, etc)
> # Thread dump (HOST:11000/oozie/admin/jvminfo.jsp)
> # Details from last n jobs and/or specific list of jobs
> #- job.properties contents
> #- XML definition
> #- verbose status for each job and each action etc
> #- Oozie logs
> #- Unfortunately, we can't get the launcher jobs' logs from Hadoop
> # Metrics/Instrumentation



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (OOZIE-3073) Do not use LogsCLI to fetch Oozie Launcher AM logs for Diagnostics bundles

[Attila Sasvari (JIRA)]

Attila Sasvari created OOZIE-3073:
-

 Summary: Do not use LogsCLI to fetch Oozie Launcher AM logs for 
Diagnostics bundles
 Key: OOZIE-3073
 URL: https://issues.apache.org/jira/browse/OOZIE-3073
 Project: Oozie
  Issue Type: Improvement
Reporter: Attila Sasvari


Fetch Oozie launcher logs based on OOZIE-2983 ("Stream the Launcher AM Logs") 
once it's done. This will also be nice in that we can get rid of the RM up 
check.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3072) oozie.service.HadoopAccessorService.action.configurations should overwrite default values set in Hadoop's configuration files

[Attila Sasvari (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16187989#comment-16187989
 ] 

Attila Sasvari commented on OOZIE-3072:
---

[~gezapeti] thanks for the patch! It looks good to me. Could you add a test 
case that includes global workflow configuration besides default action 
configuration? 

> oozie.service.HadoopAccessorService.action.configurations  should 
> overwrite default values set in Hadoop's configuration files
> --
>
> Key: OOZIE-3072
> URL: https://issues.apache.org/jira/browse/OOZIE-3072
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-3072.001.patch, OOZIE-3072.002.patch, 
> OOZIE-3072.003.patch
>
>
> When Oozie process the files defined in 
> oozie.service.HadoopAccessorService.action.configurations it uses 
> Configuration.injectDefauts(). This prevents overwriting things defined 
> inside any of the configuration files Oozie is providing default values from 
> (like yarn-site.xml, mapred-site.xml)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3072) oozie.service.HadoopAccessorService.action.configurations should overwrite default values set in Hadoop's configuration files

[Peter Cseh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh updated OOZIE-3072:
--
Attachment: OOZIE-3072.003.patch

> oozie.service.HadoopAccessorService.action.configurations  should 
> overwrite default values set in Hadoop's configuration files
> --
>
> Key: OOZIE-3072
> URL: https://issues.apache.org/jira/browse/OOZIE-3072
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-3072.001.patch, OOZIE-3072.002.patch, 
> OOZIE-3072.003.patch
>
>
> When Oozie process the files defined in 
> oozie.service.HadoopAccessorService.action.configurations it uses 
> Configuration.injectDefauts(). This prevents overwriting things defined 
> inside any of the configuration files Oozie is providing default values from 
> (like yarn-site.xml, mapred-site.xml)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3072) oozie.service.HadoopAccessorService.action.configurations should overwrite default values set in Hadoop's configuration files

[Peter Cseh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh updated OOZIE-3072:
--
Attachment: OOZIE-3072.002.patch

> oozie.service.HadoopAccessorService.action.configurations  should 
> overwrite default values set in Hadoop's configuration files
> --
>
> Key: OOZIE-3072
> URL: https://issues.apache.org/jira/browse/OOZIE-3072
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-3072.001.patch, OOZIE-3072.002.patch
>
>
> When Oozie process the files defined in 
> oozie.service.HadoopAccessorService.action.configurations it uses 
> Configuration.injectDefauts(). This prevents overwriting things defined 
> inside any of the configuration files Oozie is providing default values from 
> (like yarn-site.xml, mapred-site.xml)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3072) oozie.service.HadoopAccessorService.action.configurations should overwrite default values set in Hadoop's configuration files

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16187977#comment-16187977
 ] 

Peter Cseh commented on OOZIE-3072:
---

The change is testing that properties set in mapred-site.xml, 
mapred-default.xml and such places will be overwritten. 

1 and 4) :The first assert for maxattempts is validating that "4" is the 
default value in MR, the last assert validates that it has been changed.
2) I don't think using getInt() really matters :)
3) added.

> oozie.service.HadoopAccessorService.action.configurations  should 
> overwrite default values set in Hadoop's configuration files
> --
>
> Key: OOZIE-3072
> URL: https://issues.apache.org/jira/browse/OOZIE-3072
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-3072.001.patch
>
>
> When Oozie process the files defined in 
> oozie.service.HadoopAccessorService.action.configurations it uses 
> Configuration.injectDefauts(). This prevents overwriting things defined 
> inside any of the configuration files Oozie is providing default values from 
> (like yarn-site.xml, mapred-site.xml)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3072) oozie.service.HadoopAccessorService.action.configurations should overwrite default values set in Hadoop's configuration files

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16187967#comment-16187967
 ] 

Peter Bacsko commented on OOZIE-3072:
-

{noformat}
+assertEquals("MapReduce's default value changed", "4", 
conf.get("mapreduce.map.maxattempts"));
+ae.setupActionConf(conf, context, actionXml, new 
Path(context.getWorkflow().getAppPath()));
+assertEquals("action.bar", conf.get("action.foo"));
+assertEquals("action.barbar", 
conf.get("oozie.launcher.action.foofoo"));
+assertEquals("Maxattempts should've been overwritten", "1",
+conf.get("mapreduce.map.maxattempts"));
{noformat}

1. First message says "value changed", but isn't that what you're expecting 
here?
2. You can use {{conf.getInt()}} (not that it really matters)
3. Assertion message is missing for the next two {{assertEquals()}} calls
4. Minor: you verify maxAttempts twice, perhaps a little bit more consistent 
assertion messages would be better

> oozie.service.HadoopAccessorService.action.configurations  should 
> overwrite default values set in Hadoop's configuration files
> --
>
> Key: OOZIE-3072
> URL: https://issues.apache.org/jira/browse/OOZIE-3072
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-3072.001.patch
>
>
> When Oozie process the files defined in 
> oozie.service.HadoopAccessorService.action.configurations it uses 
> Configuration.injectDefauts(). This prevents overwriting things defined 
> inside any of the configuration files Oozie is providing default values from 
> (like yarn-site.xml, mapred-site.xml)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62352: OOZIE-2406 Completely rewrite GraphGenerator code

[András Piros]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62352/
---

(Updated Oct. 2, 2017, 11:59 a.m.)


Review request for oozie and Robert Kanter.


Changes
---

Updating based on review comments.


Repository: oozie-git


Description
---

OOZIE-2406 Completely rewrite GraphGenerator code


Diffs (updated)
-

  client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
74843dc9a3cbb8cef38e97f81e34731f191d9aab 
  core/pom.xml 6f9adb66af9344ac7d2212cdc31aa203ec06c286 
  core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 
059d3cf6dc251b49940af29d82cbdd817043a176 
  core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 
10812c675ebc0cc4aefca9f4a85ef2fc26d143e1 
  core/src/main/java/org/apache/oozie/util/GraphGenerator.java 
6ded2c6dc15c9e8453ff800407ff0324be185f41 
  core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/GraphRenderer.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/OutputFormat.java PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/WorkflowActionNode.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/servlet/TestV1JobServlet.java 
ee9ab556c9b6930c406ca5dcd54065cbbebb86c9 
  core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 
002e925b57cd830ea6d83a87cea4383165116b80 
  core/src/test/java/org/apache/oozie/util/graph/TestGraphGenerator.java 
PRE-CREATION 
  core/src/test/resources/graphWF.xml 6a7b0427a9951835a7533a04b66258ded369d5bf 
  core/src/test/resources/graphWF_26_actions.xml 
a091be0f3559ede195ccc3339adee4478a8da8c0 
  core/src/test/resources/graphWF_50_actions.xml PRE-CREATION 
  docs/src/site/twiki/WebServicesAPI.twiki 
ef3e60242512decd48beb3d8c9ac747b7d128eda 
  examples/src/main/apps/java-main/workflow.xml 
98e01ca92187e1c567686f6c2b4f689bb2a5ef6a 
  pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
  webapp/src/main/webapp/oozie-console.js 
72c8a198a4ffe60f74a9f700831f65efcb3066c4 


Diff: https://reviews.apache.org/r/62352/diff/3/

Changes: https://reviews.apache.org/r/62352/diff/2-3/


Testing
---

`TestGraphGenerator`, `TestV1JobServlet`


Thanks,

András Piros

[jira] [Updated] (OOZIE-2406) Completely rewrite GraphGenerator code

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2406:

Attachment: Screen Shot 2017-10-02 at 1.55.58 PM.png
Screen Shot 2017-10-02 at 1.56.33 PM.png

{{Instrumentation.Cron}} and {{Instrumentation.Counter}} instances.

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png, 
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png, 
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch, 
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch, Screen Shot 
> 2017-10-02 at 1.55.58 PM.png, Screen Shot 2017-10-02 at 1.56.33 PM.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16187895#comment-16187895
 ] 

Andras Piros commented on OOZIE-2406:
-

Thanks [~rkanter] for the {{Instrumentation}} intro :)

Following changes in this patchset:
* {{Instrumentation.Counter}}, {{Instrumentation.Cron}} events added for both 
{{OutputFormat}}-specific and aggregated cases
* {{OutputFormat.SVG}} added, it's used as the default one from UI
* now-unnecessary {{jung}} and obscure {{commons-collections}} dependencies 
removed
* because of {{graphviz-java}}'s {{GraphvizV8Engine}} uses 
{{com.eclipsesource.v8.V8}}, which uses thread local variables, we need that 
all rendering be run using the very same thread. This is achieved w/ a 
single-thread {{ExecutorService}} inside {{GraphivizRenderer#EXECUTOR_SERVICE}}

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png, 
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png, 
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch, 
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-2963 PreCommit Build #59

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-2963
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/59/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.59 MB...]
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 77 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/pig].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2032
Tests rerun: 115
Tests failed at first run: 
org.apache.oozie.service.TestCoordMaterializeTriggerService,org.apache.oozie.service.TestStatusTransitService,org.apache.oozie.command.coord.TestCoordActionsKillXCommand,org.apache.oozie.command.coord.TestCoordActionInputCheckXCommandNonUTC,org.apache.oozie.action.hadoop.TestJavaActionExecutor,
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/59/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100 3706k  100 3706k0 0  4216k  0 --:--:-- --:--:-- --:--:-- 8659k
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-2963
Archiving artifacts
[Fast Archiver] Compressed 1.80 MB of artifacts by 59.2% relative to #54
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Updated] (OOZIE-3072) oozie.service.HadoopAccessorService.action.configurations should overwrite default values set in Hadoop's configuration files

[Peter Cseh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh updated OOZIE-3072:
--
Attachment: OOZIE-3072.001.patch

> oozie.service.HadoopAccessorService.action.configurations  should 
> overwrite default values set in Hadoop's configuration files
> --
>
> Key: OOZIE-3072
> URL: https://issues.apache.org/jira/browse/OOZIE-3072
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-3072.001.patch
>
>
> When Oozie process the files defined in 
> oozie.service.HadoopAccessorService.action.configurations it uses 
> Configuration.injectDefauts(). This prevents overwriting things defined 
> inside any of the configuration files Oozie is providing default values from 
> (like yarn-site.xml, mapred-site.xml)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2963) Use net.hydromatic instead of org.pentaho aggdesigner-algorithmin pomfiles

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16187889#comment-16187889
 ] 

Hadoop QA commented on OOZIE-2963:
--

Testing JIRA OOZIE-2963

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2032
.Tests rerun: 115
.Tests failed at first run: 
org.apache.oozie.service.TestCoordMaterializeTriggerService,org.apache.oozie.service.TestStatusTransitService,org.apache.oozie.command.coord.TestCoordActionsKillXCommand,org.apache.oozie.command.coord.TestCoordActionInputCheckXCommandNonUTC,org.apache.oozie.action.hadoop.TestJavaActionExecutor,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/59/

> Use net.hydromatic instead of org.pentaho aggdesigner-algorithmin pomfiles
> --
>
> Key: OOZIE-2963
> URL: https://issues.apache.org/jira/browse/OOZIE-2963
> Project: Oozie
>  Issue Type: Improvement
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
>  Labels: newbie
> Fix For: 5.0.0
>
> Attachments: OOZIE-2963-0.patch, OOZIE-2963-1.patch
>
>
> getting the following message in build output
> {noformat}
> [ERROR] Artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde 
> not found.
> org.apache.maven.artifact.resolver.ArtifactNotFoundException: Could not find 
> artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde in 
> repository.cloudera.com 
> (https://repository.cloudera.com/artifactory/ext-release-local)
> Try downloading the file manually from the project website.
> Then, install it using the command: 
> mvn install:install-file -DgroupId=org.pentaho 
> -DartifactId=pentaho-aggdesigner-algorithm -Dversion=5.1.5-jhyde 
> -Dpackaging=jar -Dfile=/path/to/file
> Alternatively, if you host your own repository you can deploy the file there: 
> mvn deploy:deploy-file -DgroupId=org.pentaho 
> -DartifactId=pentaho-aggdesigner-algorithm -Dversion=5.1.5-jhyde 
> -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
>   org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde
> from 

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

[Attila Sasvari]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/
---

(Updated Oct. 2, 2017, 9:39 a.m.)


Review request for oozie.


Repository: oozie-git


Description
---

A diagnostic tool that collects a bunch of job and other information from Oozie 
in a zip file.


Diffs (updated)
-

  docs/src/site/twiki/DG_CommandLineTool.twiki 
d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
  pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
  tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
  tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
  
tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCollectorDriver.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCompressor.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleEntryWriter.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/DiagOozieClient.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/62459/diff/8/

Changes: https://reviews.apache.org/r/62459/diff/7-8/


Testing
---

- new unit tests: TestOozieDiagBundleCollector
- started Oozie with a pseudo hadoop cluster, submitted a couple workflows, and 
executed the following commands: 
-- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
-- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
http://localhost:11000/oozie -output /tmp``, 
-- ``bin/oozie-diag-bundle-collector.sh  -jobs 
001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
-output .`` (verified zip the tool generated).


Thanks,

Attila Sasvari

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

[Attila Sasvari]

> On Sept. 29, 2017, 8:17 a.m., András Piros wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 97-98 (patched)
> > 
> >
> > Put `resolvedActionsDir` and `configEntryWriter` to fields, and you got 
> > two parameters less.

Resolved action dir varies from workflow action to workflow action, and right 
now DiagBundleWriter constructor uses it.


> On Sept. 29, 2017, 8:17 a.m., András Piros wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 99-119 (patched)
> > 
> >
> > Using a `StringBuilder`, maybe?

It is a writer not where we write to.


> On Sept. 29, 2017, 8:17 a.m., András Piros wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 276-302 (patched)
> > 
> >
> > `StringBuffer`?

We are writing to a writer (StringBuffer's thread-safety is not needed).


> On Sept. 29, 2017, 8:17 a.m., András Piros wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/BundleCollectorDriver.java
> > Lines 43-45 (patched)
> > 
> >
> > `int`

Findbugs would complain about boxing / unboxing.


> On Sept. 29, 2017, 8:17 a.m., András Piros wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/BundleEntryWriter.java
> > Lines 85 (patched)
> > 
> >
> > What about try-with-resources?

Please expand. The writer object is open for a period time and write is called 
multiple times.


> On Sept. 29, 2017, 8:17 a.m., András Piros wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java
> > Lines 58-64 (patched)
> > 
> >
> > `writeSection()`

You cannot easily reuse such function in this class, look at function calls and 
types.


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186568
---


On Sept. 27, 2017, 2:12 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Sept. 27, 2017, 2:12 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml efccc346932514ada578a3462eb3c3cfe519a323 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/BundleCollectorDriver.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/BundleCompressor.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/BundleEntryWriter.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/Client.java PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/7/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

[jira] [Updated] (OOZIE-2963) Use net.hydromatic instead of org.pentaho aggdesigner-algorithmin pomfiles

[Peter Cseh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh updated OOZIE-2963:
--
Summary: Use net.hydromatic instead of org.pentaho aggdesigner-algorithmin 
pomfiles  (was: Use net.hydromatic instead of org.pentaho 
aggdesigner-algorithmin pomfile)

> Use net.hydromatic instead of org.pentaho aggdesigner-algorithmin pomfiles
> --
>
> Key: OOZIE-2963
> URL: https://issues.apache.org/jira/browse/OOZIE-2963
> Project: Oozie
>  Issue Type: Improvement
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
>  Labels: newbie
> Fix For: trunk
>
> Attachments: OOZIE-2963-0.patch, OOZIE-2963-1.patch
>
>
> getting the following message in build output
> {noformat}
> [ERROR] Artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde 
> not found.
> org.apache.maven.artifact.resolver.ArtifactNotFoundException: Could not find 
> artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde in 
> repository.cloudera.com 
> (https://repository.cloudera.com/artifactory/ext-release-local)
> Try downloading the file manually from the project website.
> Then, install it using the command: 
> mvn install:install-file -DgroupId=org.pentaho 
> -DartifactId=pentaho-aggdesigner-algorithm -Dversion=5.1.5-jhyde 
> -Dpackaging=jar -Dfile=/path/to/file
> Alternatively, if you host your own repository you can deploy the file there: 
> mvn deploy:deploy-file -DgroupId=org.pentaho 
> -DartifactId=pentaho-aggdesigner-algorithm -Dversion=5.1.5-jhyde 
> -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
>   org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde
> from the specified remote repositories:
>   repository.cloudera.com 
> (https://repository.cloudera.com/artifactory/ext-release-local, 
> releases=true, snapshots=false),
>   central (https://repo.maven.apache.org/maven2, releases=true, 
> snapshots=false),
>   central (http://repo1.maven.org/maven2, releases=true, snapshots=false),
>   apache.snapshots.repo 
> (https://repository.apache.org/content/groups/snapshots, releases=true, 
> snapshots=true),
>   datanucleus (http://www.datanucleus.org/downloads/maven2, releases=true, 
> snapshots=false),
>   Nexus (http://repository.apache.org/snapshots, releases=false, 
> snapshots=true)
>   at 
> org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:221)
>   at 
> org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:154)
>   at 
> org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:562)
>   at 
> org.apache.maven.report.projectinfo.dependencies.RepositoryUtils.resolve(RepositoryUtils.java:153)
>   at 
> org.apache.maven.report.projectinfo.dependencies.renderer.DependenciesRenderer.resolveAtrifacts(DependenciesRenderer.java:1280)
>   at 
> org.apache.maven.report.projectinfo.dependencies.renderer.DependenciesRenderer.renderSectionDependencyFileDetails(DependenciesRenderer.java:513)
>   at 
> org.apache.maven.report.projectinfo.dependencies.renderer.DependenciesRenderer.renderBody(DependenciesRenderer.java:240)
>   at 
> org.apache.maven.reporting.AbstractMavenReportRenderer.render(AbstractMavenReportRenderer.java:83)
>   at 
> org.apache.maven.report.projectinfo.DependenciesReport.executeReport(DependenciesReport.java:201)
>   at 
> org.apache.maven.reporting.AbstractMavenReport.generate(AbstractMavenReport.java:255)
>   at 
> org.apache.maven.report.projectinfo.AbstractProjectInfoReport.execute(AbstractProjectInfoReport.java:234)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:309)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194)
>   at 

[jira] [Commented] (OOZIE-2963) getting error in build ArtifactNotFoundException: Could not find artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16187797#comment-16187797
 ] 

Peter Cseh commented on OOZIE-2963:
---

+1!

> getting error in build ArtifactNotFoundException: Could not find artifact 
> org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde
> ---
>
> Key: OOZIE-2963
> URL: https://issues.apache.org/jira/browse/OOZIE-2963
> Project: Oozie
>  Issue Type: Improvement
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
>  Labels: newbie
> Fix For: trunk
>
> Attachments: OOZIE-2963-0.patch, OOZIE-2963-1.patch
>
>
> getting the following message in build output
> {noformat}
> [ERROR] Artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde 
> not found.
> org.apache.maven.artifact.resolver.ArtifactNotFoundException: Could not find 
> artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde in 
> repository.cloudera.com 
> (https://repository.cloudera.com/artifactory/ext-release-local)
> Try downloading the file manually from the project website.
> Then, install it using the command: 
> mvn install:install-file -DgroupId=org.pentaho 
> -DartifactId=pentaho-aggdesigner-algorithm -Dversion=5.1.5-jhyde 
> -Dpackaging=jar -Dfile=/path/to/file
> Alternatively, if you host your own repository you can deploy the file there: 
> mvn deploy:deploy-file -DgroupId=org.pentaho 
> -DartifactId=pentaho-aggdesigner-algorithm -Dversion=5.1.5-jhyde 
> -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
>   org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde
> from the specified remote repositories:
>   repository.cloudera.com 
> (https://repository.cloudera.com/artifactory/ext-release-local, 
> releases=true, snapshots=false),
>   central (https://repo.maven.apache.org/maven2, releases=true, 
> snapshots=false),
>   central (http://repo1.maven.org/maven2, releases=true, snapshots=false),
>   apache.snapshots.repo 
> (https://repository.apache.org/content/groups/snapshots, releases=true, 
> snapshots=true),
>   datanucleus (http://www.datanucleus.org/downloads/maven2, releases=true, 
> snapshots=false),
>   Nexus (http://repository.apache.org/snapshots, releases=false, 
> snapshots=true)
>   at 
> org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:221)
>   at 
> org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:154)
>   at 
> org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:562)
>   at 
> org.apache.maven.report.projectinfo.dependencies.RepositoryUtils.resolve(RepositoryUtils.java:153)
>   at 
> org.apache.maven.report.projectinfo.dependencies.renderer.DependenciesRenderer.resolveAtrifacts(DependenciesRenderer.java:1280)
>   at 
> org.apache.maven.report.projectinfo.dependencies.renderer.DependenciesRenderer.renderSectionDependencyFileDetails(DependenciesRenderer.java:513)
>   at 
> org.apache.maven.report.projectinfo.dependencies.renderer.DependenciesRenderer.renderBody(DependenciesRenderer.java:240)
>   at 
> org.apache.maven.reporting.AbstractMavenReportRenderer.render(AbstractMavenReportRenderer.java:83)
>   at 
> org.apache.maven.report.projectinfo.DependenciesReport.executeReport(DependenciesReport.java:201)
>   at 
> org.apache.maven.reporting.AbstractMavenReport.generate(AbstractMavenReport.java:255)
>   at 
> org.apache.maven.report.projectinfo.AbstractProjectInfoReport.execute(AbstractProjectInfoReport.java:234)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:309)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194)
>   at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:107)
>  

[jira] [Updated] (OOZIE-2963) Use net.hydromatic instead of org.pentaho aggdesigner-algorithmin pomfile

[Peter Cseh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh updated OOZIE-2963:
--
Summary: Use net.hydromatic instead of org.pentaho aggdesigner-algorithmin 
pomfile  (was: getting error in build ArtifactNotFoundException: Could not find 
artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde)

> Use net.hydromatic instead of org.pentaho aggdesigner-algorithmin pomfile
> -
>
> Key: OOZIE-2963
> URL: https://issues.apache.org/jira/browse/OOZIE-2963
> Project: Oozie
>  Issue Type: Improvement
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
>  Labels: newbie
> Fix For: trunk
>
> Attachments: OOZIE-2963-0.patch, OOZIE-2963-1.patch
>
>
> getting the following message in build output
> {noformat}
> [ERROR] Artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde 
> not found.
> org.apache.maven.artifact.resolver.ArtifactNotFoundException: Could not find 
> artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde in 
> repository.cloudera.com 
> (https://repository.cloudera.com/artifactory/ext-release-local)
> Try downloading the file manually from the project website.
> Then, install it using the command: 
> mvn install:install-file -DgroupId=org.pentaho 
> -DartifactId=pentaho-aggdesigner-algorithm -Dversion=5.1.5-jhyde 
> -Dpackaging=jar -Dfile=/path/to/file
> Alternatively, if you host your own repository you can deploy the file there: 
> mvn deploy:deploy-file -DgroupId=org.pentaho 
> -DartifactId=pentaho-aggdesigner-algorithm -Dversion=5.1.5-jhyde 
> -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
>   org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde
> from the specified remote repositories:
>   repository.cloudera.com 
> (https://repository.cloudera.com/artifactory/ext-release-local, 
> releases=true, snapshots=false),
>   central (https://repo.maven.apache.org/maven2, releases=true, 
> snapshots=false),
>   central (http://repo1.maven.org/maven2, releases=true, snapshots=false),
>   apache.snapshots.repo 
> (https://repository.apache.org/content/groups/snapshots, releases=true, 
> snapshots=true),
>   datanucleus (http://www.datanucleus.org/downloads/maven2, releases=true, 
> snapshots=false),
>   Nexus (http://repository.apache.org/snapshots, releases=false, 
> snapshots=true)
>   at 
> org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:221)
>   at 
> org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:154)
>   at 
> org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:562)
>   at 
> org.apache.maven.report.projectinfo.dependencies.RepositoryUtils.resolve(RepositoryUtils.java:153)
>   at 
> org.apache.maven.report.projectinfo.dependencies.renderer.DependenciesRenderer.resolveAtrifacts(DependenciesRenderer.java:1280)
>   at 
> org.apache.maven.report.projectinfo.dependencies.renderer.DependenciesRenderer.renderSectionDependencyFileDetails(DependenciesRenderer.java:513)
>   at 
> org.apache.maven.report.projectinfo.dependencies.renderer.DependenciesRenderer.renderBody(DependenciesRenderer.java:240)
>   at 
> org.apache.maven.reporting.AbstractMavenReportRenderer.render(AbstractMavenReportRenderer.java:83)
>   at 
> org.apache.maven.report.projectinfo.DependenciesReport.executeReport(DependenciesReport.java:201)
>   at 
> org.apache.maven.reporting.AbstractMavenReport.generate(AbstractMavenReport.java:255)
>   at 
> org.apache.maven.report.projectinfo.AbstractProjectInfoReport.execute(AbstractProjectInfoReport.java:234)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:309)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194)
>   

Re: Review Request 62352: OOZIE-2406 Completely rewrite GraphGenerator code

[András Piros]

> On Sept. 18, 2017, 11:39 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java
> > Lines 47 (patched)
> > 
> >
> > Should we make this configurable via oozie-site?

Removed totally.


> On Sept. 18, 2017, 11:39 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java
> > Lines 150 (patched)
> > 
> >
> > Isn't this going to cause an NPE when using Jung because JungRenderer 
> > returns ``null`` for ``renderDot()``?  We should probably check if ``dot == 
> > null`` and do something nicer.

As `JungRenderer` is removed, this is not an issue anymore. Anyway, introducing 
nullcheck.


- András


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62352/#review185629
---


On Sept. 17, 2017, 5:28 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62352/
> ---
> 
> (Updated Sept. 17, 2017, 5:28 p.m.)
> 
> 
> Review request for oozie and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2406 Completely rewrite GraphGenerator code
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 74843dc9a3cbb8cef38e97f81e34731f191d9aab 
>   core/pom.xml b0809546d048c2acbcbea8af5f8947eb0eaece9e 
>   core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 
> 10812c675ebc0cc4aefca9f4a85ef2fc26d143e1 
>   core/src/main/java/org/apache/oozie/util/GraphGenerator.java 
> 6ded2c6dc15c9e8453ff800407ff0324be185f41 
>   core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/GraphRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/JungRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/OutputFormat.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/WorkflowActionNode.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java 
> PRE-CREATION 
>   core/src/test/java/org/apache/oozie/servlet/TestV1JobServlet.java 
> ee9ab556c9b6930c406ca5dcd54065cbbebb86c9 
>   core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 
> 002e925b57cd830ea6d83a87cea4383165116b80 
>   core/src/test/java/org/apache/oozie/util/graph/TestGraphGenerator.java 
> PRE-CREATION 
>   core/src/test/resources/graphWF.xml 
> 6a7b0427a9951835a7533a04b66258ded369d5bf 
>   core/src/test/resources/graphWF_26_actions.xml 
> a091be0f3559ede195ccc3339adee4478a8da8c0 
>   core/src/test/resources/graphWF_50_actions.xml PRE-CREATION 
>   docs/src/site/twiki/WebServicesAPI.twiki 
> ef3e60242512decd48beb3d8c9ac747b7d128eda 
>   pom.xml db18f30814b9b6a73ba872c2cd7946692d0b876b 
> 
> 
> Diff: https://reviews.apache.org/r/62352/diff/2/
> 
> 
> Testing
> ---
> 
> `TestGraphGenerator`, `TestV1JobServlet`
> 
> 
> Thanks,
> 
> András Piros
> 
>

[jira] [Assigned] (OOZIE-3072) oozie.service.HadoopAccessorService.action.configurations should overwrite default values set in Hadoop's configuration files

[Peter Cseh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh reassigned OOZIE-3072:
-

Assignee: Peter Cseh

> oozie.service.HadoopAccessorService.action.configurations  should 
> overwrite default values set in Hadoop's configuration files
> --
>
> Key: OOZIE-3072
> URL: https://issues.apache.org/jira/browse/OOZIE-3072
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
>
> When Oozie process the files defined in 
> oozie.service.HadoopAccessorService.action.configurations it uses 
> Configuration.injectDefauts(). This prevents overwriting things defined 
> inside any of the configuration files Oozie is providing default values from 
> (like yarn-site.xml, mapred-site.xml)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3072) oozie.service.HadoopAccessorService.action.configurations should overwrit default values set in Hadoop's configuration files

[Peter Cseh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh updated OOZIE-3072:
--
Summary: oozie.service.HadoopAccessorService.action.configurations   should 
overwrit default values set in Hadoop's configuration files  (was: 
oozie.service.HadoopAccessorService.action.configurations should overwrite 
define default values set in Hadoop's configuration files)

> oozie.service.HadoopAccessorService.action.configurations  should 
> overwrit default values set in Hadoop's configuration files
> -
>
> Key: OOZIE-3072
> URL: https://issues.apache.org/jira/browse/OOZIE-3072
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>
> When Oozie process the files defined in 
> oozie.service.HadoopAccessorService.action.configurations it uses 
> Configuration.injectDefauts(). This prevents overwriting things defined 
> inside any of the configuration files Oozie is providing default values from 
> (like yarn-site.xml, mapred-site.xml)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3072) oozie.service.HadoopAccessorService.action.configurations should overwrite default values set in Hadoop's configuration files

[Peter Cseh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh updated OOZIE-3072:
--
Summary: oozie.service.HadoopAccessorService.action.configurations   should 
overwrite default values set in Hadoop's configuration files  (was: 
oozie.service.HadoopAccessorService.action.configurationsshould overwrit 
default values set in Hadoop's configuration files)

> oozie.service.HadoopAccessorService.action.configurations  should 
> overwrite default values set in Hadoop's configuration files
> --
>
> Key: OOZIE-3072
> URL: https://issues.apache.org/jira/browse/OOZIE-3072
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>
> When Oozie process the files defined in 
> oozie.service.HadoopAccessorService.action.configurations it uses 
> Configuration.injectDefauts(). This prevents overwriting things defined 
> inside any of the configuration files Oozie is providing default values from 
> (like yarn-site.xml, mapred-site.xml)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (108 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3071  Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 
than Spark 2.2.0
https://issues.apache.org/jira/browse/OOZIE-3071
OOZIE-3063  Sanitizing variables that are part of openjpa.ConnectionProperties
https://issues.apache.org/jira/browse/OOZIE-3063
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues.apache.org/jira/browse/OOZIE-3062
OOZIE-3031  Coord job with only unresolved dependencies doesn't timeout
https://issues.apache.org/jira/browse/OOZIE-3031
OOZIE-3022  fix for warning has no file and won't be listed in dependency files 
details
https://issues.apache.org/jira/browse/OOZIE-3022
OOZIE-3002  address findbugs errors in client lib
https://issues.apache.org/jira/browse/OOZIE-3002
OOZIE-3001  core library has many instances of warnings with trailing spaces 
and lines longer than 132 chars
https://issues.apache.org/jira/browse/OOZIE-3001
OOZIE-2997  files contain trailing white spaces in client lib
https://issues.apache.org/jira/browse/OOZIE-2997
OOZIE-2996  add option for -UseGCOverheadLimit to maven opts as sometimes local 
testing fails
https://issues.apache.org/jira/browse/OOZIE-2996
OOZIE-2978  Remove code that handles Pig versions before 0.8 
https://issues.apache.org/jira/browse/OOZIE-2978
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues.apache.org/jira/browse/OOZIE-2975
OOZIE-2969  Drop support for Java 1.7
https://issues.apache.org/jira/browse/OOZIE-2969
OOZIE-2964  Add -Xdoclint:none to javadoc opts to avoid warnings
https://issues.apache.org/jira/browse/OOZIE-2964
OOZIE-2963  getting error in build ArtifactNotFoundException: Could not find 
artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde
https://issues.apache.org/jira/browse/OOZIE-2963
OOZIE-2962  bump maven-javadoc-plugin to 2.10.4
https://issues.apache.org/jira/browse/OOZIE-2962
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues.apache.org/jira/browse/OOZIE-2934
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2914  Consolidate Trim 
https://issues.apache.org/jira/browse/OOZIE-2914
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues.apache.org/jira/browse/OOZIE-2867
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues.apache.org/jira/browse/OOZIE-2829
OOZIE-2826  Falcon feed fails to aws s3; Oozie joda time version does not meet 
required jar version 2.2 or later
https://issues.apache.org/jira/browse/OOZIE-2826
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions