[jira] [Commented] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action

[Raghavi Ravi (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168796#comment-16168796
 ] 

Raghavi Ravi commented on OOZIE-3057:
-

[~rkanter] 

> Custom Partitioner not working in Oozie Mapreduce action
> 
>
> Key: OOZIE-3057
> URL: https://issues.apache.org/jira/browse/OOZIE-3057
> Project: Oozie
>  Issue Type: Bug
>  Components: action, workflow
>Affects Versions: 4.1.0
> Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo)
> Linux version 3.10.0-327.10.1.el7.x86_64 
> (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
> Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
> oozie version - 4.1.0
> cdh version - 5.10.1
> Hue™ 3.11 - The Hadoop UI
>Reporter: Raghavi Ravi
>Priority: Critical
>
> I implemented secondary sort in mapreduce using old API 
> (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue).
> Though I have set the partitioner class in the properties, the partitioner is 
> not being executed. So, I'm not getting output as expected.
> The same code runs fine when run using hadoop command from CLI.
> And here is my workflow.xml
> 
> 
> 
> Action failed, error 
> message[${wf:errorMessage(wf:lastErrorNode())}]
> 
> 
> 
> ${jobTracker}
> ${nameNode}
> 
> 
> mapred.output.dir
> /test_1109_3
> 
> 
> mapred.input.dir
> 
> /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/
> 
> 
> mapred.input.format.class
> org.apache.hadoop.hive.ql.io.RCFileInputFormat
> 
> 
> mapred.mapper.class
> PonRankMapper
> 
> 
> mapred.reducer.class
> PonRankReducer
> 
> 
> mapred.output.value.comparator.class
> PonRankGroupingComparator
> 
> 
> mapred.mapoutput.key.class
> PonRankPair
> 
> 
> mapred.mapoutput.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.output.key.class
> org.apache.hadoop.io.NullWritable
> 
> 
> mapred.reduce.output.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.tasks
> 1
> 
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> 
> mapred.mapper.new-api
> False
> 
> 
> 
> 
> 
> 
> 
> When running using hadoop jar command, I set the partitioner class using 
> JobConf.setPartitionerClass API.
>  Partitioner is not executed when using old API . Inspite of adding the 
> property.
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> Executed the same logic using new API's (org.apache.hadoop.mapreduce) and 
> added mapreduce.partitioner.class property in workflow.
> Partitioner was executed and desired outcome was seen.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62350: OOZIE-3054 Disable erasure coding for sharelib if Oozie runs on Hadoop 3

[Robert Kanter]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62350/#review185504
---




tools/src/main/java/org/apache/oozie/tools/Hadoop3Support.java
Lines 42 (patched)


It would be good to have a unit test for this; but if that's not practical, 
I'm okay with you just verifying this on a cluster.


- Robert Kanter


On Sept. 15, 2017, 12:07 p.m., Peter Bacsko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62350/
> ---
> 
> (Updated Sept. 15, 2017, 12:07 p.m.)
> 
> 
> Review request for oozie, András Piros, Peter Cseh, and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> See https://issues.apache.org/jira/browse/OOZIE-3054
> 
> 
> Diffs
> -
> 
>   tools/src/main/java/org/apache/oozie/tools/Hadoop3Support.java PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java a844aa0f8 
> 
> 
> Diff: https://reviews.apache.org/r/62350/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Peter Bacsko
> 
>

Re: Review Request 62350: OOZIE-3054 Disable erasure coding for sharelib if Oozie runs on Hadoop 3

[Robert Kanter]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62350/#review185501
---




tools/src/main/java/org/apache/oozie/tools/Hadoop3Support.java
Lines 35 (patched)


Let's call this something more specific to what it's doing.  Maybe 
"ECPolicyDisabler" or something like that.



tools/src/main/java/org/apache/oozie/tools/Hadoop3Support.java
Lines 44 (patched)


"Found Hadoop that supports Erasure Coding.  Trying to disable Erasure 
Coding for " + path



tools/src/main/java/org/apache/oozie/tools/Hadoop3Support.java
Lines 58 (patched)


System.out.println("Done");



tools/src/main/java/org/apache/oozie/tools/Hadoop3Support.java
Lines 61 (patched)


"Found Hadoop that does not support Erasure Coding.  Not taking any action."



tools/src/main/java/org/apache/oozie/tools/Hadoop3Support.java
Lines 66 (patched)


Let's rename this to ``supportsErasureCoding()``



tools/src/main/java/org/apache/oozie/tools/Hadoop3Support.java
Lines 68 (patched)


This is for the sharelib upload tool, so it will only happen once per JVM.  
So I don't think we need to bother caching it.


- Robert Kanter


On Sept. 15, 2017, 12:07 p.m., Peter Bacsko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62350/
> ---
> 
> (Updated Sept. 15, 2017, 12:07 p.m.)
> 
> 
> Review request for oozie, András Piros, Peter Cseh, and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> See https://issues.apache.org/jira/browse/OOZIE-3054
> 
> 
> Diffs
> -
> 
>   tools/src/main/java/org/apache/oozie/tools/Hadoop3Support.java PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java a844aa0f8 
> 
> 
> Diff: https://reviews.apache.org/r/62350/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Peter Bacsko
> 
>

Re: Review Request 62352: OOZIE-2406 Completely rewrite GraphGenerator code

[Robert Kanter]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62352/#review185497
---




core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java
Lines 46 (patched)


?



core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java
Lines 56 (patched)


?



core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java
Lines 57-58 (patched)


Add descriptions like the above constructor



core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java
Lines 75 (patched)


description



core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java
Lines 81 (patched)


Should this go in the ``newXMLReader()`` method?



core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java
Lines 83-84 (patched)


Remove



core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java
Lines 86 (patched)


Do we need to close this?



pom.xml
Lines 1510 (patched)


Looks like the newest version is 0.2.2.  Any reason why we're using 0.1.6?

http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22guru.nidi%22%20AND%20a%3A%22graphviz-java%22

(That said, 0.1.6 is only from May 2017, so it's not actually that old)


- Robert Kanter


On Sept. 15, 2017, 12:45 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62352/
> ---
> 
> (Updated Sept. 15, 2017, 12:45 p.m.)
> 
> 
> Review request for oozie and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2406 Completely rewrite GraphGenerator code
> 
> 
> Diffs
> -
> 
>   core/pom.xml b0809546d048c2acbcbea8af5f8947eb0eaece9e 
>   core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 
> 10812c675ebc0cc4aefca9f4a85ef2fc26d143e1 
>   core/src/main/java/org/apache/oozie/util/GraphGenerator.java 
> 6ded2c6dc15c9e8453ff800407ff0324be185f41 
>   core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/GraphRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/JungRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/WorkflowActionNode.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java 
> PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 
> 002e925b57cd830ea6d83a87cea4383165116b80 
>   core/src/test/resources/graph-with-many-nodes.png PRE-CREATION 
>   core/src/test/resources/graphWF_100_actions.xml PRE-CREATION 
>   pom.xml db18f30814b9b6a73ba872c2cd7946692d0b876b 
>   sharelib/oozie/pom.xml c74c06df5313b340e27747dfdf9126b3479674af 
> 
> 
> Diff: https://reviews.apache.org/r/62352/diff/1/
> 
> 
> Testing
> ---
> 
> `TestGraphGenerator`
> 
> 
> Thanks,
> 
> András Piros
> 
>

Re: Review Request 62352: OOZIE-2406 Completely rewrite GraphGenerator code

[Robert Kanter]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62352/#review185498
---




sharelib/oozie/pom.xml
Line 59 (original), 59 (patched)


Unrelated change.


- Robert Kanter


On Sept. 15, 2017, 12:45 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62352/
> ---
> 
> (Updated Sept. 15, 2017, 12:45 p.m.)
> 
> 
> Review request for oozie and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2406 Completely rewrite GraphGenerator code
> 
> 
> Diffs
> -
> 
>   core/pom.xml b0809546d048c2acbcbea8af5f8947eb0eaece9e 
>   core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 
> 10812c675ebc0cc4aefca9f4a85ef2fc26d143e1 
>   core/src/main/java/org/apache/oozie/util/GraphGenerator.java 
> 6ded2c6dc15c9e8453ff800407ff0324be185f41 
>   core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/GraphRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/JungRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/WorkflowActionNode.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java 
> PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 
> 002e925b57cd830ea6d83a87cea4383165116b80 
>   core/src/test/resources/graph-with-many-nodes.png PRE-CREATION 
>   core/src/test/resources/graphWF_100_actions.xml PRE-CREATION 
>   pom.xml db18f30814b9b6a73ba872c2cd7946692d0b876b 
>   sharelib/oozie/pom.xml c74c06df5313b340e27747dfdf9126b3479674af 
> 
> 
> Diff: https://reviews.apache.org/r/62352/diff/1/
> 
> 
> Testing
> ---
> 
> `TestGraphGenerator`
> 
> 
> Thanks,
> 
> András Piros
> 
>

[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168306#comment-16168306
 ] 

Robert Kanter commented on OOZIE-2406:
--

I left some specific comments on RB.  A few additional things: 
# I think we can safely remove Jung and the Commons-Collections fork given that 
you got the graphviz version.  It still produces an image, and I think it's 
reasonable to assume nobody is somehow parsing those images, so it's okay if we 
change the look of the images by using a different library.  One of the goals 
of this JIRA was to remove the jung library and the Commons-Collections fork 
because they're super old, abandoned, and potentially security liabilities.
# Have you done any performance testing on the graphviz library?  The Jung 
library was slow and ate up a ton of memory, which is why we limited the number 
of nodes it could process.  If graphviz is way better, we could increase this 
number to something higher; though I think we should still have _some_ limit 
just to be safe.  Another thing to check is what happens if Oozie gets a lot of 
requests for images at the same time - can the graphviz library handle that 
well?
# Can we make the "end" circle a darker color?  It's sort of hard to see in the 
image you attached.
# I just thought of an interesting idea: it might be a good idea to create a 
cache for the generated images.  Some kind of LRU cache in memory and/or just 
writing them to the local disk.  We'd only do this for completed jobs (because 
the images will change as the job runs).  This is more of a nice to have, which 
may or may not even be necessary, and could be a lot of work, so I think this 
should be a separate JIRA.  Can you file a JIRA to look into this idea?
# It might be a neat improvement to add a way to return the dot file (I assume 
there's a way to do that from the graphviz library?).  We have to keep the 
current API returning an image for compatibility 
({{job-3?show=graph\[=true\]}}), but we could add a new parameter 
like job-3?show=graph\[=true\]\[=dot\]}} (the default 
being {{=png}}).  This would allow the caller to use the dot file in 
various ways (for example, perhaps Hue might use this).  
# Can you file a new JIRA for adding support for getting the image (whether 
that's png or dot format, or both) to the Oozie CLI?  It's currently only 
supported in the REST API.

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png, 
> OOZIE-2406.004.patch, OOZIE-2406.004.patch, OOZIE-2406.005.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (0 issues)

Subscriber: ooziedaily

You may edit this subscription at:
https://issues.apache.org/jira/secure/FilterSubscription!default.jspa?subId=16327=12325939

[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168257#comment-16168257
 ] 

Hadoop QA commented on OOZIE-2406:
--

Testing JIRA OOZIE-2406

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [tools].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2015
.Tests rerun: 82
.Tests failed at first run: 
org.apache.oozie.action.hadoop.TestJavaActionExecutor,org.apache.oozie.service.TestStatusTransitService,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/25/

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png, 
> OOZIE-2406.004.patch, OOZIE-2406.004.patch, OOZIE-2406.005.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Success: OOZIE-2406 PreCommit Build #25

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-2406
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/25/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.94 MB...]
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 77 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [tools].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2015
Tests rerun: 82
Tests failed at first run: 
org.apache.oozie.action.hadoop.TestJavaActionExecutor,org.apache.oozie.service.TestStatusTransitService,
+1 DISTRO
+1 distro tarball builds with the patch 


+1 Overall result, good!, no -1s

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/25/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100 3706k  100 3706k0 0  2735k  0  0:00:01  0:00:01 --:--:-- 7019k
Adding comment to JIRA
Comment added.

test-patch exit code: 0

[description-setter] Description set: OOZIE-2406
Archiving artifacts
[Fast Archiver] Compressed 2.14 MB of artifacts by 36.5% relative to #10
Recording test results
Email was triggered for: Success
Sending email for trigger: Success



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Updated] (OOZIE-2406) Completely rewrite GraphGenerator code

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2406:

Attachment: (was: OOZIE-2406.005.patch)

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png, 
> OOZIE-2406.004.patch, OOZIE-2406.004.patch, OOZIE-2406.005.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2406) Completely rewrite GraphGenerator code

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2406:

Attachment: OOZIE-2406.005.patch

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png, 
> OOZIE-2406.004.patch, OOZIE-2406.004.patch, OOZIE-2406.005.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action

[Raghavi Ravi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghavi Ravi updated OOZIE-3057:

Priority: Critical  (was: Major)

> Custom Partitioner not working in Oozie Mapreduce action
> 
>
> Key: OOZIE-3057
> URL: https://issues.apache.org/jira/browse/OOZIE-3057
> Project: Oozie
>  Issue Type: Bug
>  Components: action, workflow
>Affects Versions: 4.1.0
> Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo)
> Linux version 3.10.0-327.10.1.el7.x86_64 
> (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
> Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
> oozie version - 4.1.0
> cdh version - 5.10.1
> Hue™ 3.11 - The Hadoop UI
>Reporter: Raghavi Ravi
>Priority: Critical
>
> I implemented secondary sort in mapreduce using old API 
> (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue).
> Though I have set the partitioner class in the properties, the partitioner is 
> not being executed. So, I'm not getting output as expected.
> The same code runs fine when run using hadoop command from CLI.
> And here is my workflow.xml
> 
> 
> 
> Action failed, error 
> message[${wf:errorMessage(wf:lastErrorNode())}]
> 
> 
> 
> ${jobTracker}
> ${nameNode}
> 
> 
> mapred.output.dir
> /test_1109_3
> 
> 
> mapred.input.dir
> 
> /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/
> 
> 
> mapred.input.format.class
> org.apache.hadoop.hive.ql.io.RCFileInputFormat
> 
> 
> mapred.mapper.class
> PonRankMapper
> 
> 
> mapred.reducer.class
> PonRankReducer
> 
> 
> mapred.output.value.comparator.class
> PonRankGroupingComparator
> 
> 
> mapred.mapoutput.key.class
> PonRankPair
> 
> 
> mapred.mapoutput.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.output.key.class
> org.apache.hadoop.io.NullWritable
> 
> 
> mapred.reduce.output.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.tasks
> 1
> 
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> 
> mapred.mapper.new-api
> False
> 
> 
> 
> 
> 
> 
> 
> When running using hadoop jar command, I set the partitioner class using 
> JobConf.setPartitionerClass API.
>  Partitioner is not executed when using old API . Inspite of adding the 
> property.
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> Executed the same logic using new API's (org.apache.hadoop.mapreduce) and 
> added mapreduce.partitioner.class property in workflow.
> Partitioner was executed and desired outcome was seen.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action

[Raghavi Ravi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghavi Ravi updated OOZIE-3057:

Environment: 
Red Hat Enterprise Linux Server release 7.2 (Maipo)
Linux version 3.10.0-327.10.1.el7.x86_64 
(mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
oozie version - 4.1.0
cdh version - 5.10.1
Hue™ 3.11 - The Hadoop UI

  was:
Red Hat Enterprise Linux Server release 7.2 (Maipo)
Linux version 3.10.0-327.10.1.el7.x86_64 
(mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
oozie version - 4.1.0
Hue™ 3.11 - The Hadoop UI


> Custom Partitioner not working in Oozie Mapreduce action
> 
>
> Key: OOZIE-3057
> URL: https://issues.apache.org/jira/browse/OOZIE-3057
> Project: Oozie
>  Issue Type: Bug
>  Components: action, workflow
>Affects Versions: 4.1.0
> Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo)
> Linux version 3.10.0-327.10.1.el7.x86_64 
> (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
> Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
> oozie version - 4.1.0
> cdh version - 5.10.1
> Hue™ 3.11 - The Hadoop UI
>Reporter: Raghavi Ravi
>
> I implemented secondary sort in mapreduce using old API 
> (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue).
> Though I have set the partitioner class in the properties, the partitioner is 
> not being executed. So, I'm not getting output as expected.
> The same code runs fine when run using hadoop command from CLI.
> And here is my workflow.xml
> 
> 
> 
> Action failed, error 
> message[${wf:errorMessage(wf:lastErrorNode())}]
> 
> 
> 
> ${jobTracker}
> ${nameNode}
> 
> 
> mapred.output.dir
> /test_1109_3
> 
> 
> mapred.input.dir
> 
> /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/
> 
> 
> mapred.input.format.class
> org.apache.hadoop.hive.ql.io.RCFileInputFormat
> 
> 
> mapred.mapper.class
> PonRankMapper
> 
> 
> mapred.reducer.class
> PonRankReducer
> 
> 
> mapred.output.value.comparator.class
> PonRankGroupingComparator
> 
> 
> mapred.mapoutput.key.class
> PonRankPair
> 
> 
> mapred.mapoutput.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.output.key.class
> org.apache.hadoop.io.NullWritable
> 
> 
> mapred.reduce.output.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.tasks
> 1
> 
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> 
> mapred.mapper.new-api
> False
> 
> 
> 
> 
> 
> 
> 
> When running using hadoop jar command, I set the partitioner class using 
> JobConf.setPartitionerClass API.
>  Partitioner is not executed when using old API . Inspite of adding the 
> property.
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> Executed the same logic using new API's (org.apache.hadoop.mapreduce) and 
> added mapreduce.partitioner.class property in workflow.
> Partitioner was executed and desired outcome was seen.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167940#comment-16167940
 ] 

Hadoop QA commented on OOZIE-2406:
--

Testing JIRA OOZIE-2406

Cleaning local git workspace



{color:red}-1{color} Patch failed to apply to head of branch



> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png, 
> OOZIE-2406.004.patch, OOZIE-2406.004.patch, OOZIE-2406.005.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-2406 PreCommit Build #24

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-2406
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/24/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 672.13 KB...]
[INFO] Apache Oozie Share Lib . SUCCESS [  0.030 s]
[INFO] Apache Oozie Docs .. SUCCESS [  5.672 s]
[INFO] Apache Oozie WebApp  SUCCESS [  4.065 s]
[INFO] Apache Oozie Tools . SUCCESS [ 19.098 s]
[INFO] Apache Oozie MiniOozie . SUCCESS [  3.746 s]
[INFO] Apache Oozie Server  SUCCESS [ 14.974 s]
[INFO] Apache Oozie Distro  SUCCESS [  2.525 s]
[INFO] Apache Oozie ZooKeeper Security Tests .. SUCCESS [  4.850 s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 05:43 min
[INFO] Finished at: 2017-09-15T14:29:18Z
[INFO] Final Memory: 621M/1938M
[INFO] 
[TRACE] FindBugs output in HEAD verified and saved
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO

Applying patch

Checking patch core/pom.xml...
Checking patch core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java...
Checking patch core/src/main/java/org/apache/oozie/util/GraphGenerator.java...
Checking patch 
core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java...
Checking patch 
core/src/main/java/org/apache/oozie/util/graph/GraphRenderer.java...
Checking patch 
core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java...
Checking patch 
core/src/main/java/org/apache/oozie/util/graph/JungRenderer.java...
Checking patch 
core/src/main/java/org/apache/oozie/util/graph/WorkflowActionNode.java...
Checking patch 
core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java...
Checking patch core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 
=> core/src/test/java/org/apache/oozie/util/graph/TestGraphGenerator.java...
Checking patch core/src/test/resources/graph-with-many-nodes.png...
error: missing binary patch data for 
'core/src/test/resources/graph-with-many-nodes.png'
error: binary patch does not apply to 
'core/src/test/resources/graph-with-many-nodes.png'
error: core/src/test/resources/graph-with-many-nodes.png: patch does not apply
Checking patch core/src/test/resources/graphWF_100_actions.xml...
Checking patch pom.xml...
Checking patch sharelib/oozie/pom.xml...
fatal: git apply: bad git-diff - inconsistent old filename on line 1900
Patch failed to apply to head of branch

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100 3706k  100 3706k0 0  3938k  0 --:--:-- --:--:-- --:--:-- 5649k
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-2406
Archiving artifacts
[Fast Archiver] Compressed 834.83 KB of artifacts by 88.2% relative to #10
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
No tests ran.

[jira] [Updated] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action

[Raghavi Ravi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghavi Ravi updated OOZIE-3057:

Environment: 
Red Hat Enterprise Linux Server release 7.2 (Maipo)
Linux version 3.10.0-327.10.1.el7.x86_64 
(mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
oozie version - 4.1.0
Hue™ 3.11 - The Hadoop UI

  was:
Red Hat Enterprise Linux Server release 7.2 (Maipo)
Linux version 3.10.0-327.10.1.el7.x86_64 
(mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
cdh version - 5.10.1
oozie version - 4.1.0
Hue™ 3.11 - The Hadoop UI


> Custom Partitioner not working in Oozie Mapreduce action
> 
>
> Key: OOZIE-3057
> URL: https://issues.apache.org/jira/browse/OOZIE-3057
> Project: Oozie
>  Issue Type: Bug
>  Components: action, workflow
>Affects Versions: 4.1.0
> Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo)
> Linux version 3.10.0-327.10.1.el7.x86_64 
> (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
> Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
> oozie version - 4.1.0
> Hue™ 3.11 - The Hadoop UI
>Reporter: Raghavi Ravi
>
> I implemented secondary sort in mapreduce using old API 
> (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue).
> Though I have set the partitioner class in the properties, the partitioner is 
> not being executed. So, I'm not getting output as expected.
> The same code runs fine when run using hadoop command from CLI.
> And here is my workflow.xml
> 
> 
> 
> Action failed, error 
> message[${wf:errorMessage(wf:lastErrorNode())}]
> 
> 
> 
> ${jobTracker}
> ${nameNode}
> 
> 
> mapred.output.dir
> /test_1109_3
> 
> 
> mapred.input.dir
> 
> /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/
> 
> 
> mapred.input.format.class
> org.apache.hadoop.hive.ql.io.RCFileInputFormat
> 
> 
> mapred.mapper.class
> PonRankMapper
> 
> 
> mapred.reducer.class
> PonRankReducer
> 
> 
> mapred.output.value.comparator.class
> PonRankGroupingComparator
> 
> 
> mapred.mapoutput.key.class
> PonRankPair
> 
> 
> mapred.mapoutput.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.output.key.class
> org.apache.hadoop.io.NullWritable
> 
> 
> mapred.reduce.output.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.tasks
> 1
> 
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> 
> mapred.mapper.new-api
> False
> 
> 
> 
> 
> 
> 
> 
> When running using hadoop jar command, I set the partitioner class using 
> JobConf.setPartitionerClass API.
>  Partitioner is not executed when using old API . Inspite of adding the 
> property.
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> Executed the same logic using new API's (org.apache.hadoop.mapreduce) and 
> added mapreduce.partitioner.class property in workflow.
> Partitioner was executed and desired outcome was seen.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action

[Raghavi Ravi (JIRA)]

Raghavi Ravi created OOZIE-3057:
---

 Summary: Custom Partitioner not working in Oozie Mapreduce action
 Key: OOZIE-3057
 URL: https://issues.apache.org/jira/browse/OOZIE-3057
 Project: Oozie
  Issue Type: Bug
  Components: action, workflow
Affects Versions: 4.1.0
 Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo)
Linux version 3.10.0-327.10.1.el7.x86_64 
(mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
cdh version - 5.10.1
oozie version - 4.1.0
Hue™ 3.11 - The Hadoop UI
Reporter: Raghavi Ravi


I implemented secondary sort in mapreduce using old API 
(org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue).

Though I have set the partitioner class in the properties, the partitioner is 
not being executed. So, I'm not getting output as expected.

The same code runs fine when run using hadoop command from CLI.

And here is my workflow.xml




Action failed, error 
message[${wf:errorMessage(wf:lastErrorNode())}]



${jobTracker}
${nameNode}


mapred.output.dir
/test_1109_3


mapred.input.dir

/apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/


mapred.input.format.class
org.apache.hadoop.hive.ql.io.RCFileInputFormat


mapred.mapper.class
PonRankMapper


mapred.reducer.class
PonRankReducer


mapred.output.value.comparator.class
PonRankGroupingComparator


mapred.mapoutput.key.class
PonRankPair


mapred.mapoutput.value.class
org.apache.hadoop.io.Text


mapred.reduce.output.key.class
org.apache.hadoop.io.NullWritable


mapred.reduce.output.value.class
org.apache.hadoop.io.Text


mapred.reduce.tasks
1


mapred.partitioner.class
PonRankPartitioner


mapred.mapper.new-api
False








When running using hadoop jar command, I set the partitioner class using 
JobConf.setPartitionerClass API.

 Partitioner is not executed when using old API . Inspite of adding the 
property.


mapred.partitioner.class
PonRankPartitioner


Executed the same logic using new API's (org.apache.hadoop.mapreduce) and added 
mapreduce.partitioner.class property in workflow.

Partitioner was executed and desired outcome was seen.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2406) Completely rewrite GraphGenerator code

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2406:

Attachment: OOZIE-2406.005.patch

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png, 
> OOZIE-2406.004.patch, OOZIE-2406.004.patch, OOZIE-2406.005.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2406) Completely rewrite GraphGenerator code

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2406:

Attachment: (was: OOZIE-2406.005.patch)

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png, 
> OOZIE-2406.004.patch, OOZIE-2406.004.patch, OOZIE-2406.005.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167854#comment-16167854
 ] 

Hadoop QA commented on OOZIE-2406:
--

Testing JIRA OOZIE-2406

Cleaning local git workspace



{color:red}-1{color} Patch failed to apply to head of branch



> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png, 
> OOZIE-2406.004.patch, OOZIE-2406.004.patch, OOZIE-2406.005.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-2406 PreCommit Build #23

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-2406
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/23/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 672.13 KB...]
[INFO] Apache Oozie Share Lib . SUCCESS [  0.024 s]
[INFO] Apache Oozie Docs .. SUCCESS [  5.065 s]
[INFO] Apache Oozie WebApp  SUCCESS [  3.648 s]
[INFO] Apache Oozie Tools . SUCCESS [ 15.509 s]
[INFO] Apache Oozie MiniOozie . SUCCESS [  2.658 s]
[INFO] Apache Oozie Server  SUCCESS [ 15.472 s]
[INFO] Apache Oozie Distro  SUCCESS [  2.327 s]
[INFO] Apache Oozie ZooKeeper Security Tests .. SUCCESS [  4.539 s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 05:27 min
[INFO] Finished at: 2017-09-15T13:18:32Z
[INFO] Final Memory: 620M/1814M
[INFO] 
[TRACE] FindBugs output in HEAD verified and saved
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO

Applying patch

Checking patch core/pom.xml...
Checking patch core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java...
Checking patch core/src/main/java/org/apache/oozie/util/GraphGenerator.java...
Checking patch 
core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java...
Checking patch 
core/src/main/java/org/apache/oozie/util/graph/GraphRenderer.java...
Checking patch 
core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java...
Checking patch 
core/src/main/java/org/apache/oozie/util/graph/JungRenderer.java...
Checking patch 
core/src/main/java/org/apache/oozie/util/graph/WorkflowActionNode.java...
Checking patch 
core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java...
Checking patch core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 
=> core/src/test/java/org/apache/oozie/util/graph/TestGraphGenerator.java...
Checking patch core/src/test/resources/graph-with-many-nodes.png...
error: missing binary patch data for 
'core/src/test/resources/graph-with-many-nodes.png'
error: binary patch does not apply to 
'core/src/test/resources/graph-with-many-nodes.png'
error: core/src/test/resources/graph-with-many-nodes.png: patch does not apply
Checking patch core/src/test/resources/graphWF_100_actions.xml...
Checking patch pom.xml...
Checking patch sharelib/oozie/pom.xml...
fatal: git apply: bad git-diff - inconsistent old filename on line 1900
Patch failed to apply to head of branch

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  0 
00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
100 3706k  100 3706k0 0  4293k  0 --:--:-- --:--:-- --:--:-- 4293k
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-2406
Archiving artifacts
[Fast Archiver] Compressed 834.83 KB of artifacts by 88.2% relative to #10
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
No tests ran.

[jira] [Updated] (OOZIE-2406) Completely rewrite GraphGenerator code

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2406:

Attachment: OOZIE-2406.005.patch

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png, 
> OOZIE-2406.004.patch, OOZIE-2406.004.patch, OOZIE-2406.005.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2406) Completely rewrite GraphGenerator code

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2406:

Attachment: (was: OOZIE-2406.005.patch)

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png, 
> OOZIE-2406.004.patch, OOZIE-2406.004.patch, OOZIE-2406.005.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167832#comment-16167832
 ] 

Hadoop QA commented on OOZIE-2406:
--

Testing JIRA OOZIE-2406

Cleaning local git workspace



{color:red}-1{color} Patch failed to apply to head of branch



> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png, 
> OOZIE-2406.004.patch, OOZIE-2406.004.patch, OOZIE-2406.005.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-2406 PreCommit Build #22

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-2406
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/22/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 672.59 KB...]
[INFO] Apache Oozie Share Lib . SUCCESS [  0.029 s]
[INFO] Apache Oozie Docs .. SUCCESS [  5.818 s]
[INFO] Apache Oozie WebApp  SUCCESS [  3.586 s]
[INFO] Apache Oozie Tools . SUCCESS [ 18.901 s]
[INFO] Apache Oozie MiniOozie . SUCCESS [  2.839 s]
[INFO] Apache Oozie Server  SUCCESS [ 17.291 s]
[INFO] Apache Oozie Distro  SUCCESS [  2.782 s]
[INFO] Apache Oozie ZooKeeper Security Tests .. SUCCESS [  5.032 s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 05:40 min
[INFO] Finished at: 2017-09-15T12:59:15Z
[INFO] Final Memory: 621M/1828M
[INFO] 
[TRACE] FindBugs output in HEAD verified and saved
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO

Applying patch

Checking patch core/pom.xml...
Checking patch core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java...
Checking patch core/src/main/java/org/apache/oozie/util/GraphGenerator.java...
Checking patch 
core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java...
Checking patch 
core/src/main/java/org/apache/oozie/util/graph/GraphRenderer.java...
Checking patch 
core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java...
Checking patch 
core/src/main/java/org/apache/oozie/util/graph/JungRenderer.java...
Checking patch 
core/src/main/java/org/apache/oozie/util/graph/WorkflowActionNode.java...
Checking patch 
core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java...
Checking patch core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 
=> core/src/test/java/org/apache/oozie/util/graph/TestGraphGenerator.java...
Checking patch core/src/test/resources/graph-with-many-nodes.png...
error: missing binary patch data for 
'core/src/test/resources/graph-with-many-nodes.png'
error: binary patch does not apply to 
'core/src/test/resources/graph-with-many-nodes.png'
error: core/src/test/resources/graph-with-many-nodes.png: patch does not apply
Checking patch core/src/test/resources/graphWF_100_actions.xml...
Checking patch pom.xml...
Checking patch sharelib/oozie/pom.xml...
fatal: git apply: bad git-diff - inconsistent old filename on line 1900
Patch failed to apply to head of branch

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100 3706k  100 3706k0 0  3940k  0 --:--:-- --:--:-- --:--:-- 8291k
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-2406
Archiving artifacts
[Fast Archiver] Compressed 835.73 KB of artifacts by 84.2% relative to #10
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
No tests ran.

[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167813#comment-16167813
 ] 

Andras Piros commented on OOZIE-2406:
-

[~rkanter] here is the [*ReviewBoard link*|https://reviews.apache.org/r/62352/].

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png, 
> OOZIE-2406.004.patch, OOZIE-2406.004.patch, OOZIE-2406.005.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 62352: OOZIE-2406 Completely rewrite GraphGenerator code

[András Piros]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62352/
---

Review request for oozie and Robert Kanter.


Repository: oozie-git


Description
---

OOZIE-2406 Completely rewrite GraphGenerator code


Diffs
-

  core/pom.xml b0809546d048c2acbcbea8af5f8947eb0eaece9e 
  core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 
10812c675ebc0cc4aefca9f4a85ef2fc26d143e1 
  core/src/main/java/org/apache/oozie/util/GraphGenerator.java 
6ded2c6dc15c9e8453ff800407ff0324be185f41 
  core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/GraphRenderer.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/JungRenderer.java PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/WorkflowActionNode.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 
002e925b57cd830ea6d83a87cea4383165116b80 
  core/src/test/resources/graph-with-many-nodes.png PRE-CREATION 
  core/src/test/resources/graphWF_100_actions.xml PRE-CREATION 
  pom.xml db18f30814b9b6a73ba872c2cd7946692d0b876b 
  sharelib/oozie/pom.xml c74c06df5313b340e27747dfdf9126b3479674af 


Diff: https://reviews.apache.org/r/62352/diff/1/


Testing
---

`TestGraphGenerator`


Thanks,

András Piros

[jira] [Updated] (OOZIE-2406) Completely rewrite GraphGenerator code

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2406:

Attachment: OOZIE-2406.005.patch

Addressing FindBugs issues.

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png, 
> OOZIE-2406.004.patch, OOZIE-2406.004.patch, OOZIE-2406.005.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62350: OOZIE-3054 Disable erasure coding for sharelib if Oozie runs on Hadoop 3

[András Piros]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62350/#review185480
---



Some comments. Please also add unit tests.


tools/src/main/java/org/apache/oozie/tools/Hadoop3Support.java
Lines 50 (patched)


We should have log messages for the `if` case when we want to set, and for 
the now-unsettled `else` case when we don't want to set.



tools/src/main/java/org/apache/oozie/tools/Hadoop3Support.java
Lines 68 (patched)


Since `Class.forName()` is an expensive operation, it might be worth 
caching its result.



tools/src/main/java/org/apache/oozie/tools/Hadoop3Support.java
Lines 75 (patched)


Rename to `getReplicationPolicy()`.



tools/src/main/java/org/apache/oozie/tools/Hadoop3Support.java
Lines 80 (patched)


Would be better name all the possible `Exception` subclasses that may arise.



tools/src/main/java/org/apache/oozie/tools/Hadoop3Support.java
Lines 82 (patched)


Since we don't throw `RuntimeException` within `isHadoop3()`, and the first 
part of this method is covered there, I suggest having a common method 
`getECPoliciesClass()` that throws `ClassNotFoundException` and is used by both 
`isHadoop3()` and `getReplicationPolicy()`.



tools/src/main/java/org/apache/oozie/tools/Hadoop3Support.java
Lines 103 (patched)


`System.err.println()`



tools/src/main/java/org/apache/oozie/tools/Hadoop3Support.java
Lines 112 (patched)


`System.err.println()`


- András Piros


On Sept. 15, 2017, 12:07 p.m., Peter Bacsko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62350/
> ---
> 
> (Updated Sept. 15, 2017, 12:07 p.m.)
> 
> 
> Review request for oozie, András Piros, Peter Cseh, and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> See https://issues.apache.org/jira/browse/OOZIE-3054
> 
> 
> Diffs
> -
> 
>   tools/src/main/java/org/apache/oozie/tools/Hadoop3Support.java PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java a844aa0f8 
> 
> 
> Diff: https://reviews.apache.org/r/62350/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Peter Bacsko
> 
>

Review Request 62350: OOZIE-3054 Disable erasure coding for sharelib if Oozie runs on Hadoop 3

[Peter Bacsko]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62350/
---

Review request for oozie, András Piros, Peter Cseh, and Robert Kanter.


Repository: oozie-git


Description
---

See https://issues.apache.org/jira/browse/OOZIE-3054


Diffs
-

  tools/src/main/java/org/apache/oozie/tools/Hadoop3Support.java PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java a844aa0f8 


Diff: https://reviews.apache.org/r/62350/diff/1/


Testing
---


Thanks,

Peter Bacsko

[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (106 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3054  Disable erasure coding for sharelib if Oozie runs on Hadoop 3
https://issues.apache.org/jira/browse/OOZIE-3054
OOZIE-3031  Coord job with only unresolved dependencies doesn't timeout
https://issues.apache.org/jira/browse/OOZIE-3031
OOZIE-3022  fix for warning has no file and won't be listed in dependency files 
details
https://issues.apache.org/jira/browse/OOZIE-3022
OOZIE-3002  address findbugs errors in client lib
https://issues.apache.org/jira/browse/OOZIE-3002
OOZIE-3001  core library has many instances of warnings with trailing spaces 
and lines longer than 132 chars
https://issues.apache.org/jira/browse/OOZIE-3001
OOZIE-2997  files contain trailing white spaces in client lib
https://issues.apache.org/jira/browse/OOZIE-2997
OOZIE-2996  add option for -UseGCOverheadLimit to maven opts as sometimes local 
testing fails
https://issues.apache.org/jira/browse/OOZIE-2996
OOZIE-2978  Remove code that handles Pig versions before 0.8 
https://issues.apache.org/jira/browse/OOZIE-2978
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues.apache.org/jira/browse/OOZIE-2975
OOZIE-2969  Drop support for Java 1.7
https://issues.apache.org/jira/browse/OOZIE-2969
OOZIE-2964  Add -Xdoclint:none to javadoc opts to avoid warnings
https://issues.apache.org/jira/browse/OOZIE-2964
OOZIE-2963  getting error in build ArtifactNotFoundException: Could not find 
artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde
https://issues.apache.org/jira/browse/OOZIE-2963
OOZIE-2962  bump maven-javadoc-plugin to 2.10.4
https://issues.apache.org/jira/browse/OOZIE-2962
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues.apache.org/jira/browse/OOZIE-2934
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2914  Consolidate Trim 
https://issues.apache.org/jira/browse/OOZIE-2914
OOZIE-2909  LauncherAM: rewrite UGI calls
https://issues.apache.org/jira/browse/OOZIE-2909
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues.apache.org/jira/browse/OOZIE-2867
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues.apache.org/jira/browse/OOZIE-2829
OOZIE-2826  Falcon feed fails to aws s3; Oozie joda time version does not meet 
required jar version 2.2 or later
https://issues.apache.org/jira/browse/OOZIE-2826
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2795  Create lib directory or symlink for Oozie CLI during packaging
https://issues.apache.org/jira/browse/OOZIE-2795
OOZIE-2791  ShareLib