[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203850#comment-16203850
]
Robert Kanter commented on OOZIE-2406:
--
Yay!
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-2406.004.patch, OOZIE-2406.004.patch,
> OOZIE-2406.005.patch, OOZIE-2406.006.patch, OOZIE-2406.007.patch,
> OOZIE-2406.008.patch, OOZIE-2406.009.patch, OOZIE-2406.010.patch,
> OOZIE-2406.011.patch, OOZIE-2406.012.patch, OOZIE-2406.012.patch,
> OOZIE-2406.012.patch, OOZIE-2406.012.patch, OOZIE-2406.012.patch,
> OOZIE-2406.013.patch, Screen Shot 2017-10-02 at 1.55.58 PM.png, Screen Shot
> 2017-10-02 at 1.56.33 PM.png, graphviz-10-action-nodes.png,
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png,
> graphviz-many-nodes.png, jung-many-nodes.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and
> remove Jung and the Commons-Collections fork. Whatever we choose, it should
> an svg which will draw the image on the browser.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203544#comment-16203544
]
Peter Cseh commented on OOZIE-2406:
---
+1 Thanks for fixing this!
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-2406.004.patch, OOZIE-2406.004.patch,
> OOZIE-2406.005.patch, OOZIE-2406.006.patch, OOZIE-2406.007.patch,
> OOZIE-2406.008.patch, OOZIE-2406.009.patch, OOZIE-2406.010.patch,
> OOZIE-2406.011.patch, OOZIE-2406.012.patch, OOZIE-2406.012.patch,
> OOZIE-2406.012.patch, OOZIE-2406.012.patch, OOZIE-2406.012.patch,
> OOZIE-2406.013.patch, Screen Shot 2017-10-02 at 1.55.58 PM.png, Screen Shot
> 2017-10-02 at 1.56.33 PM.png, graphviz-10-action-nodes.png,
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png,
> graphviz-many-nodes.png, jung-many-nodes.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and
> remove Jung and the Commons-Collections fork. Whatever we choose, it should
> an svg which will draw the image on the browser.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203209#comment-16203209
]
Andras Piros commented on OOZIE-2406:
-
Thanks [~gezapeti] [~rkanter] [~pbacsko] for the reviews! Hopefully everything
is addressed. Can you have a look?
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-2406.004.patch, OOZIE-2406.004.patch,
> OOZIE-2406.005.patch, OOZIE-2406.006.patch, OOZIE-2406.007.patch,
> OOZIE-2406.008.patch, OOZIE-2406.009.patch, OOZIE-2406.010.patch,
> OOZIE-2406.011.patch, OOZIE-2406.012.patch, OOZIE-2406.012.patch,
> OOZIE-2406.012.patch, OOZIE-2406.012.patch, OOZIE-2406.012.patch,
> OOZIE-2406.013.patch, Screen Shot 2017-10-02 at 1.55.58 PM.png, Screen Shot
> 2017-10-02 at 1.56.33 PM.png, graphviz-10-action-nodes.png,
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png,
> graphviz-many-nodes.png, jung-many-nodes.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and
> remove Jung and the Commons-Collections fork. Whatever we choose, it should
> an svg which will draw the image on the browser.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202596#comment-16202596
]
Hadoop QA commented on OOZIE-2406:
--
Testing JIRA OOZIE-2406
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 5 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:orange}0{color} There are [1] new bugs found in total that would be nice
to have fixed.
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:orange}0{color} There are [1] new bugs found in [core] that would be
nice to have fixed.
. You can find the FindBugs diff here: core/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [examples].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2041
.Tests rerun: 59
.Tests failed at first run:
org.apache.oozie.service.TestStatusTransitService,org.apache.oozie.service.TestJMSAccessorService,org.apache.oozie.event.TestEventGeneration,org.apache.oozie.service.TestCoordMaterializeTriggerService,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:green}*+1 Overall result, good!, no -1s*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/105/
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-2406.004.patch, OOZIE-2406.004.patch,
> OOZIE-2406.005.patch, OOZIE-2406.006.patch, OOZIE-2406.007.patch,
> OOZIE-2406.008.patch, OOZIE-2406.009.patch, OOZIE-2406.010.patch,
> OOZIE-2406.011.patch, OOZIE-2406.012.patch, OOZIE-2406.012.patch,
> OOZIE-2406.012.patch, OOZIE-2406.012.patch, OOZIE-2406.012.patch,
> OOZIE-2406.013.patch, Screen Shot 2017-10-02 at 1.55.58 PM.png, Screen Shot
> 2017-10-02 at 1.56.33 PM.png, graphviz-10-action-nodes.png,
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png,
> graphviz-many-nodes.png, jung-many-nodes.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196728#comment-16196728
]
Hadoop QA commented on OOZIE-2406:
--
Testing JIRA OOZIE-2406
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 5 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:orange}0{color} There are [1] new bugs found in total that would be nice
to have fixed.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:orange}0{color} There are [1] new bugs found in [core] that would be
nice to have fixed.
. You can find the FindBugs diff here: core/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2036
.Tests failed: 1
.Tests errors: 1
.The patch failed the following testcases:
. testSimpleGraphPng(org.apache.oozie.util.graph.TestGraphGenerator)
.Tests failing with errors:
. testGraphWithManyNodes(org.apache.oozie.util.graph.TestGraphGenerator)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/90/
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-2406.004.patch, OOZIE-2406.004.patch,
> OOZIE-2406.005.patch, OOZIE-2406.006.patch, OOZIE-2406.007.patch,
> OOZIE-2406.008.patch, OOZIE-2406.009.patch, OOZIE-2406.010.patch,
> OOZIE-2406.011.patch, OOZIE-2406.012.patch, OOZIE-2406.012.patch,
> OOZIE-2406.012.patch, OOZIE-2406.012.patch, OOZIE-2406.012.patch, Screen Shot
> 2017-10-02 at 1.55.58 PM.png, Screen Shot 2017-10-02 at 1.56.33 PM.png,
> graphviz-10-action-nodes.png, graphviz-25-action-nodes.png,
> graphviz-50-action-nodes.png, graphviz-many-nodes.png, jung-many-nodes.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196379#comment-16196379
]
Hadoop QA commented on OOZIE-2406:
--
Testing JIRA OOZIE-2406
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 5 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:orange}0{color} There are [1] new bugs found in total that would be nice
to have fixed.
. {color:orange}0{color} There are [1] new bugs found in [core] that would be
nice to have fixed.
. You can find the FindBugs diff here: core/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [tools].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/89/
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-2406.004.patch, OOZIE-2406.004.patch,
> OOZIE-2406.005.patch, OOZIE-2406.006.patch, OOZIE-2406.007.patch,
> OOZIE-2406.008.patch, OOZIE-2406.009.patch, OOZIE-2406.010.patch,
> OOZIE-2406.011.patch, OOZIE-2406.012.patch, OOZIE-2406.012.patch,
> OOZIE-2406.012.patch, OOZIE-2406.012.patch, Screen Shot 2017-10-02 at 1.55.58
> PM.png, Screen Shot 2017-10-02 at 1.56.33 PM.png,
> graphviz-10-action-nodes.png, graphviz-25-action-nodes.png,
> graphviz-50-action-nodes.png, graphviz-many-nodes.png, jung-many-nodes.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196335#comment-16196335
]
Hadoop QA commented on OOZIE-2406:
--
Testing JIRA OOZIE-2406
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 5 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:red}-1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:red}-1{color} patch does not compile
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [tools].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:red}-1 DISTRO{color}
.{color:red}-1{color} distro tarball fails with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/88/
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-2406.004.patch, OOZIE-2406.004.patch,
> OOZIE-2406.005.patch, OOZIE-2406.006.patch, OOZIE-2406.007.patch,
> OOZIE-2406.008.patch, OOZIE-2406.009.patch, OOZIE-2406.010.patch,
> OOZIE-2406.011.patch, OOZIE-2406.012.patch, OOZIE-2406.012.patch,
> OOZIE-2406.012.patch, Screen Shot 2017-10-02 at 1.55.58 PM.png, Screen Shot
> 2017-10-02 at 1.56.33 PM.png, graphviz-10-action-nodes.png,
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png,
> graphviz-many-nodes.png, jung-many-nodes.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196322#comment-16196322
]
Hadoop QA commented on OOZIE-2406:
--
Testing JIRA OOZIE-2406
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 5 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:red}-1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:red}-1{color} patch does not compile
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [tools].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:red}-1 DISTRO{color}
.{color:red}-1{color} distro tarball fails with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/86/
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-2406.004.patch, OOZIE-2406.004.patch,
> OOZIE-2406.005.patch, OOZIE-2406.006.patch, OOZIE-2406.007.patch,
> OOZIE-2406.008.patch, OOZIE-2406.009.patch, OOZIE-2406.010.patch,
> OOZIE-2406.011.patch, OOZIE-2406.012.patch, OOZIE-2406.012.patch, Screen Shot
> 2017-10-02 at 1.55.58 PM.png, Screen Shot 2017-10-02 at 1.56.33 PM.png,
> graphviz-10-action-nodes.png, graphviz-25-action-nodes.png,
> graphviz-50-action-nodes.png, graphviz-many-nodes.png, jung-many-nodes.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196086#comment-16196086
]
Hadoop QA commented on OOZIE-2406:
--
Testing JIRA OOZIE-2406
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 1 line(s) longer than 132
characters
.{color:green}+1{color} the patch does adds/modifies 5 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:orange}0{color} There are [1] new bugs found in total that would be nice
to have fixed.
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:orange}0{color} There are [1] new bugs found in [core] that would be
nice to have fixed.
. You can find the FindBugs diff here: core/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2036
.Tests rerun: 67
.Tests failed at first run:
org.apache.oozie.command.wf.TestWorkflowActionRetryInfoXCommand,org.apache.oozie.command.coord.TestCoordPushDependencyCheckXCommand,org.apache.oozie.action.hadoop.TestJavaActionExecutor,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/85/
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-2406.004.patch, OOZIE-2406.004.patch,
> OOZIE-2406.005.patch, OOZIE-2406.006.patch, OOZIE-2406.007.patch,
> OOZIE-2406.008.patch, OOZIE-2406.009.patch, OOZIE-2406.010.patch,
> OOZIE-2406.011.patch, OOZIE-2406.012.patch, Screen Shot 2017-10-02 at 1.55.58
> PM.png, Screen Shot 2017-10-02 at 1.56.33 PM.png,
> graphviz-10-action-nodes.png, graphviz-25-action-nodes.png,
> graphviz-50-action-nodes.png, graphviz-many-nodes.png, jung-many-nodes.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
>
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190869#comment-16190869
]
Andras Piros commented on OOZIE-2406:
-
Test failure is unrelated. [~rkanter] [~gezapeti] [~pbacsko] can you please
review?
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png,
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png,
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch,
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch,
> OOZIE-2406.007.patch, OOZIE-2406.008.patch, OOZIE-2406.009.patch,
> OOZIE-2406.010.patch, Screen Shot 2017-10-02 at 1.55.58 PM.png, Screen Shot
> 2017-10-02 at 1.56.33 PM.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and
> remove Jung and the Commons-Collections fork. Whatever we choose, it should
> an svg which will draw the image on the browser.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190525#comment-16190525
]
Hadoop QA commented on OOZIE-2406:
--
Testing JIRA OOZIE-2406
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 1 line(s) longer than 132
characters
.{color:green}+1{color} the patch does adds/modifies 6 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:orange}0{color} There are [1] new bugs found in total that would be nice
to have fixed.
. {color:orange}0{color} There are [1] new bugs found in [core] that would be
nice to have fixed.
. You can find the FindBugs diff here: core/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [examples].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2039
.Tests failed: 0
.Tests errors: 1
.The patch failed the following testcases:
.
.Tests failing with errors:
. testJMXInstrumentation(org.apache.oozie.util.TestMetricsInstrumentation)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/73/
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png,
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png,
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch,
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch,
> OOZIE-2406.007.patch, OOZIE-2406.008.patch, OOZIE-2406.009.patch,
> OOZIE-2406.010.patch, Screen Shot 2017-10-02 at 1.55.58 PM.png, Screen Shot
> 2017-10-02 at 1.56.33 PM.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189035#comment-16189035
]
Rohini Palaniswamy commented on OOZIE-2406:
---
bq. I also considered the Oozie UI to have GraphGenerator render in svg format,
it should take less effort on Oozie server side, as for SVGs we don't need
BufferedImage instances
Big +1 for switching to svg. We had to turn off graph rendering for large
dags as the huge image objects caused the Oozie servers to go out of memory.
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png,
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png,
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch,
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch,
> OOZIE-2406.007.patch, OOZIE-2406.008.patch, OOZIE-2406.009.patch, Screen Shot
> 2017-10-02 at 1.55.58 PM.png, Screen Shot 2017-10-02 at 1.56.33 PM.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and
> remove Jung and the Commons-Collections fork. Whatever we choose, it should
> an svg which will draw the image on the browser.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189034#comment-16189034
]
Hadoop QA commented on OOZIE-2406:
--
Testing JIRA OOZIE-2406
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 6 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:orange}0{color} There are [1] new bugs found in total that would be nice
to have fixed.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:orange}0{color} There are [1] new bugs found in [core] that would be
nice to have fixed.
. You can find the FindBugs diff here: core/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2037
.Tests rerun: 69
.Tests failed at first run:
org.apache.oozie.command.coord.TestCoordActionInputCheckXCommandNonUTC,org.apache.oozie.action.hadoop.TestJavaActionExecutor,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:green}*+1 Overall result, good!, no -1s*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/69/
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png,
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png,
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch,
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch,
> OOZIE-2406.007.patch, OOZIE-2406.008.patch, OOZIE-2406.009.patch, Screen Shot
> 2017-10-02 at 1.55.58 PM.png, Screen Shot 2017-10-02 at 1.56.33 PM.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188627#comment-16188627
]
Hadoop QA commented on OOZIE-2406:
--
Testing JIRA OOZIE-2406
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 5 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:orange}0{color} There are [2] new bugs found in total that would be nice
to have fixed.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:orange}0{color} There are [2] new bugs found in [core] that would be
nice to have fixed.
. You can find the FindBugs diff here: core/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2036
.Tests failed: 8
.Tests errors: 2
.The patch failed the following testcases:
. testImportTablesOverflowBatchSize(org.apache.oozie.tools.TestDBLoadDump)
.
testImportInvalidDataLeavesTablesEmpty(org.apache.oozie.tools.TestDBLoadDump)
.
testImportToNonEmptyTablesCausesPrematureExit(org.apache.oozie.tools.TestDBLoadDump)
. testImportedDBIsExportedCorrectly(org.apache.oozie.tools.TestDBLoadDump)
. testMain(org.apache.oozie.action.hadoop.TestHiveMain)
. testPig_withNullExternalID(org.apache.oozie.action.hadoop.TestPigMain)
. testEmbeddedPigWithinPython(org.apache.oozie.action.hadoop.TestPigMain)
. testPigScript(org.apache.oozie.action.hadoop.TestPigMain)
.Tests failing with errors:
. testAddXIncludeFromReader(org.apache.oozie.util.TestXConfiguration)
. testAddXIncludeFromStream(org.apache.oozie.util.TestXConfiguration)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/67/
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png,
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png,
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch,
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch,
> OOZIE-2406.007.patch, OOZIE-2406.008.patch, Screen Shot 2017-10-02 at 1.55.58
> PM.png, Screen Shot 2017-10-02 at 1.56.33 PM.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188178#comment-16188178
]
Hadoop QA commented on OOZIE-2406:
--
Testing JIRA OOZIE-2406
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 5 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [1] new bugs found below threshold in total that
must be fixed.
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:red}-1{color} There are [1] new bugs found below threshold in [core]
that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones):
core/findbugs-new.html
. The most important FindBugs errors are:
. At GraphGenerator.java:[line 81]: The use of XMLReader.parse(...) is
vulnerable to XML External Entity attacks
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2036
.Tests rerun: 51
.Tests failed at first run:
org.apache.oozie.action.hadoop.TestJavaActionExecutor,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/62/
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png,
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png,
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch,
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch,
> OOZIE-2406.007.patch, Screen Shot 2017-10-02 at 1.55.58 PM.png, Screen Shot
> 2017-10-02 at 1.56.33 PM.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
>
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188091#comment-16188091
]
Hadoop QA commented on OOZIE-2406:
--
Testing JIRA OOZIE-2406
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 5 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [1] new bugs found below threshold in total that
must be fixed.
. {color:red}-1{color} There are [1] new bugs found below threshold in [core]
that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones):
core/findbugs-new.html
. The most important FindBugs errors are:
. At GraphGenerator.java:[line 81]: The use of XMLReader.parse(...) is
vulnerable to XML External Entity attacks
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [tools].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2036
.Tests failed: 0
.Tests errors: 1
.The patch failed the following testcases:
.
.Tests failing with errors:
. testJMXInstrumentation(org.apache.oozie.util.TestMetricsInstrumentation)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/61/
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png,
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png,
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch,
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch,
> OOZIE-2406.007.patch, Screen Shot 2017-10-02 at 1.55.58 PM.png, Screen Shot
> 2017-10-02 at 1.56.33 PM.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16187895#comment-16187895
]
Andras Piros commented on OOZIE-2406:
-
Thanks [~rkanter] for the {{Instrumentation}} intro :)
Following changes in this patchset:
* {{Instrumentation.Counter}}, {{Instrumentation.Cron}} events added for both
{{OutputFormat}}-specific and aggregated cases
* {{OutputFormat.SVG}} added, it's used as the default one from UI
* now-unnecessary {{jung}} and obscure {{commons-collections}} dependencies
removed
* because of {{graphviz-java}}'s {{GraphvizV8Engine}} uses
{{com.eclipsesource.v8.V8}}, which uses thread local variables, we need that
all rendering be run using the very same thread. This is achieved w/ a
single-thread {{ExecutorService}} inside {{GraphivizRenderer#EXECUTOR_SERVICE}}
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png,
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png,
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch,
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and
> remove Jung and the Commons-Collections fork. Whatever we choose, it should
> an svg which will draw the image on the browser.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172354#comment-16172354
]
Robert Kanter commented on OOZIE-2406:
--
We could add an svg format too :)
For the metrics stuff, you can do this to get an {{Instrumentation}} object (it
will do the old instrumentation or the newer metrics depending on what's
configured - no need for you to worry about that):
{code:java}
Instrumentation instrumentation =
Services.get().get(InstrumentationService.class).get();
{code}
There's 4 types of metrics you can use:
# Counter: good for things that need to be counted (e.g. how many graphs we've
generated)
{code:java}
instrumentation.incr("group", "name", someAmount);
{code}
# Variable: good for static values (you probably don't need this)
{code:java}
instrumentation.addVariable("group", "name", new Instrumentation.Variable() {
@Override
public T getValue() {
return someValue;
}
});
{code}
# Cron: good for measuring how long something takes (e.g. how long it takes to
generate a graph)
{code:java}
Instrumentation.Cron cron = new Instrumentation.Cron();
try {
cron.start();
// do some work
} finally {
cron.stop();
}
instrumentation.addCron("group", "name", cron);
{code}
# Sampler: these are more complicated and rarely used (you probably shouldn't
use this)
The "group" and "name" are whatever you want and are concatenated as
"group.name".
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png,
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png,
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch,
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and
> remove Jung and the Commons-Collections fork. Whatever we choose, it should
> an svg which will draw the image on the browser.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171948#comment-16171948
]
Andras Piros commented on OOZIE-2406:
-
Hi [~rkanter],
I agree w/ 1), I want to remove {{JungRenderer}} and its dependencies from the
project within this JIRA. I only want to make sure in beforehand that
{{GraphvizRenderer}} is not performing much worse. I also considered the Oozie
UI to have {{GraphGenerator}} render in {{svg}} format, it should take less
effort on Oozie server side, as for SVGs we don't need {{BufferedImage}}
instances, and it's also supported by {{graphviz-java}}.
As for {{MetricsInstrumentationService}} usage, can you please point to a
servlet where it's correctly used, and give also example usage REST calls?
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png,
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png,
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch,
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and
> remove Jung and the Commons-Collections fork. Whatever we choose, it should
> an svg which will draw the image on the browser.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170899#comment-16170899
]
Robert Kanter commented on OOZIE-2406:
--
1. The advantage of removing the {{JungRenderer}} is that we could remove the
jung and Commons-Collections fork dependencies. I'm okay with leaving it for
now if we want to give the {{GraphvizRenderer}} more time to bake. However, it
looks like {{GraphGenerator}} is hardcoded to use {{GraphvizRenderer}} so you
can't use the {{JungRenderer}} anyway.
2. We should add Metrics/Instrumentation in key areas of the generator code.
That might be helpful for performance testing, and would be a useful thing to
expose anyway. e.g. how long it takes to generate the graph, how long it takes
to render the png, etc
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png,
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png,
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch,
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and
> remove Jung and the Commons-Collections fork. Whatever we choose, it should
> an svg which will draw the image on the browser.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169363#comment-16169363
]
Hadoop QA commented on OOZIE-2406:
--
Testing JIRA OOZIE-2406
Cleaning local git workspace
{color:red}-1{color} Patch failed to apply to head of branch
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png,
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png,
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch,
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and
> remove Jung and the Commons-Collections fork. Whatever we choose, it should
> an svg which will draw the image on the browser.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169359#comment-16169359
]
Andras Piros commented on OOZIE-2406:
-
Hi [~rkanter],
thanks for the review!
Addressing your points:
# I'd still wait until we've measured {{GraphvizRenderer}} is not much slower /
consumes not much more CPU or heap than {{JungRenderer}}. In any case, writing
a new {{GraphRenderer}} implementation w/ a new PNG generation library is not
much of a story by now
# I have the test case {{TestGraphGenerator#testSingleThreadedPerformance}}
that does rendering of a 53-node graph 5 times in a row - that takes now about
8 secs. I can extend that to measure performance more, or have a real cluster
w/ real workflows, JMeter set up etc.
# color handling of edges and nodes is modified to fit documentation
{{WebServicesAPI.twiki}}
# yes, filing separate JIRA for temp file based PNG caching
# it's implemented
# filing separate JIRA for extending Oozie CLI w/ graph generation and
downloading
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png,
> OOZIE-2406.004.patch, OOZIE-2406.004.patch, OOZIE-2406.005.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and
> remove Jung and the Commons-Collections fork. Whatever we choose, it should
> an svg which will draw the image on the browser.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168306#comment-16168306
]
Robert Kanter commented on OOZIE-2406:
--
I left some specific comments on RB. A few additional things:
# I think we can safely remove Jung and the Commons-Collections fork given that
you got the graphviz version. It still produces an image, and I think it's
reasonable to assume nobody is somehow parsing those images, so it's okay if we
change the look of the images by using a different library. One of the goals
of this JIRA was to remove the jung library and the Commons-Collections fork
because they're super old, abandoned, and potentially security liabilities.
# Have you done any performance testing on the graphviz library? The Jung
library was slow and ate up a ton of memory, which is why we limited the number
of nodes it could process. If graphviz is way better, we could increase this
number to something higher; though I think we should still have _some_ limit
just to be safe. Another thing to check is what happens if Oozie gets a lot of
requests for images at the same time - can the graphviz library handle that
well?
# Can we make the "end" circle a darker color? It's sort of hard to see in the
image you attached.
# I just thought of an interesting idea: it might be a good idea to create a
cache for the generated images. Some kind of LRU cache in memory and/or just
writing them to the local disk. We'd only do this for completed jobs (because
the images will change as the job runs). This is more of a nice to have, which
may or may not even be necessary, and could be a lot of work, so I think this
should be a separate JIRA. Can you file a JIRA to look into this idea?
# It might be a neat improvement to add a way to return the dot file (I assume
there's a way to do that from the graphviz library?). We have to keep the
current API returning an image for compatibility
({{job-3?show=graph\[=true\]}}), but we could add a new parameter
like job-3?show=graph\[=true\]\[=dot\]}} (the default
being {{=png}}). This would allow the caller to use the dot file in
various ways (for example, perhaps Hue might use this).
# Can you file a new JIRA for adding support for getting the image (whether
that's png or dot format, or both) to the Oozie CLI? It's currently only
supported in the REST API.
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png,
> OOZIE-2406.004.patch, OOZIE-2406.004.patch, OOZIE-2406.005.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and
> remove Jung and the Commons-Collections fork. Whatever we choose, it should
> an svg which will draw the image on the browser.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168257#comment-16168257
]
Hadoop QA commented on OOZIE-2406:
--
Testing JIRA OOZIE-2406
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [tools].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2015
.Tests rerun: 82
.Tests failed at first run:
org.apache.oozie.action.hadoop.TestJavaActionExecutor,org.apache.oozie.service.TestStatusTransitService,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:green}*+1 Overall result, good!, no -1s*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/25/
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png,
> OOZIE-2406.004.patch, OOZIE-2406.004.patch, OOZIE-2406.005.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and
> remove Jung and the Commons-Collections fork. Whatever we choose, it should
> an svg which will draw the image on the browser.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167940#comment-16167940
]
Hadoop QA commented on OOZIE-2406:
--
Testing JIRA OOZIE-2406
Cleaning local git workspace
{color:red}-1{color} Patch failed to apply to head of branch
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png,
> OOZIE-2406.004.patch, OOZIE-2406.004.patch, OOZIE-2406.005.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and
> remove Jung and the Commons-Collections fork. Whatever we choose, it should
> an svg which will draw the image on the browser.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167854#comment-16167854
]
Hadoop QA commented on OOZIE-2406:
--
Testing JIRA OOZIE-2406
Cleaning local git workspace
{color:red}-1{color} Patch failed to apply to head of branch
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png,
> OOZIE-2406.004.patch, OOZIE-2406.004.patch, OOZIE-2406.005.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and
> remove Jung and the Commons-Collections fork. Whatever we choose, it should
> an svg which will draw the image on the browser.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167832#comment-16167832
]
Hadoop QA commented on OOZIE-2406:
--
Testing JIRA OOZIE-2406
Cleaning local git workspace
{color:red}-1{color} Patch failed to apply to head of branch
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png,
> OOZIE-2406.004.patch, OOZIE-2406.004.patch, OOZIE-2406.005.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and
> remove Jung and the Commons-Collections fork. Whatever we choose, it should
> an svg which will draw the image on the browser.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167813#comment-16167813
]
Andras Piros commented on OOZIE-2406:
-
[~rkanter] here is the [*ReviewBoard link*|https://reviews.apache.org/r/62352/].
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png,
> OOZIE-2406.004.patch, OOZIE-2406.004.patch, OOZIE-2406.005.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and
> remove Jung and the Commons-Collections fork. Whatever we choose, it should
> an svg which will draw the image on the browser.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167142#comment-16167142
]
Robert Kanter commented on OOZIE-2406:
--
[~andras.piros], thanks for working on this; I'm glad to see us move forward on
this issue. The graphviz diagram looks better too :)
Can you post your patch to ReviewBoard?
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png,
> OOZIE-2406.004.patch, OOZIE-2406.004.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and
> remove Jung and the Commons-Collections fork. Whatever we choose, it should
> an svg which will draw the image on the browser.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167081#comment-16167081
]
Hadoop QA commented on OOZIE-2406:
--
Testing JIRA OOZIE-2406
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 5 line(s) longer than 132
characters
.{color:green}+1{color} the patch does adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [17] new bugs found below threshold in total
that must be fixed.
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:red}-1{color} There are [17] new bugs found below threshold in [core]
that must be fixed, listing only the first [5] ones.
. You can find the FindBugs diff here (look for the red and orange ones):
core/findbugs-new.html
. The top [5] most important FindBugs errors are:
. At GraphGenerator.java:[line 84]: The use of XMLReader.parse(...) is
vulnerable to XML External Entity attacks
. At JungRenderer.java:[line 61]: Unread field:
org.apache.oozie.util.graph.JungRenderer.xIncr; should this field be static?
. At JungRenderer.java:[line 59]: Unread field:
org.apache.oozie.util.graph.JungRenderer.xMargin; should this field be static?
. At JungRenderer.java:[line 62]: Unread field:
org.apache.oozie.util.graph.JungRenderer.yIncr; should this field be static?
. At JungRenderer.java:[line 60]: Unread field:
org.apache.oozie.util.graph.JungRenderer.yMargin; should this field be static?
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2015
.Tests rerun: 43
.Tests failed at first run:
org.apache.oozie.action.hadoop.TestJavaActionExecutor,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/21/
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png,
> OOZIE-2406.004.patch, OOZIE-2406.004.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166553#comment-16166553
]
Hadoop QA commented on OOZIE-2406:
--
Testing JIRA OOZIE-2406
Cleaning local git workspace
{color:red}-1{color} Patch failed to apply to head of branch
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-many-nodes.png, jung-many-nodes.png,
> OOZIE-2406.004.patch
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and
> remove Jung and the Commons-Collections fork. Whatever we choose, it should
> an svg which will draw the image on the browser.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928434#comment-15928434
]
Robert Kanter commented on OOZIE-2406:
--
One thing that may be a problem is that we currently expose (and document)
getting the DAG graph image as a png file via the REST API
(http://oozie.apache.org/docs/4.3.0/WebServicesAPI.html#Job_graph). If we
switch to a client side library like d3, we'd be serving some kind of JSON blog
or something, and not an image; which would mean we'd have to break this REST
call.
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and
> remove Jung and the Commons-Collections fork. Whatever we choose, it should
> an svg which will draw the image on the browser.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556563#comment-15556563
]
Robert Kanter commented on OOZIE-2406:
--
Another option (which might be the best) is to have the Oozie Server send out
the DAG as JSON and use a client-side library for rendering the graph (there
must be many Javascript libraries for this).
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and
> remove Jung and the Commons-Collections fork. Whatever we choose, it should
> an svg which will draw the image on the browser.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code
[
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041832#comment-15041832
]
Robert Kanter commented on OOZIE-2406:
--
I haven't looked too much into it, but it looks like [Apache
Bitak|http://xmlgraphics.apache.org/batik/] would be a good choice of library
to use.
> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
> Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>
> The Web UI currently generates a graph of the workflow DAG as a png image
> that it can show to the user ({{GraphGenerator}} class). Unfortunately,
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which
> hasn't been updated since 2010.
> #- This library also has a dependency on a fork of Commons-Collections
> ({{net.sourceforge.collections}} from
> http://sourceforge.net/projects/collections/), which similarly hasn't been
> updated since 2010. This is a problem because we can't update
> Commons-Collections when there are security concerns such as COLLECTIONS-580
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and
> remove Jung and the Commons-Collections fork. Whatever we choose, it should
> an svg which will draw the image on the browser.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
