[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041832#comment-15041832
 ] 

Robert Kanter commented on OOZIE-2406:
--

I haven't looked too much into it, but it looks like [Apache 
Bitak|http://xmlgraphics.apache.org/batik/] would be a good choice of library 
to use.

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OOZIE-1445) Job DAG causes OOM. Reimplement using client side javascript

[Rohini Palaniswamy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohini Palaniswamy resolved OOZIE-1445.
---
Resolution: Duplicate

> Job DAG causes OOM. Reimplement using client side javascript
> 
>
> Key: OOZIE-1445
> URL: https://issues.apache.org/jira/browse/OOZIE-1445
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 3.3.2
>Reporter: Rohini Palaniswamy
>
> Job DAG tab in the UI shows the workflow in a graphical representation. The 
> way it is implemented it constructs image objects on the server side and 
> serves it which is a bad idea and leads to OOM often on the server. Has to be 
> reimplemented using client side java script.
> OOZIE-1444 removes the refresh option for now which mitigates the problem a 
> bit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042256#comment-15042256
 ] 

Robert Kanter commented on OOZIE-2402:
--

Thanks for making the changes.  Though it looks like my #4 comment was missed:
{quote}
It would be nice if checkCopyResults could print out the exception for each 
failure, instead of just one. Otherwise, if there are multiple problems, the 
user will have to keep trying after resolving each issue.
{quote}
Can you do that too?  I think if you might just have to remove the {{break;}} 
statements.

> oozie-setup.sh sharelib create takes a long time on large clusters
> --
>
> Key: OOZIE-2402
> URL: https://issues.apache.org/jira/browse/OOZIE-2402
> Project: Oozie
>  Issue Type: Improvement
>  Components: tools
>Affects Versions: 4.2.0
>Reporter: Illya Yalovyy
>Assignee: Illya Yalovyy
> Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch, 
> OOZIE-2402-3.patch
>
>
> When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. 
> Copy the tarball itself takes only around 10 seconds. It seems like 
> performance could be improved by loading files concurrently in many threads.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2413) Kerberos credentials can expire if the KDC is slow to respond

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042382#comment-15042382
 ] 

Rohini Palaniswamy commented on OOZIE-2413:
---

This should be done in JavaActionExecutor.setCredentialTokens() so that the 
check is done only once for any action even if there are multiple credentials.

{code}
if (context != null && action != null && credPropertiesMap != null) {
+LOG.debug("About to relogin from keytab");
+UserGroupInformation.getLoginUser().checkTGTAndReloginFromKeytab();
+LOG.debug("Relogin from keytab successful");
for (Entry entry : 
credPropertiesMap.entrySet()) {
{code}

> Kerberos credentials can expire if the KDC is slow to respond
> -
>
> Key: OOZIE-2413
> URL: https://issues.apache.org/jira/browse/OOZIE-2413
> Project: Oozie
>  Issue Type: Bug
>  Components: security
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: trunk
>
> Attachments: OOZIE-2413.001.patch
>
>
> We've seen some very rare cases where Oozie gets a Kerberos error when trying 
> to get delegation tokens via the {{Credentials}} mechanism (e.g. getting HS2 
> delegation tokens).
> We finally narrowed it down to slow KDC responses, so Oozie's Kerberos 
> credentials have expired when it tries to get the delegation token.  The 
> reason we don't see this with Hadoop clients (DFSClient for HDFS, JobClient 
> for MR, etc) is because they call 
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before trying to 
> connect.  
> We should do a similar fix by calling 
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before using a 
> Credentials implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2413) Kerberos credentials can expire if the KDC is slow to respond

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042172#comment-15042172
 ] 

Robert Kanter commented on OOZIE-2413:
--

Test failures unrelated.

> Kerberos credentials can expire if the KDC is slow to respond
> -
>
> Key: OOZIE-2413
> URL: https://issues.apache.org/jira/browse/OOZIE-2413
> Project: Oozie
>  Issue Type: Bug
>  Components: security
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: trunk
>
> Attachments: OOZIE-2413.001.patch
>
>
> We've seen some very rare cases where Oozie gets a Kerberos error when trying 
> to get delegation tokens via the {{Credentials}} mechanism (e.g. getting HS2 
> delegation tokens).
> We finally narrowed it down to slow KDC responses, so Oozie's Kerberos 
> credentials have expired when it tries to get the delegation token.  The 
> reason we don't see this with Hadoop clients (DFSClient for HDFS, JobClient 
> for MR, etc) is because they call 
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before trying to 
> connect.  
> We should do a similar fix by calling 
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before using a 
> Credentials implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2185) Make oozie cli source /etc/oozie/conf/oozie-env.sh (or some other env script)

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042296#comment-15042296
 ] 

Robert Kanter commented on OOZIE-2185:
--

Sorry, I've been bogged down with other work.  I'll try to get to this soon.

> Make oozie cli source /etc/oozie/conf/oozie-env.sh (or some other env script)
> -
>
> Key: OOZIE-2185
> URL: https://issues.apache.org/jira/browse/OOZIE-2185
> Project: Oozie
>  Issue Type: New Feature
>  Components: client
>Affects Versions: 4.3.0
>Reporter: Robert Justice
>Assignee: Mike Grimes
>Priority: Minor
>  Labels: client, feature, newbie
> Attachments: OOZIE-2185-1.patch, OOZIE-2185-2.patch, 
> OOZIE-2185-3.patch
>
>
> Currently, the oozie cli doesn't source any environment script.   Users may 
> wish to define environment variables such as OOZIE_URL outside of normal 
> shell environment and have the oozie cli source /etc/oozie/conf/oozie-env.sh. 
>  This is somewhat confusing when users see hadoop commands sourcing 
> hadoop-env.sh, mapred-env.sh, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (OOZIE-2185) Make oozie cli source /etc/oozie/conf/oozie-env.sh (or some other env script)

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042296#comment-15042296
 ] 

Robert Kanter edited comment on OOZIE-2185 at 12/4/15 9:53 PM:
---

Sorry, I've been bogged down with other work.  I'll try to get to this soon.  
It's on my todo list.


was (Author: rkanter):
Sorry, I've been bogged down with other work.  I'll try to get to this soon.

> Make oozie cli source /etc/oozie/conf/oozie-env.sh (or some other env script)
> -
>
> Key: OOZIE-2185
> URL: https://issues.apache.org/jira/browse/OOZIE-2185
> Project: Oozie
>  Issue Type: New Feature
>  Components: client
>Affects Versions: 4.3.0
>Reporter: Robert Justice
>Assignee: Mike Grimes
>Priority: Minor
>  Labels: client, feature, newbie
> Attachments: OOZIE-2185-1.patch, OOZIE-2185-2.patch, 
> OOZIE-2185-3.patch
>
>
> Currently, the oozie cli doesn't source any environment script.   Users may 
> wish to define environment variables such as OOZIE_URL outside of normal 
> shell environment and have the oozie cli source /etc/oozie/conf/oozie-env.sh. 
>  This is somewhat confusing when users see hadoop commands sourcing 
> hadoop-env.sh, mapred-env.sh, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-1908) Create Crunch Action

[Ferenc Denes (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042195#comment-15042195
 ] 

Ferenc Denes commented on OOZIE-1908:
-

Overall this looks good, however there are missing
- Putting the xsd files into the oozie-default.xml
- Putting the action type into oozie-default.xml as well
- Test cases
- Documentation

> Create Crunch Action
> 
>
> Key: OOZIE-1908
> URL: https://issues.apache.org/jira/browse/OOZIE-1908
> Project: Oozie
>  Issue Type: New Feature
>  Components: action
>Reporter: Micah Whitacre
> Attachments: OOZIE-1908.patch
>
>
> Launching an Apache Crunch Pipeline[1] is currently done through the Java 
> Action.  While this works it, it however loses traceability between the 
> action and any child processes that get launched.
> [1] - http://crunch.apache.org/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters

[Illya Yalovyy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Illya Yalovyy updated OOZIE-2402:
-
Attachment: OOZIE-2402-4.patch

> oozie-setup.sh sharelib create takes a long time on large clusters
> --
>
> Key: OOZIE-2402
> URL: https://issues.apache.org/jira/browse/OOZIE-2402
> Project: Oozie
>  Issue Type: Improvement
>  Components: tools
>Affects Versions: 4.2.0
>Reporter: Illya Yalovyy
>Assignee: Illya Yalovyy
> Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch, 
> OOZIE-2402-3.patch, OOZIE-2402-4.patch
>
>
> When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. 
> Copy the tarball itself takes only around 10 seconds. It seems like 
> performance could be improved by loading files concurrently in many threads.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2413) Kerberos credentials can expire if the KDC is slow to respond

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042417#comment-15042417
 ] 

Rohini Palaniswamy commented on OOZIE-2413:
---

You can put code in a new CredentialProvider method too to maintain the 
abstraction, but need to call before this loop to avoid checking TGT multiple 
times. 

> Kerberos credentials can expire if the KDC is slow to respond
> -
>
> Key: OOZIE-2413
> URL: https://issues.apache.org/jira/browse/OOZIE-2413
> Project: Oozie
>  Issue Type: Bug
>  Components: security
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: trunk
>
> Attachments: OOZIE-2413.001.patch
>
>
> We've seen some very rare cases where Oozie gets a Kerberos error when trying 
> to get delegation tokens via the {{Credentials}} mechanism (e.g. getting HS2 
> delegation tokens).
> We finally narrowed it down to slow KDC responses, so Oozie's Kerberos 
> credentials have expired when it tries to get the delegation token.  The 
> reason we don't see this with Hadoop clients (DFSClient for HDFS, JobClient 
> for MR, etc) is because they call 
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before trying to 
> connect.  
> We should do a similar fix by calling 
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before using a 
> Credentials implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: oozie-trunk-precommit-build #2618

[Apache Jenkins Server]

See 

Changes:

[rkanter] OOZIE-2411 Add BCC to oozie email action (fdenes via rkanter)

--
[...truncated 7600 lines...]
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-tools 
---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-tools ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.3.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-mini ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-mini ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-mini ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-mini ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-mini ---
[INFO] Reading assembly descriptor: src/main/assemblies/empty.xml
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Distro 4.3.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-distro 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-distro 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-distro ---
[INFO] No sources to 

[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042567#comment-15042567
 ] 

Hadoop QA commented on OOZIE-2402:
--

Testing JIRA OOZIE-2402

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/2618/

> oozie-setup.sh sharelib create takes a long time on large clusters
> --
>
> Key: OOZIE-2402
> URL: https://issues.apache.org/jira/browse/OOZIE-2402
> Project: Oozie
>  Issue Type: Improvement
>  Components: tools
>Affects Versions: 4.2.0
>Reporter: Illya Yalovyy
>Assignee: Illya Yalovyy
> Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch, 
> OOZIE-2402-3.patch, OOZIE-2402-4.patch
>
>
> When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. 
> Copy the tarball itself takes only around 10 seconds. It seems like 
> performance could be improved by loading files concurrently in many threads.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2413) Kerberos credentials can expire if the KDC is slow to respond

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-2413:
-
Attachment: OOZIE-2413.002.patch

It shouldn't be a problem to call the method more frequently than needed 
because it does some checks to see if it makes sense to actually relogin at 
this point or if it should do a no-op.  However, it's probably cleaner to 
abstract it out like you said and just call it once per action.

The 002 patch does that.

> Kerberos credentials can expire if the KDC is slow to respond
> -
>
> Key: OOZIE-2413
> URL: https://issues.apache.org/jira/browse/OOZIE-2413
> Project: Oozie
>  Issue Type: Bug
>  Components: security
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: trunk
>
> Attachments: OOZIE-2413.001.patch, OOZIE-2413.002.patch
>
>
> We've seen some very rare cases where Oozie gets a Kerberos error when trying 
> to get delegation tokens via the {{Credentials}} mechanism (e.g. getting HS2 
> delegation tokens).
> We finally narrowed it down to slow KDC responses, so Oozie's Kerberos 
> credentials have expired when it tries to get the delegation token.  The 
> reason we don't see this with Hadoop clients (DFSClient for HDFS, JobClient 
> for MR, etc) is because they call 
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before trying to 
> connect.  
> We should do a similar fix by calling 
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before using a 
> Credentials implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters

[Illya Yalovyy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042643#comment-15042643
 ] 

Illya Yalovyy commented on OOZIE-2402:
--

Results of the build don't make sense:
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 04:00 min
[INFO] Finished at: 2015-12-05T01:48:14+00:00
[INFO] Final Memory: 69M/989M
[INFO] 
...
-1 TESTS - patch does not compile, cannot run testcases

Could you please suggest a way to troubleshoot the issue?

> oozie-setup.sh sharelib create takes a long time on large clusters
> --
>
> Key: OOZIE-2402
> URL: https://issues.apache.org/jira/browse/OOZIE-2402
> Project: Oozie
>  Issue Type: Improvement
>  Components: tools
>Affects Versions: 4.2.0
>Reporter: Illya Yalovyy
>Assignee: Illya Yalovyy
> Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch, 
> OOZIE-2402-3.patch, OOZIE-2402-4.patch
>
>
> When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. 
> Copy the tarball itself takes only around 10 seconds. It seems like 
> performance could be improved by loading files concurrently in many threads.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OOZIE-2415) Build Error for oozie-docs

[Adebiy Abdurrahman (JIRA)]

Adebiy Abdurrahman created OOZIE-2415:
-

 Summary: Build Error for oozie-docs
 Key: OOZIE-2415
 URL: https://issues.apache.org/jira/browse/OOZIE-2415
 Project: Oozie
  Issue Type: Improvement
  Components: build
Affects Versions: 4.2.0
 Environment: Darwin Kernel Version 15.0.0: Sat Sep 19 15:53:46 PDT 
2015; root:xnu-3247.10.11~1/RELEASE_X86_64 x86_64

Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9; 
2014-02-14T18:37:52+01:00)
Java version: 1.8.0_25, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.11.1", arch: "x86_64", family: "mac"

MacBook Pro (13-inch, Early 2011)

Reporter: Adebiy Abdurrahman
Priority: Minor
 Fix For: 4.3.0


Building oozie with the advised command.

bin/mkdistro.sh -DskipTests

[INFO] Apache Oozie Share Lib Hive 2 . SUCCESS [  8.926 s]
[INFO] Apache Oozie Share Lib Sqoop .. SUCCESS [  2.571 s]
[INFO] Apache Oozie Examples . SUCCESS [ 13.693 s]
[INFO] Apache Oozie Share Lib Spark .. SUCCESS [ 20.181 s]
[INFO] Apache Oozie Share Lib  SUCCESS [ 30.188 s]
[INFO] Apache Oozie Docs . FAILURE [  9.458 s]

[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-assembly-plugin:2.2.1:single (default-cli) on 
project oozie-distro: Failed to create assembly: Error adding file to archive: 
/Users/abd_adebiyi/sw/apache/oozie/oozie-4.2.0/distro/./../client/target/oozie-client-4.2.0-client.tar.gz
 isn't a file. -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :oozie-distro



Failure was due to the termination of Codehause services . See more details at 
http://www.codehaus.org

The maven pom.xml file needs mohave the repository removed.

i.e 
 
Codehaus repository
http://repository.codehaus.org/

  false

  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (62 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2413  Kerberos credentials can expire if the KDC is slow to respond
https://issues.apache.org/jira/browse/OOZIE-2413
OOZIE-2410  Fork collections-generic
https://issues.apache.org/jira/browse/OOZIE-2410
OOZIE-2407  AbandonedService should not send mail if there is no abandoned coord
https://issues.apache.org/jira/browse/OOZIE-2407
OOZIE-2402  oozie-setup.sh sharelib create takes a long time on large clusters
https://issues.apache.org/jira/browse/OOZIE-2402
OOZIE-2400  Workflow xml configuration parser cannot deal with namespace prefix
https://issues.apache.org/jira/browse/OOZIE-2400
OOZIE-2394  Oozie can execute command without holding lock
https://issues.apache.org/jira/browse/OOZIE-2394
OOZIE-2390  Rerun with failed option removing completed output data
https://issues.apache.org/jira/browse/OOZIE-2390
OOZIE-2380  Oozie Hive action failed with wrong tmp path
https://issues.apache.org/jira/browse/OOZIE-2380
OOZIE-2362  SQL injection in BulkJPAExecutor
https://issues.apache.org/jira/browse/OOZIE-2362
OOZIE-2357  CoordRerun should queue CoordPushDependencyCheckXCommand if there 
is push dependencies
https://issues.apache.org/jira/browse/OOZIE-2357
OOZIE-2349  Method getCoordJobInfo(String jobId, String filter, int offset, int 
length, boolean desc) is not present in LocalOozieClientCoord
https://issues.apache.org/jira/browse/OOZIE-2349
OOZIE-2348  Recovery service keeps on recovering coord action of suspended jobs
https://issues.apache.org/jira/browse/OOZIE-2348
OOZIE-2338  Invalid configuration defined reported for some valid configs
https://issues.apache.org/jira/browse/OOZIE-2338
OOZIE-2312  oozie doesn't purge audit and error log
https://issues.apache.org/jira/browse/OOZIE-2312
OOZIE-2273  MiniOozie does not work outside of Oozie
https://issues.apache.org/jira/browse/OOZIE-2273
OOZIE-2259  Create a callback action 
https://issues.apache.org/jira/browse/OOZIE-2259
OOZIE-2258  Introducing a new counter in the instrumentation log to distinguish 
between the reasons for launcher failure
https://issues.apache.org/jira/browse/OOZIE-2258
OOZIE-2253  Spark Job is failing when it is running in standalone server
https://issues.apache.org/jira/browse/OOZIE-2253
OOZIE-2244  Oozie should mask passwords in the logs when logging command 
arguments
https://issues.apache.org/jira/browse/OOZIE-2244
OOZIE-2243  Kill Command does not kill the child job for java action
https://issues.apache.org/jira/browse/OOZIE-2243
OOZIE-  Oozie UI parent job should be clickable
https://issues.apache.org/jira/browse/OOZIE-
OOZIE-2203  Fix the login example
https://issues.apache.org/jira/browse/OOZIE-2203
OOZIE-2196  Create Local Client for Bundle
https://issues.apache.org/jira/browse/OOZIE-2196
OOZIE-2185  Make oozie cli source /etc/oozie/conf/oozie-env.sh (or some other 
env script)
https://issues.apache.org/jira/browse/OOZIE-2185
OOZIE-2172  ZooKeeper Security Tests failed with JVM IBM JAVA
https://issues.apache.org/jira/browse/OOZIE-2172
OOZIE-2134  Remove references to Services.get().getConf() in code
https://issues.apache.org/jira/browse/OOZIE-2134
OOZIE-2106  Make tomcat download url configurable in the pom file
https://issues.apache.org/jira/browse/OOZIE-2106
OOZIE-2105  Make version of submodules configurable with parent version 
https://issues.apache.org/jira/browse/OOZIE-2105
OOZIE-2099  Add test-patch support for patches generated without --no-prefix
https://issues.apache.org/jira/browse/OOZIE-2099
OOZIE-2081  WorkflowJob notification to include coordinator action id 
https://issues.apache.org/jira/browse/OOZIE-2081
OOZIE-2060  Incorrect documentation of Java action config XML filename
https://issues.apache.org/jira/browse/OOZIE-2060
OOZIE-2044  ssh action succeed with a not exists command which should be fail.
https://issues.apache.org/jira/browse/OOZIE-2044
OOZIE-2030  Configuration properties from global section is not getting set in 
Hadoop job conf when using sub-workflow action in Oozie workflow.xml 
https://issues.apache.org/jira/browse/OOZIE-2030
OOZIE-2020  Rerun all Failed/killed/timedout coordinator actions rather than 
specifying action numbers
https://issues.apache.org/jira/browse/OOZIE-2020
OOZIE-1980  Sql error should not fail coord job
https://issues.apache.org/jira/browse/OOZIE-1980
OOZIE-1977  Display patch analysis issues
https://issues.apache.org/jira/browse/OOZIE-1977
OOZIE-1936  Queuedump command should display queue information for all server.

[jira] [Commented] (OOZIE-2413) Kerberos credentials can expire if the KDC is slow to respond

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042722#comment-15042722
 ] 

Hadoop QA commented on OOZIE-2413:
--

Testing JIRA OOZIE-2413

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/2619/

> Kerberos credentials can expire if the KDC is slow to respond
> -
>
> Key: OOZIE-2413
> URL: https://issues.apache.org/jira/browse/OOZIE-2413
> Project: Oozie
>  Issue Type: Bug
>  Components: security
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: trunk
>
> Attachments: OOZIE-2413.001.patch, OOZIE-2413.002.patch
>
>
> We've seen some very rare cases where Oozie gets a Kerberos error when trying 
> to get delegation tokens via the {{Credentials}} mechanism (e.g. getting HS2 
> delegation tokens).
> We finally narrowed it down to slow KDC responses, so Oozie's Kerberos 
> credentials have expired when it tries to get the delegation token.  The 
> reason we don't see this with Hadoop clients (DFSClient for HDFS, JobClient 
> for MR, etc) is because they call 
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before trying to 
> connect.  
> We should do a similar fix by calling 
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before using a 
> Credentials implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2185) Make oozie cli source /etc/oozie/conf/oozie-env.sh (or some other env script)

[Mike Grimes (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041893#comment-15041893
 ] 

Mike Grimes commented on OOZIE-2185:


[~rkanter] are you able to look at this again when you get a chance?

> Make oozie cli source /etc/oozie/conf/oozie-env.sh (or some other env script)
> -
>
> Key: OOZIE-2185
> URL: https://issues.apache.org/jira/browse/OOZIE-2185
> Project: Oozie
>  Issue Type: New Feature
>  Components: client
>Affects Versions: 4.3.0
>Reporter: Robert Justice
>Assignee: Mike Grimes
>Priority: Minor
>  Labels: client, feature, newbie
> Attachments: OOZIE-2185-1.patch, OOZIE-2185-2.patch, 
> OOZIE-2185-3.patch
>
>
> Currently, the oozie cli doesn't source any environment script.   Users may 
> wish to define environment variables such as OOZIE_URL outside of normal 
> shell environment and have the oozie cli source /etc/oozie/conf/oozie-env.sh. 
>  This is somewhat confusing when users see hadoop commands sourcing 
> hadoop-env.sh, mapred-env.sh, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)