date:20171015

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-10-15 Thread Attila Sasvari via Review Board



> On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 151 (patched)
> > 
> >
> > Should we skip control types?  That's probably fine for the start 
> > action, but what about a decision action?  That has some useful info.  
> > Maybe we should still print these out, but with fewer fields or something 
> > (e.g. no need for a console url).
> 
> Robert Kanter wrote:
> Thinking about this more, I think we should include control types.  It 
> will be helpful for following the flow of a workflow when there's decision 
> and fork nodes.  Please update the patch to handle this.  We don't need to 
> print out all of the fields, so you'll have to figure out which ones make 
> sense.
> 
> Attila Sasvari wrote:
> Sure, I will do so.

I believe only the decision node might interesting to see execution path, but I 
am not sure. ``info.txt`` already contains execution / transition info:
```
Action Id  : "004-171015215418832-oozie-asas-W@ctrl"
Name   : "ctrl"
Type   : "switch"
Status : "OK"
Transition : "forking"
Start Time : "2017-10-15 20:20:36 GMT"
End Time   : "2017-10-15 20:20:36 GMT"
Error Code : null
Error Message  : null
Console URL: "-"
Tracker URI: "-"
External Child Ids : null
External Id: "-"
External Status: "forking"
Data   : null
Stats  : null
Credentials: null
Retries: "0"
User Retry Int : "10"
User Retry Count   : "0"
User Retry Max : "0"
```

If I store resolved action defintion, it would not add too much information:
```
# cat 004-171015215418832-oozie-asas-W/resolved-actions/ctrl.xml  

  true
  
% 
```

I created a workflow with a fork node and also with a decision node. I saw that 
resolved action config xml were empty for control nodes other than the decision 
node.

```
0  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/:start:.xml
  108  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/ctrl.xml
0  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/end.xml
0  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/forking.xml
0  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/join.xml
83834  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/launcher_mr-node.log
83843  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/launcher_mr-node2.log
47965  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/launcher_shell1.log
47965  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/launcher_shell2.log
 2045  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/mr-node.xml
 2045  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/mr-node2.xml
  544  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/shell1.xml
  544  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/shell2.xml
```
Zero byte xml-s are not interesting. I will remove non-decision action types.

Other thing: it would also good to add later an option to download the final 
job graph (thanks to OOZIE-2406 now we can download it as a Dot file too). If 
they have many action, it might help a lot when taking a quick look. 

Misc: Would it better to have a different pattern for the filename? For 
example: it might be good to see where the bundle was actually taken (i.e. 
include fqdn-hostname, there are existing diagnostic tools that use similar 
pattern).


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186754
---


On Oct. 4, 2017, 2:18 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Oct. 4, 2017, 2:18 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.

[jira] [Updated] (OOZIE-2296) Add an Oozie diagnostic bundle tool

2017-10-15 Thread Attila Sasvari (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2296:
--
Attachment: OOZIE-2296-014.patch

> Add an Oozie diagnostic bundle tool
> ---
>
> Key: OOZIE-2296
> URL: https://issues.apache.org/jira/browse/OOZIE-2296
> Project: Oozie
>  Issue Type: New Feature
>  Components: tools
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Attachments: OOZIE-2296-002.patch, OOZIE-2296-003.patch, 
> OOZIE-2296-004.patch, OOZIE-2296-005.patch, OOZIE-2296-006.patch, 
> OOZIE-2296-009.patch, OOZIE-2296-010.patch, OOZIE-2296-011.patch, 
> OOZIE-2296-012.patch, OOZIE-2296-013.patch, OOZIE-2296-014.patch
>
>
> To help with our support cases, I've built a tool that collects a bunch of 
> job and other information from Oozie that throws it all in a tarball.  The 
> idea is that the user can just click a button in Cloudera Manager, and it 
> will run this tool.  
> This tool could be useful for others as an easy way to get information out of 
> Oozie, so I thought I'd contribute it here.  It's built as a "tool" (so it 
> sits next to the sharelib and database tools), and simply uses the Oozie 
> client for getting pretty much everything, so it doesn't require anything 
> special.
> Here's the information that it can get:
> # Sharelib: {{ooze admin -shareliblist}} and {{oozie admin -shareliblist 
> }}
> # Oozie Server's resolved loaded configuration (from admin endpoint)
> # Other admin commands output (queue dump, env vars, etc)
> # Thread dump (HOST:11000/oozie/admin/jvminfo.jsp)
> # Details from last n jobs and/or specific list of jobs
> #- job.properties contents
> #- XML definition
> #- verbose status for each job and each action etc
> #- Oozie logs
> #- Unfortunately, we can't get the launcher jobs' logs from Hadoop
> # Metrics/Instrumentation



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-2296 PreCommit Build #113

2017-10-15 Thread Apache Jenkins Server

Jira: https://issues.apache.org/jira/browse/OOZIE-2296
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/113/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.54 MB...]
+1 the patch does adds/modifies 4 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 77 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
0 There are [3] new bugs found in total that would be nice to have fixed.
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [examples].
 0 There are [3] new bugs found in [tools] that would be nice to have fixed.
 You can find the FindBugs diff here: tools/findbugs-new.html
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/pig].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS - patch does not compile, cannot run testcases
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/113/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100 3706k  100 3706k0 0  4065k  0 --:--:-- --:--:-- --:--:-- 9576k
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-2296
Archiving artifacts
[Fast Archiver] Compressed 1.80 MB of artifacts by 52.0% relative to #108
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Commented] (OOZIE-2296) Add an Oozie diagnostic bundle tool

2017-10-15 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16205353#comment-16205353
 ] 

Hadoop QA commented on OOZIE-2296:
--

Testing JIRA OOZIE-2296

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 4 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:orange}0{color} There are [3] new bugs found in total that would be nice 
to have fixed.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:orange}0{color} There are [3] new bugs found in [tools] that would be 
nice to have fixed.
. You can find the FindBugs diff here: tools/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/113/

> Add an Oozie diagnostic bundle tool
> ---
>
> Key: OOZIE-2296
> URL: https://issues.apache.org/jira/browse/OOZIE-2296
> Project: Oozie
>  Issue Type: New Feature
>  Components: tools
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Attachments: OOZIE-2296-002.patch, OOZIE-2296-003.patch, 
> OOZIE-2296-004.patch, OOZIE-2296-005.patch, OOZIE-2296-006.patch, 
> OOZIE-2296-009.patch, OOZIE-2296-010.patch, OOZIE-2296-011.patch, 
> OOZIE-2296-012.patch, OOZIE-2296-013.patch, OOZIE-2296-014.patch
>
>
> To help with our support cases, I've built a tool that collects a bunch of 
> job and other information from Oozie that throws it all in a tarball.  The 
> idea is that the user can just click a button in Cloudera Manager, and it 
> will run this tool.  
> This tool could be useful for others as an easy way to get information out of 
> Oozie, so I thought I'd contribute it here.  It's built as a "tool" (so it 
> sits next to the sharelib and database tools), and simply uses the Oozie 
> client for getting pretty much everything, so it doesn't require anything 
> special.
> Here's the information that it can get:
> # Sharelib: {{ooze admin -shareliblist}} and {{oozie admin -shareliblist 
> }}
> # Oozie Server's resolved loaded configuration (from admin endpoint)
> # Other admin commands output (queue dump, env vars, etc)
> # Thread dump (HOST:11000/oozie/admin/jvminfo.jsp)
> # Details from last n jobs and/or specific list of jobs
> #- job.properties contents
> #- XML definition
> #- verbose status for each job and each action etc
> #- Oozie logs
> #- Unfortunately, we can't get th

[jira] Subscription: Oozie Patch Available

2017-10-15 Thread jira

Issue Subscription
Filter: Oozie Patch Available (107 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3084  Add missing JVM properties to Jetty startup script
https://issues.apache.org/jira/browse/OOZIE-3084
OOZIE-3083  Make improved version Info backward compatible
https://issues.apache.org/jira/browse/OOZIE-3083
OOZIE-3071  Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 
than Spark 2.2.0
https://issues.apache.org/jira/browse/OOZIE-3071
OOZIE-3063  Sanitizing variables that are part of openjpa.ConnectionProperties
https://issues.apache.org/jira/browse/OOZIE-3063
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues.apache.org/jira/browse/OOZIE-3062
OOZIE-3022  fix for warning has no file and won't be listed in dependency files 
details
https://issues.apache.org/jira/browse/OOZIE-3022
OOZIE-3002  address findbugs errors in client lib
https://issues.apache.org/jira/browse/OOZIE-3002
OOZIE-3001  core library has many instances of warnings with trailing spaces 
and lines longer than 132 chars
https://issues.apache.org/jira/browse/OOZIE-3001
OOZIE-2997  files contain trailing white spaces in client lib
https://issues.apache.org/jira/browse/OOZIE-2997
OOZIE-2996  add option for -UseGCOverheadLimit to maven opts as sometimes local 
testing fails
https://issues.apache.org/jira/browse/OOZIE-2996
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues.apache.org/jira/browse/OOZIE-2975
OOZIE-2969  Drop support for Java 1.7
https://issues.apache.org/jira/browse/OOZIE-2969
OOZIE-2964  Add -Xdoclint:none to javadoc opts to avoid warnings
https://issues.apache.org/jira/browse/OOZIE-2964
OOZIE-2962  bump maven-javadoc-plugin to 2.10.4
https://issues.apache.org/jira/browse/OOZIE-2962
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues.apache.org/jira/browse/OOZIE-2934
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2914  Consolidate Trim 
https://issues.apache.org/jira/browse/OOZIE-2914
OOZIE-2896  Ensure compatibility for existing LauncherMapper settings
https://issues.apache.org/jira/browse/OOZIE-2896
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues.apache.org/jira/browse/OOZIE-2867
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues.apache.org/jira/browse/OOZIE-2829
OOZIE-2826  Falcon feed fails to aws s3; Oozie joda time version does not meet 
required jar version 2.2 or later
https://issues.apache.org/jira/browse/OOZIE-2826
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2795  Create lib directory or symlink for Oozie CLI during packaging

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

[jira] [Updated] (OOZIE-2296) Add an Oozie diagnostic bundle tool

Failed: OOZIE-2296 PreCommit Build #113

[jira] [Commented] (OOZIE-2296) Add an Oozie diagnostic bundle tool

[jira] Subscription: Oozie Patch Available

5 matches

Site Navigation

Mail list logo

Footer information