[jira] Subscription: Oozie Patch Available

2017-09-21 Thread jira 
Issue Subscription
Filter: Oozie Patch Available (107 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues.apache.org/jira/browse/OOZIE-3062
OOZIE-3054  Disable erasure coding for sharelib if Oozie runs on Hadoop 3
https://issues.apache.org/jira/browse/OOZIE-3054
OOZIE-3031  Coord job with only unresolved dependencies doesn't timeout
https://issues.apache.org/jira/browse/OOZIE-3031
OOZIE-3022  fix for warning has no file and won't be listed in dependency files 
details
https://issues.apache.org/jira/browse/OOZIE-3022
OOZIE-3002  address findbugs errors in client lib
https://issues.apache.org/jira/browse/OOZIE-3002
OOZIE-3001  core library has many instances of warnings with trailing spaces 
and lines longer than 132 chars
https://issues.apache.org/jira/browse/OOZIE-3001
OOZIE-2997  files contain trailing white spaces in client lib
https://issues.apache.org/jira/browse/OOZIE-2997
OOZIE-2996  add option for -UseGCOverheadLimit to maven opts as sometimes local 
testing fails
https://issues.apache.org/jira/browse/OOZIE-2996
OOZIE-2978  Remove code that handles Pig versions before 0.8 
https://issues.apache.org/jira/browse/OOZIE-2978
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues.apache.org/jira/browse/OOZIE-2975
OOZIE-2969  Drop support for Java 1.7
https://issues.apache.org/jira/browse/OOZIE-2969
OOZIE-2964  Add -Xdoclint:none to javadoc opts to avoid warnings
https://issues.apache.org/jira/browse/OOZIE-2964
OOZIE-2963  getting error in build ArtifactNotFoundException: Could not find 
artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde
https://issues.apache.org/jira/browse/OOZIE-2963
OOZIE-2962  bump maven-javadoc-plugin to 2.10.4
https://issues.apache.org/jira/browse/OOZIE-2962
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues.apache.org/jira/browse/OOZIE-2934
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2914  Consolidate Trim 
https://issues.apache.org/jira/browse/OOZIE-2914
OOZIE-2885  Running Spark actions should not need Hive on the classpath
https://issues.apache.org/jira/browse/OOZIE-2885
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues.apache.org/jira/browse/OOZIE-2867
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues.apache.org/jira/browse/OOZIE-2829
OOZIE-2826  Falcon feed fails to aws s3; Oozie joda time version does not meet 
required jar version 2.2 or later
https://issues.apache.org/jira/browse/OOZIE-2826
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2795  Create lib di

[jira] [Commented] (OOZIE-2296) Add an Oozie diagnostic bundle tool

2017-09-21 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175307#comment-16175307
 ] 

Hadoop QA commented on OOZIE-2296:
--

Testing JIRA OOZIE-2296

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 1 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1770
.Tests rerun: 10
.Tests failed at first run: 
org.apache.oozie.TestSLAEventBean,org.apache.oozie.TestDagELFunctions,org.apache.oozie.TestDagEngine,org.apache.oozie.TestCoordinatorEngineStreamLog,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/43/

> Add an Oozie diagnostic bundle tool
> ---
>
> Key: OOZIE-2296
> URL: https://issues.apache.org/jira/browse/OOZIE-2296
> Project: Oozie
>  Issue Type: New Feature
>  Components: tools
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Attachments: OOZIE-2296-002.patch, OOZIE-2296-003.patch, 
> OOZIE-2296-004.patch
>
>
> To help with our support cases, I've built a tool that collects a bunch of 
> job and other information from Oozie that throws it all in a tarball.  The 
> idea is that the user can just click a button in Cloudera Manager, and it 
> will run this tool.  
> This tool could be useful for others as an easy way to get information out of 
> Oozie, so I thought I'd contribute it here.  It's built as a "tool" (so it 
> sits next to the sharelib and database tools), and simply uses the Oozie 
> client for getting pretty much everything, so it doesn't require anything 
> special.
> Here's the information that it can get:
> # Sharelib: {{ooze admin -shareliblist}} and {{oozie admin -shareliblist 
> }}
> # Oozie Server's resolved loaded configuration (from admin endpoint)
> # Other admin commands output (queue dump, env vars, etc)
> # Thread dump (HOST:11000/oozie/admin/jvminfo.jsp)
> # Details from last n jobs and/or specific list of jobs
> #- job.properties contents
> #- XML definition
> #- verbose status for each job and each action etc
> #- Oozie logs
> #- Unfortunately, we can't get the launcher jobs' logs from Hadoop
> # Metrics/Instrumentation



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-2296 PreCommit Build #43

2017-09-21 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-2296
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/43/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.57 MB...]
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 77 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/pig].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 1770
Tests rerun: 10
Tests failed at first run: 
org.apache.oozie.TestSLAEventBean,org.apache.oozie.TestDagELFunctions,org.apache.oozie.TestDagEngine,org.apache.oozie.TestCoordinatorEngineStreamLog,
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/43/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  5 3706k5  189k0 0   281k  0  0:00:13 --:--:--  0:00:13  
281k100 3706k  100 3706k0 0  4766k  0 --:--:-- --:--:-- --:--:-- 
33.3M
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-2296
Archiving artifacts
[Fast Archiver] Compressed 1.80 MB of artifacts by 60.8% relative to #41
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Created] (OOZIE-3064) Add Spark SQL test for spark action

2017-09-21 Thread Satish Subhashrao Saley (JIRA) 
Satish Subhashrao Saley created OOZIE-3064:
--

 Summary: Add Spark SQL test for spark action
 Key: OOZIE-3064
 URL: https://issues.apache.org/jira/browse/OOZIE-3064
 Project: Oozie
  Issue Type: Bug
Reporter: Satish Subhashrao Saley


Add Spark SQL test for spark action. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2554) Spark action tests on yarn-cluster, yarn-client mode

2017-09-21 Thread Peter Cseh (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175136#comment-16175136
 ] 

Peter Cseh commented on OOZIE-2554:
---

Pig did something along the lines of this in PIG-5305, we could investigate 
their solution.

> Spark action tests on yarn-cluster, yarn-client mode
> 
>
> Key: OOZIE-2554
> URL: https://issues.apache.org/jira/browse/OOZIE-2554
> Project: Oozie
>  Issue Type: New Feature
>Reporter: Satish Subhashrao Saley
> Attachments: OOZIE-2554.poc.patch
>
>
> Currently, we have unit test cases for Spark Action in local mode. To verify 
> other functionality, we should have tests in yarn-cluster and yarn-client 
> mode.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3062) Set HADOOP_CONF_DIR for spark action

2017-09-21 Thread Satish Subhashrao Saley (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-3062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-3062:
---
Description: 
OOZIE-2569 created core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml 
for spark action. But for spark to consider these files ([spark code 
ref|https://github.com/apache/spark/blob/83fe3b5e10f8dc62245ea37143abb96be0f39805/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala#L284]),
 we need to set HADOOP_CONF_DIR environmental variable, otherwise it will point 
to default HADOOP_CONF_DIR configured on the node. Therefore, Oozie should set 
HADOOP_CONF_DIR.
Due to a bug YARN-4727, yarn was not considering user specified value for 
whiltelisted environment variables including HADOOP_CONF_DIR. So, this will 
work on specified hadoop fix versions in YARN-4727.

This fix in Oozie will be no-op on Hadoop without YARN-4727 fix and Oozie will 
behave as it is.

To verify this on one node machine. 
1. In hadoop mapred-site.xml, set 
{code}

   mapreduce.fileoutputcommitter.marksuccessfuljobs
   false

{code}
2. Run any spark file copy example from Oozie. Check output directory, _SUCCESS 
flag is missing. It should be there because we explicitly set 
conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "true"); in 
https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L293

  was:
OOZIE-2569 created core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml 
for spark action. But for spark to consider these files ([spark code 
ref|https://github.com/apache/spark/blob/83fe3b5e10f8dc62245ea37143abb96be0f39805/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala#L284]),
 we need to set HADOOP_CONF_DIR environmental variable, otherwise it will point 
to default HADOOP_CONF_DIR configured on the node. Therefore, Oozie should set 
HADOOP_CONF_DIR.
Due to a bug YARN-4727, yarn was not considering user specified value for 
whiltelisted environment variables including HADOOP_CONF_DIR. So, this will 
work on specified hadoop fix versions in YARN-4727.

This fix in Oozie will be no-op on Hadoop without YARN-4727 fix and will behave 
as it is.

To verify this on one node machine. 
1. In hadoop mapred-site.xml, set 
{code}

   mapreduce.fileoutputcommitter.marksuccessfuljobs
   false

{code}
2. Run any spark file copy example from Oozie. Check output directory, _SUCCESS 
flag is missing. It should be there because we explicitly set 
conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "true"); in 
https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L293


> Set HADOOP_CONF_DIR for spark action
> 
>
> Key: OOZIE-3062
> URL: https://issues.apache.org/jira/browse/OOZIE-3062
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3062-1.patch
>
>
> OOZIE-2569 created core-site.xml, hdfs-site.xml, mapred-site.xml, 
> yarn-site.xml for spark action. But for spark to consider these files ([spark 
> code 
> ref|https://github.com/apache/spark/blob/83fe3b5e10f8dc62245ea37143abb96be0f39805/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala#L284]),
>  we need to set HADOOP_CONF_DIR environmental variable, otherwise it will 
> point to default HADOOP_CONF_DIR configured on the node. Therefore, Oozie 
> should set HADOOP_CONF_DIR.
> Due to a bug YARN-4727, yarn was not considering user specified value for 
> whiltelisted environment variables including HADOOP_CONF_DIR. So, this will 
> work on specified hadoop fix versions in YARN-4727.
> This fix in Oozie will be no-op on Hadoop without YARN-4727 fix and Oozie 
> will behave as it is.
> To verify this on one node machine. 
> 1. In hadoop mapred-site.xml, set 
> {code}
> 
>mapreduce.fileoutputcommitter.marksuccessfuljobs
>false
> 
> {code}
> 2. Run any spark file copy example from Oozie. Check output directory, 
> _SUCCESS flag is missing. It should be there because we explicitly set 
> conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "true"); in 
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L293



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3062) Set HADOOP_CONF_DIR for spark action

2017-09-21 Thread Satish Subhashrao Saley (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-3062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-3062:
---
Description: 
OOZIE-2569 created core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml 
for spark action. But for spark to consider these files ([spark code 
ref|https://github.com/apache/spark/blob/83fe3b5e10f8dc62245ea37143abb96be0f39805/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala#L284]),
 we need to set HADOOP_CONF_DIR environmental variable, otherwise it will point 
to default HADOOP_CONF_DIR configured on the node. Therefore, Oozie should set 
HADOOP_CONF_DIR.
Due to a bug YARN-4727, yarn was not considering user specified value for 
whiltelisted environment variables including HADOOP_CONF_DIR. So, this will 
work on specified hadoop fix versions in YARN-4727.

This fix in Oozie will be no-op on Hadoop without YARN-4727 fix and will behave 
as it is.

To verify this on one node machine. 
1. In hadoop mapred-site.xml, set 
{code}

   mapreduce.fileoutputcommitter.marksuccessfuljobs
   false

{code}
2. Run any spark file copy example from Oozie. Check output directory, _SUCCESS 
flag is missing. It should be there because we explicitly set 
conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "true"); in 
https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L293

  was:
OOZIE-2569 created core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml 
for spark action. But for spark to consider these files ([spark code 
ref|https://github.com/apache/spark/blob/83fe3b5e10f8dc62245ea37143abb96be0f39805/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala#L284]),
 we need to set HADOOP_CONF_DIR environmental variable, otherwise it will point 
to default HADOOP_CONF_DIR configured on the node. Therefore, Oozie should set 
HADOOP_CONF_DIR.
Due to a bug YARN-4727, yarn was not considering user specified value for 
whiltelisted environment variables including HADOOP_CONF_DIR. So, this will 
work on specified hadoop fix versions in YARN-4727.

To verify this on one node machine. 
1. In hadoop mapred-site.xml, set 
{code}

   mapreduce.fileoutputcommitter.marksuccessfuljobs
   false

{code}
2. Run any spark file copy example from Oozie. Check output directory, _SUCCESS 
flag is missing. It should be there because we explicitly set 
conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "true"); in 
https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L293


> Set HADOOP_CONF_DIR for spark action
> 
>
> Key: OOZIE-3062
> URL: https://issues.apache.org/jira/browse/OOZIE-3062
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3062-1.patch
>
>
> OOZIE-2569 created core-site.xml, hdfs-site.xml, mapred-site.xml, 
> yarn-site.xml for spark action. But for spark to consider these files ([spark 
> code 
> ref|https://github.com/apache/spark/blob/83fe3b5e10f8dc62245ea37143abb96be0f39805/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala#L284]),
>  we need to set HADOOP_CONF_DIR environmental variable, otherwise it will 
> point to default HADOOP_CONF_DIR configured on the node. Therefore, Oozie 
> should set HADOOP_CONF_DIR.
> Due to a bug YARN-4727, yarn was not considering user specified value for 
> whiltelisted environment variables including HADOOP_CONF_DIR. So, this will 
> work on specified hadoop fix versions in YARN-4727.
> This fix in Oozie will be no-op on Hadoop without YARN-4727 fix and will 
> behave as it is.
> To verify this on one node machine. 
> 1. In hadoop mapred-site.xml, set 
> {code}
> 
>mapreduce.fileoutputcommitter.marksuccessfuljobs
>false
> 
> {code}
> 2. Run any spark file copy example from Oozie. Check output directory, 
> _SUCCESS flag is missing. It should be there because we explicitly set 
> conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "true"); in 
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L293



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3062) Set HADOOP_CONF_DIR for spark action

2017-09-21 Thread Rohini Palaniswamy (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175127#comment-16175127
 ] 

Rohini Palaniswamy commented on OOZIE-3062:
---

env setting is usually separated by , in hadoop configs and not 
File.pathSeparator. Can you verify?

> Set HADOOP_CONF_DIR for spark action
> 
>
> Key: OOZIE-3062
> URL: https://issues.apache.org/jira/browse/OOZIE-3062
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3062-1.patch
>
>
> OOZIE-2569 created core-site.xml, hdfs-site.xml, mapred-site.xml, 
> yarn-site.xml for spark action. But for spark to consider these files ([spark 
> code 
> ref|https://github.com/apache/spark/blob/83fe3b5e10f8dc62245ea37143abb96be0f39805/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala#L284]),
>  we need to set HADOOP_CONF_DIR environmental variable, otherwise it will 
> point to default HADOOP_CONF_DIR configured on the node. Therefore, Oozie 
> should set HADOOP_CONF_DIR.
> Due to a bug YARN-4727, yarn was not considering user specified value for 
> whiltelisted environment variables including HADOOP_CONF_DIR. So, this will 
> work on specified hadoop fix versions in YARN-4727.
> To verify this on one node machine. 
> 1. In hadoop mapred-site.xml, set 
> {code}
> 
>mapreduce.fileoutputcommitter.marksuccessfuljobs
>false
> 
> {code}
> 2. Run any spark file copy example from Oozie. Check output directory, 
> _SUCCESS flag is missing. It should be there because we explicitly set 
> conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "true"); in 
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L293



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-21 Thread Attila Sasvari 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/
---

(Updated Sept. 21, 2017, 5:10 p.m.)


Review request for oozie.


Repository: oozie-git


Description
---

A diagnostic tool that collects a bunch of job and other information from Oozie 
in a zip file.


Diffs (updated)
-

  docs/src/site/twiki/DG_CommandLineTool.twiki 
d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
  pom.xml efccc346932514ada578a3462eb3c3cfe519a323 
  tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
  tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/TestOozieDiagBundleCollector.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/62459/diff/2/

Changes: https://reviews.apache.org/r/62459/diff/1-2/


Testing
---

- new unit tests: TestOozieDiagBundleCollector
- started Oozie with a pseudo hadoop cluster, submitted a couple workflows, and 
executed the following commands: 
-- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
-- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
http://localhost:11000/oozie -output /tmp``, 
-- ``bin/oozie-diag-bundle-collector.sh  -jobs 
001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
-output .`` (verified zip the tool generated).


Thanks,

Attila Sasvari

[jira] [Updated] (OOZIE-2296) Add an Oozie diagnostic bundle tool

2017-09-21 Thread Attila Sasvari (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2296:
--
Attachment: OOZIE-2296-004.patch

> Add an Oozie diagnostic bundle tool
> ---
>
> Key: OOZIE-2296
> URL: https://issues.apache.org/jira/browse/OOZIE-2296
> Project: Oozie
>  Issue Type: New Feature
>  Components: tools
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Attachments: OOZIE-2296-002.patch, OOZIE-2296-003.patch, 
> OOZIE-2296-004.patch
>
>
> To help with our support cases, I've built a tool that collects a bunch of 
> job and other information from Oozie that throws it all in a tarball.  The 
> idea is that the user can just click a button in Cloudera Manager, and it 
> will run this tool.  
> This tool could be useful for others as an easy way to get information out of 
> Oozie, so I thought I'd contribute it here.  It's built as a "tool" (so it 
> sits next to the sharelib and database tools), and simply uses the Oozie 
> client for getting pretty much everything, so it doesn't require anything 
> special.
> Here's the information that it can get:
> # Sharelib: {{ooze admin -shareliblist}} and {{oozie admin -shareliblist 
> }}
> # Oozie Server's resolved loaded configuration (from admin endpoint)
> # Other admin commands output (queue dump, env vars, etc)
> # Thread dump (HOST:11000/oozie/admin/jvminfo.jsp)
> # Details from last n jobs and/or specific list of jobs
> #- job.properties contents
> #- XML definition
> #- verbose status for each job and each action etc
> #- Oozie logs
> #- Unfortunately, we can't get the launcher jobs' logs from Hadoop
> # Metrics/Instrumentation



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2885) Running Spark actions should not need Hive on the classpath

2017-09-21 Thread Rohini Palaniswamy (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175120#comment-16175120
 ] 

Rohini Palaniswamy commented on OOZIE-2885:
---

bq. This depends on OOZIE-2554
  [~satishsaley], can you create a separate jira for adding a Spark SQL 
testcase making it dependent on OOZIE-2554 and relating to this jira. Testing 
spark sql should automatically cover using hive-site.xml which is the actual 
use case we have in production. 

> Running Spark actions should not need Hive on the classpath
> ---
>
> Key: OOZIE-2885
> URL: https://issues.apache.org/jira/browse/OOZIE-2885
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Satish Subhashrao Saley
>Priority: Critical
> Attachments: OOZIE-2885-1.patch
>
>
> SparkMain references HiveConf so we need hive-common jar on the classpath 
> even if the Spark job has nothing to do with Hive. 
> This shouldn't be the case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (OOZIE-2896) Ensure compatibility for existing LauncherMapper settings

2017-09-21 Thread Peter Cseh (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh reassigned OOZIE-2896:
-

Assignee: Andras Piros  (was: Peter Cseh)

> Ensure compatibility for existing LauncherMapper settings
> -
>
> Key: OOZIE-2896
> URL: https://issues.apache.org/jira/browse/OOZIE-2896
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Bacsko
>Assignee: Andras Piros
>Priority: Blocker
>
> There are a couple of LauncherMapper settings that we should support for 
> backward compatibility reasons.
> For example, {{oozie.launcher.mapreduce.map.memory.mb}}, 
> {{mapred.child.env}}, {{mapred.map.env}}, etc.
> We also have to discuss the list of properties we're going to support, since 
> there are many of them.
> List of hadoop properties under this task:
> mapreduce.job.queuename
> mapreduce.map.memory.mb
> mapreduce.map.java.opts
> mapreduce.map.env
> mapreduce.job.priority
> yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms
> yarn.app.mapreduce.am.resource.mb
> yarn.app.mapreduce.am.command-opts
> yarn.app.mapreduce.am.env
> ACL related properties should go under OOZIE-2897:
> mapreduce.job.acl-view-job
> mapreduce.job.acl-modfy-job



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2885) Running Spark actions should not need Hive on the classpath

2017-09-21 Thread Rohini Palaniswamy (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175096#comment-16175096
 ] 

Rohini Palaniswamy commented on OOZIE-2885:
---

bq. Please extract the code in the try-catch block to a method w/ a describing 
name
  It is already in a setupHiveSite method. Please do not add another method for 
this. 

[~satishsaley],
   I think you can actually remove
{code}
HiveConf.setHiveSiteLocation(HiveConf.class.getClassLoader().getResource("hive-site.xml"));
{code}

and it should still work fine.

bq. and add a unit test to TestSparkMain 
  This depends on OOZIE-2554. Any normal test in cluster mode would have caught 
this. We need to fix OOZIE-2554 soon.

> Running Spark actions should not need Hive on the classpath
> ---
>
> Key: OOZIE-2885
> URL: https://issues.apache.org/jira/browse/OOZIE-2885
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Satish Subhashrao Saley
>Priority: Critical
> Attachments: OOZIE-2885-1.patch
>
>
> SparkMain references HiveConf so we need hive-common jar on the classpath 
> even if the Spark job has nothing to do with Hive. 
> This shouldn't be the case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 62470: OOZIE-3062 Set HADOOP_CONF_DIR for spark action

2017-09-21 Thread Satish Saley 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62470/
---

Review request for oozie.


Bugs: OOZIE-3062
https://issues.apache.org/jira/browse/OOZIE-3062


Repository: oozie-git


Description
---

OOZIE-3062 Set HADOOP_CONF_DIR for spark action


Diffs
-

  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
9d1afb51 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
80d64ec8 
  
sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java
 d97f1f06 


Diff: https://reviews.apache.org/r/62470/diff/1/


Testing
---


Thanks,

Satish Saley

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-21 Thread Peter Cseh 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review185887
---




tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 869 (patched)


I think we should distinguish "null" values from empty strings as this can 
be an important thing in configurations


- Peter Cseh


On Sept. 21, 2017, 10:31 a.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Sept. 21, 2017, 10:31 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/TestOozieDiagBundleCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/1/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-21 Thread Peter Cseh 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review185884
---




tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 141-148 (patched)


Can these be fields so the argument lists are not that long?


- Peter Cseh


On Sept. 21, 2017, 10:31 a.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Sept. 21, 2017, 10:31 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/TestOozieDiagBundleCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/1/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

[jira] [Commented] (OOZIE-2296) Add an Oozie diagnostic bundle tool

2017-09-21 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174658#comment-16174658
 ] 

Hadoop QA commented on OOZIE-2296:
--

Testing JIRA OOZIE-2296

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 3 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:red}-1{color} There are [6] new bugs found below threshold in total that 
must be fixed.
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:red}-1{color} There are [6] new bugs found below threshold in [tools] 
that must be fixed, listing only the first [5] ones.
. You can find the FindBugs diff here (look for the red and orange ones): 
tools/findbugs-new.html
. The top [5] most important FindBugs errors are:
. At OozieDiagBundleCollector.java:[line 178]: 
org.apache.oozie.tools.OozieDiagBundleCollector.parseCommandLineArgs(String[]) 
invokes System.exit(...), which shuts down the entire virtual machine
. Method invoked at OozieDiagBundleCollector.java:[line 740]: Null passed for 
non-null parameter of new java.io.File(File, String) in 
org.apache.oozie.tools.OozieDiagBundleCollector.getBundleJob(OozieClient, File, 
String, int)
. File(...) reads a file whose location might be specified by user input: Known 
null at OozieDiagBundleCollector.java:[line 734]
. At OozieDiagBundleCollector.java:[line 145]: At 
OozieDiagBundleCollector.java:[line 740]
. At OozieDiagBundleCollector.java:[line 200]: At 
OozieDiagBundleCollector.java:[line 156]
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/42/

> Add an Oozie diagnostic bundle tool
> ---
>
> Key: OOZIE-2296
> URL: https://issues.apache.org/jira/browse/OOZIE-2296
> Project: Oozie
>  Issue Type: New Feature
>  Components: tools
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Attachments: OOZIE-2296-002.patch, OOZIE-2296-003.patch
>
>
> To help with our support cases, I've built a tool that collects a bunch of 
> job and other information from Oozie that throws it all in a tarball.  The 
> idea is that the user can just click a button in Cloudera Manager, and it 
> will run this tool.  
> This tool could be useful for others as an easy way to get information out of 
> Oozie, so I thought I'd contribute it here.  It's built as a "tool" (so it 
> sits next to the sharelib and

Failed: OOZIE-2296 PreCommit Build #42

2017-09-21 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-2296
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/42/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.54 MB...]
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
-1 There are [6] new bugs found below threshold in total that must be fixed.
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/oozie].
 -1 There are [6] new bugs found below threshold in [tools] that must be fixed, 
listing only the first [5] ones.
 You can find the FindBugs diff here (look for the red and orange ones): 
tools/findbugs-new.html
 The top [5] most important FindBugs errors are:
 At OozieDiagBundleCollector.java:[line 178]: 
org.apache.oozie.tools.OozieDiagBundleCollector.parseCommandLineArgs(String[]) 
invokes System.exit(...), which shuts down the entire virtual machine
 Method invoked at OozieDiagBundleCollector.java:[line 740]: Null passed for 
non-null parameter of new java.io.File(File, String) in 
org.apache.oozie.tools.OozieDiagBundleCollector.getBundleJob(OozieClient, File, 
String, int)
 File(...) reads a file whose location might be specified by user input: Known 
null at OozieDiagBundleCollector.java:[line 734]
 At OozieDiagBundleCollector.java:[line 145]: At 
OozieDiagBundleCollector.java:[line 740]
 At OozieDiagBundleCollector.java:[line 200]: At 
OozieDiagBundleCollector.java:[line 156]
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS - patch does not compile, cannot run testcases
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/42/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100 3706k  100 3706k0 0  3293k  0  0:00:01  0:00:01 --:--:-- 5083k
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-2296
Archiving artifacts
[Fast Archiver] Compressed 1.81 MB of artifacts by 77.8% relative to #41
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Commented] (OOZIE-3063) Sanitizing variables that are part of openjpa.ConnectionProperties

2017-09-21 Thread Peter Cseh (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174628#comment-16174628
 ] 

Peter Cseh commented on OOZIE-3063:
---

I've tried to poke around this in OOZIE-2608, you might find interesting stuff 
in that issue's comments.

> Sanitizing variables that are part of openjpa.ConnectionProperties
> --
>
> Key: OOZIE-3063
> URL: https://issues.apache.org/jira/browse/OOZIE-3063
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.2.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>
> There are values from oozie-site.xml which are not properly checked before 
> they are passed into a comma-separated string property in JPAService class. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (OOZIE-3063) Sanitizing variables that are part of openjpa.ConnectionProperties

2017-09-21 Thread Denes Bodo (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denes Bodo reassigned OOZIE-3063:
-

Assignee: Denes Bodo

> Sanitizing variables that are part of openjpa.ConnectionProperties
> --
>
> Key: OOZIE-3063
> URL: https://issues.apache.org/jira/browse/OOZIE-3063
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.2.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>
> There are values from oozie-site.xml which are not properly checked before 
> they are passed into a comma-separated string property in JPAService class. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3063) Sanitizing variables that are part of openjpa.ConnectionProperties

2017-09-21 Thread Denes Bodo (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174576#comment-16174576
 ] 

Denes Bodo commented on OOZIE-3063:
---

[~andras.piros] I'll provide code later today. Oozie should check for integer 
values and url to avoid accidentally written SQL statements in connection 
properties.

> Sanitizing variables that are part of openjpa.ConnectionProperties
> --
>
> Key: OOZIE-3063
> URL: https://issues.apache.org/jira/browse/OOZIE-3063
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.2.0
>Reporter: Denes Bodo
>
> There are values from oozie-site.xml which are not properly checked before 
> they are passed into a comma-separated string property in JPAService class. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3063) Sanitizing variables that are part of openjpa.ConnectionProperties

2017-09-21 Thread Andras Piros (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174567#comment-16174567
 ] 

Andras Piros commented on OOZIE-3063:
-

Good idea [~dionusos]! Any ideas / wishes what to filter?

> Sanitizing variables that are part of openjpa.ConnectionProperties
> --
>
> Key: OOZIE-3063
> URL: https://issues.apache.org/jira/browse/OOZIE-3063
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.2.0
>Reporter: Denes Bodo
>
> There are values from oozie-site.xml which are not properly checked before 
> they are passed into a comma-separated string property in JPAService class. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-21 Thread Attila Sasvari 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/
---

Review request for oozie.


Repository: oozie-git


Description
---

A diagnostic tool that collects a bunch of job and other information from Oozie 
in a zip file.


Diffs
-

  docs/src/site/twiki/DG_CommandLineTool.twiki 
d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
  tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/TestOozieDiagBundleCollector.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/62459/diff/1/


Testing
---

- new unit tests: TestOozieDiagBundleCollector
- started Oozie with a pseudo hadoop cluster, submitted a couple workflows, and 
executed the following commands: 
-- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
-- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
http://localhost:11000/oozie -output /tmp``, 
-- ``bin/oozie-diag-bundle-collector.sh  -jobs 
001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
-output .`` (verified zip the tool generated).


Thanks,

Attila Sasvari

[jira] [Created] (OOZIE-3063) Sanitizing variables that are part of openjpa.ConnectionProperties

2017-09-21 Thread Denes Bodo (JIRA) 
Denes Bodo created OOZIE-3063:
-

 Summary: Sanitizing variables that are part of 
openjpa.ConnectionProperties
 Key: OOZIE-3063
 URL: https://issues.apache.org/jira/browse/OOZIE-3063
 Project: Oozie
  Issue Type: Bug
  Components: core
Affects Versions: 4.2.0
Reporter: Denes Bodo


There are values from oozie-site.xml which are not properly checked before they 
are passed into a comma-separated string property in JPAService class. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2296) Add an Oozie diagnostic bundle tool

2017-09-21 Thread Attila Sasvari (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2296:
--
Attachment: OOZIE-2296-003.patch

So far I only added just a couple of unit tests for validating user input.

Unfortunately, in its current form, MiniOozieTestCase cannot be used for 
testing the tool. {{getClient()}} of {{LocalOozie}} can only be used to run 
workflows and get wf statuses. It does not provide full oozie API. Many REST 
API calls (such as listing environment variables, sys props, sharelibs, etc.) 
result in errors / no-op. If TestOozieDiagBundleCollector extended 
MiniOozieTestCase, other tool tests like TestOozieDBCLI would have impact on 
these tests too (resources contain {{fake-oozie-site.xml}} and 
{{hsqldb-oozie-site.xml}}).

> Add an Oozie diagnostic bundle tool
> ---
>
> Key: OOZIE-2296
> URL: https://issues.apache.org/jira/browse/OOZIE-2296
> Project: Oozie
>  Issue Type: New Feature
>  Components: tools
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Attachments: OOZIE-2296-002.patch, OOZIE-2296-003.patch
>
>
> To help with our support cases, I've built a tool that collects a bunch of 
> job and other information from Oozie that throws it all in a tarball.  The 
> idea is that the user can just click a button in Cloudera Manager, and it 
> will run this tool.  
> This tool could be useful for others as an easy way to get information out of 
> Oozie, so I thought I'd contribute it here.  It's built as a "tool" (so it 
> sits next to the sharelib and database tools), and simply uses the Oozie 
> client for getting pretty much everything, so it doesn't require anything 
> special.
> Here's the information that it can get:
> # Sharelib: {{ooze admin -shareliblist}} and {{oozie admin -shareliblist 
> }}
> # Oozie Server's resolved loaded configuration (from admin endpoint)
> # Other admin commands output (queue dump, env vars, etc)
> # Thread dump (HOST:11000/oozie/admin/jvminfo.jsp)
> # Details from last n jobs and/or specific list of jobs
> #- job.properties contents
> #- XML definition
> #- verbose status for each job and each action etc
> #- Oozie logs
> #- Unfortunately, we can't get the launcher jobs' logs from Hadoop
> # Metrics/Instrumentation



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)