Re: Review Request 36512: OOZIE-2306 Oozie health check

2016-08-23 Thread Jaydeep Vishwakarma

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36512/#review146563
---




client/src/main/java/org/apache/oozie/client/OozieClient.java (line 2215)


responce typo



client/src/main/java/org/apache/oozie/client/OozieClient.java (line 2218)


use constant for line.seperator



core/src/main/java/org/apache/oozie/healthcheck/HdfsHealthCheck.java (line 71)


dont close the file system , as you are not opening it.



core/src/main/java/org/apache/oozie/healthcheck/SharelibMetaFileHealthCheck.java
 (line 100)


I am seeing the constant SHARELIB_META_FILE_HEALTH_CHECK have same value. 
Either we should change the constant value or, use the constant from here to 
return the value? I feel in future it can be misused.



core/src/main/java/org/apache/oozie/healthcheck/SharelibMetaFileHealthCheck.java
 (line 262)


final?



core/src/main/java/org/apache/oozie/servlet/V1AdminServlet.java (line 177)


new line



core/src/main/resources/oozie-default.xml (line 2344)


newline


- Jaydeep Vishwakarma


On July 6, 2016, 9:20 p.m., Purshotam Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36512/
> ---
> 
> (Updated July 6, 2016, 9:20 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2306
> https://issues.apache.org/jira/browse/OOZIE-2306
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Oozie can be configured to check the health of its components. 
> http://localhost:11000/oozie/v2/admin/health will report the health of 
> configured components and time taken to compute health check (this can be 
> used to find component slowness).
> Health check component can be configured with below properties.
> 
> 
> oozie.healthCheck.components
> org.apache.oozie.healthcheck.DataBaseHealthCheck
> 
> 
> 
> 
> Support components are DataBaseHealthCheck, HdfsHealthCheck, 
> SharelibHealthCheck, SharelibMetaFileHealthCheck and ZKHealthCheck.
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 65291ff1303cb0e8419e2930521d26c0b3cd7142 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java 
> 67a62c622a9ef86afb82b492735d146faba427de 
>   client/src/main/java/org/apache/oozie/client/rest/JsonTags.java 
> b3b148acb8e0b71b6233937b793453009a4faf2a 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 3c6931915a1afb3ead4b12e1b9ea3c02835c0b5d 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 4fc0c52396165ea9e8add98368e360e3ec989b77 
>   core/src/main/java/org/apache/oozie/healthcheck/DataBaseHealthCheck.java 
> e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
>   core/src/main/java/org/apache/oozie/healthcheck/HdfsHealthCheck.java 
> e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
>   core/src/main/java/org/apache/oozie/healthcheck/HealthCheckExecutor.java 
> e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
>   
> core/src/main/java/org/apache/oozie/healthcheck/HealthCheckExecutorHTML.java 
> e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
>   
> core/src/main/java/org/apache/oozie/healthcheck/HealthCheckExecutorJson.java 
> e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
>   core/src/main/java/org/apache/oozie/healthcheck/HealthCheckStatus.java 
> e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
>   
> core/src/main/java/org/apache/oozie/healthcheck/OozieHealthCheckComponent.java
>  e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
>   core/src/main/java/org/apache/oozie/healthcheck/SharelibHealthCheck.java 
> e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
>   
> core/src/main/java/org/apache/oozie/healthcheck/SharelibMetaFileHealthCheck.java
>  e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
>   core/src/main/java/org/apache/oozie/healthcheck/ZKHealthCheck.java 
> e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
>   core/src/main/java/org/apache/oozie/service/ShareLibService.java 
> 81a9f2d5da781ff79b9bcb1cb5cca02575d57b49 
>   core/src/main/java/org/apache/oozie/servlet/BaseAdminServlet.java 
> 64d3f1f40c5c7b3746ad410cf43f80925424d094 
>   core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 
> 059d3cf6dc251b49940af29d82cbdd817043a176 
>   core/src/main/java/org/apache/oozie/servlet/V0AdminServlet.java 
> fd573d51b77d86c77e944b4d49e1f77f30eb6a26 
>   core/src/main/java/org/apache/oozie/servlet/V1AdminServlet.java 
> 9

[jira] Subscription: Oozie Patch Available

2016-08-23 Thread jira
Issue Subscription
Filter: Oozie Patch Available (90 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2653  Close active connection to hcat server in fs action
https://issues.apache.org/jira/browse/OOZIE-2653
OOZIE-2650  Retry coord start on database exceptions
https://issues.apache.org/jira/browse/OOZIE-2650
OOZIE-2642  Extend TestSubWorkflowActionExecutor to the test if subworkflow 
action's parameter is properly set
https://issues.apache.org/jira/browse/OOZIE-2642
OOZIE-2641  rerunning an oozie coordinator job stucks on waiting
https://issues.apache.org/jira/browse/OOZIE-2641
OOZIE-2635  TimeZone.getTimeZone has performance issue.
https://issues.apache.org/jira/browse/OOZIE-2635
OOZIE-2632  Provide database dump/load functionality to make database migration 
easier
https://issues.apache.org/jira/browse/OOZIE-2632
OOZIE-2630  Oozie Coordinator EL Functions to get first day of the week/month
https://issues.apache.org/jira/browse/OOZIE-2630
OOZIE-2619  Make  Hive action defaults to match hive defaults when running from 
command line
https://issues.apache.org/jira/browse/OOZIE-2619
OOZIE-2618  Include hive-common in hive-beeline as compile component for 
hive-server2 (hive2 action)
https://issues.apache.org/jira/browse/OOZIE-2618
OOZIE-2617  Read properties files in action configurations
https://issues.apache.org/jira/browse/OOZIE-2617
OOZIE-2616  Add Tez profile for Hive and Pig sharelibs
https://issues.apache.org/jira/browse/OOZIE-2616
OOZIE-2613  Upgrade hive version from 0.13.1 to 1.2.0
https://issues.apache.org/jira/browse/OOZIE-2613
OOZIE-2606  Set spark.yarn.jars to fix Spark 2.0 with Oozie
https://issues.apache.org/jira/browse/OOZIE-2606
OOZIE-2588  Support getting credentials for same cluster hcat when credentials 
config is empty
https://issues.apache.org/jira/browse/OOZIE-2588
OOZIE-2585  Disable or remove test case 
TestPartitionDependencyManagerEhcache.testMemoryUsageAndSpeedOverflowToDisk and 
testMemoryUsageAndSpeed
https://issues.apache.org/jira/browse/OOZIE-2585
OOZIE-2584  Eliminate Thread.sleep() calls in TestMemoryLocks
https://issues.apache.org/jira/browse/OOZIE-2584
OOZIE-2583  oozie throws EL Exception when reference variable name containing 
dot
https://issues.apache.org/jira/browse/OOZIE-2583
OOZIE-2582  Populating external child Ids for action failures
https://issues.apache.org/jira/browse/OOZIE-2582
OOZIE-2573  dataset url  which contains spaces can not be handled rightly
https://issues.apache.org/jira/browse/OOZIE-2573
OOZIE-2572  SLA DURATION miss not shown when job is running for longer than 
expected time
https://issues.apache.org/jira/browse/OOZIE-2572
OOZIE-2571  Add scala.binary.version Maven property so that Scala 2.11 can be 
used
https://issues.apache.org/jira/browse/OOZIE-2571
OOZIE-2569  Adding yarn-site, core-site, hdfs-site and mapred-site into spark 
launcher 
https://issues.apache.org/jira/browse/OOZIE-2569
OOZIE-2568  SSH  action pretends to retry automaticly when it failed
https://issues.apache.org/jira/browse/OOZIE-2568
OOZIE-2564  Create new log4j config for unit tests so that logged messages 
contain threads
https://issues.apache.org/jira/browse/OOZIE-2564
OOZIE-2552  Update ActiveMQ version for security and other fixes
https://issues.apache.org/jira/browse/OOZIE-2552
OOZIE-2538  Update HttpClient versions to close security vulnerabilities
https://issues.apache.org/jira/browse/OOZIE-2538
OOZIE-2534  Versioned action libs (similar to how SharedLibs works)
https://issues.apache.org/jira/browse/OOZIE-2534
OOZIE-2530  Remove jline from parent and sharelib pig pom
https://issues.apache.org/jira/browse/OOZIE-2530
OOZIE-2528  Print out environment variables in LauncherMapper
https://issues.apache.org/jira/browse/OOZIE-2528
OOZIE-2525  SchemaChecker fails with NPE
https://issues.apache.org/jira/browse/OOZIE-2525
OOZIE-2518  Oozie is creating many files and directories in /tmp per day
https://issues.apache.org/jira/browse/OOZIE-2518
OOZIE-2513  log.scan.duration should not be used for error and audit logs
https://issues.apache.org/jira/browse/OOZIE-2513
OOZIE-2501  ZK reentrant lock doesn't work for few cases
https://issues.apache.org/jira/browse/OOZIE-2501
OOZIE-2500  -DtestJarSimple option mentioned in minioozie doc does not work
https://issues.apache.org/jira/browse/OOZIE-2500
OOZIE-2499  map-reduce launcher does not need distributed files, archives 
except  jar of input/outputformat class
https://issues.apache.org/jira/browse/OOZIE-2499
OOZIE-2498  Oozie CallerId configuration for downstream components 
https://issues.apache.org/ji

[jira] [Commented] (OOZIE-2517) Add support for startCreatedTime and endCreatedTime filters for coord and bundles

2016-08-23 Thread Abhishek Bafna (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15434162#comment-15434162
 ] 

Abhishek Bafna commented on OOZIE-2517:
---

Thanks [~jaydeepvishwakarma] for review and committing.

> Add support for startCreatedTime and endCreatedTime filters for coord and 
> bundles
> -
>
> Key: OOZIE-2517
> URL: https://issues.apache.org/jira/browse/OOZIE-2517
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
> Fix For: 4.3.0
>
> Attachments: OOZIE-2517-001.patch, OOZIE-2517-003.patch, 
> OOZIE-2517-005.patch, OOZIE-2517-007.patch, OOZIE-2517-008.patch, 
> OOZIE-2517-009.patch, OOZIE-2517-010.patch
>
>
> Support for the 'startCreatedTime' and 'endCreatedTime' is not implemented 
> for the coordinators and bundle job. 
> Curl command with jobtype coordinator and bundle fails where as jobtype=wf 
> executes successfully.
> https://oozie.apache.org/docs/4.2.0/WebServicesAPI.html#Jobs_Information
> {noformat}
> curl 
> "http://host:11000/oozie/v2/jobs?filter=startCreatedTime%3D2016-04-28T00%3A00Z&jobtype=bundle";
> curl 
> "http://host:11000/oozie/v2/jobs?filter=startCreatedTime%3D2016-04-28T00%3A00Z&jobtype=coordinator";
> curl 
> "http://host:11000/oozie/v2/jobs?filter=startCreatedTime%3D2016-04-28T00%3A00Z&jobtype=wf";
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (OOZIE-2654) Zookeeper dependent services should not depend on Connectionstate to be valid before cleaning up

2016-08-23 Thread Venkat Ranganathan (JIRA)
Venkat Ranganathan created OOZIE-2654:
-

 Summary: Zookeeper dependent services should not depend on 
Connectionstate to be valid before cleaning up
 Key: OOZIE-2654
 URL: https://issues.apache.org/jira/browse/OOZIE-2654
 Project: Oozie
  Issue Type: Bug
  Components: HA
Affects Versions: 4.2.0
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan


Currently in ZKUtils, ZKLocks and ZKJobsConcurrency services, we don't properly 
teardown the zookeeper connections when the callback was not received from 
zookeeper to change the connection state.

We can get into this situation if the ZK session for example was closed by ZK 
before any callback was received to update the connection state. This can cause 
the oozie server in a HA mode to not terminate  with one or more sockets in 
close_wait state.

Here is an instance of this issue

>From the network connections, we have one connection still on close_wait with 
>indefinite wait.
{quote} tcp6 143 0 x.x.x.1:46710 x.x.x.2:2181 CLOSE_WAIT 4688/java off 
(0.00/0/0)
{quote}

>From the zookeeper logs,
{quote}
016-08-18 20:45:29,921 - INFO 
NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868 - Client 
attempting to establish new session at /x.x.x.1:46710 2016-08-18 20:45:29,926 - 
INFO CommitProcessor:1:ZooKeeperServer@617 - Established session 
0x1569f576843000e with negotiated timeout 4 for client /x.x.x.1:46710
{quote}
and later
{quote}
2016-08-18 20:46:34,008 - INFO CommitProcessor:1:NIOServerCnxn@1007 - Closed 
socket connection for client /x.x.x.1:46710 which had sessionid 
0x1569f576843000e
{quote}
The fix is to not check for the connectionstate during service destroy and  
teardown the zk connections.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (OOZIE-2653) Close active connection to hcat server in fs action

2016-08-23 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15433939#comment-15433939
 ] 

Hadoop QA commented on OOZIE-2653:
--

Testing JIRA OOZIE-2653

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1798
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3237/

> Close active connection to hcat server in fs action
> ---
>
> Key: OOZIE-2653
> URL: https://issues.apache.org/jira/browse/OOZIE-2653
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2653-1.patch
>
>
> In FsActionExecutor.java, after deleting the uri, Oozie does not close the 
> hcat connection. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Build failed in Jenkins: oozie-trunk-precommit-build #3237

2016-08-23 Thread Apache Jenkins Server
See 

Changes:

[jvishwakarma] OOZIE-2517 Add support for startCreatedTime and endCreatedTime 
filters

--
[...truncated 6233 lines...]
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-tools ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.3.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-mini ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-mini ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-mini ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-mini ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-mini ---
[INFO] Reading assembly descriptor: src/main/assemblies/empty.xml
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Distro 4.3.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-distro 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-distro 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-distro ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ 

[jira] [Updated] (OOZIE-2653) Close active connection to hcat server in fs action

2016-08-23 Thread Satish Subhashrao Saley (JIRA)

 [ 
https://issues.apache.org/jira/browse/OOZIE-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2653:
---
Attachment: OOZIE-2653-1.patch

> Close active connection to hcat server in fs action
> ---
>
> Key: OOZIE-2653
> URL: https://issues.apache.org/jira/browse/OOZIE-2653
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2653-1.patch
>
>
> In FsActionExecutor.java, after deleting the uri, Oozie does not close the 
> hcat connection. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Assigned] (OOZIE-2653) Close active connection to hcat server in fs action

2016-08-23 Thread Satish Subhashrao Saley (JIRA)

 [ 
https://issues.apache.org/jira/browse/OOZIE-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley reassigned OOZIE-2653:
--

Assignee: Satish Subhashrao Saley

> Close active connection to hcat server in fs action
> ---
>
> Key: OOZIE-2653
> URL: https://issues.apache.org/jira/browse/OOZIE-2653
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>
> In FsActionExecutor.java, after deleting the uri, Oozie does not close the 
> hcat connection. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (OOZIE-2653) Close active connection to hcat server in fs action

2016-08-23 Thread Satish Subhashrao Saley (JIRA)
Satish Subhashrao Saley created OOZIE-2653:
--

 Summary: Close active connection to hcat server in fs action
 Key: OOZIE-2653
 URL: https://issues.apache.org/jira/browse/OOZIE-2653
 Project: Oozie
  Issue Type: Bug
Reporter: Satish Subhashrao Saley


In FsActionExecutor.java, after deleting the uri, Oozie does not close the hcat 
connection. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Jenkins build is back to normal : oozie-trunk-find-patches-available #130787

2016-08-23 Thread Apache Jenkins Server
See 



Build failed in Jenkins: oozie-trunk-find-patches-available #130786

2016-08-23 Thread Apache Jenkins Server
See 

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H1 (yahoo-not-h2 Hadoop Mapreduce Hdfs Pig) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/oozie.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/oozie.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/oozie.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 0754451daf5a7c3bdbc401bb245611a1da13731e 
(refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0754451daf5a7c3bdbc401bb245611a1da13731e
 > git rev-list 0754451daf5a7c3bdbc401bb245611a1da13731e # timeout=10
[oozie-trunk-find-patches-available] $ /bin/bash 
/tmp/hudson5205401109739720360.sh
mkdir: cannot create directory 
':
 File exists
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed

  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
100  184k0  184k0 0   193k  0 --:--:-- --:--:-- --:--:--  193k
100  262k0  262k0 0   123k  0 --:--:--  0:00:02 --:--:--  123k
100  448k0  448k0 0   150k  0 --:--:--  0:00:02 --:--:--  150k
100  736k0  736k0 0   185k  0 --:--:--  0:00:03 --:--:--  185k
100  789k0  789k0 0   158k  0 --:--:--  0:00:04 --:--:--  158k
100 1009k0 1009k0 0   152k  0 --:--:--  0:00:06 --:--:--  145k
100 1009k0 1009k0 0   140k  0 --:--:--  0:00:07 --:--:--  147k
100 1061k0 1061k0 0   128k  0 --:--:--  0:00:08 --:--:--  115k
100 1123k0 1123k0 0   125k  0 --:--:--  0:00:08 --:--:-- 80035
100 1474k0 1474k0 0   148k  0 --:--:--  0:00:09 --:--:--  138k
100 1509k0 1509k0 0   148k  0 --:--:--  0:00:10 --:--:--  141k
curl: (18) transfer closed with outstanding read data remaining
Could not retrieve available patches from JIRA
Build step 'Execute shell' marked build as failure


[jira] [Commented] (OOZIE-2650) Retry coord start on database exceptions

2016-08-23 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15433488#comment-15433488
 ] 

Hadoop QA commented on OOZIE-2650:
--

Testing JIRA OOZIE-2650

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1794
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3236/

> Retry coord start on database exceptions
> 
>
> Key: OOZIE-2650
> URL: https://issues.apache.org/jira/browse/OOZIE-2650
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2650-1.patch
>
>
> {code:title=CoordActionStartXCommand.java}
> updateList.add(new UpdateEntry(
> WorkflowJobQuery.UPDATE_WORKFLOW_PARENT_MODIFIED, 
> wfJob));
> updateList.add(new UpdateEntry(
> CoordActionQuery.UPDATE_COORD_ACTION_FOR_START, 
> coordAction));
> try {
> executor.executeBatchInsertUpdateDelete(insertList, 
> updateList, null);
> queue(new 
> CoordActionNotificationXCommand(coordAction), 100);
> if (EventHandlerService.isEnabled()) {
> generateEvent(coordAction, user, appName, 
> wfJob.getStartTime());
> }
> }
> catch (JPAExecutorException je) {
> throw new CommandException(je);
> }
> .
> ...
> 
>  finally {
> if (makeFail == true) { // No DB exception occurs
> 
> 
> 
> queue(new 
> CoordActionReadyXCommand(coordAction.getJobId()));
> }
> }
> {code}
> If there is any Database issue while starting coord action, we fail the coord 
> action. We should retry. 
> CoordActionStartXCommand submits workflow. Workflow gets linked to the coord 
> action if workflow submission succeeds. But if coord action update fails due 
> to database exception, recovery service should be able to recover it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Jenkins build is back to normal : oozie-trunk-precommit-build #3236

2016-08-23 Thread Apache Jenkins Server
See 



[jira] [Updated] (OOZIE-2651) Set javax.xml.parsers.DocumentBuilderFactory sys prop to make XML handling faster

2016-08-23 Thread Robert Kanter (JIRA)

 [ 
https://issues.apache.org/jira/browse/OOZIE-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-2651:
-
Attachment: OOZIE-2651.001.patch

The 001 patch adds a new config property, 
{{oozie.javax.xml.parsers.DocumentBuilderFactory}}, that lets you specify the 
Factory to use (default {{org.apache.xerces.jaxp.DocumentBuilderFactoryImpl}}). 
 A blank/whitespace value tells Oozie not to set this.

Unfortunately, I'll need to look into this some more because it breaks the 
Oozie Web UI (Tomcat can't find 
org.apache.xerces.jaxp.DocumentBuilderFactoryImpl).  I probably need to add the 
Xerces jar to Tomcat somewhere.

> Set javax.xml.parsers.DocumentBuilderFactory sys prop to make XML handling 
> faster
> -
>
> Key: OOZIE-2651
> URL: https://issues.apache.org/jira/browse/OOZIE-2651
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 4.3.0
>
> Attachments: OOZIE-2651.001.patch
>
>
> We noticed from some jstacks that Oozie was spending a lot of time looking 
> for XML related jars (we do a lot of XML parsing in Oozie and Hadoop code).  
> For instance, we'd see hundreds of threads blocked here
> {noformat}
> "pool-5-thread-57" prio=10 tid=0x7f9579923000 nid=0x6b7f waiting for 
> monitor entry [0x7f94225e3000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at 
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
> - waiting to lock <0x000500026148> (a 
> org.apache.catalina.loader.WebappClassLoader)
> at 
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
> at 
> javax.xml.parsers.FactoryFinder.getProviderClass(FactoryFinder.java:117)
> at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:178)
> at 
> javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:333)
> at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255)
> at 
> javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
> at 
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2526)
> at 
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
> at 
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
> - locked <0x0007eac5ea28> (a org.apache.hadoop.conf.Configuration)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1144)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1116)
> at org.apache.oozie.util.XConfiguration.copy(XConfiguration.java:206)
> ...
> {noformat}
> There's nothing we can do about the lock at the {{WebappClassLoader}}, but by 
> setting the {{javax.xml.parsers.DocumentBuilderFactory}} system property to 
> {{org.apache.xerces.jaxp.DocumentBuilderFactoryImpl}}, we can speed this up a 
> lot by getting rid of all of the searching that it's doing to find the 
> implementation.  
> See 
> https://docs.oracle.com/javase/7/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#newInstance()
>  
> and 
> https://www.databasesandlife.com/java-always-explicitly-specify-which-xml-parser-to-use/
> In our testing, we had a metric around creating a new {{JobClient}}, which 
> also calls {{DocumentBuilderFactory.newInstance}}, and it dropped from ~3min 
> max to ~2sec max (not an exact test of the savings here).  And we also didn't 
> see so much blocking on this in the jstacks anymore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (OOZIE-2517) Add support for startCreatedTime and endCreatedTime filters for coord and bundles

2016-08-23 Thread Jaydeep Vishwakarma (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15433327#comment-15433327
 ] 

Jaydeep Vishwakarma commented on OOZIE-2517:


+1

> Add support for startCreatedTime and endCreatedTime filters for coord and 
> bundles
> -
>
> Key: OOZIE-2517
> URL: https://issues.apache.org/jira/browse/OOZIE-2517
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
> Fix For: 4.3.0
>
> Attachments: OOZIE-2517-001.patch, OOZIE-2517-003.patch, 
> OOZIE-2517-005.patch, OOZIE-2517-007.patch, OOZIE-2517-008.patch, 
> OOZIE-2517-009.patch, OOZIE-2517-010.patch
>
>
> Support for the 'startCreatedTime' and 'endCreatedTime' is not implemented 
> for the coordinators and bundle job. 
> Curl command with jobtype coordinator and bundle fails where as jobtype=wf 
> executes successfully.
> https://oozie.apache.org/docs/4.2.0/WebServicesAPI.html#Jobs_Information
> {noformat}
> curl 
> "http://host:11000/oozie/v2/jobs?filter=startCreatedTime%3D2016-04-28T00%3A00Z&jobtype=bundle";
> curl 
> "http://host:11000/oozie/v2/jobs?filter=startCreatedTime%3D2016-04-28T00%3A00Z&jobtype=coordinator";
> curl 
> "http://host:11000/oozie/v2/jobs?filter=startCreatedTime%3D2016-04-28T00%3A00Z&jobtype=wf";
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: Review Request 48469: OOZIE-2517 Add support for startCreatedTime and endCreatedTime filters for coord and bundles

2016-08-23 Thread Jaydeep Vishwakarma

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48469/#review146553
---


Ship it!




Ship It!

- Jaydeep Vishwakarma


On July 27, 2016, 4:38 a.m., Abhishek Bafna wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48469/
> ---
> 
> (Updated July 27, 2016, 4:38 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2517
> https://issues.apache.org/jira/browse/OOZIE-2517
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Add support for startCreatedTime and endCreatedTime filters for coord and 
> bundles
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/CoordinatorEngine.java 2301908 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/BundleJobInfoGetJPAExecutor.java
>  dace4e8 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/CoordJobInfoGetJPAExecutor.java
>  8fcb1e6 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/WorkflowsJobGetJPAExecutor.java
>  5cde60e 
>   core/src/main/java/org/apache/oozie/store/StoreStatusFilter.java ebdbaa4 
>   core/src/main/java/org/apache/oozie/util/JobsFilterUtils.java 90c772f 
>   core/src/test/java/org/apache/oozie/client/TestOozieCLI.java 6529bfa 
>   
> core/src/test/java/org/apache/oozie/executor/jpa/TestBundleJobInfoGetJPAExecutor.java
>  14b 
>   
> core/src/test/java/org/apache/oozie/executor/jpa/TestCoordJobInfoGetJPAExecutor.java
>  e98b146 
>   docs/src/site/twiki/WebServicesAPI.twiki 5dadafe 
> 
> Diff: https://reviews.apache.org/r/48469/diff/
> 
> 
> Testing
> ---
> 
> unit tests done. Manual testing done.
> 
> 
> Thanks,
> 
> Abhishek Bafna
> 
>



[jira] [Updated] (OOZIE-2651) Set javax.xml.parsers.DocumentBuilderFactory sys prop to make XML handling faster

2016-08-23 Thread Robert Kanter (JIRA)

 [ 
https://issues.apache.org/jira/browse/OOZIE-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-2651:
-
Description: 
We noticed from some jstacks that Oozie was spending a lot of time looking for 
XML related jars (we do a lot of XML parsing in Oozie and Hadoop code).  For 
instance, we'd see hundreds of threads blocked here
{noformat}
"pool-5-thread-57" prio=10 tid=0x7f9579923000 nid=0x6b7f waiting for 
monitor entry [0x7f94225e3000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at 
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
- waiting to lock <0x000500026148> (a 
org.apache.catalina.loader.WebappClassLoader)
at 
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
at 
javax.xml.parsers.FactoryFinder.getProviderClass(FactoryFinder.java:117)
at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:178)
at 
javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:333)
at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255)
at 
javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
at 
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2526)
at 
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
at 
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
- locked <0x0007eac5ea28> (a org.apache.hadoop.conf.Configuration)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1144)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1116)
at org.apache.oozie.util.XConfiguration.copy(XConfiguration.java:206)
...
{noformat}
There's nothing we can do about the lock at the {{WebappClassLoader}}, but by 
setting the {{javax.xml.parsers.DocumentBuilderFactory}} system property to 
{{org.apache.xerces.jaxp.DocumentBuilderFactoryImpl}}, we can speed this up a 
lot by getting rid of all of the searching that it's doing to find the 
implementation.  
See 
https://docs.oracle.com/javase/7/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#newInstance()
 
and 
https://www.databasesandlife.com/java-always-explicitly-specify-which-xml-parser-to-use/

In our testing, we had a metric around creating a new {{JobClient}}, which also 
calls {{DocumentBuilderFactory.newInstance}}, and it dropped from ~3min max to 
~2sec max (not an exact test of the savings here).  And we also didn't see so 
much blocking on this in the jstacks anymore.

  was:
We noticed from some jstacks that Oozie was spending a lot of time looking for 
XML related jars (we do a lot of XML parsing in Oozie and Hadoop code).  For 
instance, we'd see hundreds of threads blocked here
{noformat}
"pool-5-thread-57" prio=10 tid=0x7f9579923000 nid=0x6b7f waiting for 
monitor entry [0x7f94225e3000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at 
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
- waiting to lock <0x000500026148> (a 
org.apache.catalina.loader.WebappClassLoader)
at 
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
at 
javax.xml.parsers.FactoryFinder.getProviderClass(FactoryFinder.java:117)
at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:178)
at 
javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:333)
at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255)
at 
javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
at 
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2526)
at 
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
at 
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
- locked <0x0007eac5ea28> (a org.apache.hadoop.conf.Configuration)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1144)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1116)
at org.apache.oozie.util.XConfiguration.copy(XConfiguration.java:206)
...
{noformat}
There's nothing we can do about the lock at the {{WebappClassLoader}}, but by 
setting the {{javax.xml.parsers.DocumentBuilderFactory}} system property to 
{{org.apache.xerces.jaxp.DocumentBuilderFactoryImpl}}, we can speed this up a 
lot by getting rid of all of the searching that it's doing to find the 
implementation.  
See 
https://docs.oracle.com/javase/7/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#newInstance()
 
and 
https://www.databasesandlife.com/java-always-explicitly-specify-which-xml-parser-to-use/

In our testing, we had a metric around creating a n

[jira] [Created] (OOZIE-2652) Skip trash while deleting hive table partition

2016-08-23 Thread Satish Subhashrao Saley (JIRA)
Satish Subhashrao Saley created OOZIE-2652:
--

 Summary: Skip trash while deleting hive table partition
 Key: OOZIE-2652
 URL: https://issues.apache.org/jira/browse/OOZIE-2652
 Project: Oozie
  Issue Type: Bug
Reporter: Satish Subhashrao Saley
Assignee: Satish Subhashrao Saley


Currently, when we delete a partition in the work flow, the data persists in 
trash and does not get purged. This takes up namespaces.

{code}
 
${jobTracker}
${oozieNameNode}



{code}

Solution: User will pass on "skip-trash" option. Accordingly, Oozie will call 
appropriate hcat api. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (OOZIE-2651) Set javax.xml.parsers.DocumentBuilderFactory sys prop to make XML handling faster

2016-08-23 Thread Robert Kanter (JIRA)
Robert Kanter created OOZIE-2651:


 Summary: Set javax.xml.parsers.DocumentBuilderFactory sys prop to 
make XML handling faster
 Key: OOZIE-2651
 URL: https://issues.apache.org/jira/browse/OOZIE-2651
 Project: Oozie
  Issue Type: Improvement
  Components: core
Affects Versions: trunk
Reporter: Robert Kanter
Assignee: Robert Kanter
 Fix For: 4.3.0


We noticed from some jstacks that Oozie was spending a lot of time looking for 
XML related jars (we do a lot of XML parsing in Oozie and Hadoop code).  For 
instance, we'd see hundreds of threads blocked here
{noformat}
"pool-5-thread-57" prio=10 tid=0x7f9579923000 nid=0x6b7f waiting for 
monitor entry [0x7f94225e3000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at 
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
- waiting to lock <0x000500026148> (a 
org.apache.catalina.loader.WebappClassLoader)
at 
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
at 
javax.xml.parsers.FactoryFinder.getProviderClass(FactoryFinder.java:117)
at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:178)
at 
javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:333)
at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255)
at 
javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
at 
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2526)
at 
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
at 
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
- locked <0x0007eac5ea28> (a org.apache.hadoop.conf.Configuration)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1144)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1116)
at org.apache.oozie.util.XConfiguration.copy(XConfiguration.java:206)
...
{noformat}
There's nothing we can do about the lock at the {{WebappClassLoader}}, but by 
setting the {{javax.xml.parsers.DocumentBuilderFactory}} system property to 
{{org.apache.xerces.jaxp.DocumentBuilderFactoryImpl}}, we can speed this up a 
lot by getting rid of all of the searching that it's doing to find the 
implementation.  
See 
https://docs.oracle.com/javase/7/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#newInstance()
 
and 
https://www.databasesandlife.com/java-always-explicitly-specify-which-xml-parser-to-use/

In our testing, we had a metric around creating a new {{JobClient}}, which 
eventually calls this code, and it dropped from ~3min max to ~2sec max.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Assigned] (OOZIE-1355) Support all streaming options

2016-08-23 Thread Satish Subhashrao Saley (JIRA)

 [ 
https://issues.apache.org/jira/browse/OOZIE-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley reassigned OOZIE-1355:
--

Assignee: Satish Subhashrao Saley

> Support all streaming options
> -
>
> Key: OOZIE-1355
> URL: https://issues.apache.org/jira/browse/OOZIE-1355
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 3.3.2
>Reporter: Rohini Palaniswamy
>Assignee: Satish Subhashrao Saley
>
> Oozie only currently supports
> mapper>[MAPPER-PROCESS]
> [REDUCER-PROCESS]
> [RECORD-READER-CLASS]
> [NAME=VALUE]
> [NAME=VALUE]
>  We should add native support in streaming action for input, output, 
> combiner, inputformat, outputformat, partitioner, lazyOutput, numReduceTasks, 
> mapdebugand reducedebug
> http://hadoop.apache.org/docs/r1.1.2/streaming.html#Streaming+Command+Options



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (OOZIE-2616) Add Tez profile for Hive and Pig sharelibs

2016-08-23 Thread Thomas Poepping (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-2616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15433222#comment-15433222
 ] 

Thomas Poepping commented on OOZIE-2616:


The settings that work for me are -Dhadoop.version=2.7.2 -Dpig.version=0.16.0 
-Dhive.version=2.1.0 -Ptez -Phadoop-2

Let me know what you find.

> Add Tez profile for Hive and Pig sharelibs
> --
>
> Key: OOZIE-2616
> URL: https://issues.apache.org/jira/browse/OOZIE-2616
> Project: Oozie
>  Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Thomas Poepping
> Attachments: OOZIE-2616-1.patch
>
>
> More recent versions of Hive and Pig depend on Tez as their default execution 
> engine. As the sharelib for those applications stands today, those 
> applications are unable to run Tez jobs. This patch adds an optional profile 
> to the root pom that will include the proper Tez dependencies into those 
> sharelibs, while excluding problematic jars to keep the sharelibs from having 
> classpath duplication issues. *Note*: this patch will only be required for 
> those clients who specify their own, newer versions of Hive or Pig. This 
> change is not required for the current supported versions of those 
> applications (hence why the profile is off by default).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (OOZIE-2650) Retry coord start on database exceptions

2016-08-23 Thread Satish Subhashrao Saley (JIRA)

 [ 
https://issues.apache.org/jira/browse/OOZIE-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2650:
---
Attachment: OOZIE-2650-1.patch

> Retry coord start on database exceptions
> 
>
> Key: OOZIE-2650
> URL: https://issues.apache.org/jira/browse/OOZIE-2650
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2650-1.patch
>
>
> {code:title=CoordActionStartXCommand.java}
> updateList.add(new UpdateEntry(
> WorkflowJobQuery.UPDATE_WORKFLOW_PARENT_MODIFIED, 
> wfJob));
> updateList.add(new UpdateEntry(
> CoordActionQuery.UPDATE_COORD_ACTION_FOR_START, 
> coordAction));
> try {
> executor.executeBatchInsertUpdateDelete(insertList, 
> updateList, null);
> queue(new 
> CoordActionNotificationXCommand(coordAction), 100);
> if (EventHandlerService.isEnabled()) {
> generateEvent(coordAction, user, appName, 
> wfJob.getStartTime());
> }
> }
> catch (JPAExecutorException je) {
> throw new CommandException(je);
> }
> .
> ...
> 
>  finally {
> if (makeFail == true) { // No DB exception occurs
> 
> 
> 
> queue(new 
> CoordActionReadyXCommand(coordAction.getJobId()));
> }
> }
> {code}
> If there is any Database issue while starting coord action, we fail the coord 
> action. We should retry. 
> CoordActionStartXCommand submits workflow. Workflow gets linked to the coord 
> action if workflow submission succeeds. But if coord action update fails due 
> to database exception, recovery service should be able to recover it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (OOZIE-2650) Retry coord start on database exceptions

2016-08-23 Thread Satish Subhashrao Saley (JIRA)

 [ 
https://issues.apache.org/jira/browse/OOZIE-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2650:
---
Attachment: (was: OOZIE-2650-1.patch)

> Retry coord start on database exceptions
> 
>
> Key: OOZIE-2650
> URL: https://issues.apache.org/jira/browse/OOZIE-2650
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>
> {code:title=CoordActionStartXCommand.java}
> updateList.add(new UpdateEntry(
> WorkflowJobQuery.UPDATE_WORKFLOW_PARENT_MODIFIED, 
> wfJob));
> updateList.add(new UpdateEntry(
> CoordActionQuery.UPDATE_COORD_ACTION_FOR_START, 
> coordAction));
> try {
> executor.executeBatchInsertUpdateDelete(insertList, 
> updateList, null);
> queue(new 
> CoordActionNotificationXCommand(coordAction), 100);
> if (EventHandlerService.isEnabled()) {
> generateEvent(coordAction, user, appName, 
> wfJob.getStartTime());
> }
> }
> catch (JPAExecutorException je) {
> throw new CommandException(je);
> }
> .
> ...
> 
>  finally {
> if (makeFail == true) { // No DB exception occurs
> 
> 
> 
> queue(new 
> CoordActionReadyXCommand(coordAction.getJobId()));
> }
> }
> {code}
> If there is any Database issue while starting coord action, we fail the coord 
> action. We should retry. 
> CoordActionStartXCommand submits workflow. Workflow gets linked to the coord 
> action if workflow submission succeeds. But if coord action update fails due 
> to database exception, recovery service should be able to recover it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (OOZIE-2617) Read properties files in action configurations

2016-08-23 Thread Taklon Stephen Wu (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-2617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15433118#comment-15433118
 ] 

Taklon Stephen Wu commented on OOZIE-2617:
--

ping?

> Read properties files in action configurations
> --
>
> Key: OOZIE-2617
> URL: https://issues.apache.org/jira/browse/OOZIE-2617
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Affects Versions: trunk
>Reporter: Taklon Stephen Wu
>  Labels: patch
> Fix For: 4.3.0
>
> Attachments: OOZIE-2617-1.patch, OOZIE-2617-2.patch, 
> OOZIE-2617-3.patch
>
>
> Current logic of acton configuration only READ xml files within Oozie action 
> directory, but for pig and other applications, properties files are supported 
> and may be default configuration file format. To simplify the logic of 
> reusing these properties files when using Oozie with them, we add properties 
> file reading in action conf.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (OOZIE-2616) Add Tez profile for Hive and Pig sharelibs

2016-08-23 Thread Peter Cseh (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-2616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15433107#comment-15433107
 ] 

Peter Cseh commented on OOZIE-2616:
---

I'm sorry if I wasn't clear. (also for the late response)
I just wanted to try out that when the profile is active, things are working as 
they should.
I would like to ask for a command line like this:
{{mvn clean install -Dhadoop.version=2.2.0 -Dpig.version=1.2.3 
-Dhive.version=4.5.6 -Ptez}}

Also, I think it would be nice to update the DG_QuickStart.twiki page with the 
new profile information.

> Add Tez profile for Hive and Pig sharelibs
> --
>
> Key: OOZIE-2616
> URL: https://issues.apache.org/jira/browse/OOZIE-2616
> Project: Oozie
>  Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Thomas Poepping
> Attachments: OOZIE-2616-1.patch
>
>
> More recent versions of Hive and Pig depend on Tez as their default execution 
> engine. As the sharelib for those applications stands today, those 
> applications are unable to run Tez jobs. This patch adds an optional profile 
> to the root pom that will include the proper Tez dependencies into those 
> sharelibs, while excluding problematic jars to keep the sharelibs from having 
> classpath duplication issues. *Note*: this patch will only be required for 
> those clients who specify their own, newer versions of Hive or Pig. This 
> change is not required for the current supported versions of those 
> applications (hence why the profile is off by default).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (OOZIE-2616) Add Tez profile for Hive and Pig sharelibs

2016-08-23 Thread Thomas Poepping (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-2616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15433092#comment-15433092
 ] 

Thomas Poepping commented on OOZIE-2616:


Hi [~gezapeti], why wouldn't we still want the hive and pig versions to be up 
to the user? There could be multiple versions of those applications for which 
bringing in Tez is a valid use case.

> Add Tez profile for Hive and Pig sharelibs
> --
>
> Key: OOZIE-2616
> URL: https://issues.apache.org/jira/browse/OOZIE-2616
> Project: Oozie
>  Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Thomas Poepping
> Attachments: OOZIE-2616-1.patch
>
>
> More recent versions of Hive and Pig depend on Tez as their default execution 
> engine. As the sharelib for those applications stands today, those 
> applications are unable to run Tez jobs. This patch adds an optional profile 
> to the root pom that will include the proper Tez dependencies into those 
> sharelibs, while excluding problematic jars to keep the sharelibs from having 
> classpath duplication issues. *Note*: this patch will only be required for 
> those clients who specify their own, newer versions of Hive or Pig. This 
> change is not required for the current supported versions of those 
> applications (hence why the profile is off by default).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (OOZIE-2517) Add support for startCreatedTime and endCreatedTime filters for coord and bundles

2016-08-23 Thread Abhishek Bafna (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15432689#comment-15432689
 ] 

Abhishek Bafna commented on OOZIE-2517:
---

[~jaydeepvishwakarma] Ping.

> Add support for startCreatedTime and endCreatedTime filters for coord and 
> bundles
> -
>
> Key: OOZIE-2517
> URL: https://issues.apache.org/jira/browse/OOZIE-2517
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
> Fix For: 4.3.0
>
> Attachments: OOZIE-2517-001.patch, OOZIE-2517-003.patch, 
> OOZIE-2517-005.patch, OOZIE-2517-007.patch, OOZIE-2517-008.patch, 
> OOZIE-2517-009.patch, OOZIE-2517-010.patch
>
>
> Support for the 'startCreatedTime' and 'endCreatedTime' is not implemented 
> for the coordinators and bundle job. 
> Curl command with jobtype coordinator and bundle fails where as jobtype=wf 
> executes successfully.
> https://oozie.apache.org/docs/4.2.0/WebServicesAPI.html#Jobs_Information
> {noformat}
> curl 
> "http://host:11000/oozie/v2/jobs?filter=startCreatedTime%3D2016-04-28T00%3A00Z&jobtype=bundle";
> curl 
> "http://host:11000/oozie/v2/jobs?filter=startCreatedTime%3D2016-04-28T00%3A00Z&jobtype=coordinator";
> curl 
> "http://host:11000/oozie/v2/jobs?filter=startCreatedTime%3D2016-04-28T00%3A00Z&jobtype=wf";
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (OOZIE-2225) Add wild card filter for gathering jobs

2016-08-23 Thread Jaydeep Vishwakarma (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15432659#comment-15432659
 ] 

Jaydeep Vishwakarma commented on OOZIE-2225:


Can you please upload it to review board.

> Add wild card filter for gathering jobs
> ---
>
> Key: OOZIE-2225
> URL: https://issues.apache.org/jira/browse/OOZIE-2225
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 4.1.0
>Reporter: Sai Chirravuri
>Assignee: Sai Chirravuri
> Attachments: OOZIE-2225-001.patch, OOZIE-2225-002.patch, 
> OOZIE-2225-003.patch, OOZIE-2225-004.patch
>
>
> Oozie currently supports name, user, group, status, frequency and unit as 
> filter params.
> Add wild card param that matches partially with name/user or complete match 
> with ID



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (OOZIE-2203) Fix the login example

2016-08-23 Thread Jaydeep Vishwakarma (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15432656#comment-15432656
 ] 

Jaydeep Vishwakarma commented on OOZIE-2203:


+1

> Fix the login example
> -
>
> Key: OOZIE-2203
> URL: https://issues.apache.org/jira/browse/OOZIE-2203
> Project: Oozie
>  Issue Type: Bug
>  Components: security
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-2203.patch
>
>
> The login example currently doesn't compile:
> {noformat}
> # mvn clean package assembly:single -DskipTests -PloginServerExample
> Picked up _JAVA_OPTIONS: -Djava.awt.headless=true
> [INFO] Scanning for projects...
> [ERROR] The build could not read 1 project -> [Help 1]
> [ERROR]   
> [ERROR]   The project org.apache.oozie:oozie-login:4.2.0-SNAPSHOT 
> (/Users/rkanter/dev/oozie-trunk2/login/pom.xml) has 2 errors
> [ERROR] 'dependencies.dependency.version' for 
> org.apache.oozie:oozie-hadoop:jar is missing. @ line 71, column 21
> [ERROR] 'dependencies.dependency.version' for 
> org.apache.oozie:oozie-hadoop-test:jar is missing. @ line 83, column 21
> {noformat}
> We should also be able to remove our copy of 
> {{AltKerberosAuthenticationHandler}} and reference it from Hadoop.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (OOZIE-2646) Spark Action Improvement

2016-08-23 Thread Peter Cseh (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15432429#comment-15432429
 ] 

Peter Cseh commented on OOZIE-2646:
---

Isn't this the same as OOZIE-2391? 
It will be fixed in 4.3.0, coming soon. 

> Spark Action Improvement
> 
>
> Key: OOZIE-2646
> URL: https://issues.apache.org/jira/browse/OOZIE-2646
> Project: Oozie
>  Issue Type: Improvement
>  Components: action
>Affects Versions: 4.2.0
>Reporter: Rui Xue
>  Labels: easyfix
> Fix For: 4.3.0, 4.4.0
>
>
> In the current mapr release (4.2.0), Oozie's Spark action still cannot accept 
> the value of the "--conf" option with double quotation. For instance, in 
> Spark-submit, the value of --conf "xxx" is allowed, whereas in Oozie it is 
> not allowed, and this is required sometimes anyway in practice



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Jenkins build is back to normal : oozie-trunk-find-patches-available #130727

2016-08-23 Thread Apache Jenkins Server
See 



Build failed in Jenkins: oozie-trunk-find-patches-available #130726

2016-08-23 Thread Apache Jenkins Server
See 

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H1 (yahoo-not-h2 Hadoop Mapreduce Hdfs Pig) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/oozie.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/oozie.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/oozie.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 1b46fd92f10c561d14839c5a53da12d3afb46bb8 
(refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 1b46fd92f10c561d14839c5a53da12d3afb46bb8
 > git rev-list 1b46fd92f10c561d14839c5a53da12d3afb46bb8 # timeout=10
[oozie-trunk-find-patches-available] $ /bin/bash 
/tmp/hudson5983406223159387987.sh
mkdir: cannot create directory 
':
 File exists
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed

  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
100  131k0  131k0 0  83600  0 --:--:--  0:00:01 --:--:-- 83566
100  227k0  227k0 0   108k  0 --:--:--  0:00:02 --:--:--  108k
100  596k0  596k0 0   192k  0 --:--:--  0:00:03 --:--:--  192k
100  868k0  868k0 0   213k  0 --:--:--  0:00:04 --:--:--  213k
100  868k0  868k0 0   213k  0 --:--:--  0:00:04 --:--:--  216k
curl: (18) transfer closed with outstanding read data remaining
Could not retrieve available patches from JIRA
Build step 'Execute shell' marked build as failure


Jenkins build is back to normal : oozie-trunk-find-patches-available #130725

2016-08-23 Thread Apache Jenkins Server
See 



Build failed in Jenkins: oozie-trunk-find-patches-available #130724

2016-08-23 Thread Apache Jenkins Server
See 

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H1 (yahoo-not-h2 Hadoop Mapreduce Hdfs Pig) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/oozie.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/oozie.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/oozie.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 1b46fd92f10c561d14839c5a53da12d3afb46bb8 
(refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 1b46fd92f10c561d14839c5a53da12d3afb46bb8
 > git rev-list 1b46fd92f10c561d14839c5a53da12d3afb46bb8 # timeout=10
[oozie-trunk-find-patches-available] $ /bin/bash /tmp/hudson33633829648451592.sh
mkdir: cannot create directory 
':
 File exists
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed

  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
100  184k0  184k0 0   127k  0 --:--:--  0:00:01 --:--:--  127k
100  482k0  482k0 0   198k  0 --:--:--  0:00:02 --:--:--  198k
100  868k0  868k0 0   257k  0 --:--:--  0:00:03 --:--:--  257k
100 1044k0 1044k0 0   261k  0 --:--:--  0:00:03 --:--:--  261k
curl: (18) transfer closed with outstanding read data remaining
Could not retrieve available patches from JIRA
Build step 'Execute shell' marked build as failure