[jira] Subscription: Oozie Patch Available

2017-07-12 Thread jira 
Issue Subscription
Filter: Oozie Patch Available (111 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2987  Coord action missing dependencies should show URI template with 
unresolved dependencies
https://issues.apache.org/jira/browse/OOZIE-2987
OOZIE-2980  JAXB Proof of Concept
https://issues.apache.org/jira/browse/OOZIE-2980
OOZIE-2978  Remove code that handles Pig versions before 0.8 
https://issues.apache.org/jira/browse/OOZIE-2978
OOZIE-2977  error: self-closing element not allowed
https://issues.apache.org/jira/browse/OOZIE-2977
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues.apache.org/jira/browse/OOZIE-2975
OOZIE-2974  Change TestLiteWorkflowAppParser so that it works with Hadoop 3
https://issues.apache.org/jira/browse/OOZIE-2974
OOZIE-2969  Drop support for Java 1.7
https://issues.apache.org/jira/browse/OOZIE-2969
OOZIE-2964  getting at 
org.apache.maven.plugin.javadoc.AbstractJavadocMojo.executeJavadocCommandLine(AbstractJavadocMojo.java:5163)
https://issues.apache.org/jira/browse/OOZIE-2964
OOZIE-2963  getting error in build ArtifactNotFoundException: Could not find 
artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde
https://issues.apache.org/jira/browse/OOZIE-2963
OOZIE-2962  bump maven-javadoc-plugin to 2.10.4
https://issues.apache.org/jira/browse/OOZIE-2962
OOZIE-2961  build contains multiple warnings for is not a parameter name
https://issues.apache.org/jira/browse/OOZIE-2961
OOZIE-2960  many warnings for @return tag has no arguments
https://issues.apache.org/jira/browse/OOZIE-2960
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues.apache.org/jira/browse/OOZIE-2934
OOZIE-2931  multiple warnings during build for "no @param for"
https://issues.apache.org/jira/browse/OOZIE-2931
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2916  Set a job name for the MR Action's child job
https://issues.apache.org/jira/browse/OOZIE-2916
OOZIE-2914  Consolidate Trim 
https://issues.apache.org/jira/browse/OOZIE-2914
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues.apache.org/jira/browse/OOZIE-2867
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues.apache.org/jira/browse/OOZIE-2829
OOZIE-2826  Falcon feed fails to aws s3; Oozie joda time version does not meet 
required jar version 2.2 or later
https://issues.apache.org/jira/browse/OOZIE-2826
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2795  Create lib directory or symlink for Oozie CLI during packaging
https://issues.apache.org/jira/browse/OOZIE-2795
OOZIE-2791  ShareLib installation may fa

[jira] [Commented] (OOZIE-2987) Coord action missing dependencies should show URI template with unresolved dependencies

2017-07-12 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085101#comment-16085101
 ] 

Hadoop QA commented on OOZIE-2987:
--

Testing JIRA OOZIE-2987

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 1 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1100
.Tests rerun: 22
.Tests failed at first run: org.apache.oozie.action.hadoop.TestLauncherAM,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/3971/

> Coord action missing dependencies should show URI template with unresolved 
> dependencies
> ---
>
> Key: OOZIE-2987
> URL: https://issues.apache.org/jira/browse/OOZIE-2987
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-2987-V1.patch
>
>
> Patch also fixes and an issue where if there are unresolved dependencies and 
> hcat dependencies,  {{blocked on}} show unresolved dependency.
>  Missing dependency for unresolved will show as 
> {code}${coord:latestRange(-5,0)} -> 
> hcat://localhost:30083/db_c/table1/dt=${YEAR}${DAY};country=usa{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (OOZIE-2987) Coord action missing dependencies should show URI template with unresolved dependencies

2017-07-12 Thread Purshotam Shah (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah reassigned OOZIE-2987:
-

Assignee: Purshotam Shah

> Coord action missing dependencies should show URI template with unresolved 
> dependencies
> ---
>
> Key: OOZIE-2987
> URL: https://issues.apache.org/jira/browse/OOZIE-2987
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-2987-V1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2767) Oozie 4.3.0 with spark 2.0 run spark R job on yarn cluster mode failed

2017-07-12 Thread Vivek Kapoor (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085044#comment-16085044
 ] 

Vivek Kapoor commented on OOZIE-2767:
-

I'm following the same steps, but after the sparkr.zip error is resolved, 
getting this exception in the yarn logs:

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], 
main() threw exception, Application application_1499090560114_0185 finished 
with failed status
org.apache.spark.SparkException: Application application_1499090560114_0185 
finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1143)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1194)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:745)
at 
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:311)
at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:232)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:58)
at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:239)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)


> Oozie 4.3.0 with spark 2.0 run spark R job on yarn cluster mode failed 
> ---
>
> Key: OOZIE-2767
> URL: https://issues.apache.org/jira/browse/OOZIE-2767
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 4.3.0
>Reporter: Dongying Jiao
> Attachments: spark1.sh, spark2.sh
>
>
> Erro seen from yarn log:
> Warning: Master yarn-cluster is deprecated since 2.0. Please use master 
> "yarn" with specified deploy mode instead.
> Error: sparkr.zip does not exist for R application in YARN mode.
> Run with --help for usage help or --verbose for debug output
> Intercepting System.exit(1)
> Failing Oozie Launcher, Main class 
> [org.apache.oozie.action.hadoop.SparkMain], exit code [1]
> End of LogType:stderr



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2987) Coord action missing dependencies should show URI template with unresolved dependencies

2017-07-12 Thread Purshotam Shah (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah updated OOZIE-2987:
--
Description: 
Patch also fixes and an issue where if there are unresolved dependencies and 
hcat dependencies,  {{blocked on}} show unresolved dependency.

 Missing dependency for unresolved will show as 
{{${coord:latestRange(-5,0)} -> 
hcat://localhost:30083/db_c/table1/dt=${YEAR}${DAY};country=usa}}

> Coord action missing dependencies should show URI template with unresolved 
> dependencies
> ---
>
> Key: OOZIE-2987
> URL: https://issues.apache.org/jira/browse/OOZIE-2987
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-2987-V1.patch
>
>
> Patch also fixes and an issue where if there are unresolved dependencies and 
> hcat dependencies,  {{blocked on}} show unresolved dependency.
>  Missing dependency for unresolved will show as 
> {{${coord:latestRange(-5,0)} -> 
> hcat://localhost:30083/db_c/table1/dt=${YEAR}${DAY};country=usa}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2987) Coord action missing dependencies should show URI template with unresolved dependencies

2017-07-12 Thread Purshotam Shah (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah updated OOZIE-2987:
--
Description: 
Patch also fixes and an issue where if there are unresolved dependencies and 
hcat dependencies,  {{blocked on}} show unresolved dependency.

 Missing dependency for unresolved will show as 
{code}${coord:latestRange(-5,0)} -> 
hcat://localhost:30083/db_c/table1/dt=${YEAR}${DAY};country=usa{code}

  was:
Patch also fixes and an issue where if there are unresolved dependencies and 
hcat dependencies,  {{blocked on}} show unresolved dependency.

 Missing dependency for unresolved will show as 
{{${coord:latestRange(-5,0)} -> 
hcat://localhost:30083/db_c/table1/dt=${YEAR}${DAY};country=usa}}


> Coord action missing dependencies should show URI template with unresolved 
> dependencies
> ---
>
> Key: OOZIE-2987
> URL: https://issues.apache.org/jira/browse/OOZIE-2987
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-2987-V1.patch
>
>
> Patch also fixes and an issue where if there are unresolved dependencies and 
> hcat dependencies,  {{blocked on}} show unresolved dependency.
>  Missing dependency for unresolved will show as 
> {code}${coord:latestRange(-5,0)} -> 
> hcat://localhost:30083/db_c/table1/dt=${YEAR}${DAY};country=usa{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2987) Coord action missing dependencies should show URI template with unresolved dependencies

2017-07-12 Thread Purshotam Shah (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah updated OOZIE-2987:
--
Attachment: OOZIE-2987-V1.patch

> Coord action missing dependencies should show URI template with unresolved 
> dependencies
> ---
>
> Key: OOZIE-2987
> URL: https://issues.apache.org/jira/browse/OOZIE-2987
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
> Attachments: OOZIE-2987-V1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (OOZIE-2987) Coord action missing dependencies should show URI template with unresolved dependencies

2017-07-12 Thread Purshotam Shah (JIRA) 
Purshotam Shah created OOZIE-2987:
-

 Summary: Coord action missing dependencies should show URI 
template with unresolved dependencies
 Key: OOZIE-2987
 URL: https://issues.apache.org/jira/browse/OOZIE-2987
 Project: Oozie
  Issue Type: Bug
Reporter: Purshotam Shah






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : oozie-trunk-find-patches-available #163985

2017-07-12 Thread Apache Jenkins Server 
See

Build failed in Jenkins: oozie-trunk-find-patches-available #163984

2017-07-12 Thread Apache Jenkins Server 
See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H1 (Hadoop) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/oozie.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/oozie.git
 > git --version # timeout=10
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/oozie.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision a54f7c20d7c5d1d72f195d6d92001edd147bc304 
(refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a54f7c20d7c5d1d72f195d6d92001edd147bc304
 > git rev-list a54f7c20d7c5d1d72f195d6d92001edd147bc304 # timeout=10
[oozie-trunk-find-patches-available] $ /bin/bash 
/tmp/hudson7291626481992181241.sh
mkdir: cannot create directory 
':
 File exists
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100  135k0  135k0 0   153k  0 --:--:-- --:--:-- --:--:--  
153k100  623k0  623k0 0   332k  0 --:--:--  0:00:01 --:--:--  
332k100 1255k0 1255k0 0   428k  0 --:--:--  0:00:02 --:--:--  
428k100 1759k0 1759k0 0   453k  0 --:--:--  0:00:03 --:--:--  
453k100 2063k0 2063k0 0   462k  0 --:--:--  0:00:04 --:--:--  
462k
curl: (18) transfer closed with outstanding read data remaining
Could not retrieve available patches from JIRA
Build step 'Execute shell' marked build as failure

Jenkins build is back to normal : oozie-trunk-find-patches-available #163977

2017-07-12 Thread Apache Jenkins Server 
See

Build failed in Jenkins: oozie-trunk-find-patches-available #163976

2017-07-12 Thread Apache Jenkins Server 
See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H1 (Hadoop) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/oozie.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/oozie.git
 > git --version # timeout=10
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/oozie.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision a54f7c20d7c5d1d72f195d6d92001edd147bc304 
(refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a54f7c20d7c5d1d72f195d6d92001edd147bc304
 > git rev-list a54f7c20d7c5d1d72f195d6d92001edd147bc304 # timeout=10
[oozie-trunk-find-patches-available] $ /bin/bash 
/tmp/hudson8945411593838744711.sh
mkdir: cannot create directory 
':
 File exists
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0100 
 263k0  263k0 0   219k  0 --:--:--  0:00:01 --:--:--  219k100  
831k0  831k0 0   379k  0 --:--:--  0:00:02 --:--:--  379k100 
1551k0 1551k0 0   483k  0 --:--:--  0:00:03 --:--:--  483k100 
1639k0 1639k0 0   446k  0 --:--:--  0:00:03 --:--:--  446k
curl: (18) transfer closed with outstanding read data remaining
Could not retrieve available patches from JIRA
Build step 'Execute shell' marked build as failure

[jira] [Commented] (OOZIE-2601) Ability to use local paths for the sharelib

2017-07-12 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084516#comment-16084516
 ] 

Hadoop QA commented on OOZIE-2601:
--

Testing JIRA OOZIE-2601

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 1 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1598
.Tests rerun: 3
.Tests failed at first run: 
org.apache.oozie.action.hadoop.TestHdfsOperations,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/3970/

> Ability to use local paths for the sharelib
> ---
>
> Key: OOZIE-2601
> URL: https://issues.apache.org/jira/browse/OOZIE-2601
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: oya
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Fix For: oya
>
> Attachments: OOZIE-2601-01.patch, OOZIE-2601-03.patch, 
> OOZIE-2601-05.patch, OOZIE-2601-06.patch
>
>
> With OOZIE-2590, as part of OOZIE-1770 Oozie on Yarn work, Oozie now has full 
> control over the classpath given to the Launcher AM.  In a cluster where all 
> nodes have everything installed locally (in the same paths), it should be 
> possible to have the Launcher AM reference the local jars instead of having 
> to localize them from HDFS.
> For example, if you have Hive installed on all nodes at {{/usr/lib/hive/}} 
> and all Hive jars under {{/usr/lib/hive/lib/}}, we could have the Launcher AM 
> add {{/usr/lib/hive/lib}} to its classpath.  This saves on the overhead of 
> localizing the same jars from the hive sharelib in HDFS.  
> I think the best way to implement this is to augment the [Sharelib Mapping 
> File|https://oozie.apache.org/docs/4.2.0/AG_Install.html#Oozie_Share_Lib] 
> feature to accept {{file:///}} paths.
> If we had this also work with the "oozie" sharelib and the Oozie jars in the 
> individual sharelibs (e.g. have the Mapping file take comma-separated 
> dirs/jars), then in a cluster with everything installed on all of the nodes, 
> you wouldn't need to bother with the sharelib at all!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : oozie-trunk-precommit-build #3970

2017-07-12 Thread Apache Jenkins Server 
See

[jira] [Updated] (OOZIE-2601) Ability to use local paths for the sharelib

2017-07-12 Thread Attila Sasvari (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2601:
--
Attachment: OOZIE-2601-06.patch

> Ability to use local paths for the sharelib
> ---
>
> Key: OOZIE-2601
> URL: https://issues.apache.org/jira/browse/OOZIE-2601
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: oya
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Fix For: oya
>
> Attachments: OOZIE-2601-01.patch, OOZIE-2601-03.patch, 
> OOZIE-2601-05.patch, OOZIE-2601-06.patch
>
>
> With OOZIE-2590, as part of OOZIE-1770 Oozie on Yarn work, Oozie now has full 
> control over the classpath given to the Launcher AM.  In a cluster where all 
> nodes have everything installed locally (in the same paths), it should be 
> possible to have the Launcher AM reference the local jars instead of having 
> to localize them from HDFS.
> For example, if you have Hive installed on all nodes at {{/usr/lib/hive/}} 
> and all Hive jars under {{/usr/lib/hive/lib/}}, we could have the Launcher AM 
> add {{/usr/lib/hive/lib}} to its classpath.  This saves on the overhead of 
> localizing the same jars from the hive sharelib in HDFS.  
> I think the best way to implement this is to augment the [Sharelib Mapping 
> File|https://oozie.apache.org/docs/4.2.0/AG_Install.html#Oozie_Share_Lib] 
> feature to accept {{file:///}} paths.
> If we had this also work with the "oozie" sharelib and the Oozie jars in the 
> individual sharelibs (e.g. have the Mapping file take comma-separated 
> dirs/jars), then in a cluster with everything installed on all of the nodes, 
> you wouldn't need to bother with the sharelib at all!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (OOZIE-2986) API for workflows

2017-07-12 Thread Andras Piros (JIRA) 
Andras Piros created OOZIE-2986:
---

 Summary: API for workflows
 Key: OOZIE-2986
 URL: https://issues.apache.org/jira/browse/OOZIE-2986
 Project: Oozie
  Issue Type: Sub-task
Reporter: Andras Piros
Assignee: Daniel Becker


After the [*generated JAXB objects*|OOZIE-2980] and after we have an 
understanding [*how Dozer works*|OOZIE-2982], the task is:
* implement a [*fluent API*|https://en.wikipedia.org/wiki/Fluent_interface] 
that covers an easy-to-use workflow creation mechanism
* similar to the one [*Apache 
Airflow*|https://airflow.incubator.apache.org/concepts.html] uses
* {{WorkflowApp}} instances are created
* these are converted to JAXB {{Object}} s via 
[*Dozer*|http://dozer.sourceforge.net/]
* these can be serialized to XML and written to filesystem
* cross checks are not yet covered
* only workflows are covered
* schema versions of workflows are not yet covered
* only the direction API -> {{WorkflowApp}} -> JAXB {{Object}} -> XML is covered



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2980) JAXB Proof of Concept

2017-07-12 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084165#comment-16084165
 ] 

Hadoop QA commented on OOZIE-2980:
--

Testing JIRA OOZIE-2980

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 2 testcase(s)
{color:red}-1 RAT{color}
.{color:red}-1{color} the patch seems to introduce 1 new RAT warning(s)
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 1 Javadoc warning(s)
{color:red}-1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:red}-1{color} the patch seems to introduce 1 new javac warning(s)
{color:green}+1{color} There are no new bugs found in total.
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1100
.Tests rerun: 22
.Tests failed at first run: org.apache.oozie.action.hadoop.TestLauncherAM,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/3969/

> JAXB Proof of Concept
> -
>
> Key: OOZIE-2980
> URL: https://issues.apache.org/jira/browse/OOZIE-2980
> Project: Oozie
>  Issue Type: Sub-task
>  Components: client
>Affects Versions: 4.3.0
>Reporter: Andras Piros
>Assignee: Daniel Becker
> Attachments: OOZIE-2980-001-wip.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Create a proof of concept w/ JAXB that:
> * only works for workflows
> * given a workflow XSD file
> Following activities in descending complexity order:
> * auto-generates the JAXB beans w/ the 
> [*{{jaxb2-maven-plugin}}*|http://www.mojohaus.org/jaxb2-maven-plugin/Documentation/v2.2/]
> * code / configs are stored in a Maven subproject
> * can read a {{workflow.xml}} into JAXB beans
> * can write a JAXB bean into {{workflow.xml}}
> * make unit tests for XML -> JAXB and JAXB -> XML use cases
> * parametrization ({{job.properties}} key / value pairs), as well as EL 
> functions (like {{$\{wf:currentUser()\}}}) are stored as plain strings



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: oozie-trunk-precommit-build #3969

2017-07-12 Thread Apache Jenkins Server 
See 


Changes:

[gezapeti] OOZIE-2854 Oozie should handle transient database problems 
(andras.piros

--
[...truncated 414.54 KB...]
[INFO] Copying jackson-core-asl-1.9.13.jar to 

[INFO] Copying commons-jexl-2.1.1.jar to 

[INFO] Copying oozie-sharelib-hcatalog-5.0.0-SNAPSHOT.jar to 

[INFO] Copying hadoop-mapreduce-client-jobclient-2.6.0.jar to 

[INFO] Copying gson-2.7.jar to 

[INFO] Copying oozie-core-5.0.0-SNAPSHOT.jar to 

[INFO] Copying commons-pool-1.5.4.jar to 

[INFO] Copying htrace-core-3.0.4.jar to 

[INFO] Copying jaxb-impl-2.2.3-1.jar to 

[INFO] Copying jets3t-0.9.0.jar to 

[INFO] Copying datanucleus-rdbms-3.2.9.jar to 

[INFO] Copying hive-common-1.2.0.jar to 

[INFO] Copying snappy-java-1.0.4.1.jar to 

[INFO] Copying javax.annotation-api-1.2.jar to 

[INFO] Copying commons-compiler-2.7.6.jar to 

[INFO] Copying ant-1.9.1.jar to 

[INFO] Copying hadoop-annotations-2.6.0.jar to 

[INFO] Copying janino-2.7.6.jar to 

[INFO] Copying activemq-client-5.13.3.jar to 

[INFO] Copying guice-3.0.jar to 

[INFO] Copying httpclient-4.3.6.jar to 

[INFO] Copying xz-1.0.jar to 

[INFO] Copying oozie-client-5.0.0-SNAPSHOT.jar to 

[INFO] Copying hadoop-mapreduce-client-common-2.6.0.jar to 

[INFO] Copying leveldbjni-all-1.8.jar to 

[INFO] Copying jetty-util-6.1.26.jar to 

[INFO] Copying velocity-1.5.jar to 

[INFO] Copying jackson-core-2.4.2.jar to 

[INFO] Copying apache-log4j-extras-1.2.17.jar to 

[INFO] Copying netty-3.7.0.Final.jar to 

[INFO] Copying xmlenc-0.52.jar to

[jira] [Updated] (OOZIE-2982) Investigate Dozer

2017-07-12 Thread Andras Piros (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2982:

Description: 
Investigate [*Dozer*|http://dozer.sourceforge.net/]:
* only for workflows
* whether it's capable of converting between our generated JAXB beans and API 
artifacts ({{WorkflowBuilder}}
 and {{WorkflowActionBuilder}} instances)
* decide whether to take the per-configuration or the per-code approach
* what is the minimum level of configuration / coding we have to take when 
there is a new XSD version
* how we can support multiple XSD versions simultaneously

  was:
Investigate [*Dozer*|http://dozer.sourceforge.net/]:
* only for workflows
* whether it's capable of converting between our generated JAXB beans and 
entities ({{WorkflowJobBean}}
 and {{WorkflowActionBean}} instances)
* decide whether to take the per-configuration or the per-code approach
* what is the minimum level of configuration / coding we have to take when 
there is a new XSD version
* how we can support multiple XSD versions simultaneously


> Investigate Dozer
> -
>
> Key: OOZIE-2982
> URL: https://issues.apache.org/jira/browse/OOZIE-2982
> Project: Oozie
>  Issue Type: Sub-task
>  Components: client
>Reporter: Andras Piros
>Assignee: Daniel Becker
>
> Investigate [*Dozer*|http://dozer.sourceforge.net/]:
> * only for workflows
> * whether it's capable of converting between our generated JAXB beans and API 
> artifacts ({{WorkflowBuilder}}
>  and {{WorkflowActionBuilder}} instances)
> * decide whether to take the per-configuration or the per-code approach
> * what is the minimum level of configuration / coding we have to take when 
> there is a new XSD version
> * how we can support multiple XSD versions simultaneously



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (OOZIE-2985) If LauncherAM fails, Oozie is not notified in a timely manner

2017-07-12 Thread Attila Sasvari (JIRA) 
Attila Sasvari created OOZIE-2985:
-

 Summary: If LauncherAM fails, Oozie is not notified in a timely 
manner
 Key: OOZIE-2985
 URL: https://issues.apache.org/jira/browse/OOZIE-2985
 Project: Oozie
  Issue Type: Bug
Reporter: Attila Sasvari


I've noticed if LauncherAM fails, Oozie is notified about the launcher's 
failure with a lot of delay. It gives the impression that the workflow is 
running.

{{oozie job -oozie http://localhost:11000/oozie -config 
examples/apps/datelist-java-main/job.properties  -info  
000-170712153835057-oozie-asas-W}}
{code}
000-170712153835057-oozie-asas-W@java1
RUNNING   application_1499866588585_0001RUNNING- 
{code}

I've looked at yarn logs for the launcher and seen that the launcher failed. 
For example, in my case , during development, oozie-sharelib launcher was not 
found:  
{code}
Error: Could not find or load main class 
org.apache.oozie.action.hadoop.LauncherAM
{code}

The problem is only after the specified timeout (by default 10 minutes) we see 
that the workflow has actually failed /errored.

{code}
Created   : 2017-07-12 13:38 GMT
Started   : 2017-07-12 13:38 GMT
Last Modified : 2017-07-12 13:49 GMT
...
000-170712153835057-oozie-asas-W@java1
ERROR application_1499866588585_0001FAILED/KILLED- 
{code} 

The problem might be that in {{JavaActionExecutor}} in the {{start()}} method 
the check is too fast.

{code}
LOG.debug("Starting action " + action.getId() + " getting Action File System");
FileSystem actionFs = context.getAppFileSystem();
LOG.debug("Preparing action Dir through copying " + 
context.getActionDir());
prepareActionDir(actionFs, context);
LOG.debug("Action Dir is ready. Submitting the action ");
submitLauncher(actionFs, context, action);
LOG.debug("Action submit completed. Performing check ");
check(context, action);
LOG.debug("Action check is done after submission
{code}

There should be some delay after {{submitLauncher()}} before {{check()}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2980) JAXB Proof of Concept

2017-07-12 Thread Daniel Becker (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Becker updated OOZIE-2980:
-
Attachment: OOZIE-2980-001-wip.patch

The patch contains a new maven subproject with some tests.
Please review it [~andras.piros].
Thanks.

Also added to review board:
[https://reviews.apache.org/r/60800].

> JAXB Proof of Concept
> -
>
> Key: OOZIE-2980
> URL: https://issues.apache.org/jira/browse/OOZIE-2980
> Project: Oozie
>  Issue Type: Sub-task
>  Components: client
>Affects Versions: 4.3.0
>Reporter: Andras Piros
>Assignee: Daniel Becker
> Attachments: OOZIE-2980-001-wip.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Create a proof of concept w/ JAXB that:
> * only works for workflows
> * given a workflow XSD file
> Following activities in descending complexity order:
> * auto-generates the JAXB beans w/ the 
> [*{{jaxb2-maven-plugin}}*|http://www.mojohaus.org/jaxb2-maven-plugin/Documentation/v2.2/]
> * code / configs are stored in a Maven subproject
> * can read a {{workflow.xml}} into JAXB beans
> * can write a JAXB bean into {{workflow.xml}}
> * make unit tests for XML -> JAXB and JAXB -> XML use cases
> * parametrization ({{job.properties}} key / value pairs), as well as EL 
> functions (like {{$\{wf:currentUser()\}}}) are stored as plain strings



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 60544: OOZIE-2854 Oozie should handle transient database problems

2017-07-12 Thread Peter Cseh 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60544/#review180283
---


Ship it!




Ship It!

- Peter Cseh


On July 11, 2017, 2:41 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60544/
> ---
> 
> (Updated July 11, 2017, 2:41 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and 
> Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/OOZIE-2854
> 
> 
> Diffs
> -
> 
>   core/pom.xml acddf349a89cf09a7fc4f384ebcaec56dfd0ab48 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/JsonBeanPersisterExecutor.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/executor/jpa/QueryExecutor.java 
> 8d94c23e40d1281864db40e141b200ca207a6324 
>   core/src/main/java/org/apache/oozie/service/JPAService.java 
> 028381d3b72bcc3b8c2cd27cacb3e0ac6d48d146 
>   core/src/main/java/org/apache/oozie/sla/SLASummaryBean.java 
> cfe1522a4b1f89085eb29e7f1281c2abd631bdc2 
>   core/src/main/java/org/apache/oozie/store/WorkflowStore.java 
> c565e74893b863caef6c93015cfe38fe520d04ec 
>   core/src/main/java/org/apache/oozie/util/db/BasicDataSourceWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/DatabaseRetryPredicate.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingHSQLDBDriverWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingMySQLDriverWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/OperationRetryHandler.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/util/db/PersistenceExceptionSubclassFilterRetryPredicate.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/RetryAttemptState.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/RuntimeExceptionInjector.java 
> PRE-CREATION 
>   core/src/main/resources/META-INF/persistence.xml 
> bad9278597fcd4f93b4cc482afae8af14beaa922 
>   core/src/main/resources/oozie-default.xml 
> 5629a897b4a1834554a1a5f7a21984ea857ed133 
>   core/src/main/resources/oozie-log4j.properties 
> c065f3cd4c5a3df1308b69d7c16e8fcfa8796efc 
>   core/src/test/java/org/apache/oozie/command/SkipCommitFaultInjection.java  
>   core/src/test/java/org/apache/oozie/test/XTestCase.java 
> 2816e5d80efd89151606b364376120a76b5c5a4a 
>   
> core/src/test/java/org/apache/oozie/util/db/TestOozieDmlStatementPredicate.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/db/TestOperationRetryHandler.java 
> PRE-CREATION 
>   
> core/src/test/java/org/apache/oozie/util/db/TestPersistenceExceptionSubclassFilterRetryPredicate.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/db/TestRetryAttemptState.java 
> PRE-CREATION 
>   docs/src/site/twiki/AG_Install.twiki 
> 2125442c273661d54f3566cbe41ead822c1f7baa 
>   minitest/pom.xml 9515284bb5f32c279a93161c10e6571680e4f9fc 
>   
> minitest/src/test/java/org/apache/oozie/test/TestParallelJPAOperationRetries.java
>  PRE-CREATION 
>   minitest/src/test/java/org/apache/oozie/test/TestWorkflowRetries.java 
> PRE-CREATION 
>   minitest/src/test/java/org/apache/oozie/test/WorkflowTest.java 
> 2845f0af6efb9ef75fdbfcb326115c62e6fb3bdd 
>   minitest/src/test/resources/hsqldb-oozie-site.xml 
> fa5fe9c3185e973e8247d7bf10b126119d9c02c9 
>   minitest/src/test/resources/oozie-log4j.properties 
> c142d725140930bfa89cd2b163d0768a4c3a750a 
>   minitest/src/test/resources/parallel-fs-and-shell.xml PRE-CREATION 
>   minitest/src/test/resources/wf-test.xml 
> 20c4946862039a65c76ed7f49991345e90a694de 
>   pom.xml 16c5137d44d7db891da46f80adb51c85e4c1b214 
> 
> 
> Diff: https://reviews.apache.org/r/60544/diff/9/
> 
> 
> Testing
> ---
> 
> Tests covered in code:
> 
> Unit tests
> ==
> 
> * testing the retry handler, the retry predicate filter, and parallel calls 
> to JPA `EntityManager` (mostly Oozie database reads and writes) when 
> injecting failures
> 
> Integration tests
> =
> 
> * using the `MiniOozieTestCase` framework
> * fixing it so that also asynchronous workflow applications (the ones that 
> use `CallableQueueService`) can be run
> * following workflow scenarios:
> * a very simple one consisting only of a `` and an `` node
> * a more sophisticated one consisting of multiple synchronous `` nodes 
> and a `` node
> * the ultimate one consisting of a `` node, and two branches of an 
> `` and an asynchronous `` nodes
> 
> Test cases run:
> ```
> mvn clean test 
> -Dtest=TestOperationRetryHandler,TestPersistenceExceptionSubclassFilterRetryP

[jira] [Commented] (OOZIE-2978) Remove code that handles Pig versions before 0.8

2017-07-12 Thread Attila Sasvari (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083664#comment-16083664
 ] 

Attila Sasvari commented on OOZIE-2978:
---

[~pbacsko], [~gezapeti] are there anything else that is missing / issue needs 
to be addressed?

> Remove code that handles Pig versions before 0.8 
> -
>
> Key: OOZIE-2978
> URL: https://issues.apache.org/jira/browse/OOZIE-2978
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2978-01.patch, OOZIE-2978-02.patch, 
> OOZIE-2978-03.patch, OOZIE-2978-04.patch, OOZIE-2978-05.patch
>
>
> Oozie submits Pig jobs according to given Pig version. For example, Pig 
> versions before 0.8 are treated in the following way:
>  
> [Main.main(args)|https://github.com/apache/oozie/blob/678f1b2b2cc9a2456c7e3d9f693d2637f1f03257/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java#L382]
> However, we dropped support for *Hadoop 1*, *Hadoop 0.23* support via 
> {{OOZIE-2316}}. I wonder whether submitting jobs using Pig version before 0.8 
> will work at all:
> - 
> https://pig.apache.org/releases.html#13+May%2C+2010%3A+release+0.7.0+available
>  says "This release works with Hadoop 0.20". Pig *0.10.1* is the first pig 
> release that  works with Hadoop 2.
> - Looking at [PIG-2125|https://issues.apache.org/jira/browse/PIG-2125], I 
> believe the answer is NO.  
> Therefore we can safely delete corresponding parts from the codebase. There 
> is also another class {{PigMainWithOldAPI}} that is only used by tests in 
> {{TestPigMainWithOldAPI}}. So we can remove these too.
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (OOZIE-2984) Parse spark-defaults.conf values with spaces without needing the quotes.

2017-07-12 Thread Andras Piros (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083680#comment-16083680
 ] 

Andras Piros edited comment on OOZIE-2984 at 7/12/17 9:09 AM:
--

[~gezapeti] such quoting scenario is possible when calling {{spark-submit}}:
{noformat}
spark-submit \
...
--conf \
spark.executor.extraJavaOptions="-Dlog4j.configuration=spark-log4j.properties 
-Dsome.executor.key=\"some value\"" \
--conf \
spark.driver.extraJavaOptions="-Dsome.driver.key=some.value 
-Dlog4j.configuration=spark-log4j.properties" \
...
{noformat}

Producing following output:
{noformat}
Using properties file: /usr/lib/spark/conf/spark-defaults.conf
  ...
  driverExtraJavaOptions  -Dsome.driver.key=some.value 
-Dlog4j.configuration=spark-log4j.properties
  ...
Spark properties used, including those specified through
 --conf and those from the properties file 
/usr/lib/spark/conf/spark-defaults.conf:
  ...
  (spark.executor.extraJavaOptions,-Dlog4j.configuration=spark-log4j.properties 
-Dsome.executor.key="some value")
  ...
  (spark.driver.extraJavaOptions,-Dsome.driver.key=some.value 
-Dlog4j.configuration=spark-log4j.properties)
...
System properties:
...
(spark.executor.extraJavaOptions,-Dlog4j.configuration=spark-log4j.properties 
-Dsome.executor.key="some value")
...
(spark.driver.extraJavaOptions,-Dsome.driver.key=some.value 
-Dlog4j.configuration=spark-log4j.properties)
...
{noformat}

Here we can see it's possible to (both are understood by {{spark-submit}}, so 
possibly used by Spark users):
* quote a {{--conf}} option fully to have multiple options within an option
* escape-quote a sub-option inside an already quoted {{--conf}} option


was (Author: andras.piros):
[~gezapeti] such quoting scenario is possible when calling {{spark-submit}}:
{noformat}
spark-submit \
...
--conf \
spark.executor.extraJavaOptions="-Dlog4j.configuration=spark-log4j.properties 
-Dsome.executor.key=\"some value\"" \
--conf \
spark.driver.extraJavaOptions="-Dsome.driver.key=some.value 
-Dlog4j.configuration=spark-log4j.properties" \
...
{noformat}

Producing following output:
{noformat}
Using properties file: /usr/lib/spark/conf/spark-defaults.conf
  ...
  driverExtraJavaOptions  -Dsome.driver.key=some.value 
-Dlog4j.configuration=spark-log4j.properties
  ...
Spark properties used, including those specified through
 --conf and those from the properties file 
/usr/lib/spark/conf/spark-defaults.conf:
  ...
  (spark.executor.extraJavaOptions,-Dlog4j.configuration=spark-log4j.properties 
-Dsome.executor.key="some value")
  ...
  (spark.driver.extraJavaOptions,-Dsome.driver.key=some.value 
-Dlog4j.configuration=spark-log4j.properties)
...
System properties:
...
(spark.executor.extraJavaOptions,-Dlog4j.configuration=spark-log4j.properties 
-Dsome.executor.key="some value")
...
(spark.driver.extraJavaOptions,-Dsome.driver.key=some.value 
-Dlog4j.configuration=spark-log4j.properties)
...
{noformat}

Here we can see it's possible to (both are understood by {{spark-shell}}, so 
possibly used by Spark users):
* quote a {{--conf}} option fully to have multiple options within an option
* escape-quote a sub-option inside an already quoted {{--conf}} option

> Parse spark-defaults.conf values with spaces without needing the quotes.
> 
>
> Key: OOZIE-2984
> URL: https://issues.apache.org/jira/browse/OOZIE-2984
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Fahd Siddiqui
>Assignee: Andras Piros
>
> Oozie requires multiple java opts to be enclosed in quotes. For example:
> {code}
> spark.driver.extraJavaOptions="-Xmn2703m -XX:SurvivorRatio=2 
> -XX:ParallelGCThreads=20"
> {code}
> However the above breaks spark-shell. 
> {code}
> $ ./spark2-shell
> Invalid initial young generation size: -Xmn2212m -XX:SurvivorRatio=2 
> -XX:ParallelGCThreads=20
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> {code}
> To fix spark-shell, we have to remove the quotes, which errors out Oozie:
> {code}
> Error: Unrecognized option '-XX:SurvivorRatio=2'
> ...
> --conf
> spark.executor.extraJavaOptions=-Xmn2703m 
> -Dlog4j.configuration=spark-log4j.properties
> -XX:SurvivorRatio=2
> {code}
> Oozie should be able to parse spark-defaults.conf values with spaces without 
> needing the quotes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2984) Parse spark-defaults.conf values with spaces without needing the quotes.

2017-07-12 Thread Andras Piros (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083680#comment-16083680
 ] 

Andras Piros commented on OOZIE-2984:
-

[~gezapeti] such quoting scenario is possible when calling {{spark-submit}}:
{noformat}
spark-submit \
...
--conf \
spark.executor.extraJavaOptions="-Dlog4j.configuration=spark-log4j.properties 
-Dsome.executor.key=\"some value\"" \
--conf \
spark.driver.extraJavaOptions="-Dsome.driver.key=some.value 
-Dlog4j.configuration=spark-log4j.properties" \
...
{noformat}

Producing following output:
{noformat}
Using properties file: /usr/lib/spark/conf/spark-defaults.conf
  ...
  driverExtraJavaOptions  -Dsome.driver.key=some.value 
-Dlog4j.configuration=spark-log4j.properties
  ...
Spark properties used, including those specified through
 --conf and those from the properties file 
/usr/lib/spark/conf/spark-defaults.conf:
  ...
  (spark.executor.extraJavaOptions,-Dlog4j.configuration=spark-log4j.properties 
-Dsome.executor.key="some value")
  ...
  (spark.driver.extraJavaOptions,-Dsome.driver.key=some.value 
-Dlog4j.configuration=spark-log4j.properties)
...
System properties:
...
(spark.executor.extraJavaOptions,-Dlog4j.configuration=spark-log4j.properties 
-Dsome.executor.key="some value")
...
(spark.driver.extraJavaOptions,-Dsome.driver.key=some.value 
-Dlog4j.configuration=spark-log4j.properties)
...
{noformat}

Here we can see it's possible to (both are understood by {{spark-shell}}, so 
possibly used by Spark users):
* quote a {{--conf}} option fully to have multiple options within an option
* escape-quote a sub-option inside an already quoted {{--conf}} option

> Parse spark-defaults.conf values with spaces without needing the quotes.
> 
>
> Key: OOZIE-2984
> URL: https://issues.apache.org/jira/browse/OOZIE-2984
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Fahd Siddiqui
>Assignee: Andras Piros
>
> Oozie requires multiple java opts to be enclosed in quotes. For example:
> {code}
> spark.driver.extraJavaOptions="-Xmn2703m -XX:SurvivorRatio=2 
> -XX:ParallelGCThreads=20"
> {code}
> However the above breaks spark-shell. 
> {code}
> $ ./spark2-shell
> Invalid initial young generation size: -Xmn2212m -XX:SurvivorRatio=2 
> -XX:ParallelGCThreads=20
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> {code}
> To fix spark-shell, we have to remove the quotes, which errors out Oozie:
> {code}
> Error: Unrecognized option '-XX:SurvivorRatio=2'
> ...
> --conf
> spark.executor.extraJavaOptions=-Xmn2703m 
> -Dlog4j.configuration=spark-log4j.properties
> -XX:SurvivorRatio=2
> {code}
> Oozie should be able to parse spark-defaults.conf values with spaces without 
> needing the quotes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2949) Escape quotes whitespaces in Sqoop field

2017-07-12 Thread Peter Cseh (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083617#comment-16083617
 ] 

Peter Cseh commented on OOZIE-2949:
---

We should use the same logic for splitting up pieces of string and keeping 
together quoted parts as in Spark so we avoid code duplication. You can 
move/rename SparkOptionsSplitter to a place where it's accessible by 
SqoopActionExecutor as well.

> Escape quotes whitespaces in Sqoop  field
> --
>
> Key: OOZIE-2949
> URL: https://issues.apache.org/jira/browse/OOZIE-2949
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 4.3.0
>Reporter: Peter Cseh
>Assignee: Artem Ervits
> Fix For: 5.0.0
>
> Attachments: OOZIE-2949-0.patch
>
>
> The current behavior of the Sqoop action is:
> {noformat}
> The Sqoop command can be specified either using the command element or 
> multiple arg elements.
> When using the command element, Oozie will split the command on every space 
> into multiple arguments.
> When using the arg elements, Oozie will pass each argument value as an 
> argument to Sqoop.
> {noformat}
> This prevents the user to simply copy-paste the command worked in the shell 
> into the workflow.xml.
> We should split the  field by taking quotes into account, similar to 
> what OOZIE-2391
> did for the Spark action's  field.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (OOZIE-2004) Improve Oozie version info output

2017-07-12 Thread Peter Cseh (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083603#comment-16083603
 ] 

Peter Cseh edited comment on OOZIE-2004 at 7/12/17 7:53 AM:


Thnaks for the improvement!

In OozieCLI you've accidentally escaped the newline charactes: {quote} 
append("\\nSource code repository: ") {quote}
So I get the output:
{code}
>> [61] 09:41 : oozie-client-5.0.0-SNAPSHOT (master) :: ./bin/oozie version
Oozie client build version: 5.0.0-SNAPSHOT\nSource code repository: 
https://git-wip-us.apache.org/repos/asf/oozie.git\nCompiled by gezapeti on 
2017.07.12-07:33:26GMT\nFrom source with checksum: master@a54f7c20
{code}



was (Author: gezapeti):
Thnaks for the improvement!

In OozieCLI you've accidentally escaped the newline charactes: {{ 
.append("\\nSource code repository: ")}}
So I get the output:
{code}
>> [61] 09:41 : oozie-client-5.0.0-SNAPSHOT (master) :: ./bin/oozie version
Oozie client build version: 5.0.0-SNAPSHOT\nSource code repository: 
https://git-wip-us.apache.org/repos/asf/oozie.git\nCompiled by gezapeti on 
2017.07.12-07:33:26GMT\nFrom source with checksum: master@a54f7c20
{code}


> Improve Oozie version info output
> -
>
> Key: OOZIE-2004
> URL: https://issues.apache.org/jira/browse/OOZIE-2004
> Project: Oozie
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 4.0.1
>Reporter: Chunjun Xiao
>Assignee: Artem Ervits
>Priority: Minor
> Fix For: 5.0.0
>
> Attachments: HBASE-2004-2.patch, OOZIE-2004-0.patch, 
> OOZIE-2004-1.patch
>
>
> Current oozie version info output is quite simple:
> {code}
> [root@svm5 ~] # oozie version
> Oozie client build version: 4.0.1
> [root@svm5 ~] #
> {code}
> Can we improve it to follow the convention of  Hadoop/HBase, as:
> {code}
> [root@svm5 ~] # oozie version
> Oozie 4.0.1
> Source code repository: ssh://git@XXX/oozie.git -r 
> 3171cb656a4e4c1b2c1e9c6a47abed7df5bbcc8d
> Compiled by root on Sun Jul  6 04:38:08 PDT 2014
> From source with checksum XXX
> [root@svm5 ~] #
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (OOZIE-2004) Improve Oozie version info output

2017-07-12 Thread Peter Cseh (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083603#comment-16083603
 ] 

Peter Cseh edited comment on OOZIE-2004 at 7/12/17 7:54 AM:


Thnaks for the improvement!

In OozieCLI you've accidentally escaped the newline charactes: {noformat} 
append("\\nSource code repository: ") {noformat}
So I get the output:
{code}
>> [61] 09:41 : oozie-client-5.0.0-SNAPSHOT (master) :: ./bin/oozie version
Oozie client build version: 5.0.0-SNAPSHOT\nSource code repository: 
https://git-wip-us.apache.org/repos/asf/oozie.git\nCompiled by gezapeti on 
2017.07.12-07:33:26GMT\nFrom source with checksum: master@a54f7c20
{code}



was (Author: gezapeti):
Thnaks for the improvement!

In OozieCLI you've accidentally escaped the newline charactes: {quote} 
append("\\nSource code repository: ") {quote}
So I get the output:
{code}
>> [61] 09:41 : oozie-client-5.0.0-SNAPSHOT (master) :: ./bin/oozie version
Oozie client build version: 5.0.0-SNAPSHOT\nSource code repository: 
https://git-wip-us.apache.org/repos/asf/oozie.git\nCompiled by gezapeti on 
2017.07.12-07:33:26GMT\nFrom source with checksum: master@a54f7c20
{code}


> Improve Oozie version info output
> -
>
> Key: OOZIE-2004
> URL: https://issues.apache.org/jira/browse/OOZIE-2004
> Project: Oozie
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 4.0.1
>Reporter: Chunjun Xiao
>Assignee: Artem Ervits
>Priority: Minor
> Fix For: 5.0.0
>
> Attachments: HBASE-2004-2.patch, OOZIE-2004-0.patch, 
> OOZIE-2004-1.patch
>
>
> Current oozie version info output is quite simple:
> {code}
> [root@svm5 ~] # oozie version
> Oozie client build version: 4.0.1
> [root@svm5 ~] #
> {code}
> Can we improve it to follow the convention of  Hadoop/HBase, as:
> {code}
> [root@svm5 ~] # oozie version
> Oozie 4.0.1
> Source code repository: ssh://git@XXX/oozie.git -r 
> 3171cb656a4e4c1b2c1e9c6a47abed7df5bbcc8d
> Compiled by root on Sun Jul  6 04:38:08 PDT 2014
> From source with checksum XXX
> [root@svm5 ~] #
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2004) Improve Oozie version info output

2017-07-12 Thread Peter Cseh (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083603#comment-16083603
 ] 

Peter Cseh commented on OOZIE-2004:
---

Thnaks for the improvement!

In OozieCLI you've accidentally escaped the newline charactes: {{ 
.append("\\nSource code repository: ")}}
So I get the output:
{code}
>> [61] 09:41 : oozie-client-5.0.0-SNAPSHOT (master) :: ./bin/oozie version
Oozie client build version: 5.0.0-SNAPSHOT\nSource code repository: 
https://git-wip-us.apache.org/repos/asf/oozie.git\nCompiled by gezapeti on 
2017.07.12-07:33:26GMT\nFrom source with checksum: master@a54f7c20
{code}


> Improve Oozie version info output
> -
>
> Key: OOZIE-2004
> URL: https://issues.apache.org/jira/browse/OOZIE-2004
> Project: Oozie
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 4.0.1
>Reporter: Chunjun Xiao
>Assignee: Artem Ervits
>Priority: Minor
> Fix For: 5.0.0
>
> Attachments: HBASE-2004-2.patch, OOZIE-2004-0.patch, 
> OOZIE-2004-1.patch
>
>
> Current oozie version info output is quite simple:
> {code}
> [root@svm5 ~] # oozie version
> Oozie client build version: 4.0.1
> [root@svm5 ~] #
> {code}
> Can we improve it to follow the convention of  Hadoop/HBase, as:
> {code}
> [root@svm5 ~] # oozie version
> Oozie 4.0.1
> Source code repository: ssh://git@XXX/oozie.git -r 
> 3171cb656a4e4c1b2c1e9c6a47abed7df5bbcc8d
> Compiled by root on Sun Jul  6 04:38:08 PDT 2014
> From source with checksum XXX
> [root@svm5 ~] #
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)