date:20170816

[jira] [Commented] (OOZIE-2687) Create XML schema for launcher configurations

2017-08-16 Thread Robert Kanter (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129360#comment-16129360
 ] 

Robert Kanter commented on OOZIE-2687:
--

I left some specific feedback on RB.  Here's some more general comments:
- If we do these major non-forwards compatible changes to the schemas, perhaps 
it would be a good idea to jump all schema versions to 1.0?  Or at least a 
common 0.x (e.g. 0.10)?  From looking at the patch, we have hive 0.7, hive2 
0.3, oozie-common 0.1, workflow 0.6, shell 0.4, etc.  It's all over the place 
and users will have a hard time keeping track of what versions to put to use 
all of the new features.  
- Another question to consider: should we deprecate the older schemas?  We 
could then remove them in Oozie 6.
- We currently use {{}} for most things in the schema.  This means 
that it all has to be in the "correct" order.  Should we think about changing 
that to allow any order?  My concern would be if we're making any assumptions 
about the order when parsing things.  Changing this also makes things more 
complicated, so we might be best off leaving it for now.

> Create XML schema for launcher configurations
> -
>
> Key: OOZIE-2687
> URL: https://issues.apache.org/jira/browse/OOZIE-2687
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
>Priority: Critical
> Attachments: OOZIE-2687-001.patch, OOZIE-2687-002.patch
>
>
> It was possible to manipulate the MapperLauncher's environment through 
> properties like:
> # mapreduce.map.memory.mb 
> # mapreduce.map.cpu.vcores
> # mapred.child.env
> # mapred.child.java.opts  
> # mapred.job.queue.name  - ability to set launcher queue
> E.g. We were using mapred.child.env to pass SPARK_HOME to the LauncherMapper 
> and make PySpark work. 
> Fixing OOZIE-2596 added a hack. We should decide how we support or break 
> compatibility and how we allow the manipulation of the Launcher environment.
> Verify if the new launcher section in global applies to actions in 
> sub-workflows as well. It did not use to work before and was only fixed in 
> OOZIE-2030. It would be good to have that testcase 
> (TestSubWorkflowActionExecutor. testParentGlobalConf) updated with the new 
> launcher section as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher configurations

2017-08-16 Thread Robert Kanter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review183065
---




client/src/main/resources/hive-action-0.7.xsd
Lines 31 (patched)


Should we do "resource-manager" to be consistent with "job-tracker", 
"name-node", "job-xml", "capture-output", etc?



client/src/main/resources/hive-action-0.7.xsd
Lines 49 (patched)


Can we put CONFIGURATION into oozie-common?

Same with PREPARE, DELETE, and MKDIR?



client/src/main/resources/oozie-common-0.1.xsd
Lines 25-26 (patched)


Something to think about: YARN is currently working on supporting custom 
resource types.  This allows admins to add other things like GPUs or really 
anything.  It's probably fine to not worry about that for now, but it's 
something we should think about how to support in the future.



core/src/main/resources/oozie-default.xml
Lines 1612-1616 (original), 1613-1617 (patched)


Don't you need to add all of the other new schemas?


- Robert Kanter


On Aug. 10, 2017, 9:40 a.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> ---
> 
> (Updated Aug. 10, 2017, 9:40 a.m.)
> 
> 
> Review request for oozie and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> New workflow xsd to allow changing launcher configuration (e.g. setting java 
> system properties)
> 
> 
> Diffs
> -
> 
>   client/src/main/resources/hive-action-0.7.xsd PRE-CREATION 
>   client/src/main/resources/hive2-action-0.3.xsd PRE-CREATION 
>   client/src/main/resources/oozie-common-0.1.xsd PRE-CREATION 
>   client/src/main/resources/oozie-workflow-0.6.xsd PRE-CREATION 
>   client/src/main/resources/shell-action-0.4.xsd PRE-CREATION 
>   client/src/main/resources/spark-action-0.3.xsd PRE-CREATION 
>   client/src/main/resources/sqoop-action-0.5.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> dc17950418ada9b4848ed5597702a29e7124ab38 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711 
>   core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java 
> ffc29af2f834da6d0890ea7215c5a62fd7cd693e 
>   core/src/main/java/org/apache/oozie/util/WritableUtils.java 
> aa027e37ba23d4f481698e0bcd93c26c763a0b1f 
>   
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java 
> a74e5c759fe3e336e7e98c61b8f5ac52efa4100c 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 
> 23df0867e5984c518e8424d79365997bec28f619 
>   core/src/main/resources/oozie-default.xml 
> 0d174b5069aff36ca3a9d48722c5bba69623a046 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 
> d74160a09ee237f4067e00ded526e7ec94528a6b 
>   core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 
> 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> c51c64a18c74c73aa9907027992033a9975bf4d7 
>   
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java
>  21332404bea79fe0a3cea045d57b3e52a1f15cae 
>   core/src/test/resources/wf-schema-global-launcherconf-override.xml 
> PRE-CREATION 
>   core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
> 
> 
> Diff: https://reviews.apache.org/r/61529/diff/2/
> 
> 
> Testing
> ---
> 
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that 
> verbose class loading information is printed out of stdout of the launcher job
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

[jira] [Commented] (OOZIE-3038) Make all Oozie JUnit tests pass on dist_test

2017-08-16 Thread Andras Piros (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129032#comment-16129032
 ] 

Andras Piros commented on OOZIE-3038:
-

[~gezapeti] can you please review?

> Make all Oozie JUnit tests pass on dist_test
> 
>
> Key: OOZIE-3038
> URL: https://issues.apache.org/jira/browse/OOZIE-3038
> Project: Oozie
>  Issue Type: Improvement
>  Components: tests
>Affects Versions: 4.3.0
>Reporter: Andras Piros
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-3038.001.patch
>
>
> We have Oozie JUnit test cases above 2000 pieces right now. To run all the 
> JUnit tests before a patch submission is an overkill since these take more 
> than two hours.
> As a workaround we can use the 
> [*dist_test*|https://github.com/cloudera/dist_test] framework that allows for 
> parallel test execution - the load is distributed across possibly hundreds of 
> cloud engine slaves. So the JUnit tests run time will equal to the longest 
> JUnit test run, that is, in the order of 10-ish minutes for the whole test 
> suite across all the Oozie components.
> Nevertheless, some test cases that try to read files with relative access 
> (e.g. {{src/test/resources/oozie-site.xml}}), or get system properties that 
> are not present on cloud slaves, will certainly fail. This improvement 
> addresses that exact issue by making those tests pass also on the cloud.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3038) Make all Oozie JUnit tests pass on dist_test

2017-08-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129022#comment-16129022
 ] 

Hadoop QA commented on OOZIE-3038:
--

Testing JIRA OOZIE-3038

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 5 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2009
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/4041/

> Make all Oozie JUnit tests pass on dist_test
> 
>
> Key: OOZIE-3038
> URL: https://issues.apache.org/jira/browse/OOZIE-3038
> Project: Oozie
>  Issue Type: Improvement
>  Components: tests
>Affects Versions: 4.3.0
>Reporter: Andras Piros
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-3038.001.patch
>
>
> We have Oozie JUnit test cases above 2000 pieces right now. To run all the 
> JUnit tests before a patch submission is an overkill since these take more 
> than two hours.
> As a workaround we can use the 
> [*dist_test*|https://github.com/cloudera/dist_test] framework that allows for 
> parallel test execution - the load is distributed across possibly hundreds of 
> cloud engine slaves. So the JUnit tests run time will equal to the longest 
> JUnit test run, that is, in the order of 10-ish minutes for the whole test 
> suite across all the Oozie components.
> Nevertheless, some test cases that try to read files with relative access 
> (e.g. {{src/test/resources/oozie-site.xml}}), or get system properties that 
> are not present on cloud slaves, will certainly fail. This improvement 
> addresses that exact issue by making those tests pass also on the cloud.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : oozie-trunk-precommit-build #4041

2017-08-16 Thread Apache Jenkins Server

See

[jira] [Commented] (OOZIE-3005) Cover all workflow action types and workflow / action SLA with JAXB

2017-08-16 Thread Andras Piros (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128894#comment-16128894
 ] 

Andras Piros commented on OOZIE-3005:
-

* get {{maven-jaxb2-plugin}} to generate {{equals()}} and {{hashCode()}} methods
* rename {{action.email.ACTION}} to {{action.email.EMAIL_ACTION}}, for all the 
external action types
* make SLA point to the good JAXB location

> Cover all workflow action types and workflow / action SLA with JAXB
> ---
>
> Key: OOZIE-3005
> URL: https://issues.apache.org/jira/browse/OOZIE-3005
> Project: Oozie
>  Issue Type: Sub-task
>  Components: client
>Reporter: Daniel Becker
>Assignee: Andras Piros
>
> All the Oozie workflow actions have to be covered, not only MapReduce ones, 
> considering following:
> * no XSD change should be performed
> * XJB mapping file has to be extended to incorporate all the workflow action 
> types, also the ones inside 
> {{//xs:complexType[@name='ACTION']/xs:sequence/xs:choice/xs:any[@namespace='##other']}}
> * XJB mapping file has to be extended to incorporate SLA related stuff inside 
> {{//xs:complexType[@name='ACTION']/xs:sequence/xs:any[@namespace='uri:oozie:sla:0.1
>  uri:oozie:sla:0.2']}}
> * autogenerate sources from different XSDs to different packages, resolve 
> namespace clashes that way. Note that it's not a goal to have all namespace 
> clashes resolve within XSDs
> * extend {{TestJAXBWorkflow}} to handle all those different action types, and 
> SLA, as well



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3038) Make all Oozie JUnit tests pass on dist_test

2017-08-16 Thread Andras Piros (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-3038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-3038:

Attachment: OOZIE-3038.001.patch

> Make all Oozie JUnit tests pass on dist_test
> 
>
> Key: OOZIE-3038
> URL: https://issues.apache.org/jira/browse/OOZIE-3038
> Project: Oozie
>  Issue Type: Improvement
>  Components: tests
>Affects Versions: 4.3.0
>Reporter: Andras Piros
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-3038.001.patch
>
>
> We have Oozie JUnit test cases above 2000 pieces right now. To run all the 
> JUnit tests before a patch submission is an overkill since these take more 
> than two hours.
> As a workaround we can use the 
> [*dist_test*|https://github.com/cloudera/dist_test] framework that allows for 
> parallel test execution - the load is distributed across possibly hundreds of 
> cloud engine slaves. So the JUnit tests run time will equal to the longest 
> JUnit test run, that is, in the order of 10-ish minutes for the whole test 
> suite across all the Oozie components.
> Nevertheless, some test cases that try to read files with relative access 
> (e.g. {{src/test/resources/oozie-site.xml}}), or get system properties that 
> are not present on cloud slaves, will certainly fail. This improvement 
> addresses that exact issue by making those tests pass also on the cloud.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (OOZIE-3038) Make all Oozie JUnit tests pass on dist_test

2017-08-16 Thread Andras Piros (JIRA)

Andras Piros created OOZIE-3038:
---

 Summary: Make all Oozie JUnit tests pass on dist_test
 Key: OOZIE-3038
 URL: https://issues.apache.org/jira/browse/OOZIE-3038
 Project: Oozie
  Issue Type: Improvement
  Components: tests
Affects Versions: 4.3.0
Reporter: Andras Piros
Assignee: Andras Piros
 Fix For: 5.0.0


We have Oozie JUnit test cases above 2000 pieces right now. To run all the 
JUnit tests before a patch submission is an overkill since these take more than 
two hours.

As a workaround we can use the 
[*dist_test*|https://github.com/cloudera/dist_test] framework that allows for 
parallel test execution - the load is distributed across possibly hundreds of 
cloud engine slaves. So the JUnit tests run time will equal to the longest 
JUnit test run, that is, in the order of 10-ish minutes for the whole test 
suite across all the Oozie components.

Nevertheless, some test cases that try to read files with relative access (e.g. 
{{src/test/resources/oozie-site.xml}}), or get system properties that are not 
present on cloud slaves, will certainly fail. This improvement addresses that 
exact issue by making those tests pass also on the cloud.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2858) HiveMain, ShellMain and SparkMain should not overwrite properties and config files locally

2017-08-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128770#comment-16128770
 ] 

Hadoop QA commented on OOZIE-2858:
--

Testing JIRA OOZIE-2858

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:red}-1{color} There are [2] new bugs found below threshold in total that 
must be fixed.
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:red}-1{color} There are [2] new bugs found below threshold in 
[sharelib/oozie] that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones): 
sharelib/oozie/findbugs-new.html
. The most important FindBugs errors are:
. At LauncherMain.java:[line 471]: File(...) reads a file whose location might 
be specified by user input
. At LauncherMain.java:[line 461]: File(...) reads a file whose location might 
be specified by user input
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2012
.Tests rerun: 15
.Tests failed at first run: org.apache.oozie.jms.TestJMSJobEventListener,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/4040/

> HiveMain, ShellMain and SparkMain should not overwrite properties and config 
> files locally
> --
>
> Key: OOZIE-2858
> URL: https://issues.apache.org/jira/browse/OOZIE-2858
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-2858-001.patch, OOZIE-2858-002.patch, 
> OOZIE-2858-003.patch
>
>
> We're generating files on the fly in many of our *Main classes.
> These are usually log4j.properties and hive-site.xml files (there may be 
> others, but these are the ones I managed to find).
> We should check if these files are there in advance and don't overwrite them 
> if they are present. We could log a warning in this case or merge the 
> contents of the files with the things we want to write into it. There is an 
> example for this in PigMain and log4j.properties.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: oozie-trunk-precommit-build #4040

2017-08-16 Thread Apache Jenkins Server

See 


Changes:

[gezapeti] OOZIE-3036 Spark 2.2.0 support: tell Spark not to get any delegation

[gezapeti] OOZIE-2940 Possible NullPointerException in WorkflowActionBean 
(dionusos

--
[...truncated 1.58 MB...]
[INFO] Copying slf4j-log4j12-1.6.6.jar to 

[INFO] Copying jackson-core-2.4.2.jar to 

[INFO] Copying postgresql-9.0-801.jdbc4.jar to 

[INFO] Copying datanucleus-rdbms-3.2.9.jar to 

[INFO] Copying jettison-1.1.jar to 

[INFO] Copying guice-servlet-3.0.jar to 

[INFO] Copying ant-launcher-1.9.1.jar to 

[INFO] Copying jpam-1.1.jar to 

[INFO] Copying hive-service-1.2.0.jar to 

[INFO] Copying hive-hcatalog-server-extensions-1.2.0.jar to 

[INFO] Copying hadoop-mapreduce-client-core-2.6.0.jar to 

[INFO] Copying json-20090211.jar to 

[INFO] Copying avro-1.7.4.jar to 

[INFO] Copying jetty-util-9.2.19.v20160908.jar to 

[INFO] Copying apache-curator-2.6.0.pom to 

[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-server ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/empty.xml
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Distro 5.0.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-distro 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-distro 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-distro ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-distro ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-distro ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-distro ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-distro ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/distro.xml
[INFO] Copying files to 

[INFO] Building tar : 

[INFO] 
[INFO]

[jira] [Assigned] (OOZIE-2713) Provide an option to log out contents of the localized jar files

2017-08-16 Thread Peter Cseh (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh reassigned OOZIE-2713:
-

Assignee: (was: Peter Cseh)

> Provide an option to log out contents of the localized jar files
> 
>
> Key: OOZIE-2713
> URL: https://issues.apache.org/jira/browse/OOZIE-2713
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Peter Cseh
>  Labels: newbie
> Attachments: OOZIE-2713.01.patch
>
>
> We had several issues with user jars involved in workflows containing 
> conflicting third party libraries.
> It looks like creating uberjars for UDFs in hive is a bad practice, but it's 
> sometimes hard to get the needed information to handle these issues. 
> We should introduce a new property for the actions e.g 
> oozie.action.list.jar.contents which is false by default, but if set to true, 
> list the contents of the jar files after the file list in the launcher logs. 
> This way we could see which classes are present on the classpath multiple 
> times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (OOZIE-2554) Spark action tests on yarn-cluster, yarn-client mode

2017-08-16 Thread Peter Cseh (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh reassigned OOZIE-2554:
-

Assignee: (was: Peter Cseh)

> Spark action tests on yarn-cluster, yarn-client mode
> 
>
> Key: OOZIE-2554
> URL: https://issues.apache.org/jira/browse/OOZIE-2554
> Project: Oozie
>  Issue Type: New Feature
>Reporter: Satish Subhashrao Saley
> Attachments: OOZIE-2554.poc.patch
>
>
> Currently, we have unit test cases for Spark Action in local mode. To verify 
> other functionality, we should have tests in yarn-cluster and yarn-client 
> mode.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2713) Provide an option to log out contents of the localized jar files

2017-08-16 Thread Peter Cseh (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh updated OOZIE-2713:
--
Labels: newbie  (was: )

> Provide an option to log out contents of the localized jar files
> 
>
> Key: OOZIE-2713
> URL: https://issues.apache.org/jira/browse/OOZIE-2713
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Peter Cseh
>Assignee: Peter Cseh
>  Labels: newbie
> Attachments: OOZIE-2713.01.patch
>
>
> We had several issues with user jars involved in workflows containing 
> conflicting third party libraries.
> It looks like creating uberjars for UDFs in hive is a bad practice, but it's 
> sometimes hard to get the needed information to handle these issues. 
> We should introduce a new property for the actions e.g 
> oozie.action.list.jar.contents which is false by default, but if set to true, 
> list the contents of the jar files after the file list in the launcher logs. 
> This way we could see which classes are present on the classpath multiple 
> times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2909) LauncherAM: rewrite UGI calls

2017-08-16 Thread Peter Cseh (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128637#comment-16128637
 ] 

Peter Cseh commented on OOZIE-2909:
---

Thanks for the explanation, I'm working on a new patch

> LauncherAM: rewrite UGI calls
> -
>
> Key: OOZIE-2909
> URL: https://issues.apache.org/jira/browse/OOZIE-2909
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Cseh
> Attachments: OOZIE-2909-001.patch
>
>
> There are two problems in LauncherAM:
> 1) In a kerberized cluster, we have to use {{UGI.getLoginUser()}} even though 
> a call to {{createRemoteUser()}} should be enough, just like in 
> {{MRAppMaster.java}}. Reference code:
> https://github.com/apache/hadoop/blob/3ed3062fe3979ff55a411b730a8eee2b2c96d6b3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java#L1718-L1726
> 2) It's enough if we wrap the whole LauncherAM logic in a single {{doAs()}} 
> call. No need for multiple {{doAs()}}, just makes the whole thing more 
> complicated.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] Subscription: Oozie Patch Available

2017-08-16 Thread jira

Issue Subscription
Filter: Oozie Patch Available (110 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2987  Coord action missing dependencies should show URI template with 
unresolved dependencies
https://issues-test.apache.org/jira/browse/OOZIE-2987
OOZIE-2980  JAXB Proof of Concept
https://issues-test.apache.org/jira/browse/OOZIE-2980
OOZIE-2978  Remove code that handles Pig versions before 0.8 
https://issues-test.apache.org/jira/browse/OOZIE-2978
OOZIE-2977  error: self-closing element not allowed
https://issues-test.apache.org/jira/browse/OOZIE-2977
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues-test.apache.org/jira/browse/OOZIE-2975
OOZIE-2974  Change TestLiteWorkflowAppParser so that it works with Hadoop 3
https://issues-test.apache.org/jira/browse/OOZIE-2974
OOZIE-2969  Drop support for Java 1.7
https://issues-test.apache.org/jira/browse/OOZIE-2969
OOZIE-2964  getting at 
org.apache.maven.plugin.javadoc.AbstractJavadocMojo.executeJavadocCommandLine(AbstractJavadocMojo.java:5163)
https://issues-test.apache.org/jira/browse/OOZIE-2964
OOZIE-2963  getting error in build ArtifactNotFoundException: Could not find 
artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde
https://issues-test.apache.org/jira/browse/OOZIE-2963
OOZIE-2962  bump maven-javadoc-plugin to 2.10.4
https://issues-test.apache.org/jira/browse/OOZIE-2962
OOZIE-2961  build contains multiple warnings for is not a parameter name
https://issues-test.apache.org/jira/browse/OOZIE-2961
OOZIE-2960  many warnings for @return tag has no arguments
https://issues-test.apache.org/jira/browse/OOZIE-2960
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues-test.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues-test.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues-test.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues-test.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues-test.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues-test.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues-test.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues-test.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues-test.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues-test.apache.org/jira/browse/OOZIE-2934
OOZIE-2931  multiple warnings during build for "no @param for"
https://issues-test.apache.org/jira/browse/OOZIE-2931
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues-test.apache.org/jira/browse/OOZIE-2927
OOZIE-2916  Set a job name for the MR Action's child job
https://issues-test.apache.org/jira/browse/OOZIE-2916
OOZIE-2914  Consolidate Trim 
https://issues-test.apache.org/jira/browse/OOZIE-2914
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues-test.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues-test.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues-test.apache.org/jira/browse/OOZIE-2867
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues-test.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues-test.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues-test.apache.org/jira/browse/OOZIE-2829
OOZIE-2826  Falcon feed fails to aws s3; Oozie joda time version does not meet 
required jar version 2.2 or later
https://issues-test.apache.org/jira/browse/OOZIE-2826
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues-test.apache.org/jira/browse/OOZIE-2812

[jira] Subscription: Oozie Patch Available

2017-08-16 Thread jira

Issue Subscription
Filter: Oozie Patch Available (109 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3035  HDFS HA and log aggregation: getting HDFS delegation token from 
YARN renewer within JavaActionExecutor
https://issues.apache.org/jira/browse/OOZIE-3035
OOZIE-3031  Coord job with only unresolved dependencies doesn't timeout
https://issues.apache.org/jira/browse/OOZIE-3031
OOZIE-3026  fix openjpa enhancer stage during build for logging
https://issues.apache.org/jira/browse/OOZIE-3026
OOZIE-3022  fix for warning has no file and won't be listed in dependency files 
details
https://issues.apache.org/jira/browse/OOZIE-3022
OOZIE-3002  address findbugs errors in client lib
https://issues.apache.org/jira/browse/OOZIE-3002
OOZIE-3001  core library has many instances of warnings with trailing spaces 
and lines longer than 132 chars
https://issues.apache.org/jira/browse/OOZIE-3001
OOZIE-2997  files contain trailing white spaces in client lib
https://issues.apache.org/jira/browse/OOZIE-2997
OOZIE-2996  add option for -UseGCOverheadLimit to maven opts as sometimes local 
testing fails
https://issues.apache.org/jira/browse/OOZIE-2996
OOZIE-2980  JAXB Proof of Concept
https://issues.apache.org/jira/browse/OOZIE-2980
OOZIE-2978  Remove code that handles Pig versions before 0.8 
https://issues.apache.org/jira/browse/OOZIE-2978
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues.apache.org/jira/browse/OOZIE-2975
OOZIE-2969  Drop support for Java 1.7
https://issues.apache.org/jira/browse/OOZIE-2969
OOZIE-2964  Add -Xdoclint:none to javadoc opts to avoid warnings
https://issues.apache.org/jira/browse/OOZIE-2964
OOZIE-2963  getting error in build ArtifactNotFoundException: Could not find 
artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde
https://issues.apache.org/jira/browse/OOZIE-2963
OOZIE-2962  bump maven-javadoc-plugin to 2.10.4
https://issues.apache.org/jira/browse/OOZIE-2962
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues.apache.org/jira/browse/OOZIE-2934
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2916  Set a job name for the MR Action's child job
https://issues.apache.org/jira/browse/OOZIE-2916
OOZIE-2914  Consolidate Trim 
https://issues.apache.org/jira/browse/OOZIE-2914
OOZIE-2909  LauncherAM: rewrite UGI calls
https://issues.apache.org/jira/browse/OOZIE-2909
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues.apache.org/jira/browse/OOZIE-2867
OOZIE-2858  HiveMain, ShellMain and SparkMain should not overwrite properties 
and config files locally
https://issues.apache.org/jira/browse/OOZIE-2858
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders

[jira] [Commented] (OOZIE-2687) Create XML schema for launcher configurations

Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher configurations

[jira] [Commented] (OOZIE-3038) Make all Oozie JUnit tests pass on dist_test

[jira] [Commented] (OOZIE-3038) Make all Oozie JUnit tests pass on dist_test

Jenkins build is back to normal : oozie-trunk-precommit-build #4041

[jira] [Commented] (OOZIE-3005) Cover all workflow action types and workflow / action SLA with JAXB

[jira] [Updated] (OOZIE-3038) Make all Oozie JUnit tests pass on dist_test

[jira] [Created] (OOZIE-3038) Make all Oozie JUnit tests pass on dist_test

[jira] [Commented] (OOZIE-2858) HiveMain, ShellMain and SparkMain should not overwrite properties and config files locally

Build failed in Jenkins: oozie-trunk-precommit-build #4040

[jira] [Assigned] (OOZIE-2713) Provide an option to log out contents of the localized jar files

[jira] [Assigned] (OOZIE-2554) Spark action tests on yarn-cluster, yarn-client mode

[jira] [Updated] (OOZIE-2713) Provide an option to log out contents of the localized jar files

[jira] [Commented] (OOZIE-2909) LauncherAM: rewrite UGI calls

[jira] Subscription: Oozie Patch Available

[jira] Subscription: Oozie Patch Available

16 matches

Site Navigation

Mail list logo

Footer information