[jira] [Commented] (OOZIE-2687) Create XML schema for launcher configurations

2017-09-13 Thread Robert Kanter (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165520#comment-16165520
 ] 

Robert Kanter commented on OOZIE-2687:
--

I'm fine with doing the sharelib override in a followup JIRA.  Generally, the 
the thing with the highest priority has been the more specific one.  For 
example, config properties defined in an action's {{}} have 
priority over an action's {{}}, which has priority over the global 
section's {{}} and {{}}, which has priority over the 
action defaults in oozie-site, and so on.  However, you bring up a good point 
that it might make sense for the sharelib to be additive instead of 
overwriting.  Though I can also see issues with that too (plus, it's 
inconsistent with everything else).

> Create XML schema for launcher configurations
> -
>
> Key: OOZIE-2687
> URL: https://issues.apache.org/jira/browse/OOZIE-2687
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
>Priority: Critical
> Attachments: OOZIE-2687-001.patch, OOZIE-2687-002.patch, 
> OOZIE-2687-003.patch, OOZIE-2687-008.patch, OOZIE-2687-009.patch, 
> OOZIE-2687-010.patch, OOZIE-2687-011.patch
>
>
> It was possible to manipulate the MapperLauncher's environment through 
> properties like:
> # mapreduce.map.memory.mb 
> # mapreduce.map.cpu.vcores
> # mapred.child.env
> # mapred.child.java.opts  
> # mapred.job.queue.name  - ability to set launcher queue
> E.g. We were using mapred.child.env to pass SPARK_HOME to the LauncherMapper 
> and make PySpark work. 
> Fixing OOZIE-2596 added a hack. We should decide how we support or break 
> compatibility and how we allow the manipulation of the Launcher environment.
> Verify if the new launcher section in global applies to actions in 
> sub-workflows as well. It did not use to work before and was only fixed in 
> OOZIE-2030. It would be good to have that testcase 
> (TestSubWorkflowActionExecutor. testParentGlobalConf) updated with the new 
> launcher section as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher configurations

2017-09-13 Thread Robert Kanter 


> On Sept. 14, 2017, 12:16 a.m., Robert Kanter wrote:
> > Ship It!

Oh, we need to update the docs.  That can be a followup JIRA.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review185352
---


On Sept. 13, 2017, 1:45 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> ---
> 
> (Updated Sept. 13, 2017, 1:45 p.m.)
> 
> 
> Review request for oozie and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> New workflow xsd to allow changing launcher configuration (e.g. setting java 
> system properties)
> 
> 
> Diffs
> -
> 
>   client/src/main/resources/hive-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION 
>   client/src/main/resources/shell-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/spark-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711 
>   core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 
> 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a 
>   core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 
> 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7 
>   core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java 
> ffc29af2f834da6d0890ea7215c5a62fd7cd693e 
>   core/src/main/java/org/apache/oozie/service/SchemaService.java 
> 137e2c0b4840c6d3858683db1aa38f54bd55be92 
>   core/src/main/java/org/apache/oozie/util/WritableUtils.java 
> aa027e37ba23d4f481698e0bcd93c26c763a0b1f 
>   core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java 
> a74e5c759fe3e336e7e98c61b8f5ac52efa4100c 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 
> 23df0867e5984c518e8424d79365997bec28f619 
>   core/src/main/resources/oozie-default.xml 
> 0d174b5069aff36ca3a9d48722c5bba69623a046 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 
> d74160a09ee237f4067e00ded526e7ec94528a6b 
>   core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 
> 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> d1d78fd527a425b076edd7bb23ba56ebdba62664 
>   core/src/test/java/org/apache/oozie/service/TestSchemaService.java 
> 88a10dbead32879334c3953a90a411a3a18fc0b3 
>   core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java 
> a882c82b25154338f89e8a90f831b6f0300fbfe7 
>   
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java
>  21332404bea79fe0a3cea045d57b3e52a1f15cae 
>   core/src/test/resources/wf-schema-global-launcherconf-override.xml 
> PRE-CREATION 
>   core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION 
>   pom.xml db18f30814b9b6a73ba872c2cd7946692d0b876b 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java 
> ff999ff2cd4875fbf3ad5e1a7eacf9972e848630 
> 
> 
> Diff: https://reviews.apache.org/r/61529/diff/11/
> 
> 
> Testing
> ---
> 
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that 
> verbose class loading information is printed out of stdout of the launcher job
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher configurations

2017-09-13 Thread Robert Kanter 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review185352
---


Ship it!




Ship It!

- Robert Kanter


On Sept. 13, 2017, 1:45 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> ---
> 
> (Updated Sept. 13, 2017, 1:45 p.m.)
> 
> 
> Review request for oozie and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> New workflow xsd to allow changing launcher configuration (e.g. setting java 
> system properties)
> 
> 
> Diffs
> -
> 
>   client/src/main/resources/hive-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION 
>   client/src/main/resources/shell-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/spark-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711 
>   core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 
> 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a 
>   core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 
> 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7 
>   core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java 
> ffc29af2f834da6d0890ea7215c5a62fd7cd693e 
>   core/src/main/java/org/apache/oozie/service/SchemaService.java 
> 137e2c0b4840c6d3858683db1aa38f54bd55be92 
>   core/src/main/java/org/apache/oozie/util/WritableUtils.java 
> aa027e37ba23d4f481698e0bcd93c26c763a0b1f 
>   core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java 
> a74e5c759fe3e336e7e98c61b8f5ac52efa4100c 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 
> 23df0867e5984c518e8424d79365997bec28f619 
>   core/src/main/resources/oozie-default.xml 
> 0d174b5069aff36ca3a9d48722c5bba69623a046 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 
> d74160a09ee237f4067e00ded526e7ec94528a6b 
>   core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 
> 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> d1d78fd527a425b076edd7bb23ba56ebdba62664 
>   core/src/test/java/org/apache/oozie/service/TestSchemaService.java 
> 88a10dbead32879334c3953a90a411a3a18fc0b3 
>   core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java 
> a882c82b25154338f89e8a90f831b6f0300fbfe7 
>   
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java
>  21332404bea79fe0a3cea045d57b3e52a1f15cae 
>   core/src/test/resources/wf-schema-global-launcherconf-override.xml 
> PRE-CREATION 
>   core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION 
>   pom.xml db18f30814b9b6a73ba872c2cd7946692d0b876b 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java 
> ff999ff2cd4875fbf3ad5e1a7eacf9972e848630 
> 
> 
> Diff: https://reviews.apache.org/r/61529/diff/11/
> 
> 
> Testing
> ---
> 
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that 
> verbose class loading information is printed out of stdout of the launcher job
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher configurations

2017-09-13 Thread Peter Cseh 


> On Sept. 11, 2017, 11:36 p.m., Robert Kanter wrote:
> > client/src/main/resources/oozie-workflow-1.0.xsd
> > Line 234 (original), 234 (patched)
> > 
> >
> > Should we think about moving the Java, MR, Pig, etc actions out of the 
> > workflow XSD and into their own XSDs to be more consistent with the other 
> > action types?  The 1.0 XSD schemas is a good opportunity.
> > 
> > The downside would be backwards compatibility where you can't simply 
> > update your workflow schema version on an existing workflow like you 
> > typically could in the past - you'd also have to add the xmlns string for 
> > these actions.  
> > 
> > I think it would be okay to force people to add the xmlns.  No other 
> > changes should be needed so it's pretty minimal.  Plus, Oozie still accepts 
> > the 0.x schemas, so you don't *have* to upgrade the schemas to 1.0 anyway.
> > 
> > Attila, Geza, what do you think?
> 
> Attila Sasvari wrote:
> For this purpose I created 
> https://issues.apache.org/jira/browse/OOZIE-3033 (this patch is already quite 
> big).

As this is a breaking change I'd rather fix this in a later jira. I think we 
can get away using 1.0 as a version and doing big changes like that in 
follow-up steps.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review185128
---


On Sept. 13, 2017, 1:45 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> ---
> 
> (Updated Sept. 13, 2017, 1:45 p.m.)
> 
> 
> Review request for oozie and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> New workflow xsd to allow changing launcher configuration (e.g. setting java 
> system properties)
> 
> 
> Diffs
> -
> 
>   client/src/main/resources/hive-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION 
>   client/src/main/resources/shell-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/spark-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711 
>   core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 
> 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a 
>   core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 
> 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7 
>   core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java 
> ffc29af2f834da6d0890ea7215c5a62fd7cd693e 
>   core/src/main/java/org/apache/oozie/service/SchemaService.java 
> 137e2c0b4840c6d3858683db1aa38f54bd55be92 
>   core/src/main/java/org/apache/oozie/util/WritableUtils.java 
> aa027e37ba23d4f481698e0bcd93c26c763a0b1f 
>   core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java 
> a74e5c759fe3e336e7e98c61b8f5ac52efa4100c 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 
> 23df0867e5984c518e8424d79365997bec28f619 
>   core/src/main/resources/oozie-default.xml 
> 0d174b5069aff36ca3a9d48722c5bba69623a046 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 
> d74160a09ee237f4067e00ded526e7ec94528a6b 
>   core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 
> 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> d1d78fd527a425b076edd7bb23ba56ebdba62664 
>   core/src/test/java/org/apache/oozie/service/TestSchemaService.java 
> 88a10dbead32879334c3953a90a411a3a18fc0b3 
>   core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java 
> a882c82b25154338f89e8a90f831b6f0300fbfe7 
>   
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java
>  21332404bea79fe0a3cea045d57b3e52a1f15cae 
>   core/src/test/resources/wf-schema-global-launcherconf-override.xml 
> PRE-CREATION 
>   core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION 
>   pom.xml db18f30814b9b6a73ba872c2cd7946692d0b876b 
>

Failed: OOZIE-2687 PreCommit Build #18

2017-09-13 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-2687
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/18/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.60 MB...]
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
0 There are [2] new bugs found in total that would be nice to have fixed.
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [tools].
 0 There are [2] new bugs found in [core] that would be nice to have fixed.
 You can find the FindBugs diff here: core/findbugs-new.html
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/pig].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2029
Tests rerun: 53
Tests failed at first run: 
org.apache.oozie.command.coord.TestCoordActionsKillXCommand,org.apache.oozie.action.hadoop.TestJavaActionExecutor,
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/18/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100 3706k  100 3706k0 0  3521k  0  0:00:01  0:00:01 --:--:-- 8520k
Adding comment to JIRA
Could not find issue with key: 'OOZIE-2687'

Remote error: ; nested exception is: 
java.net.SocketException: Connection reset

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-2687
Archiving artifacts
[Fast Archiver] Compressed 1.79 MB of artifacts by 57.5% relative to #10
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Updated] (OOZIE-2687) Create XML schema for launcher configurations

2017-09-13 Thread Attila Sasvari (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2687:
--
Attachment: OOZIE-2687-011.patch

> Create XML schema for launcher configurations
> -
>
> Key: OOZIE-2687
> URL: https://issues.apache.org/jira/browse/OOZIE-2687
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
>Priority: Critical
> Attachments: OOZIE-2687-001.patch, OOZIE-2687-002.patch, 
> OOZIE-2687-003.patch, OOZIE-2687-008.patch, OOZIE-2687-009.patch, 
> OOZIE-2687-010.patch, OOZIE-2687-011.patch
>
>
> It was possible to manipulate the MapperLauncher's environment through 
> properties like:
> # mapreduce.map.memory.mb 
> # mapreduce.map.cpu.vcores
> # mapred.child.env
> # mapred.child.java.opts  
> # mapred.job.queue.name  - ability to set launcher queue
> E.g. We were using mapred.child.env to pass SPARK_HOME to the LauncherMapper 
> and make PySpark work. 
> Fixing OOZIE-2596 added a hack. We should decide how we support or break 
> compatibility and how we allow the manipulation of the Launcher environment.
> Verify if the new launcher section in global applies to actions in 
> sub-workflows as well. It did not use to work before and was only fixed in 
> OOZIE-2030. It would be good to have that testcase 
> (TestSubWorkflowActionExecutor. testParentGlobalConf) updated with the new 
> launcher section as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2687) Create XML schema for launcher configurations

2017-09-13 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164877#comment-16164877
 ] 

Hadoop QA commented on OOZIE-2687:
--

Testing JIRA OOZIE-2687

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 6 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 9 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:red}-1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:red}-1{color} the patch seems to introduce 1 new javac warning(s)
{color:red}-1{color} There are [2] new bugs found below threshold in total that 
must be fixed.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:red}-1{color} There are [2] new bugs found below threshold in [core] 
that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones): 
core/findbugs-new.html
. The most important FindBugs errors are:
. At JavaActionExecutor.java:[line 879]: Dead store to jobName in 
org.apache.oozie.action.hadoop.JavaActionExecutor.createLauncherConf(FileSystem,
 ActionExecutor$Context, WorkflowAction, Element, Configuration)
. At Input.java:[line 103]: Found reliance on default encoding in 
org.apache.oozie.util.schema.Input.getStringData(): new String(byte[])
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2029
.Tests rerun: 107
.Tests failed at first run: 
org.apache.oozie.service.TestStatusTransitService,org.apache.oozie.command.coord.TestCoordActionInputCheckXCommandNonUTC,org.apache.oozie.action.hadoop.TestJavaActionExecutor,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/17/

> Create XML schema for launcher configurations
> -
>
> Key: OOZIE-2687
> URL: https://issues.apache.org/jira/browse/OOZIE-2687
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
>Priority: Critical
> Attachments: OOZIE-2687-001.patch, OOZIE-2687-002.patch, 
> OOZIE-2687-003.patch, OOZIE-2687-008.patch, OOZIE-2687-009.patch, 
> OOZIE-2687-010.patch
>
>
> It was possible to manipulate the MapperLauncher's environment through 
> properties like:
> # mapreduce.map.memory.mb 
> # mapreduce.map.cpu.vcores
> # mapred.child.env
> # mapred.child.java.opts  
> # mapred.job.queue.name  - ability to set launcher queue
> E.g. We were using mapred.child.env to pass SPARK_HOME to the LauncherMapper 
> and make PySpark work. 
> Fixing OOZIE-2596 added a hack. We should decide how we support or break 
> compatibility and how we allow the manipulation of the Launcher environment.
> Verify if the new launcher section in global applies to

Failed: OOZIE-2687 PreCommit Build #17

2017-09-13 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-2687
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/17/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.60 MB...]
-1 COMPILE
+1 HEAD compiles
+1 patch compiles
-1 the patch seems to introduce 1 new javac warning(s)
-1 There are [2] new bugs found below threshold in total that must be fixed.
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [tools].
 -1 There are [2] new bugs found below threshold in [core] that must be fixed.
 You can find the FindBugs diff here (look for the red and orange ones): 
core/findbugs-new.html
 The most important FindBugs errors are:
 At JavaActionExecutor.java:[line 879]: Dead store to jobName in 
org.apache.oozie.action.hadoop.JavaActionExecutor.createLauncherConf(FileSystem,
 ActionExecutor$Context, WorkflowAction, Element, Configuration)
 At Input.java:[line 103]: Found reliance on default encoding in 
org.apache.oozie.util.schema.Input.getStringData(): new String(byte[])
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/pig].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2029
Tests rerun: 107
Tests failed at first run: 
org.apache.oozie.service.TestStatusTransitService,org.apache.oozie.command.coord.TestCoordActionInputCheckXCommandNonUTC,org.apache.oozie.action.hadoop.TestJavaActionExecutor,
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/17/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  0 
00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
100 3706k  100 3706k0 0  4573k  0 --:--:-- --:--:-- --:--:-- 4573k
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-2687
Archiving artifacts
[Fast Archiver] Compressed 1.80 MB of artifacts by 55.6% relative to #10
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Commented] (OOZIE-2687) Create XML schema for launcher configurations

2017-09-13 Thread Attila Sasvari (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164840#comment-16164840
 ] 

Attila Sasvari commented on OOZIE-2687:
---

I will address issues above in the next patch.

So far I have not changed {{addActionShareLib}} in {{JavaActionExecutor}} yet. 
Regarding behaviour, what should happen if a workflow has a global and an 
action level {{launcher}} and {{configuration}} tries to override the sharelib? 
I would say that {{oozie.action.sharelib.for.#ACTIONTYPE#}} is stronger than 
the sharelib setting in the launcher. Or should we take an union of those? 
[~rohini], [~rkanter], [~gezapeti], [~pbacsko]  Do you mind if we discuss and 
implement the new mechanism to override sharelib for actions via {{launcher}} 
config in another JIRA? 
 


> Create XML schema for launcher configurations
> -
>
> Key: OOZIE-2687
> URL: https://issues.apache.org/jira/browse/OOZIE-2687
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
>Priority: Critical
> Attachments: OOZIE-2687-001.patch, OOZIE-2687-002.patch, 
> OOZIE-2687-003.patch, OOZIE-2687-008.patch, OOZIE-2687-009.patch, 
> OOZIE-2687-010.patch
>
>
> It was possible to manipulate the MapperLauncher's environment through 
> properties like:
> # mapreduce.map.memory.mb 
> # mapreduce.map.cpu.vcores
> # mapred.child.env
> # mapred.child.java.opts  
> # mapred.job.queue.name  - ability to set launcher queue
> E.g. We were using mapred.child.env to pass SPARK_HOME to the LauncherMapper 
> and make PySpark work. 
> Fixing OOZIE-2596 added a hack. We should decide how we support or break 
> compatibility and how we allow the manipulation of the Launcher environment.
> Verify if the new launcher section in global applies to actions in 
> sub-workflows as well. It did not use to work before and was only fixed in 
> OOZIE-2030. It would be good to have that testcase 
> (TestSubWorkflowActionExecutor. testParentGlobalConf) updated with the new 
> launcher section as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2909) LauncherAM: rewrite UGI calls

2017-09-13 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164826#comment-16164826
 ] 

Hadoop QA commented on OOZIE-2909:
--

Testing JIRA OOZIE-2909

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 3 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 5 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:orange}0{color} There are [6] new bugs found in total that would be nice 
to have fixed.
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:orange}0{color} There are [3] new bugs found in [sharelib/oozie] that 
would be nice to have fixed.
. You can find the FindBugs diff here: sharelib/oozie/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:orange}0{color} There are [3] new bugs found in [core] that would be 
nice to have fixed.
. You can find the FindBugs diff here: core/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2013
.Tests failed: 0
.Tests errors: 2

.The patch failed the following testcases:

.  

.Tests failing with errors:
.  testShellScriptSuccess(org.apache.oozie.action.hadoop.TestShellMain)
.  testShellScriptFailure(org.apache.oozie.action.hadoop.TestShellMain)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/16/

> LauncherAM: rewrite UGI calls
> -
>
> Key: OOZIE-2909
> URL: https://issues.apache.org/jira/browse/OOZIE-2909
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Cseh
>Priority: Blocker
> Attachments: OOZIE-2909-001.patch, OOZIE-2909-002.patch, 
> OOZIE-2909-003.patch, OOZIE-2909-004.patch, OOZIE-2909-005.patch
>
>
> There are two problems in LauncherAM:
> 1) In a kerberized cluster, we have to use {{UGI.getLoginUser()}} even though 
> a call to {{createRemoteUser()}} should be enough, just like in 
> {{MRAppMaster.java}}. Reference code:
> https://github.com/apache/hadoop/blob/3ed3062fe3979ff55a411b730a8eee2b2c96d6b3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java#L1718-L1726
> 2) It's enough if we wrap the whole LauncherAM logic in a single {{doAs()}} 
> call. No need for multiple {{doAs()}}, just makes the whole thing more 
> complicated.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-2909 PreCommit Build #16

2017-09-13 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-2909
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/16/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.60 MB...]
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive2].
 0 There are [3] new bugs found in [sharelib/oozie] that would be nice to have 
fixed.
 You can find the FindBugs diff here: sharelib/oozie/findbugs-new.html
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [server].
 0 There are [3] new bugs found in [core] that would be nice to have fixed.
 You can find the FindBugs diff here: core/findbugs-new.html
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [tools].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS
Tests run: 2013
Tests failed: 0
Tests errors: 2

The patch failed the following testcases:

  

Tests failing with errors:
  testShellScriptSuccess(org.apache.oozie.action.hadoop.TestShellMain)
  testShellScriptFailure(org.apache.oozie.action.hadoop.TestShellMain)

+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/16/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  0 
00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
100 3706k  100 3706k0 0  4592k  0 --:--:-- --:--:-- --:--:-- 4592k
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-2909
Archiving artifacts
[Fast Archiver] Compressed 1.80 MB of artifacts by 43.5% relative to #10
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
2 tests failed.
FAILED:  org.apache.oozie.action.hadoop.TestShellMain.testShellScriptSuccess

Error Message:
Could not find Oozie job launch time

Stack Trace:
java.lang.RuntimeException: Could not find Oozie job launch time
at 
org.apache.oozie.action.hadoop.LauncherMain.getChildYarnJobs(LauncherMain.java:247)
at 
org.apache.oozie.action.hadoop.LauncherMain.getChildYarnJobs(LauncherMain.java:194)
at 
org.apache.oozie.action.hadoop.LauncherMain.killChildYarnJobs(LauncherMain.java:255)
at org.apache.oozie.action.hadoop.ShellMain.run(ShellMain.java:69)
at 
org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:101)
at org.apache.oozie.action.hadoop.ShellMain.main(ShellMain.java:59)
at 
org.apache.oozie.action.hadoop.TestShellMain.call(TestShellMain.java:74)
at 
org.apache.oozie.action.hadoop.TestShellMain.call(TestShellMain.java:32)
at 
org.apache.oozie.action.hadoop.MainTestCase$1.run(MainTestCase.java:33)
at 
org.apache.oozie.action.hadoop.MainTestCase$1.run(MainTestCase.java:31)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at 
org.apache.oozie.action.hadoop.MainTestCase.execute(MainTestCase.java:31)
at 
org.apache.oozie.action.hadoop.ShellTestCase.testShellScriptSuccess(ShellTestCase.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at

[jira] [Updated] (OOZIE-2687) Create XML schema for launcher configurations

2017-09-13 Thread Attila Sasvari (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2687:
--
Attachment: OOZIE-2687-010.patch

> Create XML schema for launcher configurations
> -
>
> Key: OOZIE-2687
> URL: https://issues.apache.org/jira/browse/OOZIE-2687
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
>Priority: Critical
> Attachments: OOZIE-2687-001.patch, OOZIE-2687-002.patch, 
> OOZIE-2687-003.patch, OOZIE-2687-008.patch, OOZIE-2687-009.patch, 
> OOZIE-2687-010.patch
>
>
> It was possible to manipulate the MapperLauncher's environment through 
> properties like:
> # mapreduce.map.memory.mb 
> # mapreduce.map.cpu.vcores
> # mapred.child.env
> # mapred.child.java.opts  
> # mapred.job.queue.name  - ability to set launcher queue
> E.g. We were using mapred.child.env to pass SPARK_HOME to the LauncherMapper 
> and make PySpark work. 
> Fixing OOZIE-2596 added a hack. We should decide how we support or break 
> compatibility and how we allow the manipulation of the Launcher environment.
> Verify if the new launcher section in global applies to actions in 
> sub-workflows as well. It did not use to work before and was only fixed in 
> OOZIE-2030. It would be good to have that testcase 
> (TestSubWorkflowActionExecutor. testParentGlobalConf) updated with the new 
> launcher section as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher configurations

2017-09-13 Thread Attila Sasvari 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/
---

(Updated Sept. 13, 2017, 1:45 p.m.)


Review request for oozie and Peter Cseh.


Changes
---

- addressed review comments: schema elements' order
- moved complex types to common


Repository: oozie-git


Description
---

New workflow xsd to allow changing launcher configuration (e.g. setting java 
system properties)


Diffs (updated)
-

  client/src/main/resources/hive-action-1.0.xsd PRE-CREATION 
  client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION 
  client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION 
  client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION 
  client/src/main/resources/shell-action-1.0.xsd PRE-CREATION 
  client/src/main/resources/spark-action-1.0.xsd PRE-CREATION 
  client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
49fd4b8560b21aef725c3268f0ec0e7f46a46ee9 
  core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
8fdc50cce15271c13b20d1aaab2ffb95cb8fa711 
  core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 
7e3348550e2ef44ae4bd6c5a7a417052bf9c108a 
  core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 
47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7 
  core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java 
ffc29af2f834da6d0890ea7215c5a62fd7cd693e 
  core/src/main/java/org/apache/oozie/service/SchemaService.java 
137e2c0b4840c6d3858683db1aa38f54bd55be92 
  core/src/main/java/org/apache/oozie/util/WritableUtils.java 
aa027e37ba23d4f481698e0bcd93c26c763a0b1f 
  core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java 
a74e5c759fe3e336e7e98c61b8f5ac52efa4100c 
  core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 
23df0867e5984c518e8424d79365997bec28f619 
  core/src/main/resources/oozie-default.xml 
0d174b5069aff36ca3a9d48722c5bba69623a046 
  core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 
d74160a09ee237f4067e00ded526e7ec94528a6b 
  core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 
6cee7a8948ca7f2c7aaf44d4f172d18a1308a482 
  core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
d1d78fd527a425b076edd7bb23ba56ebdba62664 
  core/src/test/java/org/apache/oozie/service/TestSchemaService.java 
88a10dbead32879334c3953a90a411a3a18fc0b3 
  core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java 
a882c82b25154338f89e8a90f831b6f0300fbfe7 
  
core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java
 21332404bea79fe0a3cea045d57b3e52a1f15cae 
  core/src/test/resources/wf-schema-global-launcherconf-override.xml 
PRE-CREATION 
  core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION 
  pom.xml db18f30814b9b6a73ba872c2cd7946692d0b876b 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
  sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java 
ff999ff2cd4875fbf3ad5e1a7eacf9972e848630 


Diff: https://reviews.apache.org/r/61529/diff/10/

Changes: https://reviews.apache.org/r/61529/diff/9-10/


Testing
---

- TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
- java-main example on pseudo hadoop succeeded, modified workflow xml so that 
verbose class loading information is printed out of stdout of the launcher job


Thanks,

Attila Sasvari

[jira] [Updated] (OOZIE-2909) LauncherAM: rewrite UGI calls

2017-09-13 Thread Peter Cseh (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh updated OOZIE-2909:
--
Attachment: OOZIE-2909-005.patch

Addressing review comments and fixing broken test.

> LauncherAM: rewrite UGI calls
> -
>
> Key: OOZIE-2909
> URL: https://issues.apache.org/jira/browse/OOZIE-2909
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Cseh
>Priority: Blocker
> Attachments: OOZIE-2909-001.patch, OOZIE-2909-002.patch, 
> OOZIE-2909-003.patch, OOZIE-2909-004.patch, OOZIE-2909-005.patch
>
>
> There are two problems in LauncherAM:
> 1) In a kerberized cluster, we have to use {{UGI.getLoginUser()}} even though 
> a call to {{createRemoteUser()}} should be enough, just like in 
> {{MRAppMaster.java}}. Reference code:
> https://github.com/apache/hadoop/blob/3ed3062fe3979ff55a411b730a8eee2b2c96d6b3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java#L1718-L1726
> 2) It's enough if we wrap the whole LauncherAM logic in a single {{doAs()}} 
> call. No need for multiple {{doAs()}}, just makes the whole thing more 
> complicated.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls

2017-09-13 Thread Peter Cseh 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/
---

(Updated Sept. 13, 2017, 1:06 p.m.)


Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and 
Rohini Palaniswamy.


Bugs: OOZIE-2909
https://issues.apache.org/jira/browse/OOZIE-2909


Repository: oozie-git


Description
---

Changing UGI calls and cleaning up things around requesting tokens.


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/ErrorCode.java 
662e1edc9c4b23b3606c751bf5ed4b531ee7ac62 
  core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProperties.java 
20f93ce57618a529b1bd83b398add2cd1d96b36e 
  
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
 5ca8d3e80942840a5bba51a06694350929044042 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
9804c7b673a899cd9c778850c9a64731cc642fb3 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
52abbf17afa71e702d7871d10a4a190a4d21e47b 
  core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
4add5f14cb20ab77c755b81eff939335c0c5fddc 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
0b495f75842b0033a3337ef6728a334bf5651770 
  core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
49fd4b8560b21aef725c3268f0ec0e7f46a46ee9 
  core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
187cee2290ba4bbe358913d28a052d9c75f2369a 
  core/src/test/java/org/apache/oozie/action/hadoop/TestHadoopTokenHelper.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
d1d78fd527a425b076edd7bb23ba56ebdba62664 
  core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 
960c2f9750062c98fdf7a2b456d3668069e07ca5 
  
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java
 b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e 
  
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 
874d371664ced3ae19a7b5d615c67d4c72e7cfdd 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java 
bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8 
  
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java
 68c0f4bafe0322ff0d1007297cd6e8f38c92020a 
  
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 
37af3dd9ed43f4504ff3909484237e2d05665c6c 


Diff: https://reviews.apache.org/r/61560/diff/5/

Changes: https://reviews.apache.org/r/61560/diff/4-5/


Testing
---

We're running tests on a Kerberized cluster with these changes.


Thanks,

Peter Cseh

[jira] [Commented] (OOZIE-3054) Disable erasure coding for sharelib if Oozie runs on Hadoop 3

2017-09-13 Thread Peter Bacsko (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164578#comment-16164578
 ] 

Peter Bacsko commented on OOZIE-3054:
-

[~gezapeti], [~andras.piros], [~rkanter] please review

> Disable erasure coding for sharelib if Oozie runs on Hadoop 3
> -
>
> Key: OOZIE-3054
> URL: https://issues.apache.org/jira/browse/OOZIE-3054
> Project: Oozie
>  Issue Type: Improvement
>  Components: client
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: OOZIE-3045-001.patch, OOZIE-3045-002.patch
>
>
> Hadoop 3 will introduce Erasure Coding support for HDFS. This means less disk 
> usage, but increased CPU load for reading/writing.
> The contents of Oozie sharelib are localized every time an asynchronous 
> action is executed. Therefore, to reduce the possible negative performance 
> impact of EC, Oozie sharelib should not utilize it.
> Since we're supporting Hadoop 2 as well, using the EC API will be done with 
> Java reflection. Later, when we drop support for Hadoop 2, this can be 
> rewritten.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3054) Disable erasure coding for sharelib if Oozie runs on Hadoop 3

2017-09-13 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164572#comment-16164572
 ] 

Hadoop QA commented on OOZIE-3054:
--

Testing JIRA OOZIE-3054

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:orange}0{color} There are [1] new bugs found in total that would be nice 
to have fixed.
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:orange}0{color} There are [1] new bugs found in [tools] that would be 
nice to have fixed.
. You can find the FindBugs diff here: tools/findbugs-new.html
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2013
.Tests failed: 0
.Tests errors: 1

.The patch failed the following testcases:

.  

.Tests failing with errors:
.  testJMXInstrumentation(org.apache.oozie.util.TestMetricsInstrumentation)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/15/

> Disable erasure coding for sharelib if Oozie runs on Hadoop 3
> -
>
> Key: OOZIE-3054
> URL: https://issues.apache.org/jira/browse/OOZIE-3054
> Project: Oozie
>  Issue Type: Improvement
>  Components: client
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: OOZIE-3045-001.patch, OOZIE-3045-002.patch
>
>
> Hadoop 3 will introduce Erasure Coding support for HDFS. This means less disk 
> usage, but increased CPU load for reading/writing.
> The contents of Oozie sharelib are localized every time an asynchronous 
> action is executed. Therefore, to reduce the possible negative performance 
> impact of EC, Oozie sharelib should not utilize it.
> Since we're supporting Hadoop 2 as well, using the EC API will be done with 
> Java reflection. Later, when we drop support for Hadoop 2, this can be 
> rewritten.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-3054 PreCommit Build #15

2017-09-13 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-3054
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/15/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.59 MB...]
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/oozie].
 0 There are [1] new bugs found in [tools] that would be nice to have fixed.
 You can find the FindBugs diff here: tools/findbugs-new.html
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS
Tests run: 2013
Tests failed: 0
Tests errors: 1

The patch failed the following testcases:

  

Tests failing with errors:
  testJMXInstrumentation(org.apache.oozie.util.TestMetricsInstrumentation)

+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/15/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100 3706k  100 3706k0 0  3367k  0  0:00:01  0:00:01 --:--:-- 6390k
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-3054
Archiving artifacts
[Fast Archiver] Compressed 1.79 MB of artifacts by 59.2% relative to #10
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
1 tests failed.
FAILED:  org.apache.oozie.util.TestMetricsInstrumentation.testJMXInstrumentation

Error Message:
Unable to open socket file: target process not responding or HotSpot VM not 
loaded

Stack Trace:
com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: 
target process not responding or HotSpot VM not loaded
at 
sun.tools.attach.LinuxVirtualMachine.(LinuxVirtualMachine.java:106)
at 
sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:78)
at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:250)
at 
org.apache.oozie.util.TestMetricsInstrumentation.testJMXInstrumentation(TestMetricsInstrumentation.java:235)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at 
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at

Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls

2017-09-13 Thread Peter Cseh 


> On Sept. 12, 2017, 5:29 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java
> > Lines 23 (patched)
> > 
> >
> > Do we still need the JT?

I've just copied over the code from HadoopAccessorService. I guess we can drop 
it.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review185217
---


On Sept. 12, 2017, 1:40 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> ---
> 
> (Updated Sept. 12, 2017, 1:40 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and 
> Rohini Palaniswamy.
> 
> 
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Changing UGI calls and cleaning up things around requesting tokens.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/ErrorCode.java 
> 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProperties.java 
> 20f93ce57618a529b1bd83b398add2cd1d96b36e 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  5ca8d3e80942840a5bba51a06694350929044042 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 9804c7b673a899cd9c778850c9a64731cc642fb3 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> 52abbf17afa71e702d7871d10a4a190a4d21e47b 
>   core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 4add5f14cb20ab77c755b81eff939335c0c5fddc 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> 0b495f75842b0033a3337ef6728a334bf5651770 
>   core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9 
>   core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
> 187cee2290ba4bbe358913d28a052d9c75f2369a 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestHadoopTokenHelper.java 
> PRE-CREATION 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> d1d78fd527a425b076edd7bb23ba56ebdba62664 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 
> 960c2f9750062c98fdf7a2b456d3668069e07ca5 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java
>  b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java
>  874d371664ced3ae19a7b5d615c67d4c72e7cfdd 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java 
> bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java
>  68c0f4bafe0322ff0d1007297cd6e8f38c92020a 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java
>  37af3dd9ed43f4504ff3909484237e2d05665c6c 
> 
> 
> Diff: https://reviews.apache.org/r/61560/diff/4/
> 
> 
> Testing
> ---
> 
> We're running tests on a Kerberized cluster with these changes.
> 
> 
> Thanks,
> 
> Peter Cseh
> 
>

Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls

2017-09-13 Thread Peter Cseh 


> On Sept. 12, 2017, 2:12 p.m., Peter Bacsko wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java
> > Lines 91 (patched)
> > 
> >
> > Not sure if this is an issue, but wondering why a proxy user + doAs is 
> > necessary here. Oozie is expected to be authenticated and should be able to 
> > retrieve a token without any sort of proxying.

I've just copied this part over from ClientCache. I guess you have to 
impersonate the user you have to get the tokens to for JHS.


> On Sept. 12, 2017, 2:12 p.m., Peter Bacsko wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
> > Lines 325 (patched)
> > 
> >
> > Is this null-check necessary?

No, I've removed it.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review185177
---


On Sept. 12, 2017, 1:40 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> ---
> 
> (Updated Sept. 12, 2017, 1:40 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and 
> Rohini Palaniswamy.
> 
> 
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Changing UGI calls and cleaning up things around requesting tokens.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/ErrorCode.java 
> 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProperties.java 
> 20f93ce57618a529b1bd83b398add2cd1d96b36e 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  5ca8d3e80942840a5bba51a06694350929044042 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 9804c7b673a899cd9c778850c9a64731cc642fb3 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> 52abbf17afa71e702d7871d10a4a190a4d21e47b 
>   core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 4add5f14cb20ab77c755b81eff939335c0c5fddc 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> 0b495f75842b0033a3337ef6728a334bf5651770 
>   core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9 
>   core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
> 187cee2290ba4bbe358913d28a052d9c75f2369a 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestHadoopTokenHelper.java 
> PRE-CREATION 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> d1d78fd527a425b076edd7bb23ba56ebdba62664 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 
> 960c2f9750062c98fdf7a2b456d3668069e07ca5 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java
>  b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java
>  874d371664ced3ae19a7b5d615c67d4c72e7cfdd 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java 
> bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java
>  68c0f4bafe0322ff0d1007297cd6e8f38c92020a 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java
>  37af3dd9ed43f4504ff3909484237e2d05665c6c 
> 
> 
> Diff: https://reviews.apache.org/r/61560/diff/4/
> 
> 
> Testing
> ---
> 
> We're running tests on a Kerberized cluster with these changes.
> 
> 
> Thanks,
> 
> Peter Cseh
> 
>

[jira] [Updated] (OOZIE-3054) Disable erasure coding for sharelib if Oozie runs on Hadoop 3

2017-09-13 Thread Peter Bacsko (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Bacsko updated OOZIE-3054:

Attachment: OOZIE-3045-002.patch

> Disable erasure coding for sharelib if Oozie runs on Hadoop 3
> -
>
> Key: OOZIE-3054
> URL: https://issues.apache.org/jira/browse/OOZIE-3054
> Project: Oozie
>  Issue Type: Improvement
>  Components: client
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: OOZIE-3045-001.patch, OOZIE-3045-002.patch
>
>
> Hadoop 3 will introduce Erasure Coding support for HDFS. This means less disk 
> usage, but increased CPU load for reading/writing.
> The contents of Oozie sharelib are localized every time an asynchronous 
> action is executed. Therefore, to reduce the possible negative performance 
> impact of EC, Oozie sharelib should not utilize it.
> Since we're supporting Hadoop 2 as well, using the EC API will be done with 
> Java reflection. Later, when we drop support for Hadoop 2, this can be 
> rewritten.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3051) Replace Oozie's custom precommit build patch finding job with the standard Apache one

2017-09-13 Thread Peter Cseh (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164392#comment-16164392
 ] 

Peter Cseh commented on OOZIE-3051:
---

Thank you for fixing this!

> Replace Oozie's custom precommit build patch finding job with the standard 
> Apache one
> -
>
> Key: OOZIE-3051
> URL: https://issues.apache.org/jira/browse/OOZIE-3051
> Project: Oozie
>  Issue Type: Test
>  Components: tests
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-3051.001.patch
>
>
> We currently use a custom script in the 
> [oozie-trunk-find-patches-available|https://builds.apache.org/job/oozie-trunk-find-patches-available/]
>  Jenkins job to find patches on JIRA and submit a build to the 
> [oozie-trunk-precommit-build|https://builds.apache.org/job/oozie-trunk-precommit-build/]
>  job.  
> There's a number of downsides to this, the main one being that we need to tie 
> oozie-trunk-find-patches-available to a specific Jenkins host so we can store 
> a file to remember what JIRAs were looked at already (H1).  Unfortunately, 
> this means that whenever H1 is down, we can't get automatic pre-commit jobs.
> Many projects, including Hadoop, Pig, Hive, etc use the 
> [PreCommit-Admin|https://builds.apache.org/job/PreCommit-Admin/] job.  We 
> should do that as well.
> More details (though it appears to be a little out of date) are at 
> http://wiki.apache.org/general/PreCommitBuilds



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] Subscription: Oozie Patch Available

2017-09-13 Thread jira 
Issue Subscription
Filter: Oozie Patch Available (105 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3054  Disable erasure coding for sharelib if Oozie runs on Hadoop 3
https://issues.apache.org/jira/browse/OOZIE-3054
OOZIE-3031  Coord job with only unresolved dependencies doesn't timeout
https://issues.apache.org/jira/browse/OOZIE-3031
OOZIE-3022  fix for warning has no file and won't be listed in dependency files 
details
https://issues.apache.org/jira/browse/OOZIE-3022
OOZIE-3002  address findbugs errors in client lib
https://issues.apache.org/jira/browse/OOZIE-3002
OOZIE-3001  core library has many instances of warnings with trailing spaces 
and lines longer than 132 chars
https://issues.apache.org/jira/browse/OOZIE-3001
OOZIE-2997  files contain trailing white spaces in client lib
https://issues.apache.org/jira/browse/OOZIE-2997
OOZIE-2996  add option for -UseGCOverheadLimit to maven opts as sometimes local 
testing fails
https://issues.apache.org/jira/browse/OOZIE-2996
OOZIE-2978  Remove code that handles Pig versions before 0.8 
https://issues.apache.org/jira/browse/OOZIE-2978
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues.apache.org/jira/browse/OOZIE-2975
OOZIE-2969  Drop support for Java 1.7
https://issues.apache.org/jira/browse/OOZIE-2969
OOZIE-2964  Add -Xdoclint:none to javadoc opts to avoid warnings
https://issues.apache.org/jira/browse/OOZIE-2964
OOZIE-2963  getting error in build ArtifactNotFoundException: Could not find 
artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde
https://issues.apache.org/jira/browse/OOZIE-2963
OOZIE-2962  bump maven-javadoc-plugin to 2.10.4
https://issues.apache.org/jira/browse/OOZIE-2962
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues.apache.org/jira/browse/OOZIE-2934
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2914  Consolidate Trim 
https://issues.apache.org/jira/browse/OOZIE-2914
OOZIE-2909  LauncherAM: rewrite UGI calls
https://issues.apache.org/jira/browse/OOZIE-2909
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues.apache.org/jira/browse/OOZIE-2867
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues.apache.org/jira/browse/OOZIE-2829
OOZIE-2826  Falcon feed fails to aws s3; Oozie joda time version does not meet 
required jar version 2.2 or later
https://issues.apache.org/jira/browse/OOZIE-2826
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2795  Create lib directory or symlink for Oozie CLI during packaging
https://issues.apache.org/jira/browse/OOZIE-2795
OOZIE-2791  ShareLib

[jira] Subscription: Oozie Patch Available

2017-09-13 Thread jira 
Issue Subscription
Filter: Oozie Patch Available (106 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3041  TestWorkflowActionRetryInfoXCommand fails in oozie core module.
https://issues-test.apache.org/jira/browse/OOZIE-3041
OOZIE-3031  Coord job with only unresolved dependencies doesn't timeout
https://issues-test.apache.org/jira/browse/OOZIE-3031
OOZIE-3022  fix for warning has no file and won't be listed in dependency files 
details
https://issues-test.apache.org/jira/browse/OOZIE-3022
OOZIE-3002  address findbugs errors in client lib
https://issues-test.apache.org/jira/browse/OOZIE-3002
OOZIE-3001  core library has many instances of warnings with trailing spaces 
and lines longer than 132 chars
https://issues-test.apache.org/jira/browse/OOZIE-3001
OOZIE-2997  files contain trailing white spaces in client lib
https://issues-test.apache.org/jira/browse/OOZIE-2997
OOZIE-2996  add option for -UseGCOverheadLimit to maven opts as sometimes local 
testing fails
https://issues-test.apache.org/jira/browse/OOZIE-2996
OOZIE-2978  Remove code that handles Pig versions before 0.8 
https://issues-test.apache.org/jira/browse/OOZIE-2978
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues-test.apache.org/jira/browse/OOZIE-2975
OOZIE-2969  Drop support for Java 1.7
https://issues-test.apache.org/jira/browse/OOZIE-2969
OOZIE-2964  Add -Xdoclint:none to javadoc opts to avoid warnings
https://issues-test.apache.org/jira/browse/OOZIE-2964
OOZIE-2963  getting error in build ArtifactNotFoundException: Could not find 
artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde
https://issues-test.apache.org/jira/browse/OOZIE-2963
OOZIE-2962  bump maven-javadoc-plugin to 2.10.4
https://issues-test.apache.org/jira/browse/OOZIE-2962
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues-test.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues-test.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues-test.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues-test.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues-test.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues-test.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues-test.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues-test.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues-test.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues-test.apache.org/jira/browse/OOZIE-2934
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues-test.apache.org/jira/browse/OOZIE-2927
OOZIE-2916  Set a job name for the MR Action's child job
https://issues-test.apache.org/jira/browse/OOZIE-2916
OOZIE-2914  Consolidate Trim 
https://issues-test.apache.org/jira/browse/OOZIE-2914
OOZIE-2909  LauncherAM: rewrite UGI calls
https://issues-test.apache.org/jira/browse/OOZIE-2909
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues-test.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues-test.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues-test.apache.org/jira/browse/OOZIE-2867
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues-test.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues-test.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues-test.apache.org/jira/browse/OOZIE-2829
OOZIE-2826  Falcon feed fails to aws s3; Oozie joda time version does not meet 
required jar version 2.2 or later
https://issues-test.apache.org/jira/browse/OOZIE-2826
OOZIE-2812  SparkConfigurationService