[jira] [Commented] (OOZIE-2687) Create XML schema for launcher configurations

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16159438#comment-16159438
 ] 

Rohini Palaniswamy commented on OOZIE-2687:
---

Haven't gotten time to look into this jira in detail. I just have one feedback. 
Keep in mind backward compatibility. Users should be able to upgrade to the new 
schema without their existing code breaking and having to fix it. 

> Create XML schema for launcher configurations
> -
>
> Key: OOZIE-2687
> URL: https://issues.apache.org/jira/browse/OOZIE-2687
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
>Priority: Critical
> Attachments: OOZIE-2687-001.patch, OOZIE-2687-002.patch, 
> OOZIE-2687-003.patch, OOZIE-2687-008.patch
>
>
> It was possible to manipulate the MapperLauncher's environment through 
> properties like:
> # mapreduce.map.memory.mb 
> # mapreduce.map.cpu.vcores
> # mapred.child.env
> # mapred.child.java.opts  
> # mapred.job.queue.name  - ability to set launcher queue
> E.g. We were using mapred.child.env to pass SPARK_HOME to the LauncherMapper 
> and make PySpark work. 
> Fixing OOZIE-2596 added a hack. We should decide how we support or break 
> compatibility and how we allow the manipulation of the Launcher environment.
> Verify if the new launcher section in global applies to actions in 
> sub-workflows as well. It did not use to work before and was only fixed in 
> OOZIE-2030. It would be good to have that testcase 
> (TestSubWorkflowActionExecutor. testParentGlobalConf) updated with the new 
> launcher section as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls

[Peter Cseh]

> On Sept. 7, 2017, 10:13 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/ErrorCode.java
> > Lines 192-193 (patched)
> > 
> >
> > In both of these, won''t should be won't

Actually, org.apache.oozie.TestErrorCode. testEscapedSingleQuotes() will fail 
if I write won't here.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review184891
---


On Sept. 6, 2017, 2:25 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> ---
> 
> (Updated Sept. 6, 2017, 2:25 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and 
> Rohini Palaniswamy.
> 
> 
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Changing UGI calls and cleaning up things around requesting tokens.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/ErrorCode.java 
> 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  5ca8d3e80942840a5bba51a06694350929044042 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 9804c7b673a899cd9c778850c9a64731cc642fb3 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 4add5f14cb20ab77c755b81eff939335c0c5fddc 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> 0b495f75842b0033a3337ef6728a334bf5651770 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> bca79aa052521ea4f6f16e76bd69f84fb16be790 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
> 187cee2290ba4bbe358913d28a052d9c75f2369a 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 
> 960c2f9750062c98fdf7a2b456d3668069e07ca5 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java
>  b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java
>  874d371664ced3ae19a7b5d615c67d4c72e7cfdd 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java 
> bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java
>  68c0f4bafe0322ff0d1007297cd6e8f38c92020a 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java
>  37af3dd9ed43f4504ff3909484237e2d05665c6c 
> 
> 
> Diff: https://reviews.apache.org/r/61560/diff/3/
> 
> 
> Testing
> ---
> 
> We're running tests on a Kerberized cluster with these changes.
> 
> 
> Thanks,
> 
> Peter Cseh
> 
>

[jira] [Commented] (OOZIE-2916) Set a job name for the MR Action's child job

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16159167#comment-16159167
 ] 

Peter Cseh commented on OOZIE-2916:
---

Some ideas:
# Can you rename jobName in JavaActionExecutor? Job is for MR jobs. It's fine 
to reference it as JobName in MapReduceAE, but let's call it appName or 
something similar in JavaAE
# please extract "oozie.action.name" as a constant
# can you add a test for this?

> Set a job name for the MR Action's child job
> 
>
> Key: OOZIE-2916
> URL: https://issues.apache.org/jira/browse/OOZIE-2916
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action
>Affects Versions: 5.0.0
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Fix For: 5.0.0
>
> Attachments: OOZIE-2889-001.patch, OOZIE-2916-002.patch
>
>
> The MR Action's child job (i.e. the MR Job) doesn't have a name.  It always 
> shows up as "N/A".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2916) Set a job name for the MR Action's child job

[Attila Sasvari (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2916:
--
Attachment: OOZIE-2916-002.patch

> Set a job name for the MR Action's child job
> 
>
> Key: OOZIE-2916
> URL: https://issues.apache.org/jira/browse/OOZIE-2916
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action
>Affects Versions: 5.0.0
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Fix For: 5.0.0
>
> Attachments: OOZIE-2889-001.patch, OOZIE-2916-002.patch
>
>
> The MR Action's child job (i.e. the MR Job) doesn't have a name.  It always 
> shows up as "N/A".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (OOZIE-3050) Parse Error if the spark job has the spark.executor.extraClassPath confguration.

[Peter Cseh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh resolved OOZIE-3050.
---
Resolution: Duplicate

> Parse Error if the spark job has the spark.executor.extraClassPath 
> confguration.
> 
>
> Key: OOZIE-3050
> URL: https://issues.apache.org/jira/browse/OOZIE-3050
> Project: Oozie
>  Issue Type: Bug
>  Components: client, core, workflow
>Affects Versions: 4.1.0
>Reporter: Tang Yan
>
> 1. create a spark job workflow in Oozie
> 2. add --conf spark.executor.extraClassPath=/XXX/XXX in job confuration
> 3. submit the job
> Results:
> the Oozie workflow failed and the error log is as follows. it has an addtion 
> error "--conf".
>  {color:red}--conf{color}
>  --conf
>  spark.executor.extraClassPath=aaa:$PWD/*
>  --conf
>  spark.driver.extraClassPath=$PWD/*
> Error: Spark config without '=': --conf
> Run with --help for usage help or --verbose for debug output
> Intercepting System.exit(1)
> Failing Oozie Launcher, Main class 
> [org.apache.oozie.action.hadoop.SparkMain], exit code [1]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2547) Add mapreduce.job.cache.files to spark action

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158577#comment-16158577
 ] 

Peter Cseh commented on OOZIE-2547:
---

Without the launcher logs it's hard to say anything. The proper files should be 
mentioned in the --files section. However, 5.10.0 was affected by OOZIE-2802 
and OOZIE-2806 which was both fixed in CDH 5.10.1. [~szhemzhitsky], can you 
provide more information about your workflow? (possibly the  part) 
please  open an other jira for this though.

> Add mapreduce.job.cache.files to spark action
> -
>
> Key: OOZIE-2547
> URL: https://issues.apache.org/jira/browse/OOZIE-2547
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Minor
> Fix For: 4.3.0
>
> Attachments: OOZIE-2547-1.patch, OOZIE-2547-4.patch, 
> OOZIE-2547-5.patch, yarn-cluster_launcher.txt
>
>
> Currently, we pass jars using --jars option while submitting spark job. Also, 
> we add spark.yarn.dist.files option in case of yarn-client mode. 
> Instead of that, we can have only --files option and pass on the files which 
> are present in mapreduce.job.cache.files. While doing so, we make sure that 
> spark won't make another copy of the files if files exist on the hdfs. We saw 
> the issues when files are getting copied multiple times and causing 
> exceptions such as :
> {code}
> Diagnostics: Resource 
> hdfs://localhost/user/saley/.sparkStaging/application_1234_123/oozie-examples.jar
>  changed on src filesystem
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2687) Create XML schema for launcher configurations

[Attila Sasvari (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158472#comment-16158472
 ] 

Attila Sasvari commented on OOZIE-2687:
---

I updated new schemas accordingly (using 1.0 as version number). Unfortunately 
I could not find a easy/flexible way that allows to define and import "common" 
schema without namespace prefixes ( {{}} is too permissive, it would allow to use any kind of 
entities). Right now I set targetNamespace of {{oozie-common-1.0}} to 
{{"uri:oozie:workflow:1.0"}} and use a reference to the launcher element with 
{{}} in 
workflow action schemas. This way we can define a launcher element in the 
following way:

{code}
...

${jobTracker}
${nameNode}

1024

...
{code}

I did not move CONFIGURATION, PREPARE, DELETE, and MKDIR into oozie-common. 
Doing so would require workflows to define add an extra name space definition 
for the element e.g. {{}}. I don't 
think we want to do this. 

Do you have some ideas that might help here?

> Create XML schema for launcher configurations
> -
>
> Key: OOZIE-2687
> URL: https://issues.apache.org/jira/browse/OOZIE-2687
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
>Priority: Critical
> Attachments: OOZIE-2687-001.patch, OOZIE-2687-002.patch, 
> OOZIE-2687-003.patch, OOZIE-2687-008.patch
>
>
> It was possible to manipulate the MapperLauncher's environment through 
> properties like:
> # mapreduce.map.memory.mb 
> # mapreduce.map.cpu.vcores
> # mapred.child.env
> # mapred.child.java.opts  
> # mapred.job.queue.name  - ability to set launcher queue
> E.g. We were using mapred.child.env to pass SPARK_HOME to the LauncherMapper 
> and make PySpark work. 
> Fixing OOZIE-2596 added a hack. We should decide how we support or break 
> compatibility and how we allow the manipulation of the Launcher environment.
> Verify if the new launcher section in global applies to actions in 
> sub-workflows as well. It did not use to work before and was only fixed in 
> OOZIE-2030. It would be good to have that testcase 
> (TestSubWorkflowActionExecutor. testParentGlobalConf) updated with the new 
> launcher section as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2687) Create XML schema for launcher configurations

[Attila Sasvari (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2687:
--
Attachment: OOZIE-2687-008.patch

> Create XML schema for launcher configurations
> -
>
> Key: OOZIE-2687
> URL: https://issues.apache.org/jira/browse/OOZIE-2687
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
>Priority: Critical
> Attachments: OOZIE-2687-001.patch, OOZIE-2687-002.patch, 
> OOZIE-2687-003.patch, OOZIE-2687-008.patch
>
>
> It was possible to manipulate the MapperLauncher's environment through 
> properties like:
> # mapreduce.map.memory.mb 
> # mapreduce.map.cpu.vcores
> # mapred.child.env
> # mapred.child.java.opts  
> # mapred.job.queue.name  - ability to set launcher queue
> E.g. We were using mapred.child.env to pass SPARK_HOME to the LauncherMapper 
> and make PySpark work. 
> Fixing OOZIE-2596 added a hack. We should decide how we support or break 
> compatibility and how we allow the manipulation of the Launcher environment.
> Verify if the new launcher section in global applies to actions in 
> sub-workflows as well. It did not use to work before and was only fixed in 
> OOZIE-2030. It would be good to have that testcase 
> (TestSubWorkflowActionExecutor. testParentGlobalConf) updated with the new 
> launcher section as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher configurations

[Attila Sasvari]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/
---

(Updated Sept. 8, 2017, 10:04 a.m.)


Review request for oozie and Peter Cseh.


Changes
---

using ``ref`` inside workflow actions to reference launcher element


Repository: oozie-git


Description
---

New workflow xsd to allow changing launcher configuration (e.g. setting java 
system properties)


Diffs (updated)
-

  client/src/main/resources/hive-action-1.0.xsd PRE-CREATION 
  client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION 
  client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION 
  client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION 
  client/src/main/resources/shell-action-1.0.xsd PRE-CREATION 
  client/src/main/resources/spark-action-1.0.xsd PRE-CREATION 
  client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
bca79aa052521ea4f6f16e76bd69f84fb16be790 
  core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
8fdc50cce15271c13b20d1aaab2ffb95cb8fa711 
  core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 
7e3348550e2ef44ae4bd6c5a7a417052bf9c108a 
  core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 
47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7 
  core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java 
ffc29af2f834da6d0890ea7215c5a62fd7cd693e 
  core/src/main/java/org/apache/oozie/service/SchemaService.java 
137e2c0b4840c6d3858683db1aa38f54bd55be92 
  core/src/main/java/org/apache/oozie/util/WritableUtils.java 
aa027e37ba23d4f481698e0bcd93c26c763a0b1f 
  core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java 
a74e5c759fe3e336e7e98c61b8f5ac52efa4100c 
  core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 
23df0867e5984c518e8424d79365997bec28f619 
  core/src/main/resources/oozie-default.xml 
0d174b5069aff36ca3a9d48722c5bba69623a046 
  core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 
d74160a09ee237f4067e00ded526e7ec94528a6b 
  core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 
6cee7a8948ca7f2c7aaf44d4f172d18a1308a482 
  core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
ce674adf9cffa20690083e6298875a6f9047a109 
  core/src/test/java/org/apache/oozie/service/TestSchemaService.java 
88a10dbead32879334c3953a90a411a3a18fc0b3 
  core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java 
a882c82b25154338f89e8a90f831b6f0300fbfe7 
  
core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java
 21332404bea79fe0a3cea045d57b3e52a1f15cae 
  core/src/test/resources/wf-schema-global-launcherconf-override.xml 
PRE-CREATION 
  core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
  sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java 
ff999ff2cd4875fbf3ad5e1a7eacf9972e848630 


Diff: https://reviews.apache.org/r/61529/diff/8/

Changes: https://reviews.apache.org/r/61529/diff/7-8/


Testing
---

- TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
- java-main example on pseudo hadoop succeeded, modified workflow xml so that 
verbose class loading information is printed out of stdout of the launcher job


Thanks,

Attila Sasvari

Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls

[Peter Bacsko]

> On szept. 6, 2017, 4:29 du, András Piros wrote:
> > core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
> > Lines 558-570 (patched)
> > 
> >
> > I'd extract those three methods, and this one, to a new service called 
> > `HadoopTokenService`. It's best readable and tested then.
> 
> Peter Cseh wrote:
> Will look into this. I don't think I will turn it into an actual Oozie 
> Service. I don't really thing there is much I can test here at the Unit level.

It's not necessarily a bad idea, although the code path we have here is fairly 
obvious. Still, it's good to have something that gives us some coverage.

I think we can create a class like DelegationTokenRetriever, which receives 
YarnClient, JHS proxy (or YarnRPC) and FS in the constructor. Then it's easy to 
unit test it.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review184696
---


On szept. 6, 2017, 2:25 du, Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> ---
> 
> (Updated szept. 6, 2017, 2:25 du)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and 
> Rohini Palaniswamy.
> 
> 
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Changing UGI calls and cleaning up things around requesting tokens.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/ErrorCode.java 
> 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  5ca8d3e80942840a5bba51a06694350929044042 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 9804c7b673a899cd9c778850c9a64731cc642fb3 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 4add5f14cb20ab77c755b81eff939335c0c5fddc 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> 0b495f75842b0033a3337ef6728a334bf5651770 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> bca79aa052521ea4f6f16e76bd69f84fb16be790 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
> 187cee2290ba4bbe358913d28a052d9c75f2369a 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 
> 960c2f9750062c98fdf7a2b456d3668069e07ca5 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java
>  b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java
>  874d371664ced3ae19a7b5d615c67d4c72e7cfdd 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java 
> bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java
>  68c0f4bafe0322ff0d1007297cd6e8f38c92020a 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java
>  37af3dd9ed43f4504ff3909484237e2d05665c6c 
> 
> 
> Diff: https://reviews.apache.org/r/61560/diff/3/
> 
> 
> Testing
> ---
> 
> We're running tests on a Kerberized cluster with these changes.
> 
> 
> Thanks,
> 
> Peter Cseh
> 
>

Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls

[Peter Cseh]

> On Sept. 6, 2017, 4:29 p.m., András Piros wrote:
> > core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
> > Lines 558-570 (patched)
> > 
> >
> > I'd extract those three methods, and this one, to a new service called 
> > `HadoopTokenService`. It's best readable and tested then.

Will look into this. I don't think I will turn it into an actual Oozie Service. 
I don't really thing there is much I can test here at the Unit level.


> On Sept. 6, 2017, 4:29 p.m., András Piros wrote:
> > core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
> > Lines 584-594 (patched)
> > 
> >
> > Getting the `Text`, and calling `credentials.addToken()` can be 
> > refactored to a common class, and to get the token for the `Text` can be 
> > done within three different `Callable` calls for the three methods.

I don't see a way to do this while increasing readability.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review184696
---


On Sept. 6, 2017, 2:25 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> ---
> 
> (Updated Sept. 6, 2017, 2:25 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and 
> Rohini Palaniswamy.
> 
> 
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Changing UGI calls and cleaning up things around requesting tokens.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/ErrorCode.java 
> 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  5ca8d3e80942840a5bba51a06694350929044042 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 9804c7b673a899cd9c778850c9a64731cc642fb3 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 4add5f14cb20ab77c755b81eff939335c0c5fddc 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> 0b495f75842b0033a3337ef6728a334bf5651770 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> bca79aa052521ea4f6f16e76bd69f84fb16be790 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
> 187cee2290ba4bbe358913d28a052d9c75f2369a 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 
> 960c2f9750062c98fdf7a2b456d3668069e07ca5 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java
>  b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java
>  874d371664ced3ae19a7b5d615c67d4c72e7cfdd 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java 
> bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java
>  68c0f4bafe0322ff0d1007297cd6e8f38c92020a 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java
>  37af3dd9ed43f4504ff3909484237e2d05665c6c 
> 
> 
> Diff: https://reviews.apache.org/r/61560/diff/3/
> 
> 
> Testing
> ---
> 
> We're running tests on a Kerberized cluster with these changes.
> 
> 
> Thanks,
> 
> Peter Cseh
> 
>

[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (107 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3051  Replace Oozie's custom precommit build patch finding job with the 
standard Apache one
https://issues.apache.org/jira/browse/OOZIE-3051
OOZIE-3041  TestWorkflowActionRetryInfoXCommand fails in oozie core module.
https://issues.apache.org/jira/browse/OOZIE-3041
OOZIE-3031  Coord job with only unresolved dependencies doesn't timeout
https://issues.apache.org/jira/browse/OOZIE-3031
OOZIE-3022  fix for warning has no file and won't be listed in dependency files 
details
https://issues.apache.org/jira/browse/OOZIE-3022
OOZIE-3002  address findbugs errors in client lib
https://issues.apache.org/jira/browse/OOZIE-3002
OOZIE-3001  core library has many instances of warnings with trailing spaces 
and lines longer than 132 chars
https://issues.apache.org/jira/browse/OOZIE-3001
OOZIE-2997  files contain trailing white spaces in client lib
https://issues.apache.org/jira/browse/OOZIE-2997
OOZIE-2996  add option for -UseGCOverheadLimit to maven opts as sometimes local 
testing fails
https://issues.apache.org/jira/browse/OOZIE-2996
OOZIE-2978  Remove code that handles Pig versions before 0.8 
https://issues.apache.org/jira/browse/OOZIE-2978
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues.apache.org/jira/browse/OOZIE-2975
OOZIE-2969  Drop support for Java 1.7
https://issues.apache.org/jira/browse/OOZIE-2969
OOZIE-2964  Add -Xdoclint:none to javadoc opts to avoid warnings
https://issues.apache.org/jira/browse/OOZIE-2964
OOZIE-2963  getting error in build ArtifactNotFoundException: Could not find 
artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde
https://issues.apache.org/jira/browse/OOZIE-2963
OOZIE-2962  bump maven-javadoc-plugin to 2.10.4
https://issues.apache.org/jira/browse/OOZIE-2962
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues.apache.org/jira/browse/OOZIE-2934
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2916  Set a job name for the MR Action's child job
https://issues.apache.org/jira/browse/OOZIE-2916
OOZIE-2914  Consolidate Trim 
https://issues.apache.org/jira/browse/OOZIE-2914
OOZIE-2909  LauncherAM: rewrite UGI calls
https://issues.apache.org/jira/browse/OOZIE-2909
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues.apache.org/jira/browse/OOZIE-2867
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues.apache.org/jira/browse/OOZIE-2829
OOZIE-2826  Falcon feed fails to aws s3; Oozie joda time version does not meet 
required jar version 2.2 or later
https://issues.apache.org/jira/browse/OOZIE-2826
OOZIE-2812  SparkConfigurationService should