date:20180605

[jira] [Commented] (OOZIE-3260) [sla] Remove stale item above max retries on JPA related errors from in-memory SLA map

2018-06-05 Thread Peter Cseh (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502495#comment-16502495
 ] 

Peter Cseh commented on OOZIE-3260:
---

The patch looks good in general. There is a star import in it, please fix that.
Are there any guaranties that these entries are not coming back again?

> [sla] Remove stale item above max retries on JPA related errors from 
> in-memory SLA map
> --
>
> Key: OOZIE-3260
> URL: https://issues.apache.org/jira/browse/OOZIE-3260
> Project: Oozie
>  Issue Type: Bug
>  Components: coordinator, core, workflow
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3260.001.patch
>
>
> Despite having implemented OOZIE-3134, there are still cases where 
> {{SLACalculatorMemory#slaMap}} and database contents still get out of sync. 
> Some possibilities including but not limited to:
>  * database contents of {{SLA_SUMMARY}} table have been purged manually from 
> DB
>  * no corresponding {{WF_JOBS}} or {{COORD_JOBS}} entries exist anymore in DB
>  * the {{WF_JOBS}} or {{COORD_JOBS}} instance that is being tracked by the 
> {{SLACalcStatus}} instances inside {{SLACalculatorMemory#slaMap}} is not yet 
> persisted to database when the SLA entry is already processed by 
> {{SLACalculatorMemory.HistoryPurgeWorker}}. Depending on e.g. how many 
> coordinator actions are being materialized, it can very well happen that 
> {{SLACalcStatus}} entries inserted to the in-memory map will be processed 
> before their corresponding {{CoordActionBean}} entries are yet to be 
> persisted to database
> In those rare cases, we see {{JPAExecutorException}} instances like:
> {noformat}
> 2017-10-09 17:00:18,185 DEBUG openjpa.jdbc.SQL: SERVER[HOST]  conn 1584126245> [0 ms] spent
> 2017-10-09 17:00:18,185 ERROR org.apache.oozie.sla.SLACalculatorMemory: 
> SERVER[tplhc01c001.iuser.iroot.adidom.com] USER[-] GROUP[-] TOKEN[-] APP[-] 
> JOB[438-170916014916144-oozie-oozi-C@556] ACTION[-] Exception in SLA 
> processing for job [438-170916014916144-oozie-oozi-C@556]
> org.apache.oozie.executor.jpa.JPAExecutorException: E0604: Job does not exist 
> [select w.eventProcessed from SLASummaryBean w where w.jobId = :id]
> at 
> org.apache.oozie.executor.jpa.SLASummaryQueryExecutor.getSingleValue(SLASummaryQueryExecutor.java:161)
> at 
> org.apache.oozie.sla.SLACalculatorMemory.updateJobSla(SLACalculatorMemory.java:480)
> at 
> org.apache.oozie.sla.SLACalculatorMemory.updateAllSlaStatus(SLACalculatorMemory.java:601)
> {noformat}
> Solution here is to track the number of times the {{SLACalcStatus}} entry has 
> not been processed successfully, and when a preconfigured 
> {{oozie.sla.service.SLAService.maximum.retry.count}} is reached, remove any 
> {{SLACalculatorMemory#slaMap}} entries that are causing those 
> {{JPAExecutorException}} instances, to not cause huge logfiles. The items to 
> be logged don't exist, anyways.
> It's still possible that multiple {{CoordActionBean}} instances being 
> inserted won't have {{SLACalcStatus}} entries inside 
> {{SLACalculatorMemory#slaMap}} by the time written to database, and thus, no 
> SLA will be tracked. In those rare cases, preconfigured maximum retry count 
> can be extended.
> Note that current implementation of 
> [*{{SLACalculatorMemory#updateJobSla()}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L238-L242]
>  already removes the stale {{SLACalcStatus}} entry. The new functionality 
> here is to introduce {{SLACalcStatus#retryCount}}, and extend the 
> {{JPAExecutorException}} {{ErrorCode}}s of interest.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66930: OOZIE-2791 ShareLib installation may fail on busy Hadoop clusters

2018-06-05 Thread Peter Cseh via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66930/#review204339
---




tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
Lines 252 (patched)


Why is a delete call here? Is it to clean up for retries in case the 
previous attempt created the file but failed during upload?


- Peter Cseh


On June 5, 2018, 8:37 a.m., Kinga Marton wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66930/
> ---
> 
> (Updated June 5, 2018, 8:37 a.m.)
> 
> 
> Review request for oozie, András Piros and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> On a busy Hadoop cluster it can happen that users cannot install properly 
> Oozie ShareLib.
> 
> 
> Diffs
> -
> 
>   tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java dce1c559 
>   tools/src/test/java/org/apache/oozie/tools/TestBlockSizeCalculator.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestConcurrentCopyFromLocal.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestCopyTaskCallable.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java 
> ccad273b 
> 
> 
> Diff: https://reviews.apache.org/r/66930/diff/4/
> 
> 
> Testing
> ---
> 
> Tested manually
> 
> 
> Thanks,
> 
> Kinga Marton
> 
>

Re: Review Request 66930: OOZIE-2791 ShareLib installation may fail on busy Hadoop clusters

2018-06-05 Thread Peter Cseh via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66930/#review204336
---




tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
Line 254 (original), 424 (patched)


We could check for failedCopyTasks.isEmpty() here, can't we?



tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
Lines 429 (patched)


the "t" in the message might be different here than the original "t" that 
was null-checked. Please use an error from the failed task list.


- Peter Cseh


On June 5, 2018, 8:37 a.m., Kinga Marton wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66930/
> ---
> 
> (Updated June 5, 2018, 8:37 a.m.)
> 
> 
> Review request for oozie, András Piros and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> On a busy Hadoop cluster it can happen that users cannot install properly 
> Oozie ShareLib.
> 
> 
> Diffs
> -
> 
>   tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java dce1c559 
>   tools/src/test/java/org/apache/oozie/tools/TestBlockSizeCalculator.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestConcurrentCopyFromLocal.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestCopyTaskCallable.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java 
> ccad273b 
> 
> 
> Diff: https://reviews.apache.org/r/66930/diff/4/
> 
> 
> Testing
> ---
> 
> Tested manually
> 
> 
> Thanks,
> 
> Kinga Marton
> 
>

[jira] [Commented] (OOZIE-2877) Oozie Git Action

2018-06-05 Thread Robert Kanter (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502088#comment-16502088
 ] 

Robert Kanter commented on OOZIE-2877:
--

I also took a quick look at the patch and left a few minor comments on 
ReviewBoard.

Overall, though; this is really good!

> Oozie Git Action
> 
>
> Key: OOZIE-2877
> URL: https://issues.apache.org/jira/browse/OOZIE-2877
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Major
>  Labels: action
> Fix For: trunk
>
> Attachments: 0001-OOZIE-2877-Oozie-Git-Action.patch, 
> 0002-OOZIE-2877-Oozie-Git-Action.patch, 
> 0003-OOZIE-2877-Oozie-Git-Action.patch, 
> 0004-OOZIE-2877-Oozie-Git-Action.patch, 
> 0005-OOZIE-2877-Oozie-Git-Action.patch, 
> 0006-OOZIE-2877-Oozie-Git-Action.patch, 
> 0007-OOZIE-2877-Oozie-Git-Action.patch, 
> 0008-OOZIE-2877-Oozie-Git-Action.patch, 
> 0009-OOZIE-2877-Oozie-Git-Action.patch, OOZIE-2877.010.patch, 
> OOZIE-2877.011.patch, OOZIE-2877.012.patch
>
>
> To aide in deploying ASCII artifacts to clusters, let's provide a tie-in for 
> a source-code management system. Git would be my preferred choice.
> Ideally, this could handle a user's key material e.g. for an ssh key to pull 
> down from a secured repository. This would free users from handling their own 
> key staging and clean-up on YARN nodes and only require them to store the key 
> secured in HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 59620: This review board request is for an action to provide a Git action for Oozie

2018-06-05 Thread Robert Kanter via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59620/#review204323
---




examples/src/main/apps/git/job.properties
Lines 9 (patched)


trailing whitespace; there's some other cases here and in other files too



sharelib/git/pom.xml
Lines 107-118 (patched)


This can be removed (It was needed some time ago in each sharelib pom file, 
but not anymore).



sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitMain.java
Lines 216 (patched)


Is this necessary?  Why not simply throw whatever exception (IOException, 
IllegalArgumentException, etc)?  We don't seem to be doing any special handling 
or anything for GitMainExceptions.



sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestGitActionExecutor.java
Lines 33 (patched)


"is" --> "in"



sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestGitActionExecutor.java
Lines 36 (patched)


As Peter Bacsko pointed out on the JIRA, we shouldn't be fetching a remote 
repo during a unit test. Perhaps JGit has some easy way of mocking this or 
doing it locally?


- Robert Kanter


On June 4, 2018, 5:44 a.m., Clay B. wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59620/
> ---
> 
> (Updated June 4, 2018, 5:44 a.m.)
> 
> 
> Review request for oozie and András Piros.
> 
> 
> Bugs: OOZIE-2877
> https://issues.apache.org/jira/browse/OOZIE-2877
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2877 - Oozie Git Action
> 
> 
> Diffs
> -
> 
>   client/src/main/resources/git-action-1.0.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/GitActionExecutor.java 
> PRE-CREATION 
>   core/src/main/resources/oozie-default.xml c54db34 
>   docs/src/site/twiki/WorkflowFunctionalSpec.twiki 76cbe21 
>   examples/src/main/apps/git/job.properties PRE-CREATION 
>   examples/src/main/apps/git/workflow.xml PRE-CREATION 
>   pom.xml 6f35868 
>   sharelib/git/pom.xml PRE-CREATION 
>   sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitMain.java 
> PRE-CREATION 
>   
> sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitOperations.java 
> PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestGitActionExecutor.java
>  PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestGitMainGetKey.java
>  PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestIntegrationGitActionExecutor.java
>  PRE-CREATION 
>   sharelib/pom.xml 6a0864d 
>   src/main/assemblies/sharelib.xml 07dc69c 
>   webapp/pom.xml 67526d9 
> 
> 
> Diff: https://reviews.apache.org/r/59620/diff/6/
> 
> 
> Testing
> ---
> 
> Tested using unit and integration tests. Still need to:
> * Test on a cluster
> * Test with an authenticated SSH hosted Git repo
> 
> Sumitted a request to the JGit community as their branch pulling code seems 
> to have an 
> [issue](https://dev.eclipse.org/mhonarc/lists/jgit-dev/msg03343.html).
> 
> Andras has tested against his patch 011 with results at 
> https://issues.apache.org/jira/browse/OOZIE-2877?focusedCommentId=16459523=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16459523
> 
> 
> File Attachments
> 
> 
> 0001-OOZIE-2877-Oozie-Git-Action.patch
>   
> https://reviews.apache.org/media/uploaded/files/2017/05/29/24f90a78-3dc1-49fe-bf29-5927a3cd5e72__0001-OOZIE-2877-Oozie-Git-Action.patch
> Patch
>   
> https://reviews.apache.org/media/uploaded/files/2017/05/29/dd23dd72-67e0-456f-9b52-e566d8d17d16__0001-OOZIE-2877-Oozie-Git-Action.patch
> 
> 
> Thanks,
> 
> Clay B.
> 
>

[jira] [Commented] (OOZIE-2877) Oozie Git Action

2018-06-05 Thread Robert Kanter (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502065#comment-16502065
 ] 

Robert Kanter commented on OOZIE-2877:
--

Makes sense to me then.  Let's stick with the original credential handling.

[~pbacsko] makes a good point.  We shouldn't be fetching a remote repo during a 
unit test.  Perhaps JGit has some easy way of mocking this or doing it locally?

> Oozie Git Action
> 
>
> Key: OOZIE-2877
> URL: https://issues.apache.org/jira/browse/OOZIE-2877
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Major
>  Labels: action
> Fix For: trunk
>
> Attachments: 0001-OOZIE-2877-Oozie-Git-Action.patch, 
> 0002-OOZIE-2877-Oozie-Git-Action.patch, 
> 0003-OOZIE-2877-Oozie-Git-Action.patch, 
> 0004-OOZIE-2877-Oozie-Git-Action.patch, 
> 0005-OOZIE-2877-Oozie-Git-Action.patch, 
> 0006-OOZIE-2877-Oozie-Git-Action.patch, 
> 0007-OOZIE-2877-Oozie-Git-Action.patch, 
> 0008-OOZIE-2877-Oozie-Git-Action.patch, 
> 0009-OOZIE-2877-Oozie-Git-Action.patch, OOZIE-2877.010.patch, 
> OOZIE-2877.011.patch, OOZIE-2877.012.patch
>
>
> To aide in deploying ASCII artifacts to clusters, let's provide a tie-in for 
> a source-code management system. Git would be my preferred choice.
> Ideally, this could handle a user's key material e.g. for an ssh key to pull 
> down from a secured repository. This would free users from handling their own 
> key staging and clean-up on YARN nodes and only require them to store the key 
> secured in HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2877) Oozie Git Action

2018-06-05 Thread Peter Bacsko (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502011#comment-16502011
 ] 

Peter Bacsko commented on OOZIE-2877:
-

I just have a question regarding this change:

{noformat}
+final String repoUrl = "https://github.com/apache/oozie;;
+final String keyUrl = "this_would_be_an_HDFS_path";
+final String destDir = "repoDir";
+final String branch = "myBranch";
+Element actionXml = XmlUtils.parseXml("" +
+"" + getJobTrackerUri() + 
"" +
+"" + getNameNodeUri() + "" +
+"" + repoUrl + ""+
+"" + branch + ""+
+"" + keyUrl + ""+
+"" + destDir + "" +
+"");
{noformat}

Do I understand correctly that this would fetch a repository during the 
integration test? If so, I'm really not sure that this is a good idea. Can't we 
do it differently? I have two ideas:

1. Separately test {{GitMain}} with unit tests and somehow mock it during 
{{TestGitActionExecutor}}
2. Start our own SSH server and simulate the execution of Git command

#1 seems to be easier. 


> Oozie Git Action
> 
>
> Key: OOZIE-2877
> URL: https://issues.apache.org/jira/browse/OOZIE-2877
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Major
>  Labels: action
> Fix For: trunk
>
> Attachments: 0001-OOZIE-2877-Oozie-Git-Action.patch, 
> 0002-OOZIE-2877-Oozie-Git-Action.patch, 
> 0003-OOZIE-2877-Oozie-Git-Action.patch, 
> 0004-OOZIE-2877-Oozie-Git-Action.patch, 
> 0005-OOZIE-2877-Oozie-Git-Action.patch, 
> 0006-OOZIE-2877-Oozie-Git-Action.patch, 
> 0007-OOZIE-2877-Oozie-Git-Action.patch, 
> 0008-OOZIE-2877-Oozie-Git-Action.patch, 
> 0009-OOZIE-2877-Oozie-Git-Action.patch, OOZIE-2877.010.patch, 
> OOZIE-2877.011.patch, OOZIE-2877.012.patch
>
>
> To aide in deploying ASCII artifacts to clusters, let's provide a tie-in for 
> a source-code management system. Git would be my preferred choice.
> Ideally, this could handle a user's key material e.g. for an ssh key to pull 
> down from a secured repository. This would free users from handling their own 
> key staging and clean-up on YARN nodes and only require them to store the key 
> secured in HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2877) Oozie Git Action

2018-06-05 Thread Andras Piros (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501891#comment-16501891
 ] 

Andras Piros commented on OOZIE-2877:
-

[~clayb] I see. It's a good idea to stick to the original request / solution 
and not use YARN's {{ApplicationSubmissionContext}}.

I'd choose {{GitActionExecutor}} to perform the private SSH key permission 
check, and in case permission is too vague, {{LOG.error()}} and not event start 
the Git action. The user can then restrict the permission and re-run the 
workflow containing the SSH action.

> Oozie Git Action
> 
>
> Key: OOZIE-2877
> URL: https://issues.apache.org/jira/browse/OOZIE-2877
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Major
>  Labels: action
> Fix For: trunk
>
> Attachments: 0001-OOZIE-2877-Oozie-Git-Action.patch, 
> 0002-OOZIE-2877-Oozie-Git-Action.patch, 
> 0003-OOZIE-2877-Oozie-Git-Action.patch, 
> 0004-OOZIE-2877-Oozie-Git-Action.patch, 
> 0005-OOZIE-2877-Oozie-Git-Action.patch, 
> 0006-OOZIE-2877-Oozie-Git-Action.patch, 
> 0007-OOZIE-2877-Oozie-Git-Action.patch, 
> 0008-OOZIE-2877-Oozie-Git-Action.patch, 
> 0009-OOZIE-2877-Oozie-Git-Action.patch, OOZIE-2877.010.patch, 
> OOZIE-2877.011.patch, OOZIE-2877.012.patch
>
>
> To aide in deploying ASCII artifacts to clusters, let's provide a tie-in for 
> a source-code management system. Git would be my preferred choice.
> Ideally, this could handle a user's key material e.g. for an ssh key to pull 
> down from a secured repository. This would free users from handling their own 
> key staging and clean-up on YARN nodes and only require them to store the key 
> secured in HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OOZIE-2877) Oozie Git Action

2018-06-05 Thread Clay B. (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501869#comment-16501869
 ] 

Clay B. edited comment on OOZIE-2877 at 6/5/18 2:34 PM:


My understanding was [~gezapeti]'s original request was to ensure the 
credential as stored on HDFS were secure; and if it was not, then we should 
warn (or error) to the user – the same as SSH does for your private SSH key 
(e.g. see [Stack Overflow 
#9270734|[https://stackoverflow.com/questions/9270734/ssh-permissions-are-too-open-error]|https://stackoverflow.com/questions/9270734/ssh-permissions-are-too-open-error]).

In the case of YARN Credentials, how would one provide that to the Oozie job 
for on-going use? Does the Oozie REST API support some notion of the YARN 
credentials or is this only at job launch and {{git clone}} time? I would 
expect if the later, then it could be useful but I do not expect more security 
unless run on an insecure cluster where all jobs are run as the same user (e.g. 
{{yarn}} instead of the user submitting the job) or where local-filesystem 
permissions are otherwise non-standard for a secure deployment?


was (Author: clayb):
My understanding was [~gezapeti]'s original request was to ensure the 
credential as stored on HDFS were secure; and if it was not, then we should 
warn (or error) to the user – the same as SSH does for your private SSH key 
(e.g. see [Stack Overflow 
#9270734|[https://stackoverflow.com/questions/9270734/ssh-permissions-are-too-open-error]).]]).

In the case of YARN Credentials, how would one provide that to the Oozie job 
for on-going use? Does the Oozie REST API support some notion of the YARN 
credentials or is this only at job launch and {{git clone}} time? I would 
expect if the later, then it could be useful but I do not expect more security 
unless run on an insecure cluster where all jobs are run as the same user (e.g. 
{{yarn}} instead of the user submitting the job) or where local-filesystem 
permissions are otherwise non-standard for a secure deployment?

> Oozie Git Action
> 
>
> Key: OOZIE-2877
> URL: https://issues.apache.org/jira/browse/OOZIE-2877
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Major
>  Labels: action
> Fix For: trunk
>
> Attachments: 0001-OOZIE-2877-Oozie-Git-Action.patch, 
> 0002-OOZIE-2877-Oozie-Git-Action.patch, 
> 0003-OOZIE-2877-Oozie-Git-Action.patch, 
> 0004-OOZIE-2877-Oozie-Git-Action.patch, 
> 0005-OOZIE-2877-Oozie-Git-Action.patch, 
> 0006-OOZIE-2877-Oozie-Git-Action.patch, 
> 0007-OOZIE-2877-Oozie-Git-Action.patch, 
> 0008-OOZIE-2877-Oozie-Git-Action.patch, 
> 0009-OOZIE-2877-Oozie-Git-Action.patch, OOZIE-2877.010.patch, 
> OOZIE-2877.011.patch, OOZIE-2877.012.patch
>
>
> To aide in deploying ASCII artifacts to clusters, let's provide a tie-in for 
> a source-code management system. Git would be my preferred choice.
> Ideally, this could handle a user's key material e.g. for an ssh key to pull 
> down from a secured repository. This would free users from handling their own 
> key staging and clean-up on YARN nodes and only require them to store the key 
> secured in HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OOZIE-2877) Oozie Git Action

2018-06-05 Thread Clay B. (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501869#comment-16501869
 ] 

Clay B. edited comment on OOZIE-2877 at 6/5/18 2:34 PM:


My understanding was [~gezapeti]'s original request was to ensure the 
credential as stored on HDFS were secure; and if it was not, then we should 
warn (or error) to the user – the same as SSH does for your private SSH key 
(e.g. see [Stack Overflow 
#9270734|[https://stackoverflow.com/questions/9270734/ssh-permissions-are-too-open-error]]).

In the case of YARN Credentials, how would one provide that to the Oozie job 
for on-going use? Does the Oozie REST API support some notion of the YARN 
credentials or is this only at job launch and {{git clone}} time? I would 
expect if the later, then it could be useful but I do not expect more security 
unless run on an insecure cluster where all jobs are run as the same user (e.g. 
{{yarn}} instead of the user submitting the job) or where local-filesystem 
permissions are otherwise non-standard for a secure deployment?


was (Author: clayb):
My understanding was [~gezapeti]'s original request was to ensure the 
credential as stored on HDFS were secure; and if it was not, then we should 
warn (or error) to the user – the same as SSH does for your private SSH key 
(e.g. see [Stack Overflow 
#9270734|[https://stackoverflow.com/questions/9270734/ssh-permissions-are-too-open-error]|https://stackoverflow.com/questions/9270734/ssh-permissions-are-too-open-error]).

In the case of YARN Credentials, how would one provide that to the Oozie job 
for on-going use? Does the Oozie REST API support some notion of the YARN 
credentials or is this only at job launch and {{git clone}} time? I would 
expect if the later, then it could be useful but I do not expect more security 
unless run on an insecure cluster where all jobs are run as the same user (e.g. 
{{yarn}} instead of the user submitting the job) or where local-filesystem 
permissions are otherwise non-standard for a secure deployment?

> Oozie Git Action
> 
>
> Key: OOZIE-2877
> URL: https://issues.apache.org/jira/browse/OOZIE-2877
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Major
>  Labels: action
> Fix For: trunk
>
> Attachments: 0001-OOZIE-2877-Oozie-Git-Action.patch, 
> 0002-OOZIE-2877-Oozie-Git-Action.patch, 
> 0003-OOZIE-2877-Oozie-Git-Action.patch, 
> 0004-OOZIE-2877-Oozie-Git-Action.patch, 
> 0005-OOZIE-2877-Oozie-Git-Action.patch, 
> 0006-OOZIE-2877-Oozie-Git-Action.patch, 
> 0007-OOZIE-2877-Oozie-Git-Action.patch, 
> 0008-OOZIE-2877-Oozie-Git-Action.patch, 
> 0009-OOZIE-2877-Oozie-Git-Action.patch, OOZIE-2877.010.patch, 
> OOZIE-2877.011.patch, OOZIE-2877.012.patch
>
>
> To aide in deploying ASCII artifacts to clusters, let's provide a tie-in for 
> a source-code management system. Git would be my preferred choice.
> Ideally, this could handle a user's key material e.g. for an ssh key to pull 
> down from a secured repository. This would free users from handling their own 
> key staging and clean-up on YARN nodes and only require them to store the key 
> secured in HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2877) Oozie Git Action

2018-06-05 Thread Clay B. (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501869#comment-16501869
 ] 

Clay B. commented on OOZIE-2877:


My understanding was [~gezapeti]'s original request was to ensure the 
credential as stored on HDFS was secure; and if it was not, then we should warn 
(or error) to the user – the same as SSH does for your private SSH key (e.g. 
see [Stack Overflow 
#9270734|[https://stackoverflow.com/questions/9270734/ssh-permissions-are-too-open-error|https://stackoverflow.com/questions/9270734/ssh-permissions-are-too-open-error]).]]).

In the case of YARN Credentials, how would one provide that to the Oozie job 
for on-going use? Does the Oozie REST API support some notion of the YARN 
credentials or is this only at job launch and {{git clone}} time? I would 
expect if the later, then it could be useful but I do not expect more security 
unless run on an insecure cluster where all jobs are run as the same user (e.g. 
{{yarn}} instead of the user submitting the job) or where local-filesystem 
permissions are otherwise non-standard for a secure deployment?

> Oozie Git Action
> 
>
> Key: OOZIE-2877
> URL: https://issues.apache.org/jira/browse/OOZIE-2877
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Major
>  Labels: action
> Fix For: trunk
>
> Attachments: 0001-OOZIE-2877-Oozie-Git-Action.patch, 
> 0002-OOZIE-2877-Oozie-Git-Action.patch, 
> 0003-OOZIE-2877-Oozie-Git-Action.patch, 
> 0004-OOZIE-2877-Oozie-Git-Action.patch, 
> 0005-OOZIE-2877-Oozie-Git-Action.patch, 
> 0006-OOZIE-2877-Oozie-Git-Action.patch, 
> 0007-OOZIE-2877-Oozie-Git-Action.patch, 
> 0008-OOZIE-2877-Oozie-Git-Action.patch, 
> 0009-OOZIE-2877-Oozie-Git-Action.patch, OOZIE-2877.010.patch, 
> OOZIE-2877.011.patch, OOZIE-2877.012.patch
>
>
> To aide in deploying ASCII artifacts to clusters, let's provide a tie-in for 
> a source-code management system. Git would be my preferred choice.
> Ideally, this could handle a user's key material e.g. for an ssh key to pull 
> down from a secured repository. This would free users from handling their own 
> key staging and clean-up on YARN nodes and only require them to store the key 
> secured in HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OOZIE-2877) Oozie Git Action

2018-06-05 Thread Clay B. (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501869#comment-16501869
 ] 

Clay B. edited comment on OOZIE-2877 at 6/5/18 2:33 PM:


My understanding was [~gezapeti]'s original request was to ensure the 
credential as stored on HDFS were secure; and if it was not, then we should 
warn (or error) to the user – the same as SSH does for your private SSH key 
(e.g. see [Stack Overflow 
#9270734|[https://stackoverflow.com/questions/9270734/ssh-permissions-are-too-open-error]).]]).

In the case of YARN Credentials, how would one provide that to the Oozie job 
for on-going use? Does the Oozie REST API support some notion of the YARN 
credentials or is this only at job launch and {{git clone}} time? I would 
expect if the later, then it could be useful but I do not expect more security 
unless run on an insecure cluster where all jobs are run as the same user (e.g. 
{{yarn}} instead of the user submitting the job) or where local-filesystem 
permissions are otherwise non-standard for a secure deployment?


was (Author: clayb):
My understanding was [~gezapeti]'s original request was to ensure the 
credential as stored on HDFS was secure; and if it was not, then we should warn 
(or error) to the user – the same as SSH does for your private SSH key (e.g. 
see [Stack Overflow 
#9270734|[https://stackoverflow.com/questions/9270734/ssh-permissions-are-too-open-error|https://stackoverflow.com/questions/9270734/ssh-permissions-are-too-open-error]).]]).

In the case of YARN Credentials, how would one provide that to the Oozie job 
for on-going use? Does the Oozie REST API support some notion of the YARN 
credentials or is this only at job launch and {{git clone}} time? I would 
expect if the later, then it could be useful but I do not expect more security 
unless run on an insecure cluster where all jobs are run as the same user (e.g. 
{{yarn}} instead of the user submitting the job) or where local-filesystem 
permissions are otherwise non-standard for a secure deployment?

> Oozie Git Action
> 
>
> Key: OOZIE-2877
> URL: https://issues.apache.org/jira/browse/OOZIE-2877
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Major
>  Labels: action
> Fix For: trunk
>
> Attachments: 0001-OOZIE-2877-Oozie-Git-Action.patch, 
> 0002-OOZIE-2877-Oozie-Git-Action.patch, 
> 0003-OOZIE-2877-Oozie-Git-Action.patch, 
> 0004-OOZIE-2877-Oozie-Git-Action.patch, 
> 0005-OOZIE-2877-Oozie-Git-Action.patch, 
> 0006-OOZIE-2877-Oozie-Git-Action.patch, 
> 0007-OOZIE-2877-Oozie-Git-Action.patch, 
> 0008-OOZIE-2877-Oozie-Git-Action.patch, 
> 0009-OOZIE-2877-Oozie-Git-Action.patch, OOZIE-2877.010.patch, 
> OOZIE-2877.011.patch, OOZIE-2877.012.patch
>
>
> To aide in deploying ASCII artifacts to clusters, let's provide a tie-in for 
> a source-code management system. Git would be my preferred choice.
> Ideally, this could handle a user's key material e.g. for an ssh key to pull 
> down from a secured repository. This would free users from handling their own 
> key staging and clean-up on YARN nodes and only require them to store the key 
> secured in HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2877) Oozie Git Action

2018-06-05 Thread Andras Piros (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501786#comment-16501786
 ] 

Andras Piros commented on OOZIE-2877:
-

[~rkanter] what an interesting idea! I didn't think of that YARN feature :)

However, in this case, since the customer has to upload the Git credential file 
to HDFS in any case, I don't think we can leverage more security not using 
{{DistributedCache}} but {{ApplicationSubmissionContext}}. We could, though, if 
the user wouldn't upload Git credential file to HDFS, but would only put to the 
local filesystem only visible to Oozie on the Oozie server host.

[~clayb] [~gezapeti] what do you think, in how much of an extent do we want to 
protect the users' Git credentials?

> Oozie Git Action
> 
>
> Key: OOZIE-2877
> URL: https://issues.apache.org/jira/browse/OOZIE-2877
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Major
>  Labels: action
> Fix For: trunk
>
> Attachments: 0001-OOZIE-2877-Oozie-Git-Action.patch, 
> 0002-OOZIE-2877-Oozie-Git-Action.patch, 
> 0003-OOZIE-2877-Oozie-Git-Action.patch, 
> 0004-OOZIE-2877-Oozie-Git-Action.patch, 
> 0005-OOZIE-2877-Oozie-Git-Action.patch, 
> 0006-OOZIE-2877-Oozie-Git-Action.patch, 
> 0007-OOZIE-2877-Oozie-Git-Action.patch, 
> 0008-OOZIE-2877-Oozie-Git-Action.patch, 
> 0009-OOZIE-2877-Oozie-Git-Action.patch, OOZIE-2877.010.patch, 
> OOZIE-2877.011.patch, OOZIE-2877.012.patch
>
>
> To aide in deploying ASCII artifacts to clusters, let's provide a tie-in for 
> a source-code management system. Git would be my preferred choice.
> Ideally, this could handle a user's key material e.g. for an ssh key to pull 
> down from a secured repository. This would free users from handling their own 
> key staging and clean-up on YARN nodes and only require them to store the key 
> secured in HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

2018-06-05 Thread Andras Piros (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501752#comment-16501752
 ] 

Andras Piros commented on OOZIE-2791:
-

Thanks for the contribution [~kmarton]! Left some more comments over on RB.

> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-2791-003.patch, OOZIE-2791-004.patch, 
> OOZIE-2791-005.patch, OOZIE-2791-006.patch, OOZIE-2791-01.patch, 
> OOZIE-2791-02.patch
>
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
> --
> ...
> {code}
> You can see file is created but it's size is 0.
> {code}
> -rw-r--r--   3 asasvari supergroup  0 2017-02-07 10:59 
> share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar
> {code}
> This behaviour is clearly wrong. 
> In case of such an exception, we should retry copying or rollback changes. We 
> should also consider throttling HDFS requests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66930: OOZIE-2791 ShareLib installation may fail on busy Hadoop clusters

2018-06-05 Thread András Piros via Review Board

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66930/#review204314
---

tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
Lines 194 (patched)

Would handle `NumberFormatException` separately.

tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
Line 223 (original), 238 (patched)

Can it be `private`, and if not, is it `@VisibleForTesting`?

tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
Lines 257 (patched)

Do we surely need to override `Object#equals()`?

tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
Lines 267-272 (patched)

`fs` and `threadPool` should not be part of `equals()`.

tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
Lines 280 (patched)

Do we surely need to override `Object#hashCode()`?

tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
Lines 281-282 (patched)

`fs` and `threadPool` should not be part of `hashCode()`.

tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
Lines 290 (patched)

Can it be `private`, and if not, is it `@VisibleForTesting`?

tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
Lines 305 (patched)

Can it be `private`, and if not, is it `@VisibleForTesting`?

tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
Line 249 (original), 417 (patched)

Why use `String#format()` here?

tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
Line 252 (original), 420-422 (patched)

This should be in `finally`.

tools/src/test/java/org/apache/oozie/tools/TestBlockSizeCalculator.java
Lines 24 (patched)

I like this test case :)

tools/src/test/java/org/apache/oozie/tools/TestConcurrentCopyFromLocal.java
Lines 31 (patched)

Would rather not extend `TestOozieSharelibCLI` (because don't want to rerun 
all those tests once more), but extract the functionality needed from 
`TestOozieSharelibCLI` and would use rather composition over inheritance in 
both cases.

tools/src/test/java/org/apache/oozie/tools/TestCopyTaskCallable.java
Lines 38 (patched)

Would rather not extend `TestOozieSharelibCLI` (because don't want to rerun 
all those tests once more), but extract the functionality needed from 
`TestOozieSharelibCLI` and would use rather composition over inheritance in 
both cases.

tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java
Line 221 (original), 230-236 (patched)

Would rather extract the functionality needed from `TestOozieSharelibCLI` 
and would use the extracted class by composition over inheritance in every case.

tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java
Line 240 (original), 255 (patched)

Would rather extract the functionality needed from `TestOozieSharelibCLI` 
and would use the extracted class by composition over inheritance in every case.

tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java
Line 244 (original), 259 (patched)

Would rather extract the functionality needed from `TestOozieSharelibCLI` 
and would use the extracted class by composition over inheritance in every case.

- András Piros

On June 5, 2018, 8:37 a.m., Kinga Marton wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66930/
> ---
> 
> (Updated June 5, 2018, 8:37 a.m.)
> 
> 
> Review request for oozie, András Piros and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> On a busy Hadoop cluster it can happen that users cannot install properly 
> Oozie ShareLib.
> 
> 
> Diffs
> -
> 
>   tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java dce1c559 
>   tools/src/test/java/org/apache/oozie/tools/TestBlockSizeCalculator.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestConcurrentCopyFromLocal.java 
>

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

2018-06-05 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501699#comment-16501699
 ] 

Hadoop QA commented on OOZIE-2791:
--


Testing JIRA OOZIE-2791

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 4 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
error(s)
.{color:red}ERROR{color}: the current HEAD has 2 Javadoc error(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2159
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/606/



> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-2791-003.patch, OOZIE-2791-004.patch, 
> OOZIE-2791-005.patch, OOZIE-2791-006.patch, OOZIE-2791-01.patch, 
> OOZIE-2791-02.patch
>
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>   at 
>

Success: OOZIE-2791 PreCommit Build #606

2018-06-05 Thread Apache Jenkins Server

Jira: https://issues.apache.org/jira/browse/OOZIE-2791
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/606/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.71 MB...]
[DEBUG] There are no new bugs found in [sharelib/pig].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/streaming].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hive].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hcatalog].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/sqoop].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/oozie].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/distcp].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/spark].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [client].
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2365 bytes
[TRACE] Full summary file size is 1314 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO


Testing JIRA OOZIE-2791

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 4 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warning(s)
+1 the patch does not seem to introduce new Javadoc error(s)
ERROR: the current HEAD has 2 Javadoc error(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [webapp].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [client].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2159
+1 DISTRO
+1 distro tarball builds with the patch 


+1 Overall result, good!, no -1s


The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/606/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100  30020 0  100  3002  0   3702 --:--:-- --:--:-- --:--:--

[jira] [Updated] (OOZIE-3274) Remove slf4j

2018-06-05 Thread JIRA



 [ 
https://issues.apache.org/jira/browse/OOZIE-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Péter Gergő Barna updated OOZIE-3274:
-
Description: 
Remove all slf4j jars from Oozie. 

I am not sure if Oozie works perfectly without slf4j jars, for me it seems that 
all kinds of hadoop2 jobs are working fine.  Any comments are welcomed, I would 
appreciate a response from the community on this. 

 

I know about this issue, which introduced slf4j as a dependency, I am not sure 
if this is relevant any more. 

https://issues.apache.org/jira/browse/OOZIE-1004

 

 

  was:
Remove all slf4j jars from Oozie. 

I am not sure if Oozie works perfectly without slf4j jars, for me it seems that 
all kinds of hadoop2 jobs are working fin.  Any comments on this are welcomed.

 

I know about this issue, which introduced slf4j as a dependency, I am not sure 
if this is relevant any more. 

https://issues.apache.org/jira/browse/OOZIE-1004

 

 


> Remove slf4j
> 
>
> Key: OOZIE-3274
> URL: https://issues.apache.org/jira/browse/OOZIE-3274
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Péter Gergő Barna
>Priority: Major
> Fix For: trunk
>
> Attachments: OOZIE-3274.patch
>
>
> Remove all slf4j jars from Oozie. 
> I am not sure if Oozie works perfectly without slf4j jars, for me it seems 
> that all kinds of hadoop2 jobs are working fine.  Any comments are welcomed, 
> I would appreciate a response from the community on this. 
>  
> I know about this issue, which introduced slf4j as a dependency, I am not 
> sure if this is relevant any more. 
> https://issues.apache.org/jira/browse/OOZIE-1004
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3274) Remove slf4j

2018-06-05 Thread JIRA



 [ 
https://issues.apache.org/jira/browse/OOZIE-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Péter Gergő Barna updated OOZIE-3274:
-
Attachment: OOZIE-3274.patch

> Remove slf4j
> 
>
> Key: OOZIE-3274
> URL: https://issues.apache.org/jira/browse/OOZIE-3274
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Péter Gergő Barna
>Priority: Major
> Fix For: trunk
>
> Attachments: OOZIE-3274.patch
>
>
> Remove all slf4j jars from Oozie. 
> I am not sure if Oozie works perfectly without slf4j jars, for me it seems 
> that all kinds of hadoop2 jobs are working fin.  Any comments on this are 
> welcomed.
>  
> I know about this issue, which introduced slf4j as a dependency, I am not 
> sure if this is relevant any more. 
> https://issues.apache.org/jira/browse/OOZIE-1004
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OOZIE-3274) Remove slf4j

2018-06-05 Thread JIRA

Péter Gergő Barna created OOZIE-3274:


 Summary: Remove slf4j
 Key: OOZIE-3274
 URL: https://issues.apache.org/jira/browse/OOZIE-3274
 Project: Oozie
  Issue Type: Bug
Affects Versions: trunk
Reporter: Péter Gergő Barna
 Fix For: trunk
 Attachments: OOZIE-3274.patch

Remove all slf4j jars from Oozie. 

I am not sure if Oozie works perfectly without slf4j jars, for me it seems that 
all kinds of hadoop2 jobs are working fin.  Any comments on this are welcomed.

 

I know about this issue, which introduced slf4j as a dependency, I am not sure 
if this is relevant any more. 

https://issues.apache.org/jira/browse/OOZIE-1004

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

2018-06-05 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501600#comment-16501600
 ] 

Hadoop QA commented on OOZIE-2791:
--

PreCommit-OOZIE-Build started


> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-2791-003.patch, OOZIE-2791-004.patch, 
> OOZIE-2791-005.patch, OOZIE-2791-006.patch, OOZIE-2791-01.patch, 
> OOZIE-2791-02.patch
>
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
> --
> ...
> {code}
> You can see file is created but it's size is 0.
> {code}
> -rw-r--r--   3 asasvari supergroup  0 2017-02-07 10:59 
> share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar
> {code}
> This behaviour is clearly wrong. 
> In case of such an exception, we should retry copying or rollback changes. We 
> should also consider throttling HDFS requests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

2018-06-05 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501566#comment-16501566
 ] 

Hadoop QA commented on OOZIE-2791:
--


Testing JIRA OOZIE-2791

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 4 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
error(s)
.{color:red}ERROR{color}: the current HEAD has 2 Javadoc error(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2159
.Tests failed: 2
.Tests errors: 6

.The patch failed the following testcases:

testOozieSharelibCLICreate(org.apache.oozie.tools.TestConcurrentCopyFromLocal)
testOozieSharelibCLICreateConcurrent(org.apache.oozie.tools.TestConcurrentCopyFromLocal)

.Tests failing with errors:
testConcurrentCopyFromLocal(org.apache.oozie.tools.TestConcurrentCopyFromLocal)
testImportTablesOverflowBatchSize(org.apache.oozie.tools.TestDBLoadDump)
testImportToNonExistingTablesSucceedsOnHsqldb(org.apache.oozie.tools.TestDBLoadDump)
testImportInvalidDataLeavesTablesEmpty(org.apache.oozie.tools.TestDBLoadDump)
testImportToNonEmptyTablesCausesPrematureExit(org.apache.oozie.tools.TestDBLoadDump)
testImportedDBIsExportedCorrectly(org.apache.oozie.tools.TestDBLoadDump)

.{color:orange}Tests failed at first run:{color}
TestCoordActionsKillXCommand#testActionKillCommandActionNumbers
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/605/



> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-2791-003.patch, OOZIE-2791-004.patch, 
> OOZIE-2791-005.patch, OOZIE-2791-006.patch, OOZIE-2791-01.patch, 
> OOZIE-2791-02.patch
>
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
>

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

2018-06-05 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501485#comment-16501485
 ] 

Hadoop QA commented on OOZIE-2791:
--

PreCommit-OOZIE-Build started


> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-2791-003.patch, OOZIE-2791-004.patch, 
> OOZIE-2791-005.patch, OOZIE-2791-006.patch, OOZIE-2791-01.patch, 
> OOZIE-2791-02.patch
>
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
> --
> ...
> {code}
> You can see file is created but it's size is 0.
> {code}
> -rw-r--r--   3 asasvari supergroup  0 2017-02-07 10:59 
> share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar
> {code}
> This behaviour is clearly wrong. 
> In case of such an exception, we should retry copying or rollback changes. We 
> should also consider throttling HDFS requests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

2018-06-05 Thread Julia Kinga Marton (JIRA)



 [ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julia Kinga Marton updated OOZIE-2791:
--
Attachment: OOZIE-2791-006.patch

> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-2791-003.patch, OOZIE-2791-004.patch, 
> OOZIE-2791-005.patch, OOZIE-2791-006.patch, OOZIE-2791-01.patch, 
> OOZIE-2791-02.patch
>
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
> --
> ...
> {code}
> You can see file is created but it's size is 0.
> {code}
> -rw-r--r--   3 asasvari supergroup  0 2017-02-07 10:59 
> share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar
> {code}
> This behaviour is clearly wrong. 
> In case of such an exception, we should retry copying or rollback changes. We 
> should also consider throttling HDFS requests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66930: OOZIE-2791 ShareLib installation may fail on busy Hadoop clusters

2018-06-05 Thread Kinga Marton via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66930/
---

(Updated June 5, 2018, 8:37 a.m.)


Review request for oozie, András Piros and Peter Cseh.


Changes
---

findbug fix


Repository: oozie-git


Description
---

On a busy Hadoop cluster it can happen that users cannot install properly 
Oozie ShareLib.


Diffs (updated)
-

  tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java dce1c559 
  tools/src/test/java/org/apache/oozie/tools/TestBlockSizeCalculator.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/TestConcurrentCopyFromLocal.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/TestCopyTaskCallable.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java ccad273b 


Diff: https://reviews.apache.org/r/66930/diff/4/

Changes: https://reviews.apache.org/r/66930/diff/3-4/


Testing
---

Tested manually


Thanks,

Kinga Marton

[jira] Subscription: Oozie Patch Available

2018-06-05 Thread jira

Issue Subscription
Filter: Oozie Patch Available (99 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3270   Upgrade Derby to 10.14.1.0
https://issues.apache.org/jira/browse/OOZIE-3270
OOZIE-3265  properties RERUN_FAIL_NODES and RERUN_SKIP_NODES should be able to 
appear together
https://issues.apache.org/jira/browse/OOZIE-3265
OOZIE-3260  [sla] Remove stale item above max retries on JPA related errors 
from in-memory SLA map
https://issues.apache.org/jira/browse/OOZIE-3260
OOZIE-3256  refactor OozieCLI class
https://issues.apache.org/jira/browse/OOZIE-3256
OOZIE-3249  [tools] Instrumentation log parser
https://issues.apache.org/jira/browse/OOZIE-3249
OOZIE-3218  Oozie Sqoop action with command splits the select clause into 
multiple parts due to delimiter being space
https://issues.apache.org/jira/browse/OOZIE-3218
OOZIE-3199  Let system property restriction configurable
https://issues.apache.org/jira/browse/OOZIE-3199
OOZIE-3196  Authorization: restrict world readability by user
https://issues.apache.org/jira/browse/OOZIE-3196
OOZIE-3194  Oozie should set proper permissions to sharelib after upload
https://issues.apache.org/jira/browse/OOZIE-3194
OOZIE-3193  Applications are not killed when submitted via subworkflow
https://issues.apache.org/jira/browse/OOZIE-3193
OOZIE-3186  Oozie is unable to use configuration linked using jceks://file/...
https://issues.apache.org/jira/browse/OOZIE-3186
OOZIE-3179  Adding a configurable config-default.xml location to a workflow
https://issues.apache.org/jira/browse/OOZIE-3179
OOZIE-3170  Oozie Diagnostic Bundle tool fails with NPE due to missing service 
class
https://issues.apache.org/jira/browse/OOZIE-3170
OOZIE-3160  PriorityDelayQueue put()/take() can cause significant CPU load due 
to busy waiting
https://issues.apache.org/jira/browse/OOZIE-3160
OOZIE-3135  Configure log4j2 in SqoopMain
https://issues.apache.org/jira/browse/OOZIE-3135
OOZIE-3109  Escape log-streaming's HTML-specific characters
https://issues.apache.org/jira/browse/OOZIE-3109
OOZIE-3091  Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: 
org/apache/avro/mapred/AvroWrapper"
https://issues.apache.org/jira/browse/OOZIE-3091
OOZIE-3071  Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 
than Spark 2.2.0
https://issues.apache.org/jira/browse/OOZIE-3071
OOZIE-3063  Sanitizing variables that are part of openjpa.ConnectionProperties
https://issues.apache.org/jira/browse/OOZIE-3063
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues.apache.org/jira/browse/OOZIE-3062
OOZIE-3061  Kill only those child jobs which are not already killed
https://issues.apache.org/jira/browse/OOZIE-3061
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  [examples] Fix Findbugs warnings
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues.apache.org/jira/browse/OOZIE-2829
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2795  Create lib directory or symlink for Oozie CLI during packaging
https://issues.apache.org/jira/browse/OOZIE-2795
OOZIE-2791  ShareLib installation may fail on busy Hadoop clusters
https://issues.apache.org/jira/browse/OOZIE-2791
OOZIE-2784  Include WEEK as a parameter in the Coordinator Expression

[jira] [Commented] (OOZIE-3260) [sla] Remove stale item above max retries on JPA related errors from in-memory SLA map

Re: Review Request 66930: OOZIE-2791 ShareLib installation may fail on busy Hadoop clusters

Re: Review Request 66930: OOZIE-2791 ShareLib installation may fail on busy Hadoop clusters

[jira] [Commented] (OOZIE-2877) Oozie Git Action

Re: Review Request 59620: This review board request is for an action to provide a Git action for Oozie

[jira] [Commented] (OOZIE-2877) Oozie Git Action

[jira] [Commented] (OOZIE-2877) Oozie Git Action

[jira] [Commented] (OOZIE-2877) Oozie Git Action

[jira] [Comment Edited] (OOZIE-2877) Oozie Git Action

[jira] [Comment Edited] (OOZIE-2877) Oozie Git Action

[jira] [Commented] (OOZIE-2877) Oozie Git Action

[jira] [Comment Edited] (OOZIE-2877) Oozie Git Action

[jira] [Commented] (OOZIE-2877) Oozie Git Action

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

Re: Review Request 66930: OOZIE-2791 ShareLib installation may fail on busy Hadoop clusters

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

Success: OOZIE-2791 PreCommit Build #606

[jira] [Updated] (OOZIE-3274) Remove slf4j

[jira] [Updated] (OOZIE-3274) Remove slf4j

[jira] [Created] (OOZIE-3274) Remove slf4j

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

[jira] [Updated] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

Re: Review Request 66930: OOZIE-2791 ShareLib installation may fail on busy Hadoop clusters

[jira] Subscription: Oozie Patch Available

26 matches

Site Navigation

Mail list logo

Footer information