[jira] [Commented] (OOZIE-2984) Parse spark-defaults.conf values with spaces without needing the quotes

2017-07-26 Thread Peter Cseh (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101473#comment-16101473
 ] 

Peter Cseh commented on OOZIE-2984:
---

Thanks for the update!
It looks fine in general, but it does not work correctly if --verbose is in the 
list. This  option has no value for it an he current code adds an empty string 
to the parameter list. I don't think Spark likes that. 
Also, I don't know if this is the only option that has no parameters or are 
there more out there, so this should be solved in a general way.

After this, DG_SparkActionExtension.twiki should be updated to say the quoting 
is not a hard requirement any more. Right now it says {quote}Values containing 
whitespaces should be enclosed by double quotes. {quote}

> Parse spark-defaults.conf values with spaces without needing the quotes
> ---
>
> Key: OOZIE-2984
> URL: https://issues.apache.org/jira/browse/OOZIE-2984
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 4.3.0
>Reporter: Fahd Siddiqui
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-2984.001.patch, OOZIE-2984.002.patch, 
> OOZIE-2984.003.patch
>
>
> Oozie requires multiple java opts to be enclosed in quotes. For example:
> {code}
> spark.driver.extraJavaOptions="-Xmn2703m -XX:SurvivorRatio=2 
> -XX:ParallelGCThreads=20"
> {code}
> However the above breaks spark-shell. 
> {code}
> $ ./spark2-shell
> Invalid initial young generation size: -Xmn2212m -XX:SurvivorRatio=2 
> -XX:ParallelGCThreads=20
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> {code}
> To fix spark-shell, we have to remove the quotes, which errors out Oozie:
> {code}
> Error: Unrecognized option '-XX:SurvivorRatio=2'
> ...
> --conf
> spark.executor.extraJavaOptions=-Xmn2703m 
> -Dlog4j.configuration=spark-log4j.properties
> -XX:SurvivorRatio=2
> {code}
> Oozie should be able to parse spark-defaults.conf values with spaces without 
> needing the quotes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


Build failed in Jenkins: oozie-trunk-find-patches-available #165556

2017-07-26 Thread Apache Jenkins Server
See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H1 (Hadoop) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/oozie.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/oozie.git
 > git --version # timeout=10
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/oozie.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 84eb493642d35b8bb1e7527ef51b6ecfa40dd592 
(refs/remotes/origin/master)
Commit message: "OOZIE-3004 Forked action retry info is not working"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 84eb493642d35b8bb1e7527ef51b6ecfa40dd592
 > git rev-list 84eb493642d35b8bb1e7527ef51b6ecfa40dd592 # timeout=10
[oozie-trunk-find-patches-available] $ /bin/bash 
/tmp/jenkins2443226925686737039.sh
mkdir: cannot create directory 
':
 File exists
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0100 
 175k0  175k0 0   102k  0 --:--:--  0:00:01 --:--:--  102k100  
543k0  543k0 0   204k  0 --:--:--  0:00:02 --:--:--  204k100  
815k0  815k0 0   225k  0 --:--:--  0:00:03 --:--:--  225k100  
831k0  831k0 0   198k  0 --:--:--  0:00:04 --:--:--  198k
curl: (18) transfer closed with outstanding read data remaining
Could not retrieve available patches from JIRA
Build step 'Execute shell' marked build as failure


Jenkins build is back to normal : oozie-trunk-find-patches-available #165557

2017-07-26 Thread Apache Jenkins Server
See 




[jira] [Commented] (OOZIE-3009) Number of Oozie tests executed dropped after OOZIE-2854

2017-07-26 Thread Andras Piros (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101683#comment-16101683
 ] 

Andras Piros commented on OOZIE-3009:
-

Root cause is following:
* there are (at least following) five test classes that are at the moment 
retrying the database operations for an extended time period, but they 
shouldn't (since these {{FaultInjection}} facility):
** {{TestBundleJobsDeleteJPAExecutor}}
** {{TestCoordJobsDeleteJPAExecutor}}
** {{TestPurgeService}}
** {{TestPurgeXCommand}}
** {{TestWorkflowJobsDeleteJPAExecutor}}
* as a consequence, these tend to timeout often
* when that happens, all the other modules are skipped because of the timeout
* the test cases of the skipped modules are not rerun on the second run

We need to extend {{PersistenceExceptionSubclassFilterRetryPredicate}} by 
following:
* if the {{Throwable}} does not have a root cause, and is either a 
{{RuntimeException}} (in case of {{FaultInjection}}), or a 
{{JPAExecutorException}} (e.g. {{Query.getResultList()}} returned an empty 
{{List}}), the predicate returns {{false}}
* for all the other cases the existing behavior kicks in

> Number of Oozie tests executed dropped after OOZIE-2854
> ---
>
> Key: OOZIE-3009
> URL: https://issues.apache.org/jira/browse/OOZIE-3009
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Andras Piros
>Priority: Blocker
>
> I noticed that the number of executed tests has been significantly dropped 
> after OOZIE-2854.
> - Tests run: *1080* https://issues.apache.org/jira/browse/OOZIE-2854
> Previous tests:
> - OOZIE-2371 - Tests run: *1965* 
> https://issues.apache.org/jira/browse/OOZIE-2371?focusedCommentId=16076996&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16076996
> - OOZIE-2911 - Tests run: *1966* 
> https://issues.apache.org/jira/browse/OOZIE-2911?focusedCommentId=16078078&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16078078
> In OOZIE-2854, we can also see that the number of test cases vary from patch 
> to patch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Comment Edited] (OOZIE-3009) Number of Oozie tests executed dropped after OOZIE-2854

2017-07-26 Thread Andras Piros (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101683#comment-16101683
 ] 

Andras Piros edited comment on OOZIE-3009 at 7/26/17 1:57 PM:
--

Root cause is following:
* there are (at least following) five test classes that are at the moment 
retrying the database operations for an extended time period, but they 
shouldn't (since these use {{FaultInjection}} facility, or perform any other 
kind of faulty {{JPAService.execute()}} call):
** {{TestBundleJobsDeleteJPAExecutor}}
** {{TestCoordJobsDeleteJPAExecutor}}
** {{TestPurgeService}}
** {{TestPurgeXCommand}}
** {{TestWorkflowJobsDeleteJPAExecutor}}
* as a consequence, these tend to timeout often
* when that happens, all the other modules are skipped because of the timeout
* the test cases of the skipped modules are not rerun on the second run

We need to extend {{PersistenceExceptionSubclassFilterRetryPredicate}} by 
following:
* if the {{Throwable}} does not have a root cause, and is either a 
{{RuntimeException}} (in case of {{FaultInjection}}), or a 
{{JPAExecutorException}} (e.g. {{Query.getResultList()}} returned an empty 
{{List}}), the predicate returns {{false}}
* for all the other cases the existing behavior kicks in


was (Author: andras.piros):
Root cause is following:
* there are (at least following) five test classes that are at the moment 
retrying the database operations for an extended time period, but they 
shouldn't (since these {{FaultInjection}} facility):
** {{TestBundleJobsDeleteJPAExecutor}}
** {{TestCoordJobsDeleteJPAExecutor}}
** {{TestPurgeService}}
** {{TestPurgeXCommand}}
** {{TestWorkflowJobsDeleteJPAExecutor}}
* as a consequence, these tend to timeout often
* when that happens, all the other modules are skipped because of the timeout
* the test cases of the skipped modules are not rerun on the second run

We need to extend {{PersistenceExceptionSubclassFilterRetryPredicate}} by 
following:
* if the {{Throwable}} does not have a root cause, and is either a 
{{RuntimeException}} (in case of {{FaultInjection}}), or a 
{{JPAExecutorException}} (e.g. {{Query.getResultList()}} returned an empty 
{{List}}), the predicate returns {{false}}
* for all the other cases the existing behavior kicks in

> Number of Oozie tests executed dropped after OOZIE-2854
> ---
>
> Key: OOZIE-3009
> URL: https://issues.apache.org/jira/browse/OOZIE-3009
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Andras Piros
>Priority: Blocker
>
> I noticed that the number of executed tests has been significantly dropped 
> after OOZIE-2854.
> - Tests run: *1080* https://issues.apache.org/jira/browse/OOZIE-2854
> Previous tests:
> - OOZIE-2371 - Tests run: *1965* 
> https://issues.apache.org/jira/browse/OOZIE-2371?focusedCommentId=16076996&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16076996
> - OOZIE-2911 - Tests run: *1966* 
> https://issues.apache.org/jira/browse/OOZIE-2911?focusedCommentId=16078078&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16078078
> In OOZIE-2854, we can also see that the number of test cases vary from patch 
> to patch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Comment Edited] (OOZIE-3009) Number of Oozie tests executed dropped after OOZIE-2854

2017-07-26 Thread Andras Piros (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101683#comment-16101683
 ] 

Andras Piros edited comment on OOZIE-3009 at 7/26/17 2:46 PM:
--

Root cause is following:
* there are (at least following) five test classes that are at the moment 
retrying the database operations for an extended time period, but they 
shouldn't (since these use {{FaultInjection}} facility, or perform any other 
kind of faulty {{JPAService.execute()}} call):
** {{TestBundleJobsDeleteJPAExecutor}}
** {{TestCoordJobsDeleteJPAExecutor}}
** {{TestPurgeService}}
** {{TestPurgeXCommand}}
** {{TestWorkflowJobsDeleteJPAExecutor}}
* as a consequence, these tend to timeout often
* when that happens, all the other modules are skipped because of the timeout
* the test cases of the skipped modules are not rerun on the second run

We need to extend {{PersistenceExceptionSubclassFilterRetryPredicate}} by 
following:
* if the {{Throwable}} does not have a root cause, and is either a 
{{RuntimeException}} (in case of {{FaultInjection}}), or a 
{{JPAExecutorException}} (e.g. {{Query.getResultList()}} returned an empty 
{{List}}), the predicate returns {{false}}
* for all the other cases the existing behavior kicks in

It's also needed to enhance 
{{PersistenceExceptionSubclassFilterRetryPredicate}} because there are lots of 
places in the codebase where causeless {{JPAExecutorException}} instances are 
thrown where there is actually a reasonable cause, e.g. {{NoResultException}}. 
For those cases we will not retry the database operation.


was (Author: andras.piros):
Root cause is following:
* there are (at least following) five test classes that are at the moment 
retrying the database operations for an extended time period, but they 
shouldn't (since these use {{FaultInjection}} facility, or perform any other 
kind of faulty {{JPAService.execute()}} call):
** {{TestBundleJobsDeleteJPAExecutor}}
** {{TestCoordJobsDeleteJPAExecutor}}
** {{TestPurgeService}}
** {{TestPurgeXCommand}}
** {{TestWorkflowJobsDeleteJPAExecutor}}
* as a consequence, these tend to timeout often
* when that happens, all the other modules are skipped because of the timeout
* the test cases of the skipped modules are not rerun on the second run

We need to extend {{PersistenceExceptionSubclassFilterRetryPredicate}} by 
following:
* if the {{Throwable}} does not have a root cause, and is either a 
{{RuntimeException}} (in case of {{FaultInjection}}), or a 
{{JPAExecutorException}} (e.g. {{Query.getResultList()}} returned an empty 
{{List}}), the predicate returns {{false}}
* for all the other cases the existing behavior kicks in

> Number of Oozie tests executed dropped after OOZIE-2854
> ---
>
> Key: OOZIE-3009
> URL: https://issues.apache.org/jira/browse/OOZIE-3009
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Andras Piros
>Priority: Blocker
>
> I noticed that the number of executed tests has been significantly dropped 
> after OOZIE-2854.
> - Tests run: *1080* https://issues.apache.org/jira/browse/OOZIE-2854
> Previous tests:
> - OOZIE-2371 - Tests run: *1965* 
> https://issues.apache.org/jira/browse/OOZIE-2371?focusedCommentId=16076996&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16076996
> - OOZIE-2911 - Tests run: *1966* 
> https://issues.apache.org/jira/browse/OOZIE-2911?focusedCommentId=16078078&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16078078
> In OOZIE-2854, we can also see that the number of test cases vary from patch 
> to patch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


Build failed in Jenkins: oozie-trunk-find-patches-available #165575

2017-07-26 Thread Apache Jenkins Server
See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H1 (Hadoop) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/oozie.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/oozie.git
 > git --version # timeout=10
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/oozie.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 84eb493642d35b8bb1e7527ef51b6ecfa40dd592 
(refs/remotes/origin/master)
Commit message: "OOZIE-3004 Forked action retry info is not working"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 84eb493642d35b8bb1e7527ef51b6ecfa40dd592
 > git rev-list 84eb493642d35b8bb1e7527ef51b6ecfa40dd592 # timeout=10
[oozie-trunk-find-patches-available] $ /bin/bash 
/tmp/jenkins2823554320941892043.sh
mkdir: cannot create directory 
':
 File exists
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100 900240 900240 0  98604  0 --:--:-- --:--:-- --:--:-- 
98602100  439k0  439k0 0   235k  0 --:--:--  0:00:01 --:--:--  
235k100  719k0  719k0 0   249k  0 --:--:--  0:00:02 --:--:--  
249k100  935k0  935k0 0   239k  0 --:--:--  0:00:03 --:--:--  
239k100 1438k0 1438k0 0   291k  0 --:--:--  0:00:04 --:--:--  
291k100 1742k0 1742k0 0   273k  0 --:--:--  0:00:06 --:--:--  
303k100 1742k0 1742k0 0   273k  0 --:--:--  0:00:06 --:--:--  
290k
curl: (18) transfer closed with outstanding read data remaining
Could not retrieve available patches from JIRA
Build step 'Execute shell' marked build as failure


Jenkins build is back to normal : oozie-trunk-find-patches-available #165576

2017-07-26 Thread Apache Jenkins Server
See 




[jira] [Updated] (OOZIE-3009) Number of Oozie tests executed dropped after OOZIE-2854

2017-07-26 Thread Andras Piros (JIRA)

 [ 
https://issues.apache.org/jira/browse/OOZIE-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-3009:

Attachment: OOZIE-3009.001.patch

> Number of Oozie tests executed dropped after OOZIE-2854
> ---
>
> Key: OOZIE-3009
> URL: https://issues.apache.org/jira/browse/OOZIE-3009
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Andras Piros
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-3009.001.patch
>
>
> I noticed that the number of executed tests has been significantly dropped 
> after OOZIE-2854.
> - Tests run: *1080* https://issues.apache.org/jira/browse/OOZIE-2854
> Previous tests:
> - OOZIE-2371 - Tests run: *1965* 
> https://issues.apache.org/jira/browse/OOZIE-2371?focusedCommentId=16076996&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16076996
> - OOZIE-2911 - Tests run: *1966* 
> https://issues.apache.org/jira/browse/OOZIE-2911?focusedCommentId=16078078&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16078078
> In OOZIE-2854, we can also see that the number of test cases vary from patch 
> to patch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


Review Request 61142: OOZIE-3009 Number of Oozie tests executed dropped

2017-07-26 Thread András Piros

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61142/
---

Review request for oozie, Attila Sasvari and Peter Cseh.


Repository: oozie-git


Description
---

OOZIE-3009 Number of Oozie tests executed dropped


Diffs
-

  core/src/main/java/org/apache/oozie/service/JPAService.java 
93fe9da1564b734831e7b988ffe760f808af49fa 
  core/src/main/java/org/apache/oozie/sla/SLARegistrationBean.java 
1b8370f01e5097300ee20e903ac745c23c999ce3 
  
core/src/main/java/org/apache/oozie/util/db/PersistenceExceptionSubclassFilterRetryPredicate.java
 b742ca78520bdea804234460c36925beb105c5cb 
  
core/src/test/java/org/apache/oozie/executor/jpa/TestBundleJobsDeleteJPAExecutor.java
 9077970fb11e10f158b1b742e6abbf4d056f92a8 
  
core/src/test/java/org/apache/oozie/executor/jpa/TestCoordActionsDeleteJPAExecutor.java
 c7e0c38426d08c824b373409a9860a420e158821 
  
core/src/test/java/org/apache/oozie/executor/jpa/TestCoordJobsDeleteJPAExecutor.java
 bff5836f209e1ec43de1b32203689909591896b7 
  
core/src/test/java/org/apache/oozie/executor/jpa/TestWorkflowJobsDeleteJPAExecutor.java
 c4ca61bf0ce678ae6bd0f6309a586f287e05537c 
  core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 
4cb2530cae634517ab833676b68deba068c7560b 
  core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java 
34011f6b6f7f7ee4c57d63a1e62af4acb0fe32ba 
  core/src/test/java/org/apache/oozie/test/XTestCase.java 
0ef4a4be6998744914e2ab56697cc2ae517070b9 
  
core/src/test/java/org/apache/oozie/util/db/TestPersistenceExceptionSubclassFilterRetryPredicate.java
 76d2edc0e538732e65556d9255f49d4d1c0980b7 


Diff: https://reviews.apache.org/r/61142/diff/1/


Testing
---

Following test cases have been executed:
```
TestOperationRetryHandler
TestPersistenceExceptionSubclassFilterRetryPredicate
TestParallelJPAOperationRetries
TestWorkflow
TestWorkflowRetries
TestJPAService
TestRetryAttemptState
TestSLACalculatorMemory
TestWorkflowJobsDeleteJPAExecutor
TestPurgeService
TestCoordJobsDeleteJPAExecutor
TestBundleJobsDeleteJPAExecutor
TestPurgeXCommand
```


Thanks,

András Piros



[jira] [Updated] (OOZIE-2984) Parse spark-defaults.conf values with spaces without needing the quotes

2017-07-26 Thread Andras Piros (JIRA)

 [ 
https://issues.apache.org/jira/browse/OOZIE-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2984:

Attachment: OOZIE-2984.004.patch

Adding support for {{--verbose}}, and extending documentation.

> Parse spark-defaults.conf values with spaces without needing the quotes
> ---
>
> Key: OOZIE-2984
> URL: https://issues.apache.org/jira/browse/OOZIE-2984
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 4.3.0
>Reporter: Fahd Siddiqui
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-2984.001.patch, OOZIE-2984.002.patch, 
> OOZIE-2984.003.patch, OOZIE-2984.004.patch
>
>
> Oozie requires multiple java opts to be enclosed in quotes. For example:
> {code}
> spark.driver.extraJavaOptions="-Xmn2703m -XX:SurvivorRatio=2 
> -XX:ParallelGCThreads=20"
> {code}
> However the above breaks spark-shell. 
> {code}
> $ ./spark2-shell
> Invalid initial young generation size: -Xmn2212m -XX:SurvivorRatio=2 
> -XX:ParallelGCThreads=20
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> {code}
> To fix spark-shell, we have to remove the quotes, which errors out Oozie:
> {code}
> Error: Unrecognized option '-XX:SurvivorRatio=2'
> ...
> --conf
> spark.executor.extraJavaOptions=-Xmn2703m 
> -Dlog4j.configuration=spark-log4j.properties
> -XX:SurvivorRatio=2
> {code}
> Oozie should be able to parse spark-defaults.conf values with spaces without 
> needing the quotes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


Re: Review Request 60963: OOZIE-2984 Parse spark-defaults.conf values with spaces without needing the quotes

2017-07-26 Thread András Piros

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60963/
---

(Updated July 26, 2017, 6:01 p.m.)


Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and Robert 
Kanter.


Changes
---

Adding support for `--verbose`, and extending documentation.


Repository: oozie-git


Description
---

Parse `spark-defaults.conf` values with spaces without needing the quotes. That 
is, quotes should not be mandatory but optional to retain backwards 
compatibility.


Diffs (updated)
-

  docs/src/site/twiki/DG_SparkActionExtension.twiki 
863bd89cf576d2fb7be76c61ca139218c33a5f66 
  
sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkOptionsSplitter.java
 30def6f05b7a17b8ce66ed766a9d64c171796ac5 
  
sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkArgsExtractor.java
 7db26a6b81083d7664bef8ff14c8e14df4572fc1 
  
sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkOptionsSplitter.java
 02786a2476fc981af77688a22aa9c3a8fc0cce2a 


Diff: https://reviews.apache.org/r/60963/diff/4/

Changes: https://reviews.apache.org/r/60963/diff/3-4/


Testing
---

Following tests are run:

* `TestSparkOptionsSplitter`
* `TestSparkArgsExtractor`
* `TestSparkMain`


Thanks,

András Piros



[jira] [Commented] (OOZIE-2984) Parse spark-defaults.conf values with spaces without needing the quotes

2017-07-26 Thread Peter Cseh (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102081#comment-16102081
 ] 

Peter Cseh commented on OOZIE-2984:
---

I'll have a deeper look tomorrow, but please use Oozie's StringUtil.isEmpty 
instead of guava's com.google.common.base.Strings. It's better to use as few 
dependencies in the sharelib as possible.

> Parse spark-defaults.conf values with spaces without needing the quotes
> ---
>
> Key: OOZIE-2984
> URL: https://issues.apache.org/jira/browse/OOZIE-2984
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 4.3.0
>Reporter: Fahd Siddiqui
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-2984.001.patch, OOZIE-2984.002.patch, 
> OOZIE-2984.003.patch, OOZIE-2984.004.patch
>
>
> Oozie requires multiple java opts to be enclosed in quotes. For example:
> {code}
> spark.driver.extraJavaOptions="-Xmn2703m -XX:SurvivorRatio=2 
> -XX:ParallelGCThreads=20"
> {code}
> However the above breaks spark-shell. 
> {code}
> $ ./spark2-shell
> Invalid initial young generation size: -Xmn2212m -XX:SurvivorRatio=2 
> -XX:ParallelGCThreads=20
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> {code}
> To fix spark-shell, we have to remove the quotes, which errors out Oozie:
> {code}
> Error: Unrecognized option '-XX:SurvivorRatio=2'
> ...
> --conf
> spark.executor.extraJavaOptions=-Xmn2703m 
> -Dlog4j.configuration=spark-log4j.properties
> -XX:SurvivorRatio=2
> {code}
> Oozie should be able to parse spark-defaults.conf values with spaces without 
> needing the quotes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


Build failed in Jenkins: oozie-trunk-precommit-build #3994

2017-07-26 Thread Apache Jenkins Server
See 


--
[...truncated 324.78 KB...]
[INFO] Copying jackson-core-asl-1.9.13.jar to 

[INFO] Copying commons-jexl-2.1.1.jar to 

[INFO] Copying oozie-sharelib-hcatalog-5.0.0-SNAPSHOT.jar to 

[INFO] Copying hadoop-mapreduce-client-jobclient-2.6.0.jar to 

[INFO] Copying gson-2.7.jar to 

[INFO] Copying oozie-core-5.0.0-SNAPSHOT.jar to 

[INFO] Copying commons-pool-1.5.4.jar to 

[INFO] Copying htrace-core-3.0.4.jar to 

[INFO] Copying jaxb-impl-2.2.3-1.jar to 

[INFO] Copying jets3t-0.9.0.jar to 

[INFO] Copying datanucleus-rdbms-3.2.9.jar to 

[INFO] Copying hive-common-1.2.0.jar to 

[INFO] Copying snappy-java-1.0.4.1.jar to 

[INFO] Copying javax.annotation-api-1.2.jar to 

[INFO] Copying commons-compiler-2.7.6.jar to 

[INFO] Copying ant-1.9.1.jar to 

[INFO] Copying hadoop-annotations-2.6.0.jar to 

[INFO] Copying janino-2.7.6.jar to 

[INFO] Copying activemq-client-5.13.3.jar to 

[INFO] Copying guice-3.0.jar to 

[INFO] Copying httpclient-4.3.6.jar to 

[INFO] Copying xz-1.0.jar to 

[INFO] Copying oozie-client-5.0.0-SNAPSHOT.jar to 

[INFO] Copying hadoop-mapreduce-client-common-2.6.0.jar to 

[INFO] Copying leveldbjni-all-1.8.jar to 

[INFO] Copying jetty-util-6.1.26.jar to 

[INFO] Copying velocity-1.5.jar to 

[INFO] Copying jackson-core-2.4.2.jar to 

[INFO] Copying apache-log4j-extras-1.2.17.jar to 

[INFO] Copying netty-3.7.0.Final.jar to 

[INFO] Copying xmlenc-0.52.jar to 

[INFO] Copyin

[jira] [Commented] (OOZIE-3009) Number of Oozie tests executed dropped after OOZIE-2854

2017-07-26 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102187#comment-16102187
 ] 

Hadoop QA commented on OOZIE-3009:
--

Testing JIRA OOZIE-3009

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 1 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 8 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 1 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1017
.Tests rerun: 22
.Tests failed at first run: org.apache.oozie.action.hadoop.TestLauncherAM,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/3994/

> Number of Oozie tests executed dropped after OOZIE-2854
> ---
>
> Key: OOZIE-3009
> URL: https://issues.apache.org/jira/browse/OOZIE-3009
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Andras Piros
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-3009.001.patch
>
>
> I noticed that the number of executed tests has been significantly dropped 
> after OOZIE-2854.
> - Tests run: *1080* https://issues.apache.org/jira/browse/OOZIE-2854
> Previous tests:
> - OOZIE-2371 - Tests run: *1965* 
> https://issues.apache.org/jira/browse/OOZIE-2371?focusedCommentId=16076996&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16076996
> - OOZIE-2911 - Tests run: *1966* 
> https://issues.apache.org/jira/browse/OOZIE-2911?focusedCommentId=16078078&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16078078
> In OOZIE-2854, we can also see that the number of test cases vary from patch 
> to patch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


Jenkins build is back to normal : oozie-trunk-precommit-build #3995

2017-07-26 Thread Apache Jenkins Server
See 




[jira] [Commented] (OOZIE-2984) Parse spark-defaults.conf values with spaces without needing the quotes

2017-07-26 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102211#comment-16102211
 ] 

Hadoop QA commented on OOZIE-2984:
--

Testing JIRA OOZIE-2984

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 1 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1600
.Tests rerun: 3
.Tests failed at first run: 
org.apache.oozie.action.hadoop.TestHdfsOperations,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/3995/

> Parse spark-defaults.conf values with spaces without needing the quotes
> ---
>
> Key: OOZIE-2984
> URL: https://issues.apache.org/jira/browse/OOZIE-2984
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 4.3.0
>Reporter: Fahd Siddiqui
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-2984.001.patch, OOZIE-2984.002.patch, 
> OOZIE-2984.003.patch, OOZIE-2984.004.patch
>
>
> Oozie requires multiple java opts to be enclosed in quotes. For example:
> {code}
> spark.driver.extraJavaOptions="-Xmn2703m -XX:SurvivorRatio=2 
> -XX:ParallelGCThreads=20"
> {code}
> However the above breaks spark-shell. 
> {code}
> $ ./spark2-shell
> Invalid initial young generation size: -Xmn2212m -XX:SurvivorRatio=2 
> -XX:ParallelGCThreads=20
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> {code}
> To fix spark-shell, we have to remove the quotes, which errors out Oozie:
> {code}
> Error: Unrecognized option '-XX:SurvivorRatio=2'
> ...
> --conf
> spark.executor.extraJavaOptions=-Xmn2703m 
> -Dlog4j.configuration=spark-log4j.properties
> -XX:SurvivorRatio=2
> {code}
> Oozie should be able to parse spark-defaults.conf values with spaces without 
> needing the quotes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] Subscription: Oozie Patch Available

2017-07-26 Thread jira
Issue Subscription
Filter: Oozie Patch Available (116 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3009  Number of Oozie tests executed dropped after OOZIE-2854
https://issues.apache.org/jira/browse/OOZIE-3009
OOZIE-3002  address findbugs errors in client lib
https://issues.apache.org/jira/browse/OOZIE-3002
OOZIE-3001  core library has many instances of warnings with trailing spaces 
and lines longer than 132 chars
https://issues.apache.org/jira/browse/OOZIE-3001
OOZIE-2999  minioozie can't compile
https://issues.apache.org/jira/browse/OOZIE-2999
OOZIE-2997  files contain trailing white spaces in client lib
https://issues.apache.org/jira/browse/OOZIE-2997
OOZIE-2996  add option for -UseGCOverheadLimit to maven opts as sometimes local 
testing fails
https://issues.apache.org/jira/browse/OOZIE-2996
OOZIE-2995  in preparation for Java 8, remove MaxPermSize=512m
https://issues.apache.org/jira/browse/OOZIE-2995
OOZIE-2984  Parse spark-defaults.conf values with spaces without needing the 
quotes
https://issues.apache.org/jira/browse/OOZIE-2984
OOZIE-2980  JAXB Proof of Concept
https://issues.apache.org/jira/browse/OOZIE-2980
OOZIE-2978  Remove code that handles Pig versions before 0.8 
https://issues.apache.org/jira/browse/OOZIE-2978
OOZIE-2977  error: self-closing element not allowed
https://issues.apache.org/jira/browse/OOZIE-2977
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues.apache.org/jira/browse/OOZIE-2975
OOZIE-2974  Change TestLiteWorkflowAppParser so that it works with Hadoop 3
https://issues.apache.org/jira/browse/OOZIE-2974
OOZIE-2969  Drop support for Java 1.7
https://issues.apache.org/jira/browse/OOZIE-2969
OOZIE-2964  getting at 
org.apache.maven.plugin.javadoc.AbstractJavadocMojo.executeJavadocCommandLine(AbstractJavadocMojo.java:5163)
https://issues.apache.org/jira/browse/OOZIE-2964
OOZIE-2963  getting error in build ArtifactNotFoundException: Could not find 
artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde
https://issues.apache.org/jira/browse/OOZIE-2963
OOZIE-2962  bump maven-javadoc-plugin to 2.10.4
https://issues.apache.org/jira/browse/OOZIE-2962
OOZIE-2961  build contains multiple warnings for is not a parameter name
https://issues.apache.org/jira/browse/OOZIE-2961
OOZIE-2960  many warnings for @return tag has no arguments
https://issues.apache.org/jira/browse/OOZIE-2960
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues.apache.org/jira/browse/OOZIE-2934
OOZIE-2931  multiple warnings during build for "no @param for"
https://issues.apache.org/jira/browse/OOZIE-2931
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2916  Set a job name for the MR Action's child job
https://issues.apache.org/jira/browse/OOZIE-2916
OOZIE-2914  Consolidate Trim 
https://issues.apache.org/jira/browse/OOZIE-2914
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues.apache.org/jira/browse/OOZIE-2867
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
   

[jira] Subscription: Oozie Patch Available

2017-07-26 Thread jira
Issue Subscription
Filter: Oozie Patch Available (110 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2987  Coord action missing dependencies should show URI template with 
unresolved dependencies
https://issues-test.apache.org/jira/browse/OOZIE-2987
OOZIE-2980  JAXB Proof of Concept
https://issues-test.apache.org/jira/browse/OOZIE-2980
OOZIE-2978  Remove code that handles Pig versions before 0.8 
https://issues-test.apache.org/jira/browse/OOZIE-2978
OOZIE-2977  error: self-closing element not allowed
https://issues-test.apache.org/jira/browse/OOZIE-2977
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues-test.apache.org/jira/browse/OOZIE-2975
OOZIE-2974  Change TestLiteWorkflowAppParser so that it works with Hadoop 3
https://issues-test.apache.org/jira/browse/OOZIE-2974
OOZIE-2969  Drop support for Java 1.7
https://issues-test.apache.org/jira/browse/OOZIE-2969
OOZIE-2964  getting at 
org.apache.maven.plugin.javadoc.AbstractJavadocMojo.executeJavadocCommandLine(AbstractJavadocMojo.java:5163)
https://issues-test.apache.org/jira/browse/OOZIE-2964
OOZIE-2963  getting error in build ArtifactNotFoundException: Could not find 
artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde
https://issues-test.apache.org/jira/browse/OOZIE-2963
OOZIE-2962  bump maven-javadoc-plugin to 2.10.4
https://issues-test.apache.org/jira/browse/OOZIE-2962
OOZIE-2961  build contains multiple warnings for is not a parameter name
https://issues-test.apache.org/jira/browse/OOZIE-2961
OOZIE-2960  many warnings for @return tag has no arguments
https://issues-test.apache.org/jira/browse/OOZIE-2960
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues-test.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues-test.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues-test.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues-test.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues-test.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues-test.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues-test.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues-test.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues-test.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues-test.apache.org/jira/browse/OOZIE-2934
OOZIE-2931  multiple warnings during build for "no @param for"
https://issues-test.apache.org/jira/browse/OOZIE-2931
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues-test.apache.org/jira/browse/OOZIE-2927
OOZIE-2916  Set a job name for the MR Action's child job
https://issues-test.apache.org/jira/browse/OOZIE-2916
OOZIE-2914  Consolidate Trim 
https://issues-test.apache.org/jira/browse/OOZIE-2914
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues-test.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues-test.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues-test.apache.org/jira/browse/OOZIE-2867
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues-test.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues-test.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues-test.apache.org/jira/browse/OOZIE-2829
OOZIE-2826  Falcon feed fails to aws s3; Oozie joda time version does not meet 
required jar version 2.2 or later
https://issues-test.apache.org/jira/browse/OOZIE-2826
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues-test.apache.org/jira/browse/OOZIE-2812
OOZIE-