date:20170316

[jira] [Updated] (OOZIE-2819) Make Oozie REST API accept multibyte characters for script Actions

2017-03-16 Thread Attila Sasvari (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2819:
--
Attachment: (was: OOZIE-2819-03-amendment.patch)

> Make Oozie REST API accept multibyte characters for script Actions
> --
>
> Key: OOZIE-2819
> URL: https://issues.apache.org/jira/browse/OOZIE-2819
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Fix For: 5.0.0
>
> Attachments: OOZIE-2819-00.patch, OOZIE-2819-01.patch, 
> OOZIE-2819-02.patch, OOZIE-2819-03-amendment.patch, OOZIE-2819-03.patch
>
>
> Submitted Pig action with client side xml failed via proxy submission when it 
> contained multibyte characters.
> {code}
> curl -i  -X POST -d @/tmp/pig.xml -H 'Content-Type: application/XML; 
> charset=UTF-8' 
> 'http://'localhost':11000/oozie/v1/jobs?jobtype=pig=start'
> {code}
> Where
> {code}
> $ hdfs dfs -cat /tmp/encoding/input.txt
> 松
> 林檎
> 松
> {code}
> {code}
> $ cat /tmp/pig.xml 
> 
> 
> fs.default.name
> hdfs://localhost:8020/
> 
> 
> mapred.job.tracker
> localhost:8032
> 
> 
> user.name
> hdfs
> 
> 
> oozie.pig.script
> 
> 
> 
> oozie.pig.script.params.size
> 0
> 
> 
> oozie.pig.script.options.size
> 0
> 
> 
> oozie.libpath
> hdfs:///user/oozie/share/lib
> 
> 
> oozie.use.system.libpath
> true
> 
> 
> oozie.proxysubmission
> true
> 
> 
> {code}
> In the Oozie launcher log, I could see
> {code}
> lines = LOAD 'hdfs:///tmp/encoding/input.txt' USING PigStorage('\n') AS 
> line;test = FILTER lines BY line == '~';STORE test INTO 
> 'hdfs:///tmp/encoding/output' USING PigStorage('\n');
> {code}
> was used instead of the intended 松



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2826) Falcon feed fails to aws s3; Oozie joda time version does not meet required jar version 2.2 or later

2017-03-16 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/OOZIE-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Péter Gergő Barna updated OOZIE-2826:
-
Attachment: OOZIE-2826.patch

> Falcon feed fails to aws s3; Oozie joda time version does not meet required 
> jar version 2.2 or later
> 
>
> Key: OOZIE-2826
> URL: https://issues.apache.org/jira/browse/OOZIE-2826
> Project: Oozie
>  Issue Type: Bug
>  Components: workflow
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Priority: Minor
> Fix For: trunk
>
> Attachments: OOZIE-2826.patch, oozie-joda.example-feed.xml, 
> oozie-joda.log.txt
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Symptom:
> 2016-11-04 17:34:50,283 WARN [main] 
> com.amazonaws.services.s3.internal.S3MetadataResponseHandler: Unable to parse 
> last modified date: Wed, 02 Nov 2016 23:05:38 GMT
> java.lang.IllegalStateException: Joda-time 2.2 or later version is 
> required, but found version: 2.1
>   at com.amazonaws.util.DateUtils.handleException(DateUtils.java:149)
>   at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:195)
>   at 
> com.amazonaws.services.s3.internal.ServiceUtils.parseRfc822Date(ServiceUtils.java:78)
>   at 
> com.amazonaws.services.s3.internal.AbstractS3ResponseHandler.populateObjectMetadata(AbstractS3ResponseHandler.java:115)
>   at 
> com.amazonaws.services.s3.internal.S3MetadataResponseHandler.handle(S3MetadataResponseHandler.java:32)
>   at 
> com.amazonaws.services.s3.internal.S3MetadataResponseHandler.handle(S3MetadataResponseHandler.java:25)
>   at 
> com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1072)
>   at 
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:746)
>   at 
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
>   at 
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
>   at 
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
>   at 
> com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1050)
>   at 
> com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1027)
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:850)
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1403)
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:1343)
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1308)
>   at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1898)
>   at 
> org.apache.falcon.util.DistCPOptionsUtil.getDistCpOptions(DistCPOptionsUtil.java:90)
>   at 
> org.apache.falcon.replication.FeedReplicator.getDistCpOptions(FeedReplicator.java:229)
>   at 
> org.apache.falcon.replication.FeedReplicator.run(FeedReplicator.java:70)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>   at 
> org.apache.falcon.replication.FeedReplicator.main(FeedReplicator.java:64)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56)
>   at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:51)
>   at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:242)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
>   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: java.lang.IllegalArgumentException: Invalid format: "Wed, 02 
> Nov

[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code

2017-03-16 Thread Robert Kanter (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928434#comment-15928434
 ] 

Robert Kanter commented on OOZIE-2406:
--

One thing that may be a problem is that we currently expose (and document) 
getting the DAG graph image as a png file via the REST API 
(http://oozie.apache.org/docs/4.3.0/WebServicesAPI.html#Job_graph).  If we 
switch to a client side library like d3, we'd be serving some kind of JSON blog 
or something, and not an image; which would mean we'd have to break this REST 
call.

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2826) Falcon feed fails to aws s3; Oozie joda time version does not meet required jar version 2.2 or later

2017-03-16 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/OOZIE-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Péter Gergő Barna updated OOZIE-2826:
-
Attachment: oozie-joda.example-feed.xml
oozie-joda.log.txt

> Falcon feed fails to aws s3; Oozie joda time version does not meet required 
> jar version 2.2 or later
> 
>
> Key: OOZIE-2826
> URL: https://issues.apache.org/jira/browse/OOZIE-2826
> Project: Oozie
>  Issue Type: Bug
>  Components: workflow
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Priority: Minor
> Fix For: trunk
>
> Attachments: oozie-joda.example-feed.xml, oozie-joda.log.txt
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Symptom:
> 2016-11-04 17:34:50,283 WARN [main] 
> com.amazonaws.services.s3.internal.S3MetadataResponseHandler: Unable to parse 
> last modified date: Wed, 02 Nov 2016 23:05:38 GMT
> java.lang.IllegalStateException: Joda-time 2.2 or later version is 
> required, but found version: 2.1
>   at com.amazonaws.util.DateUtils.handleException(DateUtils.java:149)
>   at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:195)
>   at 
> com.amazonaws.services.s3.internal.ServiceUtils.parseRfc822Date(ServiceUtils.java:78)
>   at 
> com.amazonaws.services.s3.internal.AbstractS3ResponseHandler.populateObjectMetadata(AbstractS3ResponseHandler.java:115)
>   at 
> com.amazonaws.services.s3.internal.S3MetadataResponseHandler.handle(S3MetadataResponseHandler.java:32)
>   at 
> com.amazonaws.services.s3.internal.S3MetadataResponseHandler.handle(S3MetadataResponseHandler.java:25)
>   at 
> com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1072)
>   at 
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:746)
>   at 
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
>   at 
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
>   at 
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
>   at 
> com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1050)
>   at 
> com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1027)
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:850)
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1403)
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:1343)
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1308)
>   at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1898)
>   at 
> org.apache.falcon.util.DistCPOptionsUtil.getDistCpOptions(DistCPOptionsUtil.java:90)
>   at 
> org.apache.falcon.replication.FeedReplicator.getDistCpOptions(FeedReplicator.java:229)
>   at 
> org.apache.falcon.replication.FeedReplicator.run(FeedReplicator.java:70)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>   at 
> org.apache.falcon.replication.FeedReplicator.main(FeedReplicator.java:64)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56)
>   at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:51)
>   at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:242)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
>   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: java.lang.IllegalArgumentException: Invalid

[jira] [Comment Edited] (OOZIE-2406) Completely rewrite GraphGenerator code

2017-03-16 Thread Robert Kanter (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924920#comment-15924920
 ] 

Robert Kanter edited comment on OOZIE-2406 at 3/16/17 4:57 PM:
---

Someone was suggesting the [d3|https://d3js.org/] Javascript library.  It looks 
like it can do a lot of neat things and seemly widely used (so shouldn't go 
extinct).  
Here's some examples/tutorials that would be relevant to us:
http://bl.ocks.org/d3noob/5141278
http://bl.ocks.org/mbostock/2706022
https://tylercipriani.com/blog/2016/03/21/Visualizing-Git-Merkle-DAG-with-D3.js/
http://bl.ocks.org/sathomas/11550728
http://www.coppelia.io/2014/07/an-a-to-z-of-extra-features-for-the-d3-force-layout/
http://www.d3noob.org/2013/03/d3js-force-directed-graph-example-basic.html

Looks like it's BSD license, so that should be okay.


was (Author: rkanter):
Someone was suggesting the [d3|https://d3js.org/] Javascript library.  It looks 
like it can do a lot of neat things and seemly widely used (so shouldn't go 
extinct).  
Here's some examples that would be relevant to us:
http://bl.ocks.org/d3noob/5141278
http://bl.ocks.org/mbostock/2706022
https://tylercipriani.com/blog/2016/03/21/Visualizing-Git-Merkle-DAG-with-D3.js/

Looks like it's BSD license, so that should be okay.

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2826) Falcon feed fails to aws s3; Oozie joda time version does not meet required jar version 2.2 or later

2017-03-16 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/OOZIE-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Péter Gergő Barna updated OOZIE-2826:
-
Description: 
Symptom:

2016-11-04 17:34:50,283 WARN [main] 
com.amazonaws.services.s3.internal.S3MetadataResponseHandler: Unable to parse 
last modified date: Wed, 02 Nov 2016 23:05:38 GMT
java.lang.IllegalStateException: Joda-time 2.2 or later version is 
required, but found version: 2.1
at com.amazonaws.util.DateUtils.handleException(DateUtils.java:149)
at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:195)
at 
com.amazonaws.services.s3.internal.ServiceUtils.parseRfc822Date(ServiceUtils.java:78)
at 
com.amazonaws.services.s3.internal.AbstractS3ResponseHandler.populateObjectMetadata(AbstractS3ResponseHandler.java:115)
at 
com.amazonaws.services.s3.internal.S3MetadataResponseHandler.handle(S3MetadataResponseHandler.java:32)
at 
com.amazonaws.services.s3.internal.S3MetadataResponseHandler.handle(S3MetadataResponseHandler.java:25)
at 
com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1072)
at 
com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:746)
at 
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
at 
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
at 
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
at 
com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1050)
at 
com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1027)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:850)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1403)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:1343)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1308)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1898)
at 
org.apache.falcon.util.DistCPOptionsUtil.getDistCpOptions(DistCPOptionsUtil.java:90)
at 
org.apache.falcon.replication.FeedReplicator.getDistCpOptions(FeedReplicator.java:229)
at 
org.apache.falcon.replication.FeedReplicator.run(FeedReplicator.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at 
org.apache.falcon.replication.FeedReplicator.main(FeedReplicator.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:51)
at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:242)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.IllegalArgumentException: Invalid format: "Wed, 02 Nov 
2016 23:05:38 GMT" is malformed at "GMT"
at 
org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:747)
at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:193)
... 41 more


  was:
Symptom:

2016-11-04 17:34:50,283 WARN [main] 
com.amazonaws.services.s3.internal.S3MetadataResponseHandler: Unable to parse 
last modified date: Wed, 02 Nov 2016 23:05:38 GMT
java.lang.IllegalStateException: Joda-time 2.2 or later version is required, 
but found version: 2.1
at com.amazonaws.util.DateUtils.handleException(DateUtils.java:149)
at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:195)
at 
com.amazonaws.services.s3.internal.ServiceUtils.parseRfc822Date(ServiceUtils.java:78)

[jira] [Created] (OOZIE-2826) Falcon feed fails to aws s3; Oozie joda time version does not meet required jar version 2.2 or later

2017-03-16 Thread JIRA

Péter Gergő Barna created OOZIE-2826:


 Summary: Falcon feed fails to aws s3; Oozie joda time version does 
not meet required jar version 2.2 or later
 Key: OOZIE-2826
 URL: https://issues.apache.org/jira/browse/OOZIE-2826
 Project: Oozie
  Issue Type: Bug
  Components: workflow
Affects Versions: 4.3.0
Reporter: Péter Gergő Barna
Priority: Minor
 Fix For: trunk


Symptom:

2016-11-04 17:34:50,283 WARN [main] 
com.amazonaws.services.s3.internal.S3MetadataResponseHandler: Unable to parse 
last modified date: Wed, 02 Nov 2016 23:05:38 GMT
java.lang.IllegalStateException: Joda-time 2.2 or later version is required, 
but found version: 2.1
at com.amazonaws.util.DateUtils.handleException(DateUtils.java:149)
at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:195)
at 
com.amazonaws.services.s3.internal.ServiceUtils.parseRfc822Date(ServiceUtils.java:78)
at 
com.amazonaws.services.s3.internal.AbstractS3ResponseHandler.populateObjectMetadata(AbstractS3ResponseHandler.java:115)
at 
com.amazonaws.services.s3.internal.S3MetadataResponseHandler.handle(S3MetadataResponseHandler.java:32)
at 
com.amazonaws.services.s3.internal.S3MetadataResponseHandler.handle(S3MetadataResponseHandler.java:25)
at 
com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1072)
at 
com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:746)
at 
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
at 
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
at 
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
at 
com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1050)
at 
com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1027)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:850)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1403)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:1343)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1308)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1898)
at 
org.apache.falcon.util.DistCPOptionsUtil.getDistCpOptions(DistCPOptionsUtil.java:90)
at 
org.apache.falcon.replication.FeedReplicator.getDistCpOptions(FeedReplicator.java:229)
at 
org.apache.falcon.replication.FeedReplicator.run(FeedReplicator.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at 
org.apache.falcon.replication.FeedReplicator.main(FeedReplicator.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:51)
at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:242)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.IllegalArgumentException: Invalid format: "Wed, 02 Nov 
2016 23:05:38 GMT" is malformed at "GMT"
at 
org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:747)
at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:193)
... 41 more




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2819) Make Oozie REST API accept multibyte characters for script Actions

2017-03-16 Thread Attila Sasvari (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2819:
--
Attachment: OOZIE-2819-03-amendment.patch

@rkanter amendment patch

> Make Oozie REST API accept multibyte characters for script Actions
> --
>
> Key: OOZIE-2819
> URL: https://issues.apache.org/jira/browse/OOZIE-2819
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Fix For: 5.0.0
>
> Attachments: OOZIE-2819-00.patch, OOZIE-2819-01.patch, 
> OOZIE-2819-02.patch, OOZIE-2819-03-amendment.patch, OOZIE-2819-03.patch
>
>
> Submitted Pig action with client side xml failed via proxy submission when it 
> contained multibyte characters.
> {code}
> curl -i  -X POST -d @/tmp/pig.xml -H 'Content-Type: application/XML; 
> charset=UTF-8' 
> 'http://'localhost':11000/oozie/v1/jobs?jobtype=pig=start'
> {code}
> Where
> {code}
> $ hdfs dfs -cat /tmp/encoding/input.txt
> 松
> 林檎
> 松
> {code}
> {code}
> $ cat /tmp/pig.xml 
> 
> 
> fs.default.name
> hdfs://localhost:8020/
> 
> 
> mapred.job.tracker
> localhost:8032
> 
> 
> user.name
> hdfs
> 
> 
> oozie.pig.script
> 
> 
> 
> oozie.pig.script.params.size
> 0
> 
> 
> oozie.pig.script.options.size
> 0
> 
> 
> oozie.libpath
> hdfs:///user/oozie/share/lib
> 
> 
> oozie.use.system.libpath
> true
> 
> 
> oozie.proxysubmission
> true
> 
> 
> {code}
> In the Oozie launcher log, I could see
> {code}
> lines = LOAD 'hdfs:///tmp/encoding/input.txt' USING PigStorage('\n') AS 
> line;test = FILTER lines BY line == '~';STORE test INTO 
> 'hdfs:///tmp/encoding/output' USING PigStorage('\n');
> {code}
> was used instead of the intended 松



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2820) more spelling errors in exceptions and logging

2017-03-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928259#comment-15928259
 ] 

Hadoop QA commented on OOZIE-2820:
--

Testing JIRA OOZIE-2820

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [core].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1892
.Tests failed: 3
.Tests errors: 0

.The patch failed the following testcases:

.  
testBundleStatusTransitWithLock(org.apache.oozie.service.TestStatusTransitService)
.  
testActionKillCommandDate(org.apache.oozie.command.coord.TestCoordActionsKillXCommand)
.  
testCoordMaterializeTriggerService3(org.apache.oozie.service.TestCoordMaterializeTriggerService)

.Tests failing with errors:
.  

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3697/

> more spelling errors in exceptions and logging
> --
>
> Key: OOZIE-2820
> URL: https://issues.apache.org/jira/browse/OOZIE-2820
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Reporter: Grant Sohn
>Assignee: Laszlo Zeke
>Priority: Trivial
> Attachments: OOZIE-2820-1.patch, OOZIE-2820-2.patch
>
>
> core/src/main/java/org/apache/oozie/servlet/JVMInfo.java: 
> sb.append("Deamon 
> Threads=").append(threadMXBean.getDaemonThreadCount());
> Deamon -> Daemon
> core/src/main/java/org/apache/oozie/util/db/SqlStatement.java:
> throw new RuntimeException("Need atleast 1 Table !!");
> core/src/main/java/org/apache/oozie/util/db/SqlStatement.java:
> throw new RuntimeException("Need atleast 1 Column !!");
> atleast -> at least
> core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java
> INSTACE -> INSTANCE
> core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java
> UNKOWN_ERROR -> UNKNOWN_ERROR



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: oozie-trunk-precommit-build #3697

2017-03-16 Thread Apache Jenkins Server

See 


--
[...truncated 1.33 MB...]
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/workflow/lite/DecisionNodeHandler.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/workflow/lite/DBLiteWorkflowLib.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/workflow/lite/EndNodeDef.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/workflow/lite/KillNodeDef.KillNodeHandler.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/workflow/lite/LiteWorkflowApp.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/workflow/lite/LiteWorkflowValidator.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/workflow/lite/NodeHandler.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/workflow/lite/NodeHandler.Context.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/workflow/lite/LiteWorkflowInstance.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/workflow/lite/JoinNodeDef.JoinNodeHandler.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/workflow/WorkflowInstance.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/workflow/WorkflowApp.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/DefaultConnectionContext.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/JMSExceptionListener.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/ConnectionContext.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/JMSJobEventListener.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/MessageReceiver.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/JMSSLAEventListener.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/JMSConnectionInfo.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/MessageHandler.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/store/Store.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/store/StoreStatusFilter.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/store/OozieSchema.OozieColumn.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/store/OozieSchema.OozieTable.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/store/SLAStore.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/store/OozieSchema.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/store/WorkflowStore.html
 longer than 100 characters.

[jira] [Commented] (OOZIE-2816) Strip out the first command word from Sqoop action if its "sqoop"

2017-03-16 Thread Peter Cseh (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928178#comment-15928178
 ] 

Peter Cseh commented on OOZIE-2816:
---

Good catch, I've missed the  part also...
This solution contains duplicated code-like parts, but with different type of 
lists. Can you create a method like {{removeSqoopCommandIfPresent}} and call 
that in both cases? It would need some refactoring, but having the removal 
logic at one place would be nice. Maybe we could create a List from the 
List before processing it further.

> Strip out the first command word from Sqoop action if its "sqoop"
> -
>
> Key: OOZIE-2816
> URL: https://issues.apache.org/jira/browse/OOZIE-2816
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 4.3.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Trivial
> Attachments: OOZIE-2816.000.patch, OOZIE-2816.001.patch, 
> OOZIE-2816.002.patch
>
>
> A [sorta 
> popular|https://www.google.com/search?q=%22No+such+sqoop+tool:+sqoop%22+oozie&*]
>  user error is to specify the command part of the sqoop action in the style 
> of the CLI: "sqoop import …" instead of just the required "import …". This 
> causes the Sqoop action to predictably fail with:
> {code}
> No such sqoop tool: sqoop. See 'sqoop help'.
> Intercepting System.exit(1)
> {code}
> A simple enhancement can prevent this burden on user. We can skip the first 
> command token if its "sqoop".



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2816) Strip out the first command word from Sqoop action if its "sqoop"

2017-03-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928136#comment-15928136
 ] 

Hadoop QA commented on OOZIE-2816:
--

Testing JIRA OOZIE-2816

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1896
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3696/

> Strip out the first command word from Sqoop action if its "sqoop"
> -
>
> Key: OOZIE-2816
> URL: https://issues.apache.org/jira/browse/OOZIE-2816
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 4.3.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Trivial
> Attachments: OOZIE-2816.000.patch, OOZIE-2816.001.patch, 
> OOZIE-2816.002.patch
>
>
> A [sorta 
> popular|https://www.google.com/search?q=%22No+such+sqoop+tool:+sqoop%22+oozie&*]
>  user error is to specify the command part of the sqoop action in the style 
> of the CLI: "sqoop import …" instead of just the required "import …". This 
> causes the Sqoop action to predictably fail with:
> {code}
> No such sqoop tool: sqoop. See 'sqoop help'.
> Intercepting System.exit(1)
> {code}
> A simple enhancement can prevent this burden on user. We can skip the first 
> command token if its "sqoop".



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2820) more spelling errors in exceptions and logging

2017-03-16 Thread Laszlo Zeke (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Zeke updated OOZIE-2820:
---
Attachment: OOZIE-2820-2.patch

Fixed thrown -> threw and passed exception message.

> more spelling errors in exceptions and logging
> --
>
> Key: OOZIE-2820
> URL: https://issues.apache.org/jira/browse/OOZIE-2820
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Reporter: Grant Sohn
>Assignee: Laszlo Zeke
>Priority: Trivial
> Attachments: OOZIE-2820-1.patch, OOZIE-2820-2.patch
>
>
> core/src/main/java/org/apache/oozie/servlet/JVMInfo.java: 
> sb.append("Deamon 
> Threads=").append(threadMXBean.getDaemonThreadCount());
> Deamon -> Daemon
> core/src/main/java/org/apache/oozie/util/db/SqlStatement.java:
> throw new RuntimeException("Need atleast 1 Table !!");
> core/src/main/java/org/apache/oozie/util/db/SqlStatement.java:
> throw new RuntimeException("Need atleast 1 Column !!");
> atleast -> at least
> core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java
> INSTACE -> INSTANCE
> core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java
> UNKOWN_ERROR -> UNKNOWN_ERROR



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 57680: OOZIE-2812 SparkConfigurationService should support loading configurations from multiple Spark versions

2017-03-16 Thread András Piros


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57680/#review169109
---




core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java
Lines 82 (patched)


`sharelibName` would be a better name



core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java
Lines 82-88 (original), 83-89 (patched)


tabs vs spaces



core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java
Line 83 (original), 84 (patched)


What about extracting `SparkConfigurationService` to a local variable? See 
Law of Demeter



core/src/main/java/org/apache/oozie/service/SparkConfigurationService.java
Lines 39-47 (original), 39-47 (patched)


Honestly, either more descriptive constant names, or javadoc, or a neatly 
named builder would be of great value here.



core/src/main/java/org/apache/oozie/service/SparkConfigurationService.java
Line 40 (original), 40 (patched)


Why not call that `CONFIGURATIONS_SUFFIX` instead?



core/src/main/java/org/apache/oozie/service/SparkConfigurationService.java
Line 43 (original), 43 (patched)


Why not call that rather `BLACKLIST_SUFFIX`?



core/src/main/java/org/apache/oozie/service/SparkConfigurationService.java
Line 48 (original), 48 (patched)


Every time I see something like `Map>` 
reminds me always to use Guava's `Multimap`: 
https://github.com/google/guava/wiki/NewCollectionTypesExplained#multimap



core/src/main/java/org/apache/oozie/service/SparkConfigurationService.java
Lines 54 (patched)


Are you sure `^` is not needed as a prefix, and `$` is needed at the end? 
It would be best to have some unit tests that show what regexps are accepted 
and which ones not.



core/src/main/java/org/apache/oozie/service/SparkConfigurationService.java
Lines 55 (patched)


`sparkConfigName`



core/src/main/java/org/apache/oozie/service/SparkConfigurationService.java
Lines 76 (patched)


What about using `LinkedHashSet` which preserves addition order?



core/src/main/java/org/apache/oozie/service/SparkConfigurationService.java
Lines 77 (patched)


`blacklistedPropertyName`



core/src/main/java/org/apache/oozie/service/SparkConfigurationService.java
Lines 81-102 (original), 89-114 (patched)


This functionality should really be extracted to a separate 
`SparkConfigLoader` class - `SparkConfigurationService` does way too much ATM.



core/src/main/java/org/apache/oozie/service/SparkConfigurationService.java
Lines 119-123 (original), 131-135 (patched)


Should also be extracted to a separate class.



core/src/main/java/org/apache/oozie/service/SparkConfigurationService.java
Lines 145-153 (original), 160-168 (patched)


tabs vs spaces



core/src/test/java/org/apache/oozie/service/TestSparkConfigurationService.java
Line 50 (original), 50 (patched)


This test case is really undreadable. Can you please split that up to 
multiple test cases and add new ones that cover your new regex and different 
kinds of Spark configuration mixes that cover the use cases we'd like to 
address: Spark 1.6 standalone, Spark 2.1 standalone, Spark 1.6 w/ Spark 2.0 w/ 
Spark 2.1, etc.



core/src/test/java/org/apache/oozie/service/TestSparkConfigurationService.java
Line 63 (original), 63 (patched)


Maybe extract `"spark"` to a constant?



core/src/test/java/org/apache/oozie/service/TestSparkConfigurationService.java
Lines 160 (patched)


From the test method name alone I cannot conclude what is the test case 
performing and what are the expected outcomes / end states. Please rename it.



core/src/test/java/org/apache/oozie/service/TestSparkConfigurationService.java
Lines 161-163 (patched)


Wow, I cannot really understand who is who, the sheer amount of one-letter 
constants is amazing.



core/src/test/java/org/apache/oozie/service/TestSparkConfigurationService.java
Lines 173-175 (patched)

[jira] Subscription: Oozie Patch Available

2017-03-16 Thread jira

Issue Subscription
Filter: Oozie Patch Available (86 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2619  Make  Hive action defaults to match hive defaults when running from 
command line
https://issues-test.apache.org/jira/browse/OOZIE-2619
OOZIE-2618  Include hive-common in hive-beeline as compile component for 
hive-server2 (hive2 action)
https://issues-test.apache.org/jira/browse/OOZIE-2618
OOZIE-2617  Read properties files in action configurations
https://issues-test.apache.org/jira/browse/OOZIE-2617
OOZIE-2616  Add Tez profile for Hive and Pig sharelibs
https://issues-test.apache.org/jira/browse/OOZIE-2616
OOZIE-2606  Set spark.yarn.jars to fix Spark 2.0 with Oozie
https://issues-test.apache.org/jira/browse/OOZIE-2606
OOZIE-2589  CompletedActionXCommand priority param is of no effect
https://issues-test.apache.org/jira/browse/OOZIE-2589
OOZIE-2588  Support getting credentials for same cluster hcat when credentials 
config is empty
https://issues-test.apache.org/jira/browse/OOZIE-2588
OOZIE-2585  Disable or remove test case 
TestPartitionDependencyManagerEhcache.testMemoryUsageAndSpeedOverflowToDisk and 
testMemoryUsageAndSpeed
https://issues-test.apache.org/jira/browse/OOZIE-2585
OOZIE-2584  Eliminate Thread.sleep() calls in TestMemoryLocks
https://issues-test.apache.org/jira/browse/OOZIE-2584
OOZIE-2583  oozie throws EL Exception when reference variable name containing 
dot
https://issues-test.apache.org/jira/browse/OOZIE-2583
OOZIE-2582  Populating external child Ids for action failures
https://issues-test.apache.org/jira/browse/OOZIE-2582
OOZIE-2573  dataset url  which contains spaces can not be handled rightly
https://issues-test.apache.org/jira/browse/OOZIE-2573
OOZIE-2572  SLA DURATION miss not shown when job is running for longer than 
expected time
https://issues-test.apache.org/jira/browse/OOZIE-2572
OOZIE-2571  Add scala.binary.version Maven property so that Scala 2.11 can be 
used
https://issues-test.apache.org/jira/browse/OOZIE-2571
OOZIE-2569  Adding yarn-site, core-site, hdfs-site and mapred-site into spark 
launcher 
https://issues-test.apache.org/jira/browse/OOZIE-2569
OOZIE-2568  SSH  action pretends to retry automaticly when it failed
https://issues-test.apache.org/jira/browse/OOZIE-2568
OOZIE-2564  Create new log4j config for unit tests so that logged messages 
contain threads
https://issues-test.apache.org/jira/browse/OOZIE-2564
OOZIE-2552  Update ActiveMQ version for security and other fixes
https://issues-test.apache.org/jira/browse/OOZIE-2552
OOZIE-2538  Update HttpClient versions to close security vulnerabilities
https://issues-test.apache.org/jira/browse/OOZIE-2538
OOZIE-2534  Versioned action libs (similar to how SharedLibs works)
https://issues-test.apache.org/jira/browse/OOZIE-2534
OOZIE-2530  Remove jline from parent and sharelib pig pom
https://issues-test.apache.org/jira/browse/OOZIE-2530
OOZIE-2528  Print out environment variables in LauncherMapper
https://issues-test.apache.org/jira/browse/OOZIE-2528
OOZIE-2521  Filter options are case sensitive for jobtye=bundle
https://issues-test.apache.org/jira/browse/OOZIE-2521
OOZIE-2518  Oozie is creating many files and directories in /tmp per day
https://issues-test.apache.org/jira/browse/OOZIE-2518
OOZIE-2517  Add support for startCreatedTime and endCreatedTime filters for 
coord and bundles
https://issues-test.apache.org/jira/browse/OOZIE-2517
OOZIE-2513  log.scan.duration should not be used for error and audit logs
https://issues-test.apache.org/jira/browse/OOZIE-2513
OOZIE-2512  ShareLibservice returns incorrect path for jar
https://issues-test.apache.org/jira/browse/OOZIE-2512
OOZIE-2508  Documentation change for Coord action rerun [OOZIE-1735]
https://issues-test.apache.org/jira/browse/OOZIE-2508
OOZIE-2501  ZK reentrant lock doesn't work for few cases
https://issues-test.apache.org/jira/browse/OOZIE-2501
OOZIE-2500  -DtestJarSimple option mentioned in minioozie doc does not work
https://issues-test.apache.org/jira/browse/OOZIE-2500
OOZIE-2499  map-reduce launcher does not need distributed files, archives 
except  jar of input/outputformat class
https://issues-test.apache.org/jira/browse/OOZIE-2499
OOZIE-2498  Oozie CallerId configuration for downstream components 
https://issues-test.apache.org/jira/browse/OOZIE-2498
OOZIE-2495  change action status from  ErrorType.NON_TRANSIENT to TRANSIENT 
when SSH action occurs AUTH_FAILED occasionally
https://issues-test.apache.org/jira/browse/OOZIE-2495
OOZIE-2491  oozie acl cannot specify group,it does`t work
https://issues-test.apache.org/jira/browse/OOZIE-2491

[jira] Subscription: Oozie Patch Available

2017-03-16 Thread jira

Issue Subscription
Filter: Oozie Patch Available (93 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2824  Fix typos in documentation
https://issues.apache.org/jira/browse/OOZIE-2824
OOZIE-2820  more spelling errors in exceptions and logging
https://issues.apache.org/jira/browse/OOZIE-2820
OOZIE-2817  Increase test case stability in pre-commit job
https://issues.apache.org/jira/browse/OOZIE-2817
OOZIE-2816  Strip out the first command word from Sqoop action if its "sqoop"
https://issues.apache.org/jira/browse/OOZIE-2816
OOZIE-2813  Remove tabs and trailing whitespaces from oozie-defaul.xml
https://issues.apache.org/jira/browse/OOZIE-2813
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2796  oozie.action.keep.action.dir not getting noticed
https://issues.apache.org/jira/browse/OOZIE-2796
OOZIE-2795  Create lib directory or symlink for Oozie CLI during packaging
https://issues.apache.org/jira/browse/OOZIE-2795
OOZIE-2791  ShareLib installation may fail on busy Hadoop clusters
https://issues.apache.org/jira/browse/OOZIE-2791
OOZIE-2785  Master is not compiling with hadoop-1 profile
https://issues.apache.org/jira/browse/OOZIE-2785
OOZIE-2784  Include WEEK as a parameter in the Coordinator Expression Language 
Evaulator
https://issues.apache.org/jira/browse/OOZIE-2784
OOZIE-2780  Upgrade hadoop-common version to 2.6.0
https://issues.apache.org/jira/browse/OOZIE-2780
OOZIE-2779  Mask Hive2 action Beeline JDBC password
https://issues.apache.org/jira/browse/OOZIE-2779
OOZIE-2769  Extend FS action to allow setrep on a file
https://issues.apache.org/jira/browse/OOZIE-2769
OOZIE-2753  Update Docs for Jetty Tomcat changes
https://issues.apache.org/jira/browse/OOZIE-2753
OOZIE-2751  LocalOozieClient is missing methods from OozieClient
https://issues.apache.org/jira/browse/OOZIE-2751
OOZIE-2746  Several tests failure in TestV2ValidateServlet.java
https://issues.apache.org/jira/browse/OOZIE-2746
OOZIE-2739  Remove property expansion pattern from ShellMain's log4j properties 
content
https://issues.apache.org/jira/browse/OOZIE-2739
OOZIE-2736  Reduce the number of threads during test execution
https://issues.apache.org/jira/browse/OOZIE-2736
OOZIE-2733  change org.apache.hadoop.fs.permission.AccessControlException to 
org.apache.hadoop.security.AccessControlException
https://issues.apache.org/jira/browse/OOZIE-2733
OOZIE-2726  Flaky test due to daylight saving changes
https://issues.apache.org/jira/browse/OOZIE-2726
OOZIE-2718  Improve -dryrun for bundles
https://issues.apache.org/jira/browse/OOZIE-2718
OOZIE-2713  Provide an option to log out contents of the localized jar files
https://issues.apache.org/jira/browse/OOZIE-2713
OOZIE-2701  Oozie to support Multiple HCatalog URIs
https://issues.apache.org/jira/browse/OOZIE-2701
OOZIE-2694  Add logging for FsActionExecutor 
https://issues.apache.org/jira/browse/OOZIE-2694
OOZIE-2693  SimpleHCatDependencyCache.removeMissingDependency can throw NPE
https://issues.apache.org/jira/browse/OOZIE-2693
OOZIE-2692  Oozie job submit doesn't report error message to user if there is 
any issue with job conf
https://issues.apache.org/jira/browse/OOZIE-2692
OOZIE-2684  Bad database schema error for WF_ACTIONS table
https://issues.apache.org/jira/browse/OOZIE-2684
OOZIE-2681  fix javadoc to compile on JDK8 again
https://issues.apache.org/jira/browse/OOZIE-2681
OOZIE-2670  Upgrade Hbase to 1.2
https://issues.apache.org/jira/browse/OOZIE-2670
OOZIE-2668  Status update and recovery problems when coord action and its 
children not in sync
https://issues.apache.org/jira/browse/OOZIE-2668
OOZIE-2662  DB migration fails if DB is too big
https://issues.apache.org/jira/browse/OOZIE-2662
OOZIE-2652  Skip trash while deleting hive table partition
https://issues.apache.org/jira/browse/OOZIE-2652
OOZIE-2650  Retry coord start on database exceptions
https://issues.apache.org/jira/browse/OOZIE-2650
OOZIE-2644  Skip queuing Notification Commands when there's nothing to notify
https://issues.apache.org/jira/browse/OOZIE-2644
OOZIE-2641  rerunning an oozie coordinator job stucks on waiting
https://issues.apache.org/jira/browse/OOZIE-2641
OOZIE-2629  Get OozieShareLibCLI to perform a final rename to destPath when 
creating a Sharelib
https://issues.apache.org/jira/browse/OOZIE-2629
OOZIE-2618  Include hive-common in hive-beeline as compile component for 
hive-server2 (hive2 action)
https://issues.apache.org/jira/browse/OOZIE-2618

[jira] [Commented] (OOZIE-2540) Create a PySpark example

2017-03-16 Thread Abhishek Bafna (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927587#comment-15927587
 ] 

Abhishek Bafna commented on OOZIE-2540:
---

Thanks [~rkanter] and others for review and committing it.

> Create a PySpark example
> 
>
> Key: OOZIE-2540
> URL: https://issues.apache.org/jira/browse/OOZIE-2540
> Project: Oozie
>  Issue Type: Task
>  Components: examples
>Reporter: Robert Kanter
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2540-00.patch
>
>
> Now that we have PySpark working correctly in the Spark Action, we should 
> make an example that runs a PySpark job to give users an example of how to do 
> it.  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2819) Make Oozie REST API accept multibyte characters for script Actions

[jira] [Updated] (OOZIE-2826) Falcon feed fails to aws s3; Oozie joda time version does not meet required jar version 2.2 or later

[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code

[jira] [Updated] (OOZIE-2826) Falcon feed fails to aws s3; Oozie joda time version does not meet required jar version 2.2 or later

[jira] [Comment Edited] (OOZIE-2406) Completely rewrite GraphGenerator code

[jira] [Updated] (OOZIE-2826) Falcon feed fails to aws s3; Oozie joda time version does not meet required jar version 2.2 or later

[jira] [Created] (OOZIE-2826) Falcon feed fails to aws s3; Oozie joda time version does not meet required jar version 2.2 or later

[jira] [Updated] (OOZIE-2819) Make Oozie REST API accept multibyte characters for script Actions

[jira] [Commented] (OOZIE-2820) more spelling errors in exceptions and logging

Build failed in Jenkins: oozie-trunk-precommit-build #3697

[jira] [Commented] (OOZIE-2816) Strip out the first command word from Sqoop action if its "sqoop"

[jira] [Commented] (OOZIE-2816) Strip out the first command word from Sqoop action if its "sqoop"

[jira] [Updated] (OOZIE-2820) more spelling errors in exceptions and logging

Re: Review Request 57680: OOZIE-2812 SparkConfigurationService should support loading configurations from multiple Spark versions

[jira] Subscription: Oozie Patch Available

[jira] Subscription: Oozie Patch Available

[jira] [Commented] (OOZIE-2540) Create a PySpark example

17 matches

Site Navigation

Mail list logo

Footer information