from:"Ryota Egashira"

Re: [ANNOUNCE] New Oozie PMC Chair - Robert Kanter

2015-10-14 Thread Ryota Egashira

Robert, well deserved, congratulations!
Ryota
 On Tuesday, October 13, 2015 8:55 PM, Alejandro Abdelnur 
 wrote:
   

 Robert, congratulations!!

On Tue, Oct 13, 2015 at 8:50 PM, Aravindakshan Srinivasan <
s_arav...@yahoo.com.invalid> wrote:

> Congratulations Robert! Great news. Regards,Aravind
>      From: Mohammad Islam 
>  To: Oozie ; User 
>  Sent: Wednesday, October 7, 2015 12:25 AM
>  Subject: [ANNOUNCE] New Oozie PMC Chair - Robert Kanter
>
> Hi,
>
> I'm very happy to announce that Robert Kanter is taking over as the new VP
> of the Apache Oozie project. I think I don't need any introduction of
> Robert to the Ozoie community.
>
> Please join me in congratulating Robert on this new role!
>
>
> Regards,
> Mohammad
>
>
>

[jira] [Updated] (OOZIE-2135) Partition existence check by HCatClient.getPartitions causes high load on HCatalog

2015-07-10 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2135:
--
Description: 
HCatClient.getPartitions is used to check partition existence in Oozie, but it 
is known to be inefficient for large table with many partitions. Hive team 
working on new method, partitionExists, which has performance optimization, and 
Oozie should switch to this once committed.
[~mithun]  could you please share Hive JIRA for reference? 

> Partition existence check by HCatClient.getPartitions causes high load on 
> HCatalog
> --
>
> Key: OOZIE-2135
> URL: https://issues.apache.org/jira/browse/OOZIE-2135
> Project: Oozie
>  Issue Type: Bug
>    Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
>
> HCatClient.getPartitions is used to check partition existence in Oozie, but 
> it is known to be inefficient for large table with many partitions. Hive team 
> working on new method, partitionExists, which has performance optimization, 
> and Oozie should switch to this once committed.
> [~mithun]  could you please share Hive JIRA for reference? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Jenkins is currently unable to login to JIRA

2015-07-09 Thread Ryota Egashira

Giri
https://issues.apache.org/jira/browse/INFRA-9948

"you'll need to configure the build to use Java 7, and possibly configure the 
build to target Java 6.  Java 6 is no longer compatible with the updated 
openssl and httpd deployment on the SSL proxy serving issues.apache.org"

how to make JAVA_HOME point to Java 7?

 On Tuesday, July 7, 2015 11:48 PM, Giridharan Kesavan 
 wrote:

 Somehow I missed this email. Will look into it in the morning 

Sent from Outlook

On Tue, Jul 7, 2015 at 3:55 PM -0700, "Ryota Egashira"  
wrote:

this is still an issue. opened INFRA 
ticket.https://issues.apache.org/jira/browse/INFRA-9948

Ryota

On Monday, June 22, 2015 12:52 PM, Rohini Palaniswamy  
wrote:

+Giri who might know better.

On Wed, Jun 17, 2015 at 5:35 PM, Robert Kanter  wrote:

> Recent builds have been failing, and not posting to JIRA because of this:
>
> Adding comment to JIRA
> Unable to log in to server:
> https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2 with user:
> hadoopqa.
>  Cause: ; nested exception is:
> javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate
> DH keypair
>
> test-patch exit code: 1
>
> (Taken from the end of
> https://builds.apache.org/job/oozie-trunk-precommit-build/2419/console)
>
> Any ideas?
>
>
> - Robert
>

Re: Jenkins is currently unable to login to JIRA

2015-07-07 Thread Ryota Egashira

this is still an issue. opened INFRA 
ticket.https://issues.apache.org/jira/browse/INFRA-9948

Ryota 


 On Monday, June 22, 2015 12:52 PM, Rohini Palaniswamy 
 wrote:
   

 +Giri who might know better.

On Wed, Jun 17, 2015 at 5:35 PM, Robert Kanter  wrote:

> Recent builds have been failing, and not posting to JIRA because of this:
>
> Adding comment to JIRA
> Unable to log in to server:
> https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2 with user:
> hadoopqa.
>  Cause: ; nested exception is:
> javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate
> DH keypair
>
> test-patch exit code: 1
>
> (Taken from the end of
> https://builds.apache.org/job/oozie-trunk-precommit-build/2419/console)
>
> Any ideas?
>
>
> - Robert
>

[jira] [Updated] (OOZIE-2246) CoordinatorInputCheckCommand does not behave properly when har file is one of data dependency and doesn't exist

2015-07-07 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2246:
--
Attachment: OOZIE-2246-v3.patch

> CoordinatorInputCheckCommand does not behave properly when har file is one of 
> data dependency and doesn't exist
> ---
>
> Key: OOZIE-2246
> URL: https://issues.apache.org/jira/browse/OOZIE-2246
> Project: Oozie
>  Issue Type: Bug
>Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Attachments: OOZIE-2246-v2.patch, OOZIE-2246-v3.patch, 
> OOZIE-2246.patch
>
>
> when har file doesn't exist, FileSystem.createFileSystem in 
> HadoopAccessorService throw exception, failing CoordActionInputCheck.
> thus, even thought there are other data dependencies which already exist, it 
> is not reflected on DB.
> coordinator job cannot start until the har file becomes available anyway, and 
> once available, this error doesn't happen, so basic functionality is fine, 
> but it's misleading.
> {code}
> 2014-03-13 22:00:00,051 WARN CallableQueueService$CallableWrapper:542 
> [pool-2-thread-288] - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] 
> exception callable [coord_action_input], E1021: Coord Action Input Check 
> Error: org.apache.oozie.service.HadoopAccessorException: E0902: Exception 
> occured: [Invalid path for the Har Filesystem. No index file in 
> har://:8020/data/2014031322/archive.har]
> org.apache.oozie.command.CommandException: E1021: Coord Action Input Check 
> Error: org.apache.oozie.service.HadoopAccessorException: E0902: Exception 
> occured: [Invalid path for the Har Filesystem. No index file in 
> har://:8020/data/2014031322/archive.har]
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:182)
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:67)
> at org.apache.oozie.command.XCommand.call(XCommand.java:280)
> at 
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: 
> org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: 
> [Invalid path for the Har Filesystem. No index file in 
> har://:8020/data/2014031322/archive.har]
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.pathExists(CoordActionInputCheckXCommand.java:493)
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkListOfPaths(CoordActionInputCheckXCommand.java:459)
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkResolvedUris(CoordActionInputCheckXCommand.java:429)
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkInput(CoordActionInputCheckXCommand.java:259)
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:132)
> ... 6 more
> Caused by: org.apache.oozie.service.HadoopAccessorException: E0902: Exception 
> occured: [Invalid path for the Har Filesystem. No index file in 
> har://:8020/data/2014031322/archive.har]
> at 
> org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:430)
> at 
> org.apache.oozie.dependency.FSURIHandler.getFileSystem(FSURIHandler.java:134)
> at org.apache.oozie.dependency.FSURIHandler.exists(FSURIHandler.java:99)
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.pathExists(CoordActionInputCheckXCommand.java:488)
> ... 10 more
> Caused by: java.io.IOException: Invalid path for the Har Filesystem. No index 
> file in har://:8020/data/2014031322/archive.har
> at org.apache.hadoop.fs.HarFileSystem.initialize(HarFileSystem.java:139)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2160)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:303)
> at 
> org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:422)
> at 
> org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:420)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1284)
> at 
> org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:420)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2246) CoordinatorInputCheckCommand does not behave properly when har file is one of data dependency and doesn't exist

2015-07-07 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2246:
--
Attachment: OOZIE-2246-v2.patch

yes, putting into FSURIhandler is cleaner.  changed it

> CoordinatorInputCheckCommand does not behave properly when har file is one of 
> data dependency and doesn't exist
> ---
>
> Key: OOZIE-2246
> URL: https://issues.apache.org/jira/browse/OOZIE-2246
> Project: Oozie
>  Issue Type: Bug
>Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Attachments: OOZIE-2246-v2.patch, OOZIE-2246.patch
>
>
> when har file doesn't exist, FileSystem.createFileSystem in 
> HadoopAccessorService throw exception, failing CoordActionInputCheck.
> thus, even thought there are other data dependencies which already exist, it 
> is not reflected on DB.
> coordinator job cannot start until the har file becomes available anyway, and 
> once available, this error doesn't happen, so basic functionality is fine, 
> but it's misleading.
> {code}
> 2014-03-13 22:00:00,051 WARN CallableQueueService$CallableWrapper:542 
> [pool-2-thread-288] - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] 
> exception callable [coord_action_input], E1021: Coord Action Input Check 
> Error: org.apache.oozie.service.HadoopAccessorException: E0902: Exception 
> occured: [Invalid path for the Har Filesystem. No index file in 
> har://:8020/data/2014031322/archive.har]
> org.apache.oozie.command.CommandException: E1021: Coord Action Input Check 
> Error: org.apache.oozie.service.HadoopAccessorException: E0902: Exception 
> occured: [Invalid path for the Har Filesystem. No index file in 
> har://:8020/data/2014031322/archive.har]
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:182)
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:67)
> at org.apache.oozie.command.XCommand.call(XCommand.java:280)
> at 
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: 
> org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: 
> [Invalid path for the Har Filesystem. No index file in 
> har://:8020/data/2014031322/archive.har]
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.pathExists(CoordActionInputCheckXCommand.java:493)
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkListOfPaths(CoordActionInputCheckXCommand.java:459)
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkResolvedUris(CoordActionInputCheckXCommand.java:429)
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkInput(CoordActionInputCheckXCommand.java:259)
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:132)
> ... 6 more
> Caused by: org.apache.oozie.service.HadoopAccessorException: E0902: Exception 
> occured: [Invalid path for the Har Filesystem. No index file in 
> har://:8020/data/2014031322/archive.har]
> at 
> org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:430)
> at 
> org.apache.oozie.dependency.FSURIHandler.getFileSystem(FSURIHandler.java:134)
> at org.apache.oozie.dependency.FSURIHandler.exists(FSURIHandler.java:99)
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.pathExists(CoordActionInputCheckXCommand.java:488)
> ... 10 more
> Caused by: java.io.IOException: Invalid path for the Har Filesystem. No index 
> file in har://:8020/data/2014031322/archive.har
> at org.apache.hadoop.fs.HarFileSystem.initialize(HarFileSystem.java:139)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2160)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:303)
> at 
> org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:422)
> at 
> org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:420)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1284)
> at 
> org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:420)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2284) HBaseCredentials should only add hbase-default.xml and hbase-site.xml to actionConf

2015-07-01 Thread Ryota Egashira (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14611201#comment-14611201
 ] 

Ryota Egashira commented on OOZIE-2284:
---

+1

> HBaseCredentials should only add hbase-default.xml and hbase-site.xml to 
> actionConf
> ---
>
> Key: OOZIE-2284
> URL: https://issues.apache.org/jira/browse/OOZIE-2284
> Project: Oozie
>  Issue Type: Bug
>Reporter: Rohini Palaniswamy
>Assignee: Rohini Palaniswamy
> Fix For: trunk
>
> Attachments: OOZIE-2284-1.patch
>
>
>   OOZIE-2205 added support to not automatically add *-default.xml and 
> *-site.xml to the actionConf (just pig and hive for now) and pick them from 
> classpath of the hadoop node. This makes rolling upgrades work well as it 
> will pick up latest config changes without having to upgrade Oozie server. 
> But HBaseCredentials ends up adding the *-default.xml and *-site.xml to the 
> actionConf.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2260) Only set "java.io.tmpdir" to "./tmp" for the AM

2015-06-23 Thread Ryota Egashira (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14598553#comment-14598553
 ] 

Ryota Egashira commented on OOZIE-2260:
---

amend patch committed to trunk,  thanks for review

> Only set "java.io.tmpdir" to "./tmp"  for the AM
> 
>
> Key: OOZIE-2260
> URL: https://issues.apache.org/jira/browse/OOZIE-2260
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Rohini Palaniswamy
>Assignee: Ryota Egashira
> Fix For: trunk
>
> Attachments: OOZIE-2260-amend.patch, OOZIE-2260.patch
>
>
> OOZIE-2209 sets java.io.tmpdir to ./tmp for AM, map and reduce for both 
> launcher and action jobs.
> MapReduceChildJVM already sets java.io.tmpdir to ./tmp
> {code}
> Path childTmpDir = new Path(Environment.PWD.$(),
> YarnConfiguration.DEFAULT_CONTAINER_TEMP_DIR);
> vargs.add("-Djava.io.tmpdir=" + childTmpDir);
> {code}
> So it is only needed to be set for AM in launcher because of uber mode. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2260) Only set "java.io.tmpdir" to "./tmp" for the AM

2015-06-23 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2260:
--
Attachment: OOZIE-2260-amend.patch

turned out that OOZIE-2260 caused some test cases to fail when run against 
-Phadoop-2.   this amend patch to fix them.

> Only set "java.io.tmpdir" to "./tmp"  for the AM
> 
>
> Key: OOZIE-2260
> URL: https://issues.apache.org/jira/browse/OOZIE-2260
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Rohini Palaniswamy
>Assignee: Ryota Egashira
> Fix For: trunk
>
> Attachments: OOZIE-2260-amend.patch, OOZIE-2260.patch
>
>
> OOZIE-2209 sets java.io.tmpdir to ./tmp for AM, map and reduce for both 
> launcher and action jobs.
> MapReduceChildJVM already sets java.io.tmpdir to ./tmp
> {code}
> Path childTmpDir = new Path(Environment.PWD.$(),
> YarnConfiguration.DEFAULT_CONTAINER_TEMP_DIR);
> vargs.add("-Djava.io.tmpdir=" + childTmpDir);
> {code}
> So it is only needed to be set for AM in launcher because of uber mode. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2270) Audit log doesn't get loaded when "audit log" tab is clicked for coord job

2015-06-12 Thread Ryota Egashira (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14583791#comment-14583791
 ] 

Ryota Egashira commented on OOZIE-2270:
---

+1

> Audit log doesn't get loaded when "audit log" tab is clicked for coord job
> --
>
> Key: OOZIE-2270
> URL: https://issues.apache.org/jira/browse/OOZIE-2270
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-2270-V1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2260) Only set "java.io.tmpdir" to "./tmp" for the AM

2015-06-08 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2260:
--
Attachment: OOZIE-2260.patch

> Only set "java.io.tmpdir" to "./tmp"  for the AM
> 
>
> Key: OOZIE-2260
> URL: https://issues.apache.org/jira/browse/OOZIE-2260
> Project: Oozie
>  Issue Type: Bug
>Reporter: Rohini Palaniswamy
>Assignee: Ryota Egashira
> Fix For: trunk
>
> Attachments: OOZIE-2260.patch
>
>
> OOZIE-2209 sets java.io.tmpdir to ./tmp for AM, map and reduce for both 
> launcher and action jobs.
> MapReduceChildJVM already sets java.io.tmpdir to ./tmp
> {code}
> Path childTmpDir = new Path(Environment.PWD.$(),
> YarnConfiguration.DEFAULT_CONTAINER_TEMP_DIR);
> vargs.add("-Djava.io.tmpdir=" + childTmpDir);
> {code}
> So it is only needed to be set for AM in launcher because of uber mode. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2215) Support glob in FS EL function

2015-06-08 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2215:
--
Attachment: OOZIE-2215-v4.patch

> Support glob in FS EL function
> --
>
> Key: OOZIE-2215
> URL: https://issues.apache.org/jira/browse/OOZIE-2215
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>    Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Fix For: trunk
>
> Attachments: OOZIE-2215-v4.patch
>
>
> this patch is to support glob in fs_exists EL funct similar to OOZIE-1471.  
> need to set upper-limit in number of files to check, not to slow down oozie 
> server. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 33400: OOZIE-2215 Support glob in FS EL function

2015-06-08 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33400/
---

(Updated June 8, 2015, 10:14 p.m.)


Review request for oozie.


Changes
---

final one


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-2215


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java f8af61c 
  core/src/main/resources/oozie-default.xml 8960073 
  core/src/test/java/org/apache/oozie/action/hadoop/TestFsELFunctions.java 
c81fc61 
  docs/src/site/twiki/WorkflowFunctionalSpec.twiki 02dc65b 

Diff: https://reviews.apache.org/r/33400/diff/


Testing
---


Thanks,

Ryota Egashira

Re: Review Request 33400: OOZIE-2215 Support glob in FS EL function

2015-06-05 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33400/#review86855
---


this behavior is observed in hadoop-1 only,  fixed in hadoop-2

- Ryota Egashira


On June 1, 2015, 5:49 p.m., Ryota Egashira wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33400/
> ---
> 
> (Updated June 1, 2015, 5:49 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/OOZIE-2215
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> f8af61c 
>   core/src/main/resources/oozie-default.xml 8960073 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestFsELFunctions.java 
> c81fc61 
>   docs/src/site/twiki/WorkflowFunctionalSpec.twiki 02dc65b 
> 
> Diff: https://reviews.apache.org/r/33400/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ryota Egashira
> 
>

Re: Review Request 33400: OOZIE-2215 Support glob in FS EL function

2015-06-01 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33400/
---

(Updated June 1, 2015, 5:49 p.m.)


Review request for oozie.


Changes
---

one weird thing in Pathfilter behavior is, 
fs.globStatus(path, new FSPathFilter());
when path is some non-existent single path (no glob expression),  
PathFilter.accept() is called even though it does not exist. 
if thrwoing exception at first discovery, it breaks test case..  

this version is, not throwing excpetion at first discovery, but avoid 
duplicated check done in the orignal verison.  
also added glob max check, and to support normal datapipeline application, 
raised fs glob max to 50K.


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-2215


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java f8af61c 
  core/src/main/resources/oozie-default.xml 8960073 
  core/src/test/java/org/apache/oozie/action/hadoop/TestFsELFunctions.java 
c81fc61 
  docs/src/site/twiki/WorkflowFunctionalSpec.twiki 02dc65b 

Diff: https://reviews.apache.org/r/33400/diff/


Testing
---


Thanks,

Ryota Egashira

Re: [VOTE] Release Oozie 4.2.0 (candidate 0)

2015-05-28 Thread Ryota Egashira

+1
-verified md5-successfully built-started with hadoop-1.2.1 cluster-ran 
examples(java-main, map-reduce, pig, hive, aggregator, etc) which worked
Thank Shwetha for all release effortsRyota 


 On Thursday, May 28, 2015 2:00 PM, Purshotam Shah 
 wrote:
   

 Verified following 
* compiled and build with hadoop-1 - works
* started oozie with hadoop-1.2.1- works
* checked sharelib and deployed it to hadoop-1.2.1 - works 
* verified few generated doc page - works
* Ran some examples - works

+1
      From: Rohini Palaniswamy 
 To: "dev@oozie.apache.org"  
 Sent: Thursday, May 28, 2015 12:16 PM
 Subject: Re: [VOTE] Release Oozie 4.2.0 (candidate 0)
  
+1. Checked signatures. Compiled and ran unit tests.

Regards,
Rohini



On Wed, May 27, 2015 at 2:16 AM, Harsh J  wrote:

> +1. I checked the MD5 and the RAT report, and built it
> successfully with the -Phadoop-2 flag.
>
> On Wed, May 27, 2015 at 10:09 AM, Shwetha Shivalingamurthy
>  wrote:
> > Thanks Robert for verifying.
> >
> > I didn¹t realise it was a long weekend in the US. Extending the vote by 2
> > days. Vote now closes on Friday, May 29th, 2015 12:00 PM GMT. Please
> > verify and vote.
> >
> > +1 from me.
> >
> > Thanks,
> > Shwetha
> >
> > On 27/05/15 3:15 am, "Robert Kanter"  wrote:
> >
> >>+1
> >>
> >>
> >>+ Verified md5
> >>+ RAT report looks good
> >>+ Ran some of the examples against Hadoop 1.2.0 after compiling with the
> >>hadoop-1 and uber profiles  (mvn clean package assembly:single
> -DskipTests
> >>-Phadoop-1 -Puber -DgenerateDocs)
> >>+ Ran some of the examples against the latest Hadoop 2.8-SNAPSHOT after
> >>compiling with the hadoop-2  and uber profiles (mvn clean package
> >>assembly:single -DskipTests -Phadoop-2 -Puber -DgenerateDocs)
> >>
> >>- Robert
> >>
> >>
> >>On Tue, May 26, 2015 at 8:04 AM, pavan kumar Kolamuri <
> >>pavan.kolam...@gmail.com> wrote:
> >>
> >>> Hi shwetha  there is issue regarding spark action
> >>> https://issues.apache.org/jira/browse/OOZIE-2253. Please mention in
> >>> release
> >>> notes.
> >>>
> >>> On Tue, May 26, 2015 at 12:37 PM, Shwetha Shivalingamurthy <
> >>> sshivalingamur...@hortonworks.com> wrote:
> >>>
> >>> > Just one more day for the vote to close. Need atleast 3 votes from
> >>>PMCs.
> >>> > Please review and vote
> >>> >
> >>> > Thanks,
> >>> > Shwetha
> >>> >
> >>> > On 22/05/15 11:30 pm, "Shwetha Shivalingamurthy"
> >>> >  wrote:
> >>> >
> >>> > >Hi All,
> >>> > >
> >>> > >I have created a build for Oozie 4.2.0 (candidate 0).
> >>> > >
> >>> > >Please download, test, try it out and vote:
> >>> > >http://people.apache.org/~shwethags/oozie-4.2.0-rc0/
> >>> > >
> >>> > >The release, md5 signature, gpg signature, rat report, and gpg keys
> >>>can
> >>> > >all be found at the above address.
> >>> > >
> >>> > >Vote closes on Wednesday, May 27th, 2015 12:00 PM GMT
> >>> > >
> >>> > >Thank you all for your contribution
> >>> > >
> >>> > >Regards,
> >>> > >Shwetha
> >>> >
> >>> >
> >>>
> >>>
> >>> --
> >>> Regards
> >>> Pavan Kumar Kolamuri
> >>>
> >
>
>
>
> --
> Harsh J
>

[jira] [Commented] (OOZIE-2240) add configuration to disable email attachment support

2015-05-21 Thread Ryota Egashira (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14555646#comment-14555646
 ] 

Ryota Egashira commented on OOZIE-2240:
---

oh you did for us, thanks Shwetha

> add configuration to disable email attachment support
> -
>
> Key: OOZIE-2240
> URL: https://issues.apache.org/jira/browse/OOZIE-2240
> Project: Oozie
>  Issue Type: Bug
>Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Fix For: 4.2
>
> Attachments: OOZIE-2240-v2.patch, OOZIE-2240-v3.patch, 
> OOZIE-2240.patch
>
>
> OOZIE-2160 enable email attachment support as default.
> but depending on cluster security policy,  this might need to be switched off.
> this patch is to make it configurable. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2246) CoordinatorInputCheckCommand does not behave properly when har file is one of data dependency and doesn't exist

2015-05-21 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2246:
--
Description: 
when har file doesn't exist, FileSystem.createFileSystem in 
HadoopAccessorService throw exception, failing CoordActionInputCheck.
thus, even thought there are other data dependencies which already exist, it is 
not reflected on DB.
coordinator job cannot start until the har file becomes available anyway, and 
once available, this error doesn't happen, so basic functionality is fine, but 
it's misleading.

{code}
2014-03-13 22:00:00,051 WARN CallableQueueService$CallableWrapper:542 
[pool-2-thread-288] - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] 
exception callable [coord_action_input], E1021: Coord Action Input Check Error: 
org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: 
[Invalid path for the Har Filesystem. No index file in 
har://:8020/data/2014031322/archive.har]
org.apache.oozie.command.CommandException: E1021: Coord Action Input Check 
Error: org.apache.oozie.service.HadoopAccessorException: E0902: Exception 
occured: [Invalid path for the Har Filesystem. No index file in 
har://:8020/data/2014031322/archive.har]
at 
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:182)
at 
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:67)
at org.apache.oozie.command.XCommand.call(XCommand.java:280)
at 
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: 
org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: 
[Invalid path for the Har Filesystem. No index file in 
har://:8020/data/2014031322/archive.har]
at 
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.pathExists(CoordActionInputCheckXCommand.java:493)
at 
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkListOfPaths(CoordActionInputCheckXCommand.java:459)
at 
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkResolvedUris(CoordActionInputCheckXCommand.java:429)
at 
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkInput(CoordActionInputCheckXCommand.java:259)
at 
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:132)
... 6 more
Caused by: org.apache.oozie.service.HadoopAccessorException: E0902: Exception 
occured: [Invalid path for the Har Filesystem. No index file in 
har://:8020/data/2014031322/archive.har]
at 
org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:430)
at org.apache.oozie.dependency.FSURIHandler.getFileSystem(FSURIHandler.java:134)
at org.apache.oozie.dependency.FSURIHandler.exists(FSURIHandler.java:99)
at 
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.pathExists(CoordActionInputCheckXCommand.java:488)
... 10 more
Caused by: java.io.IOException: Invalid path for the Har Filesystem. No index 
file in har://:8020/data/2014031322/archive.har
at org.apache.hadoop.fs.HarFileSystem.initialize(HarFileSystem.java:139)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2160)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:303)
at 
org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:422)
at 
org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:420)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1284)
at 
org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:420)
{code}

  was:
when har file doesn't exist, FileSystem.createFileSystem in 
HadoopAccessorService throw exception, failing CoordActionInputCheck.
thus, even thought there are other data dependencies which already exist, it is 
not reflected on DB.
coordinator job cannot start until the har file becomes available anyway, and 
once available, this error doesn't happen, so basic functionality is fine, but 
it's misleading.

{code}
2014-03-13 22:00:00,051 WARN CallableQueueService$CallableWrapper:542 
[pool-2-thread-288] - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] 
exception callable [coord_action_input], E1021: Coord Action Input Check Error: 
org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: 
[Invalid path for the Har Filesystem. No index file in 
har://hdfs-dilithiumred-nn1.red.ygrid.yahoo.com:8020/projects/tpoozie/post_tp_rm

[jira] [Updated] (OOZIE-2246) CoordinatorInputCheckCommand does not behave properly when har file is one of data dependency and doesn't exist

2015-05-21 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2246:
--
Attachment: OOZIE-2246.patch

> CoordinatorInputCheckCommand does not behave properly when har file is one of 
> data dependency and doesn't exist
> ---
>
> Key: OOZIE-2246
> URL: https://issues.apache.org/jira/browse/OOZIE-2246
> Project: Oozie
>  Issue Type: Bug
>Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Attachments: OOZIE-2246.patch
>
>
> when har file doesn't exist, FileSystem.createFileSystem in 
> HadoopAccessorService throw exception, failing CoordActionInputCheck.
> thus, even thought there are other data dependencies which already exist, it 
> is not reflected on DB.
> coordinator job cannot start until the har file becomes available anyway, and 
> once available, this error doesn't happen, so basic functionality is fine, 
> but it's misleading.
> {code}
> 2014-03-13 22:00:00,051 WARN CallableQueueService$CallableWrapper:542 
> [pool-2-thread-288] - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] 
> exception callable [coord_action_input], E1021: Coord Action Input Check 
> Error: org.apache.oozie.service.HadoopAccessorException: E0902: Exception 
> occured: [Invalid path for the Har Filesystem. No index file in 
> har://hdfs-dilithiumred-nn1.red.ygrid.yahoo.com:8020/projects/tpoozie/post_tp_rmx_joined_click_trim_har/hourly/data/2014031322/archive.har]
> org.apache.oozie.command.CommandException: E1021: Coord Action Input Check 
> Error: org.apache.oozie.service.HadoopAccessorException: E0902: Exception 
> occured: [Invalid path for the Har Filesystem. No index file in 
> har://hdfs-dilithiumred-nn1.red.ygrid.yahoo.com:8020/projects/tpoozie/post_tp_rmx_joined_click_trim_har/hourly/data/2014031322/archive.har]
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:182)
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:67)
> at org.apache.oozie.command.XCommand.call(XCommand.java:280)
> at 
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: 
> org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: 
> [Invalid path for the Har Filesystem. No index file in 
> har://hdfs-dilithiumred-nn1.red.ygrid.yahoo.com:8020/projects/tpoozie/post_tp_rmx_joined_click_trim_har/hourly/data/2014031322/archive.har]
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.pathExists(CoordActionInputCheckXCommand.java:493)
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkListOfPaths(CoordActionInputCheckXCommand.java:459)
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkResolvedUris(CoordActionInputCheckXCommand.java:429)
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkInput(CoordActionInputCheckXCommand.java:259)
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:132)
> ... 6 more
> Caused by: org.apache.oozie.service.HadoopAccessorException: E0902: Exception 
> occured: [Invalid path for the Har Filesystem. No index file in 
> har://hdfs-dilithiumred-nn1.red.ygrid.yahoo.com:8020/projects/tpoozie/post_tp_rmx_joined_click_trim_har/hourly/data/2014031322/archive.har]
> at 
> org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:430)
> at 
> org.apache.oozie.dependency.FSURIHandler.getFileSystem(FSURIHandler.java:134)
> at org.apache.oozie.dependency.FSURIHandler.exists(FSURIHandler.java:99)
> at 
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.pathExists(CoordActionInputCheckXCommand.java:488)
> ... 10 more
> Caused by: java.io.IOException: Invalid path for the Har Filesystem. No index 
> file in 
> har://hdfs-dilithiumred-nn1.red.ygrid.yahoo.com:8020/projects/tpoozie/post_tp_rmx_joined_click_trim_har/hourly/data/2014031322/archive.har
> at org.apache.hadoop.fs.HarFileSystem.initialize(HarFileSystem.java:139)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2160)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:303)
>

[jira] [Created] (OOZIE-2246) CoordinatorInputCheckCommand does not behave properly when har file is one of data dependency and doesn't exist

2015-05-21 Thread Ryota Egashira (JIRA)

Ryota Egashira created OOZIE-2246:
-

 Summary: CoordinatorInputCheckCommand does not behave properly 
when har file is one of data dependency and doesn't exist
 Key: OOZIE-2246
 URL: https://issues.apache.org/jira/browse/OOZIE-2246
 Project: Oozie
  Issue Type: Bug
Reporter: Ryota Egashira
Assignee: Ryota Egashira


when har file doesn't exist, FileSystem.createFileSystem in 
HadoopAccessorService throw exception, failing CoordActionInputCheck.
thus, even thought there are other data dependencies which already exist, it is 
not reflected on DB.
coordinator job cannot start until the har file becomes available anyway, and 
once available, this error doesn't happen, so basic functionality is fine, but 
it's misleading.

{code}
2014-03-13 22:00:00,051 WARN CallableQueueService$CallableWrapper:542 
[pool-2-thread-288] - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] 
exception callable [coord_action_input], E1021: Coord Action Input Check Error: 
org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: 
[Invalid path for the Har Filesystem. No index file in 
har://hdfs-dilithiumred-nn1.red.ygrid.yahoo.com:8020/projects/tpoozie/post_tp_rmx_joined_click_trim_har/hourly/data/2014031322/archive.har]
org.apache.oozie.command.CommandException: E1021: Coord Action Input Check 
Error: org.apache.oozie.service.HadoopAccessorException: E0902: Exception 
occured: [Invalid path for the Har Filesystem. No index file in 
har://hdfs-dilithiumred-nn1.red.ygrid.yahoo.com:8020/projects/tpoozie/post_tp_rmx_joined_click_trim_har/hourly/data/2014031322/archive.har]
at 
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:182)
at 
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:67)
at org.apache.oozie.command.XCommand.call(XCommand.java:280)
at 
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: 
org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: 
[Invalid path for the Har Filesystem. No index file in 
har://hdfs-dilithiumred-nn1.red.ygrid.yahoo.com:8020/projects/tpoozie/post_tp_rmx_joined_click_trim_har/hourly/data/2014031322/archive.har]
at 
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.pathExists(CoordActionInputCheckXCommand.java:493)
at 
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkListOfPaths(CoordActionInputCheckXCommand.java:459)
at 
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkResolvedUris(CoordActionInputCheckXCommand.java:429)
at 
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkInput(CoordActionInputCheckXCommand.java:259)
at 
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:132)
... 6 more
Caused by: org.apache.oozie.service.HadoopAccessorException: E0902: Exception 
occured: [Invalid path for the Har Filesystem. No index file in 
har://hdfs-dilithiumred-nn1.red.ygrid.yahoo.com:8020/projects/tpoozie/post_tp_rmx_joined_click_trim_har/hourly/data/2014031322/archive.har]
at 
org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:430)
at org.apache.oozie.dependency.FSURIHandler.getFileSystem(FSURIHandler.java:134)
at org.apache.oozie.dependency.FSURIHandler.exists(FSURIHandler.java:99)
at 
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.pathExists(CoordActionInputCheckXCommand.java:488)
... 10 more
Caused by: java.io.IOException: Invalid path for the Har Filesystem. No index 
file in 
har://hdfs-dilithiumred-nn1.red.ygrid.yahoo.com:8020/projects/tpoozie/post_tp_rmx_joined_click_trim_har/hourly/data/2014031322/archive.har
at org.apache.hadoop.fs.HarFileSystem.initialize(HarFileSystem.java:139)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2160)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:303)
at 
org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:422)
at 
org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:420)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1284)
at 
org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:420)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2240) add configuration to disable email attachment support

2015-05-21 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2240:
--
Attachment: OOZIE-2240-v3.patch

done. let's not include "security reason".  could be other reason to disable 
the server-side setting.  
also test case is changed a bit since '\n' converted to \n\r in mac, but 
probably it differs in OS

> add configuration to disable email attachment support
> -
>
> Key: OOZIE-2240
> URL: https://issues.apache.org/jira/browse/OOZIE-2240
> Project: Oozie
>      Issue Type: Bug
>    Reporter: Ryota Egashira
>Assignee: Ryota Egashira
> Fix For: 4.2
>
> Attachments: OOZIE-2240-v2.patch, OOZIE-2240-v3.patch, 
> OOZIE-2240.patch
>
>
> OOZIE-2160 enable email attachment support as default.
> but depending on cluster security policy,  this might need to be switched off.
> this patch is to make it configurable. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2240) add configuration to disable email attachment support

2015-05-20 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2240:
--
Attachment: OOZIE-2240-v2.patch

> add configuration to disable email attachment support
> -
>
> Key: OOZIE-2240
> URL: https://issues.apache.org/jira/browse/OOZIE-2240
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>    Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Fix For: trunk
>
> Attachments: OOZIE-2240-v2.patch, OOZIE-2240.patch
>
>
> OOZIE-2160 enable email attachment support as default.
> but depending on cluster security policy,  this might need to be switched off.
> this patch is to make it configurable. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2240) add configuration to disable email attachment support

2015-05-20 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2240:
--
Attachment: OOZIE-2240.patch

> add configuration to disable email attachment support
> -
>
> Key: OOZIE-2240
> URL: https://issues.apache.org/jira/browse/OOZIE-2240
> Project: Oozie
>  Issue Type: Bug
>    Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Attachments: OOZIE-2240.patch
>
>
> OOZIE-2160 enable email attachment support as default.
> but depending on cluster security policy,  this might need to be switched off.
> this patch is to make it configurable. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OOZIE-2240) add configuration to disable email attachment support

2015-05-20 Thread Ryota Egashira (JIRA)

Ryota Egashira created OOZIE-2240:
-

 Summary: add configuration to disable email attachment support
 Key: OOZIE-2240
 URL: https://issues.apache.org/jira/browse/OOZIE-2240
 Project: Oozie
  Issue Type: Bug
Reporter: Ryota Egashira
Assignee: Ryota Egashira


OOZIE-2160 enable email attachment support as default.
but depending on cluster security policy,  this might need to be switched off.
this patch is to make it configurable. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2229) oozie-hadoop-utils version does not conform to standard

2015-05-15 Thread Ryota Egashira (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14546471#comment-14546471
 ] 

Ryota Egashira commented on OOZIE-2229:
---

thank you for sharing the link. ok this is way to manage different src 
directories in one module.
but given profile is used to point to different sources depending on hadoop 
version,  I guess this approach end up hitting the same profile issue in 
transitive dependency.

> oozie-hadoop-utils version does not conform to standard
> ---
>
> Key: OOZIE-2229
> URL: https://issues.apache.org/jira/browse/OOZIE-2229
> Project: Oozie
>  Issue Type: Bug
>Reporter: Rohini Palaniswamy
>    Assignee: Ryota Egashira
> Fix For: trunk
>
> Attachments: OOZIE-2229-v2.patch
>
>
> The version for oozie-hadoop-utils package for different versions is of the 
> form 
> hadoop-${hadoop.majorversion}-${project.version}
> i.e
> hadoop-1-4.2.0-SNAPSHOT
> hadoop-2-4.2.0-SNAPSHOT
> hadoop-3-4.2.0-SNAPSHOT
> This is breaking the standard major.minor.revision.patch versioning for jar 
> or the ..- 
> standard for maven.
> This causes problems with maven plugins like versions and also problems with 
> some of our internal systems which do not expect alphabets and special 
> characters (- in this case) in the major version.
> The correct way to do this would be to use classifiers like how it is done in 
> Pig and Sqoop for different versions of hadoop.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2229) oozie-hadoop-utils version does not conform to standard

2015-05-14 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2229:
--
Attachment: OOZIE-2229-v2.patch

> oozie-hadoop-utils version does not conform to standard
> ---
>
> Key: OOZIE-2229
> URL: https://issues.apache.org/jira/browse/OOZIE-2229
> Project: Oozie
>  Issue Type: Bug
>Reporter: Rohini Palaniswamy
>    Assignee: Ryota Egashira
> Fix For: trunk
>
> Attachments: OOZIE-2229-v2.patch
>
>
> The version for oozie-hadoop-utils package for different versions is of the 
> form 
> hadoop-${hadoop.majorversion}-${project.version}
> i.e
> hadoop-1-4.2.0-SNAPSHOT
> hadoop-2-4.2.0-SNAPSHOT
> hadoop-3-4.2.0-SNAPSHOT
> This is breaking the standard major.minor.revision.patch versioning for jar 
> or the ..- 
> standard for maven.
> This causes problems with maven plugins like versions and also problems with 
> some of our internal systems which do not expect alphabets and special 
> characters (- in this case) in the major version.
> The correct way to do this would be to use classifiers like how it is done in 
> Pig and Sqoop for different versions of hadoop.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2229) oozie-hadoop-utils version does not conform to standard

2015-05-14 Thread Ryota Egashira (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14544097#comment-14544097
 ] 

Ryota Egashira commented on OOZIE-2229:
---

[~shwethags] thanks for info.

{quote}
we can define just one module and include different sources depending on the 
profile.
{quote}
there is class which (e..g, LauncherMainHadoopUtils.java) has different 
implementation for different versions, and need a way to differentiate it 
inside the same module(directory). it looks a bit trick, hence trying the other 
option first.

{quote}
The problem is we need to define 3 modules for each hadoop major version and 
all these modules can't have the same version 4.2.0-SNAPSHOT
{quote}

can you explain a bit more on why it didn't work with the same version?  
the patch I tried (on RB, will attach here) can still use the same version for 
each module,  but able to differentiate it using classifier.

but actually I hit another issue, when do mvn package -Phadoop-2,   seem like 
transitive dependency cannot take right profile from CLI.
just wondering you hit the same issue last time you tried.

error message i got
{code}
[ERROR] Failed to execute goal on project oozie-sharelib-hcatalog: Could not 
resolve dependencies for project 
org.apache.oozie:oozie-sharelib-hcatalog:jar:4.2.0-SNAPSHOT: Failure to find 
org.apache.oozie:oozie-hadoop-utils:jar:hadoop1:4.2.0-SNAPSHOT in 
https://repository.apache.org/content/groups/snapshots was cached in the local 
repository, resolution will not be reattempted until the update interval of 
apache.snapshots.repo has elapsed or updates are forced -> [Help 1]
{code}

oozie-sharelib-hcatalog depends on oozie-sharelib-oozie 
oozie-sharelib-oozie depends on oozie-hadoop-utils

oozie-hadoop-utils version is specified by variable {hadooplib.classifier} 
which supposed to be substitued by profile.
but -Phadoop-2 from CLI is not propated properly, always point to hadoop1.

looks like known maven issues. 
http://jira.codehaus.org/browse/MNG-1388
http://jira.codehaus.org/browse/MNG-3782

transitive dependency(oozie-hadoop-utils) seem to resolve profile information 
from top-level pom.
if we change activeByDefault true for hadoop-2 in top-level pom.xml, it works
pig.classifier or sqoop classifeir doesn't have the same issue, since it is not 
transitively resolved. 

appreciate if any information to workaround this



















> oozie-hadoop-utils version does not conform to standard
> ---
>
> Key: OOZIE-2229
> URL: https://issues.apache.org/jira/browse/OOZIE-2229
> Project: Oozie
>  Issue Type: Bug
>Reporter: Rohini Palaniswamy
>Assignee: Ryota Egashira
> Fix For: trunk
>
>
> The version for oozie-hadoop-utils package for different versions is of the 
> form 
> hadoop-${hadoop.majorversion}-${project.version}
> i.e
> hadoop-1-4.2.0-SNAPSHOT
> hadoop-2-4.2.0-SNAPSHOT
> hadoop-3-4.2.0-SNAPSHOT
> This is breaking the standard major.minor.revision.patch versioning for jar 
> or the ..- 
> standard for maven.
> This causes problems with maven plugins like versions and also problems with 
> some of our internal systems which do not expect alphabets and special 
> characters (- in this case) in the major version.
> The correct way to do this would be to use classifiers like how it is done in 
> Pig and Sqoop for different versions of hadoop.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 33948: OOZIE-2229 oozie-hadoop-utils version does not conform to standard

2015-05-14 Thread Ryota Egashira



> On May 7, 2015, 5:59 p.m., Rohini Palaniswamy wrote:
> > hadooplibs/hadoop-auth-1/pom.xml, line 47
> > <https://reviews.apache.org/r/33948/diff/1/?file=952396#file952396line47>
> >
> > Check if you can have a jar.classifier property which defaults to "" 
> > specified in parent pom.xml and its maven-jar-plugin definition and just 
> > override that property value to hadooplib.classifier here so that you don't 
> > have to define the whole plugin section everytime?

I tried this,  but when maven-jar-pluging with execution definition is placed 
on top-level pom,   oozie-client, oozie-core and other modules also creates jar 
file with classifier(oozie-core-hadoop1.jar etc)
exclusion is only applied for goal/phase, and to exclude particular 
modules(oozie-core, oozie-client, others) not to use classifier, need to add 
do-nothing definition in these modules pom. 
http://stackoverflow.com/questions/7800647/in-a-maven-multi-module-project-how-can-i-disable-a-plugin-in-one-child

so i rather chose to change hadooplibs related poms only as currently done, 
than editing other modules pom.


- Ryota


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33948/#review82856
---


On May 14, 2015, 5:01 p.m., Ryota Egashira wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33948/
> ---
> 
> (Updated May 14, 2015, 5:01 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> WIP, still rough and not fully functional, sharing for early review.
> 
> mvn compile works for hadoop-1, hadoop-2, hadoop-0.23
> mvn package works for hadoop-1, but not for hadoop-2 and 23.
> 
> also org.apache.maven.plugins definition added to all version of poms 
> repeatedly, wonder how to remove redundancy, but  without this doens't work
> 
> 
> Diffs
> -
> 
>   client/pom.xml 73f7ef5 
>   core/pom.xml c62556c 
>   hadooplibs/hadoop-auth-0.23/pom.xml 5769f66 
>   hadooplibs/hadoop-auth-1/pom.xml ac36798 
>   hadooplibs/hadoop-auth-2/pom.xml 977d49c 
>   hadooplibs/hadoop-distcp-0.23/pom.xml c13b78d 
>   hadooplibs/hadoop-distcp-1/pom.xml 5b4f96f 
>   hadooplibs/hadoop-distcp-2/pom.xml 4097cc4 
>   hadooplibs/hadoop-distcp-3/pom.xml 8d8180d 
>   hadooplibs/hadoop-utils-0.23/pom.xml 1bd243e 
>   hadooplibs/hadoop-utils-1/pom.xml 48d97b0 
>   hadooplibs/hadoop-utils-2/pom.xml 832cf5b 
>   hadooplibs/hadoop-utils-3/pom.xml e7df53b 
>   pom.xml ea9ccef 
>   sharelib/distcp/pom.xml a329b7b 
>   sharelib/hive/pom.xml 3da0bdc 
>   sharelib/hive2/pom.xml 41eade9 
>   sharelib/oozie/pom.xml 484fb45 
>   sharelib/pig/pom.xml fd3cd66 
>   sharelib/sqoop/pom.xml 5308bd5 
> 
> Diff: https://reviews.apache.org/r/33948/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ryota Egashira
> 
>

Re: Review Request 33948: OOZIE-2229 oozie-hadoop-utils version does not conform to standard

2015-05-14 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33948/
---

(Updated May 14, 2015, 5:01 p.m.)


Review request for oozie.


Changes
---

modified version.


still hit error by "bin/mkdistro.sh -Phadoop2 or  mvn package -Phadoop-2"
-
[ERROR] Failed to execute goal on project oozie-sharelib-hcatalog: Could not 
resolve dependencies for project 
org.apache.oozie:oozie-sharelib-hcatalog:jar:4.2.0-SNAPSHOT: Failure to find 
org.apache.oozie:oozie-hadoop-utils:jar:hadoop1:4.2.0-SNAPSHOT in 
https://repository.apache.org/content/groups/snapshots was cached in the local 
repository, resolution will not be reattempted until the update interval of 
apache.snapshots.repo has elapsed or updates are forced -> [Help 1]
-

reason is that profile is not properly used in transitive dependency

oozie-sharelib-hcatalog depends on oozie-sharelib-oozie 
oozie-sharelib-oozie depends on oozie-hadoop-utils

oozie-hadoop-utils version is specified by variable {hadooplib.classifier} 
which supposed to be substitued by profile.
but -Phadoop-2 from CLI is not propated properly, always point to hadoop1.

looks like maven issues. 
http://jira.codehaus.org/browse/MNG-1388
http://jira.codehaus.org/browse/MNG-3782

transitive dependency(oozie-hadoop-utils) resolve profile information from 
top-level pom.
if we change activeByDefault true for hadoop-2 in top-level pom.xml, it works. 

tried couple of possible workaround, but no luck so far.


Repository: oozie-git


Description
---

WIP, still rough and not fully functional, sharing for early review.

mvn compile works for hadoop-1, hadoop-2, hadoop-0.23
mvn package works for hadoop-1, but not for hadoop-2 and 23.

also org.apache.maven.plugins definition added to all version of poms 
repeatedly, wonder how to remove redundancy, but  without this doens't work


Diffs (updated)
-

  client/pom.xml 73f7ef5 
  core/pom.xml c62556c 
  hadooplibs/hadoop-auth-0.23/pom.xml 5769f66 
  hadooplibs/hadoop-auth-1/pom.xml ac36798 
  hadooplibs/hadoop-auth-2/pom.xml 977d49c 
  hadooplibs/hadoop-distcp-0.23/pom.xml c13b78d 
  hadooplibs/hadoop-distcp-1/pom.xml 5b4f96f 
  hadooplibs/hadoop-distcp-2/pom.xml 4097cc4 
  hadooplibs/hadoop-distcp-3/pom.xml 8d8180d 
  hadooplibs/hadoop-utils-0.23/pom.xml 1bd243e 
  hadooplibs/hadoop-utils-1/pom.xml 48d97b0 
  hadooplibs/hadoop-utils-2/pom.xml 832cf5b 
  hadooplibs/hadoop-utils-3/pom.xml e7df53b 
  pom.xml ea9ccef 
  sharelib/distcp/pom.xml a329b7b 
  sharelib/hive/pom.xml 3da0bdc 
  sharelib/hive2/pom.xml 41eade9 
  sharelib/oozie/pom.xml 484fb45 
  sharelib/pig/pom.xml fd3cd66 
  sharelib/sqoop/pom.xml 5308bd5 

Diff: https://reviews.apache.org/r/33948/diff/


Testing
---


Thanks,

Ryota Egashira

Review Request 33948: OOZIE-2229 oozie-hadoop-utils version does not conform to standard

2015-05-07 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33948/
---

Review request for oozie.


Repository: oozie-git


Description
---

WIP, still rough and not fully functional, sharing for early review.

mvn compile works for hadoop-1, hadoop-2, hadoop-0.23
mvn package works for hadoop-1, but not for hadoop-2 and 23.

also org.apache.maven.plugins definition added to all version of poms 
repeatedly, wonder how to remove redundancy, but  without this doens't work


Diffs
-

  client/pom.xml 73f7ef5 
  core/pom.xml c62556c 
  hadooplibs/hadoop-auth-0.23/pom.xml 5769f66 
  hadooplibs/hadoop-auth-1/pom.xml ac36798 
  hadooplibs/hadoop-auth-2/pom.xml 977d49c 
  hadooplibs/hadoop-distcp-0.23/pom.xml c13b78d 
  hadooplibs/hadoop-distcp-1/pom.xml 5b4f96f 
  hadooplibs/hadoop-distcp-2/pom.xml 4097cc4 
  hadooplibs/hadoop-distcp-3/pom.xml 8d8180d 
  hadooplibs/hadoop-utils-0.23/pom.xml 1bd243e 
  hadooplibs/hadoop-utils-1/pom.xml 48d97b0 
  hadooplibs/hadoop-utils-2/pom.xml 832cf5b 
  hadooplibs/hadoop-utils-3/pom.xml e7df53b 
  pom.xml ea9ccef 
  sharelib/distcp/pom.xml a329b7b 
  sharelib/hive/pom.xml 3da0bdc 
  sharelib/hive2/pom.xml 41eade9 
  sharelib/oozie/pom.xml 484fb45 
  sharelib/pig/pom.xml fd3cd66 
  sharelib/sqoop/pom.xml 5308bd5 

Diff: https://reviews.apache.org/r/33948/diff/


Testing
---


Thanks,

Ryota Egashira

[jira] [Commented] (OOZIE-2227) PartitionDependencyManagerService keeps on purging delete coord actions

2015-05-05 Thread Ryota Egashira (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529317#comment-14529317
 ] 

Ryota Egashira commented on OOZIE-2227:
---

+1
fetchLog change is about amendment to OOZIE-2140, right?
 

> PartitionDependencyManagerService keeps on purging delete coord actions
> ---
>
> Key: OOZIE-2227
> URL: https://issues.apache.org/jira/browse/OOZIE-2227
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-2227-V1.patch
>
>
> Because of that PartitionDependencyManagerService takes more than 10 min to 
> complete.
> {code}
> Long regTime = registeredCoordActionMap.get(actionId);
> if(regTime < (currentTime - timeToLive * 1000)){
> CoordinatorActionBean caBean = null;
> try {
> caBean = 
> CoordActionQueryExecutor.getInstance().get(CoordActionQuery.GET_COORD_ACTION_STATUS,
>  actionId);
> }
> catch (JPAExecutorException e) {
> LOG.warn("Error in checking coord action:" + actionId 
> + "to purge, skipping", e);
> }
> if(caBean != null && 
> !caBean.getStatus().equals(CoordinatorAction.Status.WAITING)){
> staleActions.add(actionId);
> actionItr.remove();
> }
> {code}
> Should remove actionId if JPAExecutorException = no row found



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2221) Oozie audit log has null id for some of input request

2015-04-30 Thread Ryota Egashira (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14522402#comment-14522402
 ] 

Ryota Egashira commented on OOZIE-2221:
---

+1

> Oozie audit log has null id for some of input request
> -
>
> Key: OOZIE-2221
> URL: https://issues.apache.org/jira/browse/OOZIE-2221
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-2221-V1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2221) Oozie audit log has null id for some of input request

2015-04-30 Thread Ryota Egashira (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14522381#comment-14522381
 ] 

Ryota Egashira commented on OOZIE-2221:
---

looks good,  just wonder if CoordUpdateXCommand is the only command missing 
setLogInfo ? 

> Oozie audit log has null id for some of input request
> -
>
> Key: OOZIE-2221
> URL: https://issues.apache.org/jira/browse/OOZIE-2221
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-2221-V1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 33639: OOZIE-2219 Kill pig child jobs in ActionKillXCommand when sequence file created

2015-04-28 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33639/
---

Review request for oozie.


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-2219


Diffs
-

  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
eb2dbdb 
  core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
8b2dc16 
  
core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionKillXCommand.java
 72f0114 

Diff: https://reviews.apache.org/r/33639/diff/


Testing
---


Thanks,

Ryota Egashira

[jira] [Created] (OOZIE-2219) Kill pig child jobs in ActionKillXCommand when sequence file created

2015-04-28 Thread Ryota Egashira (JIRA)

Ryota Egashira created OOZIE-2219:
-

 Summary: Kill pig child jobs in ActionKillXCommand when sequence 
file created
 Key: OOZIE-2219
 URL: https://issues.apache.org/jira/browse/OOZIE-2219
 Project: Oozie
  Issue Type: Bug
Reporter: Ryota Egashira
Assignee: Ryota Egashira


following is the log of a cleanly killed Pig launcher job. It does write the 
child ids to the sequence file. We can process the sequence file when there is 
a kill and kill the child jobs also.
{code}
Hadoop Job IDs executed by Pig: job_1429645443900_0012
<<< Invocation of Pig command completed <<<

<<< Invocation of Main class completed <<<
Oozie Launcher, uploading action data to HDFS sequence file: 
hdfs://gsbl90101.blue.ygrid.yahoo.com:8020/user/dfsload/denseb/085-150416151027813-denseb-W/pig_sleep--pig/action-data.seq
{code}
https://issues.apache.org/jira/browse/OOZIE-1025 is another jira which will 
address this issue for all scenarios by using application tags. 
In this jira we will only kill child jobs if they were managed to be written to 
the action-data.seq file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2045) Symlink support for sharelib

2015-04-24 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2045:
--
Attachment: OOZIE-2045-amend.patch

minor issue fixed in amend patch
https://github.com/apache/oozie/commit/4fea9412f1e12d97cf30c38d4693bb27290a1a16

> Symlink support for sharelib
> 
>
> Key: OOZIE-2045
> URL: https://issues.apache.org/jira/browse/OOZIE-2045
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Fix For: trunk
>
> Attachments: OOZIE-2045-V2.patch, OOZIE-2045-V3.patch, 
> OOZIE-2045-V4.patch, OOZIE-2045-amend.patch
>
>
> With metafile configuration, one can configure sharelib (eg. pig_latest) to 
> latest version available.
> If new version is available, admin will just change the pig_latest sysmlink 
> to new version. 
> There will be no change to Oozie or metafile configuration.
> Oozie should be able to detect symlink change and reloads the sharelib.
> metafile configuration can look like
> oozie.pig_latest=hdfs:///tmp/pig_latest
> where /tmp/pig_latest is a symlink pointing to latest available version.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2214) fix test case TestCoordRerunXCommand.testCoordRerunDateNeg

2015-04-22 Thread Ryota Egashira (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507582#comment-14507582
 ] 

Ryota Egashira commented on OOZIE-2214:
---

the change was introduced by OOZIE-1913 (SLA turn-off)
I confirmed with puru that the change in this part is not necessary and can be 
reverted back.

> fix test case TestCoordRerunXCommand.testCoordRerunDateNeg
> --
>
> Key: OOZIE-2214
> URL: https://issues.apache.org/jira/browse/OOZIE-2214
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>    Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Fix For: trunk
>
> Attachments: OOZIE-2214.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2214) fix test case TestCoordRerunXCommand.testCoordRerunDateNeg

2015-04-22 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2214:
--
Attachment: OOZIE-2214.patch

> fix test case TestCoordRerunXCommand.testCoordRerunDateNeg
> --
>
> Key: OOZIE-2214
> URL: https://issues.apache.org/jira/browse/OOZIE-2214
> Project: Oozie
>  Issue Type: Bug
>    Reporter: Ryota Egashira
> Attachments: OOZIE-2214.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 33400: OOZIE-2215 Support glob in FS EL function

2015-04-21 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33400/
---

Review request for oozie.


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-2215


Diffs
-

  core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java f8af61c 
  core/src/test/java/org/apache/oozie/action/hadoop/TestFsELFunctions.java 
c81fc61 
  docs/src/site/twiki/WorkflowFunctionalSpec.twiki e3790a4 

Diff: https://reviews.apache.org/r/33400/diff/


Testing
---


Thanks,

Ryota Egashira

[jira] [Created] (OOZIE-2215) Support glob in FS EL function

2015-04-21 Thread Ryota Egashira (JIRA)

Ryota Egashira created OOZIE-2215:
-

 Summary: Support glob in FS EL function
 Key: OOZIE-2215
 URL: https://issues.apache.org/jira/browse/OOZIE-2215
 Project: Oozie
  Issue Type: Bug
Reporter: Ryota Egashira
Assignee: Ryota Egashira


this patch is to support glob in fs_exists EL funct similar to OOZIE-1471.  
need to set upper-limit in number of files to check, not to slow down oozie 
server. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Moved] (OOZIE-2214) fix test case TestCoordRerunXCommand.testCoordRerunDateNeg

2015-04-20 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira moved HADOOP-11853 to OOZIE-2214:


Key: OOZIE-2214  (was: HADOOP-11853)
Project: Oozie  (was: Hadoop Common)

> fix test case TestCoordRerunXCommand.testCoordRerunDateNeg
> --
>
> Key: OOZIE-2214
> URL: https://issues.apache.org/jira/browse/OOZIE-2214
> Project: Oozie
>  Issue Type: Bug
>    Reporter: Ryota Egashira
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2206) Change Reaper mode on ChildReaper in ZKLocksService

2015-04-20 Thread Ryota Egashira (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14503304#comment-14503304
 ] 

Ryota Egashira commented on OOZIE-2206:
---

it looks to me, once REAP_INDEFINITELY,  activePaths.remove(holder.path) never 
be called. is this Curator bug? 

> Change Reaper mode on ChildReaper in ZKLocksService
> ---
>
> Key: OOZIE-2206
> URL: https://issues.apache.org/jira/browse/OOZIE-2206
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>    Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Fix For: trunk
>
> Attachments: OOZIE-2206.patch
>
>
> OOZIE-1906 added znode cleanup thread.
> currently passing Reaper.Mode.REAP_INDEFINITELY, but this enforce Oozie 
> server to keep reaping znode even after znode is cleaned up.  
> (https://github.com/apache/curator/blob/master/curator-recipes/src/main/java/org/apache/curator/framework/recipes/locks/Reaper.java)
>  
> This adds memory pressure on oozie server. Need to change to REAP_UNTIL_GONE  
> or REAP_UNTIL_DELETE
>   
> {code}
> reaper = new ChildReaper(zk.getClient(), LOCKS_NODE, 
> Reaper.Mode.REAP_INDEFINITELY, getExecutorService(), 
> ConfigurationService.getInt(services.getConf(), REAPING_THRESHOLD) * 1000, 
> REAPING_LEADER_PATH);
> {code}
> we hit one scenario where  one ZK quorum slows down for short period, causing 
> many Zk locks not released properly, right after ChildReaper (every 5 min ) 
> runs, which keep checking the list of Znode ever since, in the end, Oozie 
> server hit OOM.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2205) add option to load default/site.xml to actionConf on compute node

2015-04-17 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2205:
--
Attachment: OOZIE-2205-v2.patch

> add option to load default/site.xml to actionConf on compute node
> -
>
> Key: OOZIE-2205
> URL: https://issues.apache.org/jira/browse/OOZIE-2205
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>    Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Fix For: trunk
>
> Attachments: OOZIE-2205-v2.patch
>
>
> currently actionConf is loaded from *-default, *-site.xml (core-site, 
> hdfs-site, mapred-site, etc) on oozie server, and some default value is 
> passed into child job as it is, which we observed cause unexpected behavior 
> for pig and hive action. this patch is to add option to load default/site.xml 
> to actionConf on compute node, for pig and hive actions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 33221: OOZIE-2205 add option to load default/site.xml to actionConf on compute node

2015-04-17 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33221/
---

(Updated April 17, 2015, 5:56 p.m.)


Review request for oozie.


Changes
---

uploaded wrong version previously,  this is correct one


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-2205


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
dbb50ba 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
cf22ed6 
  core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
4d6a1b4 
  core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
18de48a 
  core/src/main/resources/oozie-default.xml 3936fca 
  
sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveActionExecutor.java
 c28839c 
  
sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java
 039bd84 

Diff: https://reviews.apache.org/r/33221/diff/


Testing
---

tested on real cluster.


Thanks,

Ryota Egashira

Re: Review Request 33221: OOZIE-2205 add option to load default/site.xml to actionConf on compute node

2015-04-17 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33221/
---

(Updated April 17, 2015, 5:37 p.m.)


Review request for oozie.


Changes
---

revised


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-2205


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
dbb50ba 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
cf22ed6 
  core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
4d6a1b4 
  core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
18de48a 
  core/src/main/resources/oozie-default.xml 3936fca 
  
sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveActionExecutor.java
 c28839c 
  
sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java
 039bd84 

Diff: https://reviews.apache.org/r/33221/diff/


Testing
---

tested on real cluster.


Thanks,

Ryota Egashira

[jira] [Updated] (OOZIE-2206) Change Reaper mode on ChildReaper in ZKLocksService

2015-04-16 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2206:
--
Attachment: OOZIE-2206.patch

> Change Reaper mode on ChildReaper in ZKLocksService
> ---
>
> Key: OOZIE-2206
> URL: https://issues.apache.org/jira/browse/OOZIE-2206
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>    Reporter: Ryota Egashira
> Fix For: trunk
>
> Attachments: OOZIE-2206.patch
>
>
> OOZIE-1906 added znode cleanup thread.
> currently passing Reaper.Mode.REAP_INDEFINITELY, but this enforce Oozie 
> server to keep reaping znode even after znode is cleaned up.  
> (https://github.com/apache/curator/blob/master/curator-recipes/src/main/java/org/apache/curator/framework/recipes/locks/Reaper.java)
>  
> This adds memory pressure on oozie server. Need to change to REAP_UNTIL_GONE  
> or REAP_UNTIL_DELETE
>   
> {code}
> reaper = new ChildReaper(zk.getClient(), LOCKS_NODE, 
> Reaper.Mode.REAP_INDEFINITELY, getExecutorService(), 
> ConfigurationService.getInt(services.getConf(), REAPING_THRESHOLD) * 1000, 
> REAPING_LEADER_PATH);
> {code}
> we hit one scenario where  one ZK quorum slows down for short period, causing 
> many Zk locks not released properly, right after ChildReaper (every 5 min ) 
> runs, which keep checking the list of Znode ever since, in the end, Oozie 
> server hit OOM.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2206) Change Reaper mode on ChildReaper in ZKLocksService

2015-04-15 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2206:
--
Description: 
OOZIE-1906 added znode cleanup thread.
currently passing Reaper.Mode.REAP_INDEFINITELY, but this enforce Oozie server 
to keep reaping znode even after znode is cleaned up.  
(https://github.com/apache/curator/blob/master/curator-recipes/src/main/java/org/apache/curator/framework/recipes/locks/Reaper.java)
 
This adds memory pressure on oozie server. Need to change to REAP_UNTIL_GONE  
or REAP_UNTIL_DELETE
  
{code}
reaper = new ChildReaper(zk.getClient(), LOCKS_NODE, 
Reaper.Mode.REAP_INDEFINITELY, getExecutorService(), 
ConfigurationService.getInt(services.getConf(), REAPING_THRESHOLD) * 1000, 
REAPING_LEADER_PATH);
{code}

we hit one scenario where  one ZK quorum slows down for short period, causing 
many Zk locks not released properly, right after ChildReaper (every 5 min ) 
runs, which keep checking the list of Znode ever since, in the end, Oozie 
server hit OOM.


  was:
OOZIE-1906 added znode cleanup thread.
currently passing Reaper.Mode.REAP_INDEFINITELY, but this enforce Oozie server 
to keep reaping znode even after znode is cleaned up.   This adds memory 
pressure on oozie server. Need to change to REAP_UNTIL_GONE  or 
REAP_UNTIL_DELETE
  
{code}
reaper = new ChildReaper(zk.getClient(), LOCKS_NODE, 
Reaper.Mode.REAP_INDEFINITELY, getExecutorService(), 
ConfigurationService.getInt(services.getConf(), REAPING_THRESHOLD) * 1000, 
REAPING_LEADER_PATH);
{code}

we hit one scenario where  one ZK quorum slows down for short period, causing 
many Zk locks not released properly, right after ChildReaper (every 5 min ) 
runs, which keep checking the list of Znode ever since, in the end, Oozie 
server hit OOM.



> Change Reaper mode on ChildReaper in ZKLocksService
> ---
>
> Key: OOZIE-2206
> URL: https://issues.apache.org/jira/browse/OOZIE-2206
> Project: Oozie
>  Issue Type: Bug
>    Reporter: Ryota Egashira
>
> OOZIE-1906 added znode cleanup thread.
> currently passing Reaper.Mode.REAP_INDEFINITELY, but this enforce Oozie 
> server to keep reaping znode even after znode is cleaned up.  
> (https://github.com/apache/curator/blob/master/curator-recipes/src/main/java/org/apache/curator/framework/recipes/locks/Reaper.java)
>  
> This adds memory pressure on oozie server. Need to change to REAP_UNTIL_GONE  
> or REAP_UNTIL_DELETE
>   
> {code}
> reaper = new ChildReaper(zk.getClient(), LOCKS_NODE, 
> Reaper.Mode.REAP_INDEFINITELY, getExecutorService(), 
> ConfigurationService.getInt(services.getConf(), REAPING_THRESHOLD) * 1000, 
> REAPING_LEADER_PATH);
> {code}
> we hit one scenario where  one ZK quorum slows down for short period, causing 
> many Zk locks not released properly, right after ChildReaper (every 5 min ) 
> runs, which keep checking the list of Znode ever since, in the end, Oozie 
> server hit OOM.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OOZIE-2206) Change Reaper mode on ChildReaper in ZKLocksService

2015-04-15 Thread Ryota Egashira (JIRA)

Ryota Egashira created OOZIE-2206:
-

 Summary: Change Reaper mode on ChildReaper in ZKLocksService
 Key: OOZIE-2206
 URL: https://issues.apache.org/jira/browse/OOZIE-2206
 Project: Oozie
  Issue Type: Bug
Reporter: Ryota Egashira


OOZIE-1906 added znode cleanup thread.
currently passing Reaper.Mode.REAP_INDEFINITELY, but this enforce Oozie server 
to keep reaping znode even after znode is cleaned up.   This adds memory 
pressure on oozie server. Need to change to REAP_UNTIL_GONE  or 
REAP_UNTIL_DELETE
  
{code}
reaper = new ChildReaper(zk.getClient(), LOCKS_NODE, 
Reaper.Mode.REAP_INDEFINITELY, getExecutorService(), 
ConfigurationService.getInt(services.getConf(), REAPING_THRESHOLD) * 1000, 
REAPING_LEADER_PATH);
{code}

we hit one scenario where  one ZK quorum slows down for short period, causing 
many Zk locks not released properly, right after ChildReaper (every 5 min ) 
runs, which keep checking the list of Znode ever since, in the end, Oozie 
server hit OOM.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 33221: OOZIE-2205 add option to load default/site.xml to actionConf on compute node

2015-04-15 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33221/
---

Review request for oozie.


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-2205


Diffs
-

  core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
704b762 
  core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
dbb50ba 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
cf22ed6 
  core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
4d6a1b4 
  core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
18de48a 
  core/src/main/resources/oozie-default.xml 3936fca 
  
sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveActionExecutor.java
 c28839c 
  
sharelib/hive2/src/test/java/org/apache/oozie/action/hadoop/TestHive2ActionExecutor.java
 16d0267 
  
sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java
 039bd84 

Diff: https://reviews.apache.org/r/33221/diff/


Testing
---

tested on real cluster.


Thanks,

Ryota Egashira

[jira] [Created] (OOZIE-2205) add option to load default/site.xml to actionConf on compute node

2015-04-15 Thread Ryota Egashira (JIRA)

Ryota Egashira created OOZIE-2205:
-

 Summary: add option to load default/site.xml to actionConf on 
compute node
 Key: OOZIE-2205
 URL: https://issues.apache.org/jira/browse/OOZIE-2205
 Project: Oozie
  Issue Type: Bug
Reporter: Ryota Egashira
Assignee: Ryota Egashira


currently actionConf is loaded from *-default, *-site.xml (core-site, 
hdfs-site, mapred-site, etc) on oozie server, and some default value is passed 
into child job as it is, which we observed cause unexpected behavior for pig 
and hive action. this patch is to add option to load default/site.xml to 
actionConf on compute node, for pig and hive actions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSSION] Oozie 4.2.0 release

2015-04-09 Thread Ryota Egashira

OOZIE-1993 need to be fixed before release.workflow rerun failed with 
oozie.wf.rerun.failnodes=true in some cases. actually because of this, we 
reverted OOZIE-1879 for our internal branch.

ThanksRyota 



 On Thursday, March 26, 2015 4:57 PM, Robert Kanter  
wrote:
   

 Another JIRA that would be good to get is OOZIE-2186 which updates the
Tomcat version.

On Thu, Mar 26, 2015 at 2:46 PM, Shwetha Shivalingamurthy <
sshivalingamur...@hortonworks.com> wrote:

> Ok, I will wait for them. Thanks
>
>
> On 26/03/15 2:21 pm, "Rohini Palaniswamy"  wrote:
>
> >We don't have a list of issues that need to be included. But Michelle ran
> >the regression with trunk yesterday and there were couple of failures. We
> >haven't investigated them yet. Those will have to be fixed though. Ryota
> >will file jiras and upload fixes next week after investigating them.
> >
> >Regards,
> >Rohini
> >
> >On Thu, Mar 26, 2015 at 1:58 PM, Harsh J  wrote:
> >
> >> I can take over OOZIE-1963. I need to test the modified hive(1) example
> >>and
> >> will then attach the change to the JIRA.
> >>
> >> On Thu, Mar 26, 2015 at 6:20 AM, Robert Kanter 
> >> wrote:
> >>
> >> > I was hoping one of our QA engineers would take care of that, but it
> >> looks
> >> > like he won't have time.  I don't think I'm proficient enough in Hive
> >> > Server 2 to make the example, so I've unassigned OOZIE-1963 from me.
> >> > Though I think it would still be good to include.
> >> >
> >> > If we include the Spark action, I think we should also include
> >> OOZIE-2170.
> >> > OOZIE-2174 would also be good; it addresses some long-standing
> >> shortcomings
> >> > of OozieClient/OozieCLI.
> >> > Finally, we should also include OOZIE-2181, which fixes some problems
> >> > retrieving job info.
> >> > I have patches available for all 3 of these.
> >> >
> >> >
> >> > - Robert
> >> >
> >> > On Wed, Mar 25, 2015 at 10:58 AM, Shwetha Shivalingamurthy <
> >> > sshivalingamur...@hortonworks.com> wrote:
> >> >
> >> > > Robert, will you be able to take up OOZIE-1963?
> >> > >
> >> > > Dev team, Are there any other un-resolved jiras that needs to be
> >>part
> >> of
> >> > > 4.2 release?
> >> > >
> >> > > Thanks,
> >> > > Shwetha
> >> > >
> >> > >
> >> > > On 11/02/15 1:44 pm, "Robert Kanter"  wrote:
> >> > >
> >> > > >I 4.2.0 release sounds good; I don't have cycles to drive it
> >>though.
> >> > > >
> >> > > >The Spark action currently has a number of problems with it.  I'd
> >>hold
> >> > off
> >> > > >on putting that into a release until we get those fixed (OOZIE-2071
> >> > > > (it's to add an
> >> > > >example,
> >> > > >but the comments discuss the problems), OOZIE-2087
> >> > > >).
> >> > > >
> >> > > >I'm all for including the Hive2 action; though it would be good if
> >>we
> >> > > >could
> >> > > >create an example workflow for it for completeness (OOZIE-1963
> >> > > >).  I can
> >>probably
> >> > take
> >> > > >care of this relatively easily.
> >> > > >
> >> > > >
> >> > > >- Robert
> >> > > >
> >> > > >On Tue, Feb 10, 2015 at 9:56 PM, Shwetha Shivalingamurthy <
> >> > > >sshivalingamur...@hortonworks.com> wrote:
> >> > > >
> >> > > >> Hi All,
> >> > > >>
> >> > > >> Oozie trunk has a couple of changes since the last oozie release
> >>-
> >> > > >>re-run
> >> > > >> fixes, config cleanup, spark action, hive2 action, external
> >> > > >>configuration
> >> > > >> for mapred action, better error reporting and many other bug
> >>fixes.
> >> > > >>
> >> > > >> Can we have a oozie release from trunk? If someone wants to
> >> volunteer
> >> > to
> >> > > >> drive the release, please take this forward. Else, I can drive
> >>it as
> >> > > >>well.
> >> > > >>
> >> > > >> If we want to do a release, let's identify the pending jiras
> >>which
> >> are
> >> > > >> required for this release.
> >> > > >>
> >> > > >> Thanks,
> >> > > >> Shwetha
> >> > > >>
> >> > >
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >>
>
>

Re: Review Request 32389: OOZIE-2140 Audit Log should be shown in Oozie UI

2015-04-03 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32389/#review78769
---


need auditlog test in TestZKXLogStreamingService for HA case, like errorlog 
(testStreamingWithMultipleOozieServers_errorLog)


core/src/main/java/org/apache/oozie/BaseEngine.java
<https://reviews.apache.org/r/32389/#comment127801>

Workflow/Coordinator Job Information --> Workflow/Coordinator/Bundle Audit 
log

(actualy the same goes to streamErrorLog javadoc)



core/src/main/java/org/apache/oozie/DagEngine.java
<https://reviews.apache.org/r/32389/#comment127802>

only starttime, endtime, lastmodifiedtime needed. no need to fetch entire 
job bean



core/src/main/java/org/apache/oozie/service/XLogService.java
<https://reviews.apache.org/r/32389/#comment127803>

just nitpick..
need to be public? other methods no modfier


- Ryota Egashira


On March 23, 2015, 8:45 a.m., Purshotam Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32389/
> ---
> 
> (Updated March 23, 2015, 8:45 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2140
> https://issues.apache.org/jira/browse/OOZIE-2140
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2140 Audit Log should be shown in Oozie UI
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 5feb360 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java 416d066 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 4c75d2a 
>   core/src/main/java/org/apache/oozie/BaseEngine.java 44074ea 
>   core/src/main/java/org/apache/oozie/BundleEngine.java e191184 
>   core/src/main/java/org/apache/oozie/CoordinatorEngine.java 1b21d87 
>   core/src/main/java/org/apache/oozie/DagEngine.java faf2d64 
>   core/src/main/java/org/apache/oozie/service/DagXLogInfoService.java 7970546 
>   core/src/main/java/org/apache/oozie/service/XLogService.java c291fa8 
>   core/src/main/java/org/apache/oozie/service/XLogStreamingService.java 
> 9066c7a 
>   core/src/main/java/org/apache/oozie/service/ZKXLogStreamingService.java 
> b28a591 
>   core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java a581f8b 
>   core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 4e9c224 
>   core/src/main/java/org/apache/oozie/servlet/V0JobServlet.java 3cb9168 
>   core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java d4564c6 
>   core/src/main/java/org/apache/oozie/servlet/V2JobServlet.java 7100c98 
>   core/src/main/java/org/apache/oozie/util/TimestampedMessageParser.java 
> d2aa02a 
>   core/src/main/java/org/apache/oozie/util/XLogAuditFilter.java e69de29 
>   core/src/main/java/org/apache/oozie/util/XLogFilter.java 5c0d1f3 
>   core/src/test/java/org/apache/oozie/service/TestXLogStreamingService.java 
> aedd35c 
>   core/src/test/resources/test-no-dash-log4j.properties 2ff353e 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 762e2f1 
>   docs/src/site/twiki/WebServicesAPI.twiki bf2bf50 
>   webapp/src/main/webapp/oozie-console.js e07680f 
> 
> Diff: https://reviews.apache.org/r/32389/diff/
> 
> 
> Testing
> ---
> 
> UTC and manul testing
> 
> $ oozie job -auditlog 000-150322000230582-oozie-puru-C -oozie 
> http://localhost:11000/oozie 
> 2015-03-22 00:04:35,494  INFO oozieaudit:520 - IP [127.0.0.1], USER 
> [purushah], GROUP [null], APP [-], JOBID 
> [000-150322000230582-oozie-puru-C], OPERATION [start], PARAMETER [null], 
> STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
> 2015-03-22 00:05:13,823  INFO oozieaudit:520 - IP [127.0.0.1], USER 
> [purushah], GROUP [null], APP [-], JOBID 
> [000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER 
> [000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], 
> ERRORCODE [null], ERRORMESSAGE [null]
> 2015-03-22 00:06:59,561  INFO oozieaudit:520 - IP [127.0.0.1], USER 
> [purushah], GROUP [null], APP [-], JOBID 
> [000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER 
> [000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], 
> ERRORCODE [null], ERRORMESSAGE [null]
> 2015-03-22 23:22:20,012  INFO oozieaudit:520 - IP [127.0.0.1], USER 
> [purushah], GROUP [null], APP [-], JOBID 
> [000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER 
> [000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], 
> ERRORCODE [null],

[jira] [Updated] (OOZIE-2191) Upgrade jackson version for hadoop-2 profile

2015-04-01 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2191:
--
Attachment: OOZIE-2191.patch

> Upgrade jackson version for hadoop-2 profile
> 
>
> Key: OOZIE-2191
> URL: https://issues.apache.org/jira/browse/OOZIE-2191
> Project: Oozie
>  Issue Type: Bug
>    Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Attachments: OOZIE-2191.patch
>
>
> in hadoop 2,  jackson version upgraded to 1.9.13
> https://issues.apache.org/jira/browse/HADOOP-10104
> oozie currently uses 1.8.8 (jackson-mapper-asl, jackson-core-asl), and when 
> both versions mixed,  cause the following exception.  this patch is to 
> upgrade version to 1.9.13 accordingly for hadoop-2 profile.
> {code}
> Caused by: java.lang.NoSuchFieldError: EXTERNAL_PROPERTY
> at 
> org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector._findTypeResolver(JacksonAnnotationIntrospector.java:781)
>  ~[na:na]
> at 
> org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findPropertyTypeResolver(JacksonAnnotationIntrospector.java:214)
>  ~[na:na]
> at 
> org.codehaus.jackson.map.ser.BeanSerializerFactory.findPropertyTypeSerializer(BeanSerializerFactory.java:374)
>  ~[na:na]
> at 
> org.codehaus.jackson.map.ser.BeanSerializerFactory._constructWriter(BeanSerializerFactory.java:778)
>  ~[na:na]
> at 
> org.codehaus.jackson.map.ser.BeanSerializerFactory.findBeanProperties(BeanSerializerFactory.java:590)
>  ~[na:na]
> at 
> org.codehaus.jackson.map.ser.BeanSerializerFactory.constructBeanSerializer(BeanSerializerFactory.java:434)
>  ~[na:na]
> at 
> org.codehaus.jackson.map.ser.BeanSerializerFactory.findBeanSerializer(BeanSerializerFactory.java:347)
>  ~[na:na]
> at 
> org.codehaus.jackson.map.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:291)
>  ~[na:na]
> at 
> org.codehaus.jackson.map.ser.StdSerializerProvider._createUntypedSerializer(StdSerializerProvider.java:782)
>  ~[na:na]
> at 
> org.codehaus.jackson.map.ser.StdSerializerProvider._createAndCacheUntypedSerializer(StdSerializerProvider.java:735)
>  ~[na:na]
> at 
> org.codehaus.jackson.map.ser.StdSerializerProvider.findValueSerializer(StdSerializerProvider.java:344)
>  ~[na:na]
> at 
> org.codehaus.jackson.map.ser.StdSerializerProvider.findTypedValueSerializer(StdSerializerProvider.java:420)
>  ~[na:na]
> at 
> org.codehaus.jackson.map.ser.StdSerializerProvider._serializeValue(StdSerializerProvider.java:601)
>  ~[na:na]
> at 
> org.codehaus.jackson.map.ser.StdSerializerProvider.serializeValue(StdSerializerProvider.java:256)
>  ~[na:na]
> at 
> org.codehaus.jackson.map.ObjectMapper._configAndWriteValue(ObjectMapper.java:2575)
>  ~[na:na]
> at 
> org.codehaus.jackson.map.ObjectMapper.writeValue(ObjectMapper.java:2065) 
> ~[na:na]
> at 
> org.apache.oozie.util.FixedJsonInstanceSerializer.serialize(FixedJsonInstanceSerializer.java:65)
>  ~[na:na]
> at 
> org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.internalRegisterService(ServiceDiscoveryImpl.java:166)
>  ~[na:na]
> at 
> org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.registerService(ServiceDiscoveryImpl.java:150)
>  ~[na:na]
> at org.apache.oozie.util.ZKUtils.advertiseService(ZKUtils.java:217) 
> ~[na:na]
> at org.apache.oozie.util.ZKUtils.(ZKUtils.java:141) ~[na:na]
> at org.apache.oozie.util.ZKUtils.register(ZKUtils.java:154) ~[na:na]
> at 
> org.apache.oozie.service.ZKLocksService.init(ZKLocksService.java:70) ~[na:na]
> at 
> org.apache.oozie.service.Services.setServiceInternal(Services.java:386) 
> ~[na:na]
> at org.apache.oozie.service.Services.setService(Services.java:372) 
> ~[na:na]
> at org.apache.oozie.service.Services.loadServices(Services.java:305) 
> ~[na:na]
> at org.apache.oozie.service.Services.init(Services.java:213) ~[na:na]
> at 
> com.yahoo.oozie.servlet.YServicesLoader.contextInitialized(YServicesLoader.java:43)
>  ~[na:na]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OOZIE-2191) Upgrade jackson version for hadoop-2 profile

2015-04-01 Thread Ryota Egashira (JIRA)

Ryota Egashira created OOZIE-2191:
-

 Summary: Upgrade jackson version for hadoop-2 profile
 Key: OOZIE-2191
 URL: https://issues.apache.org/jira/browse/OOZIE-2191
 Project: Oozie
  Issue Type: Bug
Reporter: Ryota Egashira
Assignee: Ryota Egashira


in hadoop 2,  jackson version upgraded to 1.9.13
https://issues.apache.org/jira/browse/HADOOP-10104

oozie currently uses 1.8.8 (jackson-mapper-asl, jackson-core-asl), and when 
both versions mixed,  cause the following exception.  this patch is to upgrade 
version to 1.9.13 accordingly for hadoop-2 profile.
{code}
Caused by: java.lang.NoSuchFieldError: EXTERNAL_PROPERTY
at 
org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector._findTypeResolver(JacksonAnnotationIntrospector.java:781)
 ~[na:na]
at 
org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findPropertyTypeResolver(JacksonAnnotationIntrospector.java:214)
 ~[na:na]
at 
org.codehaus.jackson.map.ser.BeanSerializerFactory.findPropertyTypeSerializer(BeanSerializerFactory.java:374)
 ~[na:na]
at 
org.codehaus.jackson.map.ser.BeanSerializerFactory._constructWriter(BeanSerializerFactory.java:778)
 ~[na:na]
at 
org.codehaus.jackson.map.ser.BeanSerializerFactory.findBeanProperties(BeanSerializerFactory.java:590)
 ~[na:na]
at 
org.codehaus.jackson.map.ser.BeanSerializerFactory.constructBeanSerializer(BeanSerializerFactory.java:434)
 ~[na:na]
at 
org.codehaus.jackson.map.ser.BeanSerializerFactory.findBeanSerializer(BeanSerializerFactory.java:347)
 ~[na:na]
at 
org.codehaus.jackson.map.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:291)
 ~[na:na]
at 
org.codehaus.jackson.map.ser.StdSerializerProvider._createUntypedSerializer(StdSerializerProvider.java:782)
 ~[na:na]
at 
org.codehaus.jackson.map.ser.StdSerializerProvider._createAndCacheUntypedSerializer(StdSerializerProvider.java:735)
 ~[na:na]
at 
org.codehaus.jackson.map.ser.StdSerializerProvider.findValueSerializer(StdSerializerProvider.java:344)
 ~[na:na]
at 
org.codehaus.jackson.map.ser.StdSerializerProvider.findTypedValueSerializer(StdSerializerProvider.java:420)
 ~[na:na]
at 
org.codehaus.jackson.map.ser.StdSerializerProvider._serializeValue(StdSerializerProvider.java:601)
 ~[na:na]
at 
org.codehaus.jackson.map.ser.StdSerializerProvider.serializeValue(StdSerializerProvider.java:256)
 ~[na:na]
at 
org.codehaus.jackson.map.ObjectMapper._configAndWriteValue(ObjectMapper.java:2575)
 ~[na:na]
at 
org.codehaus.jackson.map.ObjectMapper.writeValue(ObjectMapper.java:2065) 
~[na:na]
at 
org.apache.oozie.util.FixedJsonInstanceSerializer.serialize(FixedJsonInstanceSerializer.java:65)
 ~[na:na]
at 
org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.internalRegisterService(ServiceDiscoveryImpl.java:166)
 ~[na:na]
at 
org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.registerService(ServiceDiscoveryImpl.java:150)
 ~[na:na]
at org.apache.oozie.util.ZKUtils.advertiseService(ZKUtils.java:217) 
~[na:na]
at org.apache.oozie.util.ZKUtils.(ZKUtils.java:141) ~[na:na]
at org.apache.oozie.util.ZKUtils.register(ZKUtils.java:154) ~[na:na]
at org.apache.oozie.service.ZKLocksService.init(ZKLocksService.java:70) 
~[na:na]
at 
org.apache.oozie.service.Services.setServiceInternal(Services.java:386) ~[na:na]
at org.apache.oozie.service.Services.setService(Services.java:372) 
~[na:na]
at org.apache.oozie.service.Services.loadServices(Services.java:305) 
~[na:na]
at org.apache.oozie.service.Services.init(Services.java:213) ~[na:na]
at 
com.yahoo.oozie.servlet.YServicesLoader.contextInitialized(YServicesLoader.java:43)
 ~[na:na]
{code}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2183) provide an option to disable cleanup for output dataset in rerun

2015-04-01 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2183:
--
Attachment: OOZIE-2183-v2.patch

> provide an option to disable cleanup for output dataset in rerun 
> -
>
> Key: OOZIE-2183
> URL: https://issues.apache.org/jira/browse/OOZIE-2183
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>    Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Fix For: trunk
>
> Attachments: OOZIE-2183-v2.patch
>
>
> currently unless nocleanup option is explicitly used in CLI command when 
> rerun coordinator, output dataset is removed, default is cleanup.  This is 
> not desirable for some data pipeline use cases where output datasets are 
> shared among multiple jobs, and accidental deletion by rerun (when failing to 
> pass nocleanup option from CLI) affect the other jobs.  
> this jira is to add cleanup attribute to disable cleanup in coordinator.xml



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2146) add SLA bulk API

2015-04-01 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2146:
--
Attachment: OOZIE-2146-v4.patch

> add SLA bulk API
> 
>
> Key: OOZIE-2146
> URL: https://issues.apache.org/jira/browse/OOZIE-2146
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>    Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Fix For: trunk
>
> Attachments: OOZIE-2146-v4.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 31232: OOZIE-2146 add SLA bulk API

2015-04-01 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31232/
---

(Updated April 1, 2015, 1:42 p.m.)


Review request for oozie.


Changes
---

fixed review comments


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-2146


Diffs (updated)
-

  client/src/main/java/org/apache/oozie/client/OozieClient.java 5de25cc 
  client/src/main/java/org/apache/oozie/client/rest/JsonTags.java 1022dd7 
  
core/src/main/java/org/apache/oozie/executor/jpa/CoordinatorJobGetForUserAppnameJPAExecutor.java
 3bb2f88 
  
core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
 b55cda3 
  core/src/main/java/org/apache/oozie/servlet/V2SLAServlet.java 57170e1 
  core/src/main/java/org/apache/oozie/sla/SLASummaryBean.java a88dcf6 
  core/src/test/java/org/apache/oozie/servlet/DagServletTestCase.java 48193c7 
  core/src/test/java/org/apache/oozie/servlet/TestV2SLAServlet.java 1886f48 
  docs/src/site/twiki/DG_SLAMonitoring.twiki 1413945 

Diff: https://reviews.apache.org/r/31232/diff/


Testing
---


Thanks,

Ryota Egashira

Re: Review Request 32443: OOZIE-2183 provide an option to disable cleanup for output dataset in rerun

2015-03-31 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32443/
---

(Updated March 31, 2015, 11:36 p.m.)


Review request for oozie.


Changes
---

fixed comment


Summary (updated)
-

OOZIE-2183 provide an option to disable cleanup for output dataset in rerun 


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-2183


Diffs (updated)
-

  client/src/main/resources/oozie-coordinator-0.4.xsd 78205f6 
  core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 
050662e 
  core/src/test/java/org/apache/oozie/command/coord/TestCoordRerunXCommand.java 
45457d1 
  core/src/test/resources/coord-rerun-action4.xml PRE-CREATION 
  docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 0812bb8 
  docs/src/site/twiki/DG_CoordinatorRerun.twiki c937da5 

Diff: https://reviews.apache.org/r/32443/diff/


Testing
---


Thanks,

Ryota Egashira

[jira] [Commented] (OOZIE-1624) Exclusion pattern for sharelib.

2015-03-24 Thread Ryota Egashira (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14378181#comment-14378181
 ] 

Ryota Egashira commented on OOZIE-1624:
---

[~puru] reviewed v2 patch.  just one comment,  can we cache regex exclusion 
pattern to map(something like key:actiontype, value:pattern)? 
buildConditionalExclusionPatterns is called every time when java action runs, 
it might be costly.  

> Exclusion pattern for sharelib.
> ---
>
> Key: OOZIE-1624
> URL: https://issues.apache.org/jira/browse/OOZIE-1624
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-1624-V2.patch, OOZIE-1624-v1.patch
>
>
> Sharelib may bring some jar which might conflict with user jars.
> Ex. Sharelib hive has json-2..jar, where as some of the user use-case 
> need higher version of json jar.
> He should be able to exclude sharelib json jar and bring his own version.
> 
> oozie.action.sharelib.for.hive.exclusion
> json-\*.jar|abc-*.jar
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 32443: provide an option to disable cleanup for output dataset in rerun

2015-03-24 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32443/
---

Review request for oozie.


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-2183


Diffs
-

  client/src/main/resources/oozie-coordinator-0.4.xsd 78205f6 
  core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 
050662e 
  core/src/test/java/org/apache/oozie/command/coord/TestCoordRerunXCommand.java 
45457d1 
  docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 0812bb8 
  docs/src/site/twiki/DG_CoordinatorRerun.twiki c937da5 

Diff: https://reviews.apache.org/r/32443/diff/


Testing
---


Thanks,

Ryota Egashira

[jira] [Created] (OOZIE-2183) provide an option to disable cleanup for output dataset in rerun

2015-03-24 Thread Ryota Egashira (JIRA)

Ryota Egashira created OOZIE-2183:
-

 Summary: provide an option to disable cleanup for output dataset 
in rerun 
 Key: OOZIE-2183
 URL: https://issues.apache.org/jira/browse/OOZIE-2183
 Project: Oozie
  Issue Type: Bug
Reporter: Ryota Egashira
Assignee: Ryota Egashira


currently unless nocleanup option is explicitly used in CLI command when rerun 
coordinator, output dataset is removed, default is cleanup.  This is not 
desirable for some data pipeline use cases where output datasets are shared 
among multiple jobs, and accidental deletion by rerun (when failing to pass 
nocleanup option from CLI) affect the other jobs.  

this jira is to add cleanup attribute to disable cleanup in coordinator.xml



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 31232: OOZIE-2146 add SLA bulk API

2015-03-23 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31232/
---

(Updated March 23, 2015, 9:25 p.m.)


Review request for oozie.


Changes
---

modified patch based on review comments


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-2146


Diffs (updated)
-

  client/src/main/java/org/apache/oozie/client/OozieClient.java 5de25cc 
  client/src/main/java/org/apache/oozie/client/rest/JsonTags.java 1022dd7 
  
core/src/main/java/org/apache/oozie/executor/jpa/CoordinatorJobGetForUserAppnameJPAExecutor.java
 3bb2f88 
  
core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
 b55cda3 
  core/src/main/java/org/apache/oozie/servlet/V2SLAServlet.java 57170e1 
  core/src/main/java/org/apache/oozie/sla/SLASummaryBean.java a88dcf6 
  core/src/test/java/org/apache/oozie/servlet/DagServletTestCase.java 48193c7 
  core/src/test/java/org/apache/oozie/servlet/TestV2SLAServlet.java 1886f48 
  docs/src/site/twiki/DG_SLAMonitoring.twiki 1413945 

Diff: https://reviews.apache.org/r/31232/diff/


Testing
---


Thanks,

Ryota Egashira

[jira] [Commented] (OOZIE-2177) Parameterize javadoc plugin configuration

2015-03-20 Thread Ryota Egashira (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14371904#comment-14371904
 ] 

Ryota Egashira commented on OOZIE-2177:
---

thanks Robert

> Parameterize javadoc plugin configuration 
> --
>
> Key: OOZIE-2177
> URL: https://issues.apache.org/jira/browse/OOZIE-2177
> Project: Oozie
>  Issue Type: Bug
>Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Attachments: OOZIE-2177.patch
>
>
> this patch is to add  to org.apache.maven.plugins, and allow 
> it to be overwritten by -D option from CLI.
> this is necessary when building oozie on JDK8,  since it hit following 
> javadoc error, which fails builds
> {quote}
> [ERROR] 
> /Users/egashira/Projects/git/aoozie-cms/client/src/main/java/org/apache/oozie/cli/OozieCLI.java:205:
>  error: self-closing element not allowed
> [ERROR] * 
> {quote}
> http://stackoverflow.com/questions/26049329/javadoc-in-jdk-8-invalid-self-closing-element-not-allowed
> JDK 8 it has been decided that tags like  and  should generate 
> errors, because they are invalid (strict) HTML 4.  
> this can be avoided by disabling specific checks, by passing -Xdoclint:none 
> to javadoc plugin parameter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2177) Parameterize javadoc plugin configuration

2015-03-20 Thread Ryota Egashira (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14371850#comment-14371850
 ] 

Ryota Egashira commented on OOZIE-2177:
---

{quote}
so I will open another JIRA for real javadoc fix to keep track
{quote}
https://issues.apache.org/jira/browse/OOZIE-2178

> Parameterize javadoc plugin configuration 
> --
>
> Key: OOZIE-2177
> URL: https://issues.apache.org/jira/browse/OOZIE-2177
> Project: Oozie
>  Issue Type: Bug
>Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Attachments: OOZIE-2177.patch
>
>
> this patch is to add  to org.apache.maven.plugins, and allow 
> it to be overwritten by -D option from CLI.
> this is necessary when building oozie on JDK8,  since it hit following 
> javadoc error, which fails builds
> {quote}
> [ERROR] 
> /Users/egashira/Projects/git/aoozie-cms/client/src/main/java/org/apache/oozie/cli/OozieCLI.java:205:
>  error: self-closing element not allowed
> [ERROR] * 
> {quote}
> http://stackoverflow.com/questions/26049329/javadoc-in-jdk-8-invalid-self-closing-element-not-allowed
> JDK 8 it has been decided that tags like  and  should generate 
> errors, because they are invalid (strict) HTML 4.  
> this can be avoided by disabling specific checks, by passing -Xdoclint:none 
> to javadoc plugin parameter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OOZIE-2178) fix javadoc to compile on JDK8

2015-03-20 Thread Ryota Egashira (JIRA)

Ryota Egashira created OOZIE-2178:
-

 Summary: fix javadoc to compile on JDK8
 Key: OOZIE-2178
 URL: https://issues.apache.org/jira/browse/OOZIE-2178
 Project: Oozie
  Issue Type: Bug
Reporter: Ryota Egashira


creating this JIRA from discussion on OOZIE-2177 

JDK8 enforce strict HTML4 checking on javadoc.  
when compiling oozie on JDK8, hit following javadoc error
{quote}
[ERROR] 
/Users/egashira/Projects/git/aoozie-cms/client/src/main/java/org/apache/oozie/cli/OozieCLI.java:205:
 error: self-closing element not allowed
[ERROR] * 
{quote}

the fix is to change self-closing element ,  to , 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2177) Parameterize javadoc plugin configuration

2015-03-20 Thread Ryota Egashira (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14371832#comment-14371832
 ] 

Ryota Egashira commented on OOZIE-2177:
---

[~rkanter] yes, agree, to really fix JDK8 javadoc issue,  we have to change 
javadoc parts of code  (e.g.,  to ), but it come appear in many 
places.  profile also works, but end up suppressing doc check anyway, which is 
still workaround in a sense. so I will open another JIRA for real javadoc fix 
to keep track, (I don't have time to work at this moment, and we are just 
checking test case failure on JDK8 over jenkins build, but anybody can pick up 
when have time)   
about this patch, regardless of JDK8 or not, parameterizing javadoc parameters 
is valuable, so let me keep this separately.

> Parameterize javadoc plugin configuration 
> --
>
> Key: OOZIE-2177
> URL: https://issues.apache.org/jira/browse/OOZIE-2177
> Project: Oozie
>  Issue Type: Bug
>    Reporter: Ryota Egashira
>Assignee: Ryota Egashira
> Attachments: OOZIE-2177.patch
>
>
> this patch is to add  to org.apache.maven.plugins, and allow 
> it to be overwritten by -D option from CLI.
> this is necessary when building oozie on JDK8,  since it hit following 
> javadoc error, which fails builds
> {quote}
> [ERROR] 
> /Users/egashira/Projects/git/aoozie-cms/client/src/main/java/org/apache/oozie/cli/OozieCLI.java:205:
>  error: self-closing element not allowed
> [ERROR] * 
> {quote}
> http://stackoverflow.com/questions/26049329/javadoc-in-jdk-8-invalid-self-closing-element-not-allowed
> JDK 8 it has been decided that tags like  and  should generate 
> errors, because they are invalid (strict) HTML 4.  
> this can be avoided by disabling specific checks, by passing -Xdoclint:none 
> to javadoc plugin parameter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2177) Parameterize javadoc plugin configuration

2015-03-20 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2177:
--
Attachment: OOZIE-2177.patch

> Parameterize javadoc plugin configuration 
> --
>
> Key: OOZIE-2177
> URL: https://issues.apache.org/jira/browse/OOZIE-2177
> Project: Oozie
>  Issue Type: Bug
>    Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Attachments: OOZIE-2177.patch
>
>
> this patch is to add  to org.apache.maven.plugins, and allow 
> it to be overwritten by -D option from CLI.
> this is necessary when building oozie on JDK8,  since it hit following 
> javadoc error, which fails builds
> {quote}
> [ERROR] 
> /Users/egashira/Projects/git/aoozie-cms/client/src/main/java/org/apache/oozie/cli/OozieCLI.java:205:
>  error: self-closing element not allowed
> [ERROR] * 
> {quote}
> http://stackoverflow.com/questions/26049329/javadoc-in-jdk-8-invalid-self-closing-element-not-allowed
> JDK 8 it has been decided that tags like  and  should generate 
> errors, because they are invalid (strict) HTML 4.  
> this can be avoided by disabling specific checks, by passing -Xdoclint:none 
> to javadoc plugin parameter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OOZIE-2177) Parameterize javadoc plugin configuration

2015-03-20 Thread Ryota Egashira (JIRA)

Ryota Egashira created OOZIE-2177:
-

 Summary: Parameterize javadoc plugin configuration 
 Key: OOZIE-2177
 URL: https://issues.apache.org/jira/browse/OOZIE-2177
 Project: Oozie
  Issue Type: Bug
Reporter: Ryota Egashira
Assignee: Ryota Egashira


this patch is to add  to org.apache.maven.plugins, and allow 
it to be overwritten by -D option from CLI.

this is necessary when building oozie on JDK8,  since it hit following javadoc 
error, which fails builds
{quote}
[ERROR] 
/Users/egashira/Projects/git/aoozie-cms/client/src/main/java/org/apache/oozie/cli/OozieCLI.java:205:
 error: self-closing element not allowed
[ERROR] * 
{quote}

http://stackoverflow.com/questions/26049329/javadoc-in-jdk-8-invalid-self-closing-element-not-allowed
JDK 8 it has been decided that tags like  and  should generate 
errors, because they are invalid (strict) HTML 4.  

this can be avoided by disabling specific checks, by passing -Xdoclint:none to 
javadoc plugin parameter.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2160) Support attachment in email action

2015-03-10 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2160:
--
Attachment: OOZIE-2160-v5-nopre.patch

> Support attachment in email action
> --
>
> Key: OOZIE-2160
> URL: https://issues.apache.org/jira/browse/OOZIE-2160
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>    Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Fix For: trunk
>
> Attachments: OOZIE-2160-v4-nopre.patch, OOZIE-2160-v5-nopre.patch
>
>
> Make email action able to attach a file on HDFS.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 31687: OOZIE-2160 Support attachment in email action

2015-03-10 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31687/
---

(Updated March 10, 2015, 6:27 p.m.)


Review request for oozie.


Changes
---

chagned it to error out.


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-2160


Diffs (updated)
-

  client/src/main/resources/email-action-0.2.xsd 60ce545 
  core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java 
21c6313 
  core/src/main/java/org/apache/oozie/service/AbandonedCoordCheckerService.java 
ec8cf71 
  core/src/test/java/org/apache/oozie/action/email/TestEmailActionExecutor.java 
e1f314e 
  docs/src/site/twiki/DG_EmailActionExtension.twiki 695e16b 

Diff: https://reviews.apache.org/r/31687/diff/


Testing
---

test done. tested in QE enviornment with real HDFS


Thanks,

Ryota Egashira

[jira] [Resolved] (OOZIE-2166) modify "Unknow hadoop job" log message when checking completion of child job in Map-Reduce action

2015-03-09 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira resolved OOZIE-2166.
---
Resolution: Duplicate

thanks Rohini,  it was dup

> modify "Unknow hadoop job" log message when checking completion of child job 
> in Map-Reduce action
> -
>
> Key: OOZIE-2166
> URL: https://issues.apache.org/jira/browse/OOZIE-2166
> Project: Oozie
>  Issue Type: Bug
>Reporter: Ryota Egashira
>Assignee: Ryota Egashira
>
> {quote}
> JA017: Unknown hadoop job [job_1425692024974_1399] associated with action 
> [2186013-150307023607065-oozie_MR-W@abc].  Failing this action!
> {quote}
> this job still points to launcher job, even when it is checking child job.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2160) Support attachment in email action

2015-03-09 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2160:
--
Description: Make email action able to attach a file on HDFS.  

> Support attachment in email action
> --
>
> Key: OOZIE-2160
> URL: https://issues.apache.org/jira/browse/OOZIE-2160
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>    Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Fix For: trunk
>
> Attachments: OOZIE-2160-v4-nopre.patch
>
>
> Make email action able to attach a file on HDFS.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2160) Support attachment in email action

2015-03-09 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2160:
--
Attachment: OOZIE-2160-v4-nopre.patch

> Support attachment in email action
> --
>
> Key: OOZIE-2160
> URL: https://issues.apache.org/jira/browse/OOZIE-2160
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>    Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Fix For: trunk
>
> Attachments: OOZIE-2160-v4-nopre.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 31687: OOZIE-2160 Support attachment in email action

2015-03-09 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31687/
---

(Updated March 9, 2015, 8:45 p.m.)


Review request for oozie.


Changes
---

final version with fix


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-2160


Diffs (updated)
-

  client/src/main/resources/email-action-0.2.xsd 60ce545 
  core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java 
21c6313 
  core/src/main/java/org/apache/oozie/service/AbandonedCoordCheckerService.java 
ec8cf71 
  core/src/test/java/org/apache/oozie/action/email/TestEmailActionExecutor.java 
e1f314e 
  docs/src/site/twiki/DG_EmailActionExtension.twiki 695e16b 

Diff: https://reviews.apache.org/r/31687/diff/


Testing
---

test done. tested in QE enviornment with real HDFS


Thanks,

Ryota Egashira

[jira] [Created] (OOZIE-2166) modify "Unknow hadoop job" log message when checking completion of child job in Map-Reduce action

2015-03-09 Thread Ryota Egashira (JIRA)

Ryota Egashira created OOZIE-2166:
-

 Summary: modify "Unknow hadoop job" log message when checking 
completion of child job in Map-Reduce action
 Key: OOZIE-2166
 URL: https://issues.apache.org/jira/browse/OOZIE-2166
 Project: Oozie
  Issue Type: Bug
Reporter: Ryota Egashira
Assignee: Ryota Egashira


{quote}
JA017: Unknown hadoop job [job_1425692024974_1399] associated with action 
[2186013-150307023607065-oozie_MR-W@abc].  Failing this action!
{quote}

this job still points to launcher job, even when it is checking child job.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 31687: OOZIE-2160 Support attachment in email action

2015-03-06 Thread Ryota Egashira



> On March 6, 2015, 6:26 p.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java, 
> > line 227
> > <https://reviews.apache.org/r/31687/diff/2/?file=885745#file885745line227>
> >
> > You will have to do doAs as that user to access the hdfs file. Oozie 
> > user will not have permissions to read it.

URIDataSource is doing that


- Ryota


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31687/#review75505
-------


On March 7, 2015, 5:54 a.m., Ryota Egashira wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31687/
> ---
> 
> (Updated March 7, 2015, 5:54 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/OOZIE-2160
> 
> 
> Diffs
> -
> 
>   client/src/main/resources/email-action-0.2.xsd 60ce545 
>   core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java 
> 21c6313 
>   
> core/src/main/java/org/apache/oozie/service/AbandonedCoordCheckerService.java 
> ec8cf71 
>   
> core/src/test/java/org/apache/oozie/action/email/TestEmailActionExecutor.java 
> e1f314e 
>   docs/src/site/twiki/DG_EmailActionExtension.twiki 695e16b 
> 
> Diff: https://reviews.apache.org/r/31687/diff/
> 
> 
> Testing
> ---
> 
> test done. tested in QE enviornment with real HDFS
> 
> 
> Thanks,
> 
> Ryota Egashira
> 
>

Re: Review Request 31687: OOZIE-2160 Support attachment in email action

2015-03-06 Thread Ryota Egashira



> On March 6, 2015, 7:02 p.m., Purshotam Shah wrote:
> > core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java, 
> > line 148
> > <https://reviews.apache.org/r/31687/diff/2/?file=885745#file885745line148>
> >
> > attachments can be parameterized, so we need to evaluate it.

ensure EL funct works on that


- Ryota


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31687/#review75526
---


On March 7, 2015, 5:54 a.m., Ryota Egashira wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31687/
> ---
> 
> (Updated March 7, 2015, 5:54 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/OOZIE-2160
> 
> 
> Diffs
> -
> 
>   client/src/main/resources/email-action-0.2.xsd 60ce545 
>   core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java 
> 21c6313 
>   
> core/src/main/java/org/apache/oozie/service/AbandonedCoordCheckerService.java 
> ec8cf71 
>   
> core/src/test/java/org/apache/oozie/action/email/TestEmailActionExecutor.java 
> e1f314e 
>   docs/src/site/twiki/DG_EmailActionExtension.twiki 695e16b 
> 
> Diff: https://reviews.apache.org/r/31687/diff/
> 
> 
> Testing
> ---
> 
> test done. tested in QE enviornment with real HDFS
> 
> 
> Thanks,
> 
> Ryota Egashira
> 
>

Re: Review Request 31687: OOZIE-2160 Support attachment in email action

2015-03-06 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31687/
---

(Updated March 7, 2015, 5:54 a.m.)


Review request for oozie.


Changes
---

revised version


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-2160


Diffs (updated)
-

  client/src/main/resources/email-action-0.2.xsd 60ce545 
  core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java 
21c6313 
  core/src/main/java/org/apache/oozie/service/AbandonedCoordCheckerService.java 
ec8cf71 
  core/src/test/java/org/apache/oozie/action/email/TestEmailActionExecutor.java 
e1f314e 
  docs/src/site/twiki/DG_EmailActionExtension.twiki 695e16b 

Diff: https://reviews.apache.org/r/31687/diff/


Testing
---

test done. tested in QE enviornment with real HDFS


Thanks,

Ryota Egashira

[jira] [Commented] (OOZIE-2156) override hive.querylog.location in hive-site.xml

2015-03-05 Thread Ryota Egashira (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349180#comment-14349180
 ] 

Ryota Egashira commented on OOZIE-2156:
---

[~puru]  opened https://issues.apache.org/jira/browse/OOZIE-2161 for that.  

> override hive.querylog.location in hive-site.xml 
> -
>
> Key: OOZIE-2156
> URL: https://issues.apache.org/jira/browse/OOZIE-2156
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>    Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Fix For: trunk
>
> Attachments: OOZIE-2156.patch
>
>
> OOZIE-2068 allow users to fetch hive-site.xml from sharelib.
> but user still have to override hive.querylog.location in their workflow 
> definition (to point to current working directory on compute node), otherwise 
> it fails.  Oozie should override for user. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OOZIE-2161) Parameterize action specific default values in oozie-default.xml

2015-03-05 Thread Ryota Egashira (JIRA)

Ryota Egashira created OOZIE-2161:
-

 Summary: Parameterize action specific default values in 
oozie-default.xml
 Key: OOZIE-2161
 URL: https://issues.apache.org/jira/browse/OOZIE-2161
 Project: Oozie
  Issue Type: Bug
Reporter: Ryota Egashira


https://issues.apache.org/jira/browse/OOZIE-2156?focusedCommentId=14347542&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14347542

{quote}
We should allow setting properties like oozie.action... 
This can be defined in oozie-default/site.xml and can be overridden by user.
Whoever want to set any property or override, they can do without making any 
code change.
{quote}

examples might be  hive action (hive.querylog.location), shell action 
(PATH=.:$PATH"  in addDefaultChildEnv)...etc



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 31687: OOZIE-2160 Support attachment in email action

2015-03-05 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31687/
---

(Updated March 5, 2015, 1:45 p.m.)


Review request for oozie.


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-2160


Diffs
-

  client/src/main/resources/email-action-0.2.xsd 60ce545 
  core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java 
21c6313 
  core/src/main/java/org/apache/oozie/service/AbandonedCoordCheckerService.java 
ec8cf71 
  core/src/test/java/org/apache/oozie/action/email/TestEmailActionExecutor.java 
e1f314e 
  docs/src/site/twiki/DG_EmailActionExtension.twiki 695e16b 

Diff: https://reviews.apache.org/r/31687/diff/


Testing (updated)
---

test done. tested in QE enviornment with real HDFS


Thanks,

Ryota Egashira

Re: Review Request 31687: OOZIE-2160 Support attachment in email action

2015-03-05 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31687/
---

(Updated March 5, 2015, 1:44 p.m.)


Review request for oozie.


Changes
---

support attaching file on HDFS,  fixed review comment


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-2160


Diffs (updated)
-

  client/src/main/resources/email-action-0.2.xsd 60ce545 
  core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java 
21c6313 
  core/src/main/java/org/apache/oozie/service/AbandonedCoordCheckerService.java 
ec8cf71 
  core/src/test/java/org/apache/oozie/action/email/TestEmailActionExecutor.java 
e1f314e 
  docs/src/site/twiki/DG_EmailActionExtension.twiki 695e16b 

Diff: https://reviews.apache.org/r/31687/diff/


Testing
---

still in testing


Thanks,

Ryota Egashira

Review Request 31687: OOZIE-2160 Support attachment in email action

2015-03-03 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31687/
---

Review request for oozie.


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-2160


Diffs
-

  client/src/main/resources/email-action-0.2.xsd 60ce545 
  core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java 
21c6313 
  core/src/main/java/org/apache/oozie/service/AbandonedCoordCheckerService.java 
ec8cf71 
  core/src/test/java/org/apache/oozie/action/email/TestEmailActionExecutor.java 
e1f314e 
  docs/src/site/twiki/DG_EmailActionExtension.twiki 695e16b 

Diff: https://reviews.apache.org/r/31687/diff/


Testing
---

still in testing


Thanks,

Ryota Egashira

[jira] [Created] (OOZIE-2160) Support attachment in email action

2015-03-03 Thread Ryota Egashira (JIRA)

Ryota Egashira created OOZIE-2160:
-

 Summary: Support attachment in email action
 Key: OOZIE-2160
 URL: https://issues.apache.org/jira/browse/OOZIE-2160
 Project: Oozie
  Issue Type: Bug
Reporter: Ryota Egashira
Assignee: Ryota Egashira






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2155) Incorrect DST Shifts are occurring based on the Database timezone

2015-02-27 Thread Ryota Egashira (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14341010#comment-14341010
 ] 

Ryota Egashira commented on OOZIE-2155:
---

Robert,  all of our production servers running on oracle, and many hourly 
coordinator jobs with timezone=UTC running over DST time(2014-11-02), but 
didn't see this materialization-time-shift issue reported (normally see other 
issues when timezone is non-UTC),  we can double-check on that, this could be 
related to Oracle setting used by our environment, we can ask DBA as well.

> Incorrect DST Shifts are occurring based on the Database timezone
> -
>
> Key: OOZIE-2155
> URL: https://issues.apache.org/jira/browse/OOZIE-2155
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: trunk, 4.1.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: oracle-fix-not-working.patch
>
>
> We discovered a critical bug where incorrect Daylight Saving Time shifts were 
> occurring based on the Database timezone.
> Try running this Coordinator:
> {code:xml}
>  start="2014-11-02T04:15Z" end="2014-11-02T16:00Z" timezone=“UTC” 
> xmlns="uri:oozie:coordinator:0.1">
>
>   10
>
>
>   
>  ${appPath}
>  
> 
>jobTracker
>${jobTracker}
> 
> 
>nameNode
>${nameNode}
> 
> 
>queueName
>${queueName}
> 
>  
>   
> 
> 
> {code}
> Note that it runs over a DST shift (at least in most US timezones).
> Here's a sample of some of the actions, along with their Nominal Times:
> {noformat}
> 013-150212140838307-oozie-rkan-C@4 2014-11-02 07:15 GMT 
> 013-150212140838307-oozie-rkan-C@6 2014-11-02 09:15 GMT 
> 013-150212140838307-oozie-rkan-C@5 2014-11-02 09:15 GMT 
> 013-150212140838307-oozie-rkan-C@7 2014-11-02 10:15 GMT 
> {noformat}
> Note that actions 5 and 6 have the same time.  This is incorrect because the 
> times are in GMT (with no DST).  I've also confirmed via a debugger that 
> these dates have the same number of seconds since epoch (so it's not just a 
> rendering issue).  
> By the way, if you're in a different timezone, you'll see this problem occur 
> on different actions because the issue is related to the timezone that your 
> database is in.  It depends on when the DST shift occurs in your timezone; 
> for example, in "America/New_York", it happens with actions 2 and 3.  
> On that note, if I ask Oozie to use "America/Los_Angeles" to print the dates, 
> I get this:
> {noformat}
> 013-150212140838307-oozie-rkan-C@4 2014-11-02 00:15 PDT 
> 013-150212140838307-oozie-rkan-C@6 2014-11-02 01:15 PST 
> 013-150212140838307-oozie-rkan-C@5 2014-11-02 01:15 PST 
> 013-150212140838307-oozie-rkan-C@7 2014-11-02 02:15 PST 
> {noformat}
> Action 5's nominal time should be {{2014-11-02 01:15 PDT}}, not {{2014-11-02 
> 01:15 PST}}.  
> Using the debugger some more, I verified that Oozie is creating the nominal 
> times correctly, and writing them to the database correctly (at least, it's 
> converting them to Java's SQL TimeStamp objects correctly; OpenJPA handles 
> writing them).  But when the problematic value is read back from the 
> database, it has the wrong value!
> Here's something interesting from the [MySQL 
> documentation|http://dev.mysql.com/doc/refman/5.5/en/datetime.html]:
> {quote}MySQL converts TIMESTAMP values from the current time zone to UTC for 
> storage, and back from UTC to the current time zone for retrieval. (This does 
> not occur for other types such as DATETIME.) By default, the current time 
> zone for each connection is the server's time. The time zone can be set on a 
> per-connection basis. As long as the time zone setting remains constant, you 
> get back the same value you store. If you store a TIMESTAMP value, and then 
> change the time zone and retrieve the value, the retrieved value is different 
> from the value you stored. This occurs because the same time zone was not 
> used for conversion in both directions.
> {quote}
> So, I think what's happening is that it's interpreting everything in PST, and 
> not in PST and PDT depending on the time itself.  Ideally, it wo

[jira] [Updated] (OOZIE-2153) modify HCatCredentialHelper to use HCatClient

2015-02-26 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2153:
--
Attachment: OOZIE-2153.patch

> modify HCatCredentialHelper to use HCatClient
> -
>
> Key: OOZIE-2153
> URL: https://issues.apache.org/jira/browse/OOZIE-2153
> Project: Oozie
>  Issue Type: Bug
>    Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Attachments: OOZIE-2153.patch
>
>
> use HCatClient instead of HiveMetaStoreClient in HCatCredentialHelper.
> at rolling upgrade of hive(HIVE-9583), users of HCatClient are protected but 
> not HiveMetaStoreClient



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2156) override hive.querylog.location in hive-site.xml

2015-02-26 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2156:
--
Attachment: (was: OOZIE-2156-noprefix.patch)

> override hive.querylog.location in hive-site.xml 
> -
>
> Key: OOZIE-2156
> URL: https://issues.apache.org/jira/browse/OOZIE-2156
> Project: Oozie
>  Issue Type: Bug
>    Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Attachments: OOZIE-2156.patch
>
>
> OOZIE-2068 allow users to fetch hive-site.xml from sharelib.
> but user still have to override hive.querylog.location in their workflow 
> definition (to point to current working directory on compute node), otherwise 
> it fails.  Oozie should override for user. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2156) override hive.querylog.location in hive-site.xml

2015-02-26 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2156:
--
Attachment: OOZIE-2156.patch

previous one is wrong patch..  uploading new one

> override hive.querylog.location in hive-site.xml 
> -
>
> Key: OOZIE-2156
> URL: https://issues.apache.org/jira/browse/OOZIE-2156
> Project: Oozie
>  Issue Type: Bug
>    Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Attachments: OOZIE-2156.patch
>
>
> OOZIE-2068 allow users to fetch hive-site.xml from sharelib.
> but user still have to override hive.querylog.location in their workflow 
> definition (to point to current working directory on compute node), otherwise 
> it fails.  Oozie should override for user. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2156) override hive.querylog.location in hive-site.xml

2015-02-26 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2156:
--
Attachment: OOZIE-2156-noprefix.patch

> override hive.querylog.location in hive-site.xml 
> -
>
> Key: OOZIE-2156
> URL: https://issues.apache.org/jira/browse/OOZIE-2156
> Project: Oozie
>  Issue Type: Bug
>    Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Attachments: OOZIE-2156-noprefix.patch
>
>
> OOZIE-2068 allow users to fetch hive-site.xml from sharelib.
> but user still have to override hive.querylog.location in their workflow 
> definition (to point to current working directory on compute node), otherwise 
> it fails.  Oozie should override for user. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OOZIE-2156) override hive.querylog.location in hive-site.xml

2015-02-25 Thread Ryota Egashira (JIRA)

Ryota Egashira created OOZIE-2156:
-

 Summary: override hive.querylog.location in hive-site.xml 
 Key: OOZIE-2156
 URL: https://issues.apache.org/jira/browse/OOZIE-2156
 Project: Oozie
  Issue Type: Bug
Reporter: Ryota Egashira
Assignee: Ryota Egashira


OOZIE-2068 allow users to fetch hive-site.xml from sharelib.
but user still have to override hive.querylog.location in their workflow 
definition (to point to current working directory on compute node), otherwise 
it fails.  Oozie should override for user. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OOZIE-2153) modify HCatCredentialHelper to use HCatClient

2015-02-25 Thread Ryota Egashira (JIRA)

Ryota Egashira created OOZIE-2153:
-

 Summary: modify HCatCredentialHelper to use HCatClient
 Key: OOZIE-2153
 URL: https://issues.apache.org/jira/browse/OOZIE-2153
 Project: Oozie
  Issue Type: Bug
Reporter: Ryota Egashira
Assignee: Ryota Egashira


use HCatClient instead of HiveMetaStoreClient in HCatCredentialHelper.
at rolling upgrade of hive(HIVE-9583), users of HCatClient are protected but 
not HiveMetaStoreClient



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 31232: OOZIE-2146 add SLA bulk API

2015-02-25 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31232/
---

(Updated Feb. 25, 2015, 7:44 p.m.)


Review request for oozie.


Changes
---

fixed review comments


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-2146


Diffs (updated)
-

  client/src/main/java/org/apache/oozie/client/OozieClient.java 5de25cc 
  client/src/main/java/org/apache/oozie/client/rest/JsonTags.java 1022dd7 
  
core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
 b55cda3 
  core/src/main/java/org/apache/oozie/servlet/V2SLAServlet.java 57170e1 
  core/src/main/java/org/apache/oozie/sla/SLASummaryBean.java a88dcf6 
  core/src/test/java/org/apache/oozie/servlet/DagServletTestCase.java 48193c7 
  core/src/test/java/org/apache/oozie/servlet/TestV2SLAServlet.java 1886f48 
  docs/src/site/twiki/DG_SLAMonitoring.twiki 1413945 

Diff: https://reviews.apache.org/r/31232/diff/


Testing
---


Thanks,

Ryota Egashira

[jira] [Updated] (OOZIE-2132) FS actions are not retried

2015-02-25 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2132:
--
Attachment: OOZIE-2132-v3.patch

> FS actions are not retried
> --
>
> Key: OOZIE-2132
> URL: https://issues.apache.org/jira/browse/OOZIE-2132
> Project: Oozie
>  Issue Type: Bug
>    Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Attachments: OOZIE-2132-v2.patch, OOZIE-2132-v3.patch, 
> OOZIE-2132.patch
>
>
> fs actions are not retried even when user explicitly set retry-max/interval. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2132) FS actions are not retried

2015-02-20 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2132:
--
Attachment: OOZIE-2132-v2.patch

> FS actions are not retried
> --
>
> Key: OOZIE-2132
> URL: https://issues.apache.org/jira/browse/OOZIE-2132
> Project: Oozie
>  Issue Type: Bug
>    Reporter: Ryota Egashira
>    Assignee: Ryota Egashira
> Attachments: OOZIE-2132-v2.patch, OOZIE-2132.patch
>
>
> fs actions are not retried even when user explicitly set retry-max/interval. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 31232: OOZIE-2146 add SLA bulk API

2015-02-20 Thread Ryota Egashira


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31232/
---

Review request for oozie.


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-2146


Diffs
-

  client/src/main/java/org/apache/oozie/client/OozieClient.java e4c93cd 
  client/src/main/java/org/apache/oozie/client/rest/JsonTags.java b7cf0e7 
  
core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
 b55cda3 
  core/src/main/java/org/apache/oozie/servlet/V2SLAServlet.java a0fe1b6 
  core/src/main/java/org/apache/oozie/sla/SLASummaryBean.java 9907dd0 
  core/src/test/java/org/apache/oozie/servlet/DagServletTestCase.java 48193c7 
  core/src/test/java/org/apache/oozie/servlet/TestV2SLAServlet.java 5f51b22 
  docs/src/site/twiki/DG_SLAMonitoring.twiki acf8ac1 

Diff: https://reviews.apache.org/r/31232/diff/


Testing
---


Thanks,

Ryota Egashira

[jira] [Created] (OOZIE-2146) add SLA bulk API

2015-02-20 Thread Ryota Egashira (JIRA)

Ryota Egashira created OOZIE-2146:
-

 Summary: add SLA bulk API
 Key: OOZIE-2146
 URL: https://issues.apache.org/jira/browse/OOZIE-2146
 Project: Oozie
  Issue Type: Bug
Reporter: Ryota Egashira
Assignee: Ryota Egashira






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OOZIE-2141) Make time filter offset format consistent among web APIs

2015-02-13 Thread Ryota Egashira (JIRA)

Ryota Egashira created OOZIE-2141:
-

 Summary: Make time filter offset format consistent among web APIs
 Key: OOZIE-2141
 URL: https://issues.apache.org/jira/browse/OOZIE-2141
 Project: Oozie
  Issue Type: Bug
Reporter: Ryota Egashira


v2/jobs or log streaming support time range filter, which allow to set offset 
value format like 10h, -1d, but the notation of '-', '+' is not consistent. for 
instance, -1d supposed to be (current time -1 day), 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2108) bulk kill, suspend, resume jobs using existing filter, offset, len, and jobtype params

2015-02-11 Thread Ryota Egashira (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14316595#comment-14316595
 ] 

Ryota Egashira commented on OOZIE-2108:
---

i agree with the idea of using the same command regardless of operating either 
on single or multiple jobs, to minimize user confusion.
totally understand that having kill subcommand in jobs is reasonable and 
requires the least change in implementation, but once we open it up to users, 
probably it will become difficult to consolidate both (or deprecate one over 
the other) in the future.

> bulk kill, suspend, resume jobs using existing filter, offset, len, and 
> jobtype params
> --
>
> Key: OOZIE-2108
> URL: https://issues.apache.org/jira/browse/OOZIE-2108
> Project: Oozie
>  Issue Type: New Feature
>Reporter: Bowen Zhang
>Assignee: Bowen Zhang
> Attachments: oozie-2108.patch
>
>
> Currently, there is no bulk write operations in "jobs" API. We would like to 
> first introduce a bulk kill operation that kills all jobs which satisfy the 
> filter. The desired usage will be {noformat}oozie jobs -oozie 
> http://localhost:11000/oozie -kill -filter name=something{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OOZIE-2135) Partition existence check by HCatClient.getPartitions causes high load on HCatalog

2015-02-09 Thread Ryota Egashira (JIRA)

Ryota Egashira created OOZIE-2135:
-

 Summary: Partition existence check by HCatClient.getPartitions 
causes high load on HCatalog
 Key: OOZIE-2135
 URL: https://issues.apache.org/jira/browse/OOZIE-2135
 Project: Oozie
  Issue Type: Bug
Reporter: Ryota Egashira
Assignee: Ryota Egashira






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2127) Add created time to RecoveryService WF queries

2015-02-06 Thread Ryota Egashira (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryota Egashira updated OOZIE-2127:
--
Attachment: OOZIE-2127-amend-test.patch

amend-patch to fix test case failure 
(TestWorkflowActionQueryExecutor.testGetList) caused by query change in 
OOZIE-2127

> Add created time to RecoveryService WF queries
> --
>
> Key: OOZIE-2127
> URL: https://issues.apache.org/jira/browse/OOZIE-2127
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Fix For: trunk
>
> Attachments: OOZIE-2127-V1.patch, OOZIE-2127-V2.patch, 
> OOZIE-2127-amend-test.patch
>
>
>Scanning the whole workflow actions table takes time. Can use created time 
> to further filter and if that is defined as the partition column it gives 
> more performance. Setting the default value as 7 days as jobs don't usually 
> run longer than that and also a delegation token cannot be renewed more than 
> 7 days by default for secure clusters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

1 2 3 4 5 6 7 8 9 10 >

1 - 100 of 968 matches

Mail list logo