date:20180206

[jira] Subscription: Oozie Patch Available

2018-02-06 Thread jira

Issue Subscription
Filter: Oozie Patch Available (111 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3129  Fix test TestConfigurationService.testOozieConfig
https://issues-test.apache.org/jira/browse/OOZIE-3129
OOZIE-3127  Remove redundant check for user
https://issues-test.apache.org/jira/browse/OOZIE-3127
OOZIE-3126  Add option to allow list of users to access system config
https://issues-test.apache.org/jira/browse/OOZIE-3126
OOZIE-3121  bump all maven plugins to latest versions
https://issues-test.apache.org/jira/browse/OOZIE-3121
OOZIE-3118  fix for error: self-closing element not allowed
https://issues-test.apache.org/jira/browse/OOZIE-3118
OOZIE-3117  fix for warning: no @throws for
https://issues-test.apache.org/jira/browse/OOZIE-3117
OOZIE-3116  fix for warning: no description for @throws
https://issues-test.apache.org/jira/browse/OOZIE-3116
OOZIE-3115  fix for javadoc warning: empty  tag
https://issues-test.apache.org/jira/browse/OOZIE-3115
OOZIE-3113  Retry for ZK lock release
https://issues-test.apache.org/jira/browse/OOZIE-3113
OOZIE-3112  SparkConfigrationService overwrites properties provided via 
--properties-file option in SparkAction
https://issues-test.apache.org/jira/browse/OOZIE-3112
OOZIE-3105  testJMXInstrumentation from the 
org.apache.oozie.util.TestMetricsInstrumentation class is flaky
https://issues-test.apache.org/jira/browse/OOZIE-3105
OOZIE-3094  fix for grammar mistake
https://issues-test.apache.org/jira/browse/OOZIE-3094
OOZIE-3091  Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: 
org/apache/avro/mapred/AvroWrapper"
https://issues-test.apache.org/jira/browse/OOZIE-3091
OOZIE-3083  Make improved version Info backward compatible
https://issues-test.apache.org/jira/browse/OOZIE-3083
OOZIE-3071  Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 
than Spark 2.2.0
https://issues-test.apache.org/jira/browse/OOZIE-3071
OOZIE-3063  Sanitizing variables that are part of openjpa.ConnectionProperties
https://issues-test.apache.org/jira/browse/OOZIE-3063
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues-test.apache.org/jira/browse/OOZIE-3062
OOZIE-3002  address findbugs errors in client lib
https://issues-test.apache.org/jira/browse/OOZIE-3002
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues-test.apache.org/jira/browse/OOZIE-2975
OOZIE-2973  Make sure Oozie works with Hadoop 3 
https://issues-test.apache.org/jira/browse/OOZIE-2973
OOZIE-2969  Drop support for Java 1.7
https://issues-test.apache.org/jira/browse/OOZIE-2969
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues-test.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues-test.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues-test.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues-test.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues-test.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues-test.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues-test.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues-test.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues-test.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues-test.apache.org/jira/browse/OOZIE-2934
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues-test.apache.org/jira/browse/OOZIE-2927
OOZIE-2914  Consolidate Trim 
https://issues-test.apache.org/jira/browse/OOZIE-2914
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues-test.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues-test.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues-test.apache.org/jira/browse/OOZIE-2867
OOZIE-2834  ParameterVerifier logging

[jira] Subscription: Oozie Patch Available

2018-02-06 Thread jira

Issue Subscription
Filter: Oozie Patch Available (104 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3173  Coordinator job with frequency using cron syntax creates only one 
action in catchup mode
https://issues.apache.org/jira/browse/OOZIE-3173
OOZIE-3172  Upgrade non-transitive Jackson dependencies from 
org.codehaus.jackson to com.fasterxml.jackson
https://issues.apache.org/jira/browse/OOZIE-3172
OOZIE-3135  Configure log4j2 in SqoopMain
https://issues.apache.org/jira/browse/OOZIE-3135
OOZIE-3134  Potential inconsistency between the in-memory SLA map and the Oozie 
database
https://issues.apache.org/jira/browse/OOZIE-3134
OOZIE-3121  bump all maven plugins to latest versions
https://issues.apache.org/jira/browse/OOZIE-3121
OOZIE-3105  testJMXInstrumentation from the 
org.apache.oozie.util.TestMetricsInstrumentation class is flaky
https://issues.apache.org/jira/browse/OOZIE-3105
OOZIE-3094  fix for grammar mistake
https://issues.apache.org/jira/browse/OOZIE-3094
OOZIE-3091  Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: 
org/apache/avro/mapred/AvroWrapper"
https://issues.apache.org/jira/browse/OOZIE-3091
OOZIE-3071  Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 
than Spark 2.2.0
https://issues.apache.org/jira/browse/OOZIE-3071
OOZIE-3063  Sanitizing variables that are part of openjpa.ConnectionProperties
https://issues.apache.org/jira/browse/OOZIE-3063
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues.apache.org/jira/browse/OOZIE-3062
OOZIE-3061  Kill only those child jobs which are not already killed
https://issues.apache.org/jira/browse/OOZIE-3061
OOZIE-3002  address findbugs errors in client lib
https://issues.apache.org/jira/browse/OOZIE-3002
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues.apache.org/jira/browse/OOZIE-2975
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues.apache.org/jira/browse/OOZIE-2934
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2914  Consolidate Trim 
https://issues.apache.org/jira/browse/OOZIE-2914
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues.apache.org/jira/browse/OOZIE-2867
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues.apache.org/jira/browse/OOZIE-2829
OOZIE-2826  Falcon feed fails to aws s3; Oozie joda time version does not meet 
required jar version 2.2 or later
https://issues.apache.org/jira/browse/OOZIE-2826
OOZIE-2814  OYA: Update example workflows to newest schemas
https://issues.apache.org/jira/browse/OOZIE-2814
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions

Re: [VOTE] Release Oozie 4.3.1(candidate 4)

2018-02-06 Thread Artem Ervits

Attila, AFAIK, these properties are in my core-site.xml except for last
one, mapreduce.framework.name is in mapred-site.xml. I'll have to double
check on the proxyuser.root.hosts as I believe my service runs with user
oozie not root and get back to you.

On Tue, Feb 6, 2018 at 3:49 PM, Attila Sasvari 
wrote:

> +1 (binding)
>
> - Verified MD5 hash, signature, rat_report, release log are OK
> - Built Oozie against Hadoop 2.7.5 (did not build against other Hadoop
> versions this time)
> - Installed Oozie sharelib
> - Some example workflows I submitted to a 1-node Hadoop 2.7.5 (running in
> pseudo distributed mode for tests) succeeded: ( aggregator bundle
> coord-input-logic cron cron-schedule custom-main datelist-java-main demo
> distcp hadoop-el hcatalog java-main map-reduce no-op pig shell sla sqoop
> sqoop-freeform ssh streaming subwf )
> - Verified basic functionality of the Web UI (version, Job DAG, Job logs)
>
>
> Artem, have you specified conf/hadoop-conf/core-site.xml with all the
> details Oozie needs to communicate with a Hadoop cluster?
>
> In my local environment, I use a pseudo Hadoop 2.7.5. My
> conf/hadoop-conf/core-site.xml is as follows:
> 
> 
> 
> 
> 
>   
> fs.defaultFS
> hdfs://localhost:9000
>   
>   
> hadoop.proxyuser.root.hosts
> *
>   
>   
> hadoop.proxyuser.root.groups
> *
>   
> 
> mapreduce.framework.name
> yarn
> 
> 
>
>
> On Tue, Feb 6, 2018 at 8:10 PM, Artem Ervits 
> wrote:
>
> > jdk: 1.8.0_161
> > hadoop: 2.7.5
> > md5: OK
> > curl http://localhost:11000/oozie/v1/admin/status
> > {"systemMode":"NORMAL"}
> >
> > curl http://localhost:11000/oozie/v2/admin/status
> > {"systemMode":"NORMAL"}
> >
> >
> > I also got in oozie-ops.log
> >
> >   STARTUP MSG: Oozie BUILD_VERSION [4.3.1] compiled by [oozie] on
> > [2018.02.06-16:30:43GMT]
> >   STARTUP MSG:   revision [unavailable]@[unavailable]
> >
> > so perhaps we may want to add but not critical OOZIE-2004, OOZIE-3083,
> > OOZIE-3152.
> >
> >  submitted a job and got but that is most likely my environment.
> Otherwise
> > +1
> >
> > 2018-02-06 18:09:37,635  WARN ActionStartXCommand:523 - SERVER[
> > hadoop.example.com] USER[vagrant] GROUP[-] TOKEN[] APP[python-wf]
> > JOB[000-180206180622504-oozie-oozi-W]
> > ACTION[000-180206180622504-oozie-oozi-W@python-node] Error starting
> > action [python-node]. ErrorType [ERROR], ErrorCode
> > [UnsupportedOperationException], Message [UnsupportedOperationException:
> > Accessing local file system is not allowed]
> > org.apache.oozie.action.ActionExecutorException:
> > UnsupportedOperationException: Accessing local file system is not allowed
> > at
> > org.apache.oozie.action.ActionExecutor.convertException(Acti
> > onExecutor.java:446)
> > at
> > org.apache.oozie.action.hadoop.JavaActionExecutor.submitLaun
> > cher(JavaActionExecutor.java:1204)
> > at
> > org.apache.oozie.action.hadoop.JavaActionExecutor.start(Java
> > ActionExecutor.java:1386)
> > at
> > org.apache.oozie.command.wf.ActionStartXCommand.execute(Acti
> > onStartXCommand.java:234)
> > at
> > org.apache.oozie.command.wf.ActionStartXCommand.execute(Acti
> > onStartXCommand.java:65)
> > at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> > at
> > org.apache.oozie.service.CallableQueueService$CompositeCalla
> > ble.call(CallableQueueService.java:332)
> > at
> > org.apache.oozie.service.CallableQueueService$CompositeCalla
> > ble.call(CallableQueueService.java:261)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.oozie.service.CallableQueueService$CallableWrappe
> > r.run(CallableQueueService.java:179)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> > Executor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> > lExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > Caused by: java.lang.UnsupportedOperationException: Accessing local file
> > system is not allowed
> > at
> > org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalF
> > ileSystem.java:48)
> > at org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSys
> > tem.java:47)
> > at org.apache.hadoop.fs.FileSystem.createFileSystem(
> FileSystem.java:2667)
> > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:93)
> > at org.apache.hadoop.fs.FileSystem$Cache.getInternal(
> FileSystem.java:2701)
> > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2683)
> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:372)
> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:171)
> > at
> > org.apache.oozie.service.ShareLibService.updateShareLib(Shar
> > eLibService.java:554)
> > at
> > org.apache.oozie.service.ShareLibService.getShareLibJars(Sha
> > reLibService.java:366)
> > at
> > org.apache.oozie.service.ShareLibService.getSystemLibJars(Sh
> > areLibService.java:432)
> > at
> >

[jira] [Commented] (OOZIE-3134) Potential inconsistency between the in-memory SLA map and the Oozie database

2018-02-06 Thread Andras Piros (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354850#comment-16354850
 ] 

Andras Piros commented on OOZIE-3134:
-

Thanks [~rohini] for the feedback! 1) will then be addressed as part of 
OOZIE-1980, and only 2) here.

As for OOZIE-2854 and database retries, there is a short intro as part of 
[*AG_Install.twiki*|https://oozie.apache.org/docs/5.0.0-beta1/AG_Install.html#Database_Configuration]
 in {{5.0.0-beta1}}.

> Potential inconsistency between the in-memory SLA map and the Oozie database
> 
>
> Key: OOZIE-3134
> URL: https://issues.apache.org/jira/browse/OOZIE-3134
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: OOZIE-3134-001.patch, OOZIE-3134-002.patch
>
>
> Upon {{SLACalculatorMemory.addRegistration}}, Oozie puts an entry into an 
> in-memory concurrent hashmap ("slaMap"), and later [persists 
> it|https://github.com/apache/oozie/blob/50f4b5984832941f1341586e43fd832c293b3275/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L380]
>  in the Oozie database. 
> However, if there is a failure during the database operation, a 
> {{JPAExecutorException}} is thrown, and the entry [is not removed from the 
> SLA map| 
> https://github.com/apache/oozie/blob/50f4b5984832941f1341586e43fd832c293b3275/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L393].
> It may introduce inconsistency between the Oozie database and the SLA map.
> To prevent this, a rollback mechanism (with proper logging) should be 
> implemented.  It would also make sense to do more sanity/consistency check in 
> the Oozie server.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3134) Potential inconsistency between the in-memory SLA map and the Oozie database

2018-02-06 Thread Rohini Palaniswamy (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354633#comment-16354633
 ] 

Rohini Palaniswamy commented on OOZIE-3134:
---

bq. This issue was originally targeted to fix only scenario 2)
 Just looked at the patch. All it does is a minor change of adding the entry to 
the map after the database operation. This should be fine to go in. 1) can be 
addressed in OOZIE-1980. We started with that jira and patch when he had a 
database issue. But never got around to wrapping that up and it did not became 
a priority as we have not run into database issues after that for long (Touch 
wood). But it is something that has to be fixed.

Was not aware of OOZIE-2854 and database update retries. Will take a look at 
that sometime.

> Potential inconsistency between the in-memory SLA map and the Oozie database
> 
>
> Key: OOZIE-3134
> URL: https://issues.apache.org/jira/browse/OOZIE-3134
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: OOZIE-3134-001.patch, OOZIE-3134-002.patch
>
>
> Upon {{SLACalculatorMemory.addRegistration}}, Oozie puts an entry into an 
> in-memory concurrent hashmap ("slaMap"), and later [persists 
> it|https://github.com/apache/oozie/blob/50f4b5984832941f1341586e43fd832c293b3275/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L380]
>  in the Oozie database. 
> However, if there is a failure during the database operation, a 
> {{JPAExecutorException}} is thrown, and the entry [is not removed from the 
> SLA map| 
> https://github.com/apache/oozie/blob/50f4b5984832941f1341586e43fd832c293b3275/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L393].
> It may introduce inconsistency between the Oozie database and the SLA map.
> To prevent this, a rollback mechanism (with proper logging) should be 
> implemented.  It would also make sense to do more sanity/consistency check in 
> the Oozie server.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Release Oozie 4.3.1(candidate 4)

2018-02-06 Thread Attila Sasvari

+1 (binding)

- Verified MD5 hash, signature, rat_report, release log are OK
- Built Oozie against Hadoop 2.7.5 (did not build against other Hadoop
versions this time)
- Installed Oozie sharelib
- Some example workflows I submitted to a 1-node Hadoop 2.7.5 (running in
pseudo distributed mode for tests) succeeded: ( aggregator bundle
coord-input-logic cron cron-schedule custom-main datelist-java-main demo
distcp hadoop-el hcatalog java-main map-reduce no-op pig shell sla sqoop
sqoop-freeform ssh streaming subwf )
- Verified basic functionality of the Web UI (version, Job DAG, Job logs)


Artem, have you specified conf/hadoop-conf/core-site.xml with all the
details Oozie needs to communicate with a Hadoop cluster?

In my local environment, I use a pseudo Hadoop 2.7.5. My
conf/hadoop-conf/core-site.xml is as follows:





  
fs.defaultFS
hdfs://localhost:9000
  
  
hadoop.proxyuser.root.hosts
*
  
  
hadoop.proxyuser.root.groups
*
  

mapreduce.framework.name
yarn




On Tue, Feb 6, 2018 at 8:10 PM, Artem Ervits  wrote:

> jdk: 1.8.0_161
> hadoop: 2.7.5
> md5: OK
> curl http://localhost:11000/oozie/v1/admin/status
> {"systemMode":"NORMAL"}
>
> curl http://localhost:11000/oozie/v2/admin/status
> {"systemMode":"NORMAL"}
>
>
> I also got in oozie-ops.log
>
>   STARTUP MSG: Oozie BUILD_VERSION [4.3.1] compiled by [oozie] on
> [2018.02.06-16:30:43GMT]
>   STARTUP MSG:   revision [unavailable]@[unavailable]
>
> so perhaps we may want to add but not critical OOZIE-2004, OOZIE-3083,
> OOZIE-3152.
>
>  submitted a job and got but that is most likely my environment. Otherwise
> +1
>
> 2018-02-06 18:09:37,635  WARN ActionStartXCommand:523 - SERVER[
> hadoop.example.com] USER[vagrant] GROUP[-] TOKEN[] APP[python-wf]
> JOB[000-180206180622504-oozie-oozi-W]
> ACTION[000-180206180622504-oozie-oozi-W@python-node] Error starting
> action [python-node]. ErrorType [ERROR], ErrorCode
> [UnsupportedOperationException], Message [UnsupportedOperationException:
> Accessing local file system is not allowed]
> org.apache.oozie.action.ActionExecutorException:
> UnsupportedOperationException: Accessing local file system is not allowed
> at
> org.apache.oozie.action.ActionExecutor.convertException(Acti
> onExecutor.java:446)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLaun
> cher(JavaActionExecutor.java:1204)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(Java
> ActionExecutor.java:1386)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(Acti
> onStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(Acti
> onStartXCommand.java:65)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCalla
> ble.call(CallableQueueService.java:332)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCalla
> ble.call(CallableQueueService.java:261)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrappe
> r.run(CallableQueueService.java:179)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.UnsupportedOperationException: Accessing local file
> system is not allowed
> at
> org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalF
> ileSystem.java:48)
> at org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSys
> tem.java:47)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:93)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2701)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2683)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:372)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:171)
> at
> org.apache.oozie.service.ShareLibService.updateShareLib(Shar
> eLibService.java:554)
> at
> org.apache.oozie.service.ShareLibService.getShareLibJars(Sha
> reLibService.java:366)
> at
> org.apache.oozie.service.ShareLibService.getSystemLibJars(Sh
> areLibService.java:432)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.addSystemS
> hareLibForAction(JavaActionExecutor.java:756)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.addAllShar
> eLibs(JavaActionExecutor.java:853)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFile
> sArchives(JavaActionExecutor.java:844)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLaun
> cher(JavaActionExecutor.java:1095)
> ... 11 more
> 2018-02-06 18:09:37,661  WARN ActionStartXCommand:523 - SERVER[
> hadoop.example.com] USER[vagrant] GROUP[-] TOKEN[] APP[python-wf]
> JOB[000-180206180622504-oozie-oozi-W]
>

Re: Review Request 65535: OOZIE-3173 Coordinator job with frequency using cron syntax creates only one action in catchup mode

2018-02-06 Thread Peter Cseh via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65535/#review196925
---




core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java
Line 507 (original), 509-514 (patched)


This was probably left here during development.



core/src/test/java/org/apache/oozie/command/coord/TestCoordMaterializeTransitionXCommand.java
Lines 331 (patched)


Could you add error messages to the asserts so it's easier to understand 
what is the error if something goes wrong?



core/src/test/java/org/apache/oozie/command/coord/TestCoordMaterializeTransitionXCommand.java
Lines 344-345 (patched)


What is the purpose of this two lines?



core/src/test/java/org/apache/oozie/command/coord/TestCoordMaterializeTransitionXCommand.java
Lines 373-374 (patched)


Same as above


- Peter Cseh


On Feb. 6, 2018, 7:34 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65535/
> ---
> 
> (Updated Feb. 6, 2018, 7:34 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-3173 Coordinator job with frequency using cron syntax creates only one 
> action in catchup mode
> 
> 
> Diffs
> -
> 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java
>  91d3508d128032ea5928cd5c6de482a65008c2bc 
>   
> core/src/test/java/org/apache/oozie/command/coord/TestCoordMaterializeTransitionXCommand.java
>  2a648d7c35c1a396e099ab27ab33d91e95d119d5 
> 
> 
> Diff: https://reviews.apache.org/r/65535/diff/1/
> 
> 
> Testing
> ---
> 
> New unit tests:
> 
> * 
> `TestCoordMaterializeTransitionXCommand#testCronFrequencyCatchupThrottleLessThanDuration()`
> * 
> `TestCoordMaterializeTransitionXCommand#testCronFrequencyCatchupThrottleEqualsDuration`
> * 
> `TestCoordMaterializeTransitionXCommand#testCronFrequencyCatchupThrottleMoreThanDuration`
> 
> 
> Thanks,
> 
> András Piros
> 
>

Review Request 65535: OOZIE-3173 Coordinator job with frequency using cron syntax creates only one action in catchup mode

2018-02-06 Thread András Piros via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65535/
---

Review request for oozie.


Repository: oozie-git


Description
---

OOZIE-3173 Coordinator job with frequency using cron syntax creates only one 
action in catchup mode


Diffs
-

  
core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java
 91d3508d128032ea5928cd5c6de482a65008c2bc 
  
core/src/test/java/org/apache/oozie/command/coord/TestCoordMaterializeTransitionXCommand.java
 2a648d7c35c1a396e099ab27ab33d91e95d119d5 


Diff: https://reviews.apache.org/r/65535/diff/1/


Testing
---

New unit tests:

* 
`TestCoordMaterializeTransitionXCommand#testCronFrequencyCatchupThrottleLessThanDuration()`
* 
`TestCoordMaterializeTransitionXCommand#testCronFrequencyCatchupThrottleEqualsDuration`
* 
`TestCoordMaterializeTransitionXCommand#testCronFrequencyCatchupThrottleMoreThanDuration`


Thanks,

András Piros

[jira] [Commented] (OOZIE-3173) Coordinator job with frequency using cron syntax creates only one action in catchup mode

2018-02-06 Thread Andras Piros (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354432#comment-16354432
 ] 

Andras Piros commented on OOZIE-3173:
-

{{TestHiveActionExecutor}} passes locally. [~asasvari] can you please take a 
look?

> Coordinator job with frequency using cron syntax creates only one action in 
> catchup mode
> 
>
> Key: OOZIE-3173
> URL: https://issues.apache.org/jira/browse/OOZIE-3173
> Project: Oozie
>  Issue Type: Bug
>  Components: coordinator
>Affects Versions: 5.0.0b1
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: OOZIE-3173.001.patch
>
>
> Using a coordinator with cron based {{frequency}}, when in catchup mode (job 
> is still in {{PREP}}), {{startTime}} lies in the past, next materialize time 
> lies also in the past), would create only one action, independent of 
> {{throttle}} value.
> As many actions should be created in catchup mode, as many times 
> {{frequency}} would fit in between {{startTime}}, {{Math.min(Math.min(now, 
> endTime), startTime + frequency * throttle)}}.
> Some examples, {{frequency="0 * * * *"}}:
> * when between {{startTime}} and {{endTime}} there are 10 hours, both lie in 
> the past, and {{throttle}} is {{1}} -> one action should be created
> * when between {{startTime}} and {{endTime}} there are 10 hours, both lie in 
> the past, and {{throttle}} is {{3}} -> three actions should be created
> * when between {{startTime}} and {{endTime}} there are 10 hours, both lie in 
> the past, and {{throttle}} is {{15}} -> ten actions should be created



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Release Oozie 4.3.1(candidate 4)

2018-02-06 Thread Artem Ervits

jdk: 1.8.0_161
hadoop: 2.7.5
md5: OK
curl http://localhost:11000/oozie/v1/admin/status
{"systemMode":"NORMAL"}

curl http://localhost:11000/oozie/v2/admin/status
{"systemMode":"NORMAL"}

I also got in oozie-ops.log

  STARTUP MSG: Oozie BUILD_VERSION [4.3.1] compiled by [oozie] on
[2018.02.06-16:30:43GMT]
  STARTUP MSG:   revision [unavailable]@[unavailable]

so perhaps we may want to add but not critical OOZIE-2004, OOZIE-3083,
OOZIE-3152.

 submitted a job and got but that is most likely my environment. Otherwise
+1

2018-02-06 18:09:37,635  WARN ActionStartXCommand:523 - SERVER[
hadoop.example.com] USER[vagrant] GROUP[-] TOKEN[] APP[python-wf]
JOB[000-180206180622504-oozie-oozi-W]
ACTION[000-180206180622504-oozie-oozi-W@python-node] Error starting
action [python-node]. ErrorType [ERROR], ErrorCode
[UnsupportedOperationException], Message [UnsupportedOperationException:
Accessing local file system is not allowed]
org.apache.oozie.action.ActionExecutorException:
UnsupportedOperationException: Accessing local file system is not allowed
at
org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1204)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1386)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:332)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:261)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.UnsupportedOperationException: Accessing local file
system is not allowed
at
org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:48)
at org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSystem.java:47)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:93)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2701)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2683)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:372)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:171)
at
org.apache.oozie.service.ShareLibService.updateShareLib(ShareLibService.java:554)
at
org.apache.oozie.service.ShareLibService.getShareLibJars(ShareLibService.java:366)
at
org.apache.oozie.service.ShareLibService.getSystemLibJars(ShareLibService.java:432)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.addSystemShareLibForAction(JavaActionExecutor.java:756)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.addAllShareLibs(JavaActionExecutor.java:853)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:844)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1095)
... 11 more
2018-02-06 18:09:37,661  WARN ActionStartXCommand:523 - SERVER[
hadoop.example.com] USER[vagrant] GROUP[-] TOKEN[] APP[python-wf]
JOB[000-180206180622504-oozie-oozi-W]
ACTION[000-180206180622504-oozie-oozi-W@python-node] Setting Action
Status to [DONE]

On Tue, Feb 6, 2018 at 7:42 AM, Gézapeti Cseh  wrote:

> Hi,
>
> I have created a build for Oozie 4.3.1, candidate 4.
>
> It includes all the changes in rc3 plus OOZIE-2888 to fix sharelib
> creation with Hadoop 2.7.5. and possibly other newer versions.
>
> Keys to verify the signature of the release artifact are available at
>   http://www.apache.org/dist/oozie/KEYS
>
> Please download, test, and try it out:
>
> http://people.apache.org/~gezapeti/oozie-4.3.1-rc4/
>
> The release, md5 signature, gpg signature, and rat report can be found at
> the above address.
>
> Vote closes on 02/12/2018 11:59 PM CET.
>

[jira] [Commented] (OOZIE-3172) Upgrade non-transitive Jackson dependencies from org.codehaus.jackson to com.fasterxml.jackson

2018-02-06 Thread Andras Piros (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354333#comment-16354333
 ] 

Andras Piros commented on OOZIE-3172:
-

[~asasvari] following examples have been run on a local Oozie server, local 
pseudo-distributed Hadoop 2.6.5 successfully:
* {{aggregator}}
* {{bundle}}
* {{coord-input-logic}}
* {{cron}}
* {{cron-schedule}}
* {{custom-main}}
* {{datelist-java-main}}
* {{demo}}
* {{distcp}}
* {{hadoop-el}}
* {{java-main}}
* {{map-reduce}}
* {{no-op}}
* {{pig}}
* {{shell}}
* {{sla}}
* {{sqoop}}
* {{sqoop-freeform}}
* {{streaming}}
* {{subwf}}
* {{spark}}

Didn't try because locally components weren't available / more setup steps 
would have been needed:
* {{hcatalog}}
* {{hive}}
* {{hive2}}
* {{pyspark}}
* {{ssh}}

I think most potential {{sharelib}} conflicts have been tested with. What are 
your thoughts?

[~gezapeti] are you suggesting to switch to {{test}} inside 
{{sharelib}} for Jackson dependencies?

> Upgrade non-transitive Jackson dependencies from org.codehaus.jackson to 
> com.fasterxml.jackson
> --
>
> Key: OOZIE-3172
> URL: https://issues.apache.org/jira/browse/OOZIE-3172
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 5.0.0b1
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3172.001.patch, OOZIE-3172.002.patch
>
>
> Jackson 1.9.3 is way too old, and has several security vulnerabilities as 
> well. Jackson 2.9.2 covers most of these.
> Let's switch from {{org.codehaus.jackson}} to {{com.fasterxml.jackson}} in 
> Oozie's direct (non-transitive) dependencies.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-1980) Sql error should not fail coord job

2018-02-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-1980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354184#comment-16354184
 ] 

Hadoop QA commented on OOZIE-1980:
--


Testing JIRA OOZIE-1980

Cleaning local git workspace



{color:red}-1{color} Patch failed to apply to head of branch




> Sql error should not fail coord job
> ---
>
> Key: OOZIE-1980
> URL: https://issues.apache.org/jira/browse/OOZIE-1980
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
>Priority: Major
> Attachments: OOZIE-1980-V1.patch, OOZIE-1980-V2.patch, 
> OOZIE-1980-V3.patch, OOZIE-1980-V4.patch
>
>
> We know that sql error is transient, it should not fail the job.
> We should ignore the SQL error, so that next materialization cycle can pick 
> the job.
> {code}
> protected void materialize() throws CommandException {
> Instrumentation.Cron cron = new Instrumentation.Cron();
> cron.start();
> try {
> materializeActions(false);
> updateJobMaterializeInfo(coordJob);
> }
> catch (CommandException ex) {
> LOG.warn("Exception occurred:" + ex.getMessage() + " Making the 
> job failed ", ex);
> coordJob.setStatus(Job.Status.FAILED);
> coordJob.resetPending();
> // remove any materialized actions and slaEvents
> insertList.clear();
> }
> catch (Exception e) {
> LOG.error("Exception occurred:" + e.getMessage() + " Making the 
> job failed ", e);
> coordJob.setStatus(Job.Status.FAILED);
> try {
> 
> CoordJobQueryExecutor.getInstance().executeUpdate(CoordJobQuery.UPDATE_COORD_JOB_MATERIALIZE,
>  coordJob);
> }
> catch (JPAExecutorException jex) {
> throw new CommandException(ErrorCode.E1011, jex);
> }
> throw new CommandException(ErrorCode.E1012, e.getMessage(), e);
> } finally {
> cron.stop();
> instrumentation.addCron(INSTRUMENTATION_GROUP, getName() + 
> ".materialize", cron);
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Failed: OOZIE-1980 PreCommit Build #381

2018-02-06 Thread Apache Jenkins Server

Jira: https://issues.apache.org/jira/browse/OOZIE-1980
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/381/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 700.68 KB...]


error: patch failed: 
core/src/main/java/org/apache/oozie/sla/SLACalculator.java:19
error: core/src/main/java/org/apache/oozie/sla/SLACalculator.java: patch does 
not apply
Checking patch 
core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java...
error: while searching for:
public boolean addRegistration(String jobId, SLARegistrationBean reg) 
throws JPAExecutorException {
try {
if (slaMap.size() < capacity) {
SLACalcStatus slaCalc = new SLACalcStatus(reg);
slaCalc.setSLAStatus(SLAStatus.NOT_STARTED);
slaCalc.setJobStatus(getJobStatus(reg.getAppType()));
slaMap.put(jobId, slaCalc);
List insertList = new ArrayList();
final SLASummaryBean summaryBean = new SLASummaryBean(slaCalc);
final Timestamp currentTime = 
DateUtils.convertDateToTimestamp(new Date());
reg.setCreatedTimestamp(currentTime);
summaryBean.setCreatedTimestamp(currentTime);
insertList.add(reg);
insertList.add(summaryBean);

BatchQueryExecutor.getInstance().executeBatchInsertUpdateDelete(insertList, 
null, null);
LOG.trace("SLA Registration Event - Job:" + jobId);
return true;

error: patch failed: 
core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java:613
error: core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java: patch 
does not apply
Checking patch core/src/main/java/org/apache/oozie/sla/SLAOperations.java...
error: while searching for:

import java.text.ParseException;
import java.util.Date;

import org.apache.oozie.AppType;
import org.apache.oozie.ErrorCode;
import org.apache.oozie.client.event.SLAEvent.EventStatus;
import org.apache.oozie.command.CommandException;
import org.apache.oozie.executor.jpa.JPAExecutorException;
import org.apache.oozie.executor.jpa.SLARegistrationQueryExecutor;

error: patch failed: 
core/src/main/java/org/apache/oozie/sla/SLAOperations.java:19
error: core/src/main/java/org/apache/oozie/sla/SLAOperations.java: patch does 
not apply
Checking patch 
core/src/main/java/org/apache/oozie/sla/service/SLAService.java...
error: while searching for:
 */
package org.apache.oozie.sla.service;

import java.util.Date;

import org.apache.hadoop.conf.Configuration;
import org.apache.oozie.ErrorCode;
import org.apache.oozie.client.event.JobEvent.EventStatus;
import org.apache.oozie.executor.jpa.JPAExecutorException;
import org.apache.oozie.service.EventHandlerService;
import org.apache.oozie.service.SchedulerService;

error: patch failed: 
core/src/main/java/org/apache/oozie/sla/service/SLAService.java:17
error: core/src/main/java/org/apache/oozie/sla/service/SLAService.java: patch 
does not apply
Checking patch 
core/src/main/java/org/apache/oozie/util/db/SLADbOperations.java...
error: while searching for:
 */
package org.apache.oozie.util.db;

import java.util.Date;

import org.apache.oozie.ErrorCode;

error: patch failed: 
core/src/main/java/org/apache/oozie/util/db/SLADbOperations.java:17
error: core/src/main/java/org/apache/oozie/util/db/SLADbOperations.java: patch 
does not apply
Checking patch 
src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java...
error: 
src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java:
 No such file or directory
Checking patch src/main/java/org/apache/oozie/command/wf/ReRunXCommand.java...
error: src/main/java/org/apache/oozie/command/wf/ReRunXCommand.java: No such 
file or directory
Checking patch src/main/java/org/apache/oozie/sla/SLACalculator.java...
error: src/main/java/org/apache/oozie/sla/SLACalculator.java: No such file or 
directory
Checking patch src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java...
error: src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java: No such 
file or directory
Checking patch src/main/java/org/apache/oozie/sla/SLAOperations.java...
error: src/main/java/org/apache/oozie/sla/SLAOperations.java: No such file or 
directory
Checking patch src/main/java/org/apache/oozie/sla/service/SLAService.java...
error: src/main/java/org/apache/oozie/sla/service/SLAService.java: No such file 
or directory
Checking patch src/main/java/org/apache/oozie/util/db/SLADbOperations.java...
error: src/main/java/org/apache/oozie/util/db/SLADbOperations.java: No such 
file or directory
Patch failed to apply to head of branch

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00

[jira] [Commented] (OOZIE-1980) Sql error should not fail coord job

2018-02-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-1980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354176#comment-16354176
 ] 

Hadoop QA commented on OOZIE-1980:
--

PreCommit-OOZIE-Build started


> Sql error should not fail coord job
> ---
>
> Key: OOZIE-1980
> URL: https://issues.apache.org/jira/browse/OOZIE-1980
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
>Priority: Major
> Attachments: OOZIE-1980-V1.patch, OOZIE-1980-V2.patch, 
> OOZIE-1980-V3.patch, OOZIE-1980-V4.patch
>
>
> We know that sql error is transient, it should not fail the job.
> We should ignore the SQL error, so that next materialization cycle can pick 
> the job.
> {code}
> protected void materialize() throws CommandException {
> Instrumentation.Cron cron = new Instrumentation.Cron();
> cron.start();
> try {
> materializeActions(false);
> updateJobMaterializeInfo(coordJob);
> }
> catch (CommandException ex) {
> LOG.warn("Exception occurred:" + ex.getMessage() + " Making the 
> job failed ", ex);
> coordJob.setStatus(Job.Status.FAILED);
> coordJob.resetPending();
> // remove any materialized actions and slaEvents
> insertList.clear();
> }
> catch (Exception e) {
> LOG.error("Exception occurred:" + e.getMessage() + " Making the 
> job failed ", e);
> coordJob.setStatus(Job.Status.FAILED);
> try {
> 
> CoordJobQueryExecutor.getInstance().executeUpdate(CoordJobQuery.UPDATE_COORD_JOB_MATERIALIZE,
>  coordJob);
> }
> catch (JPAExecutorException jex) {
> throw new CommandException(ErrorCode.E1011, jex);
> }
> throw new CommandException(ErrorCode.E1012, e.getMessage(), e);
> } finally {
> cron.stop();
> instrumentation.addCron(INSTRUMENTATION_GROUP, getName() + 
> ".materialize", cron);
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3134) Potential inconsistency between the in-memory SLA map and the Oozie database

2018-02-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354175#comment-16354175
 ] 

Hadoop QA commented on OOZIE-3134:
--


Testing JIRA OOZIE-3134

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [server].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2093
.Tests failed: 4
.Tests errors: 10

.The patch failed the following testcases:

testPartitionDependency(org.apache.oozie.service.TestPartitionDependencyManagerService)
testCoordActionRecoveryServiceForWaitingRegisterPartition(org.apache.oozie.service.TestRecoveryService)
testConnectionDrop(org.apache.oozie.jms.TestJMSJobEventListener)
testPartitionDependency(org.apache.oozie.service.TestPartitionDependencyManagerEhcache)

.Tests failing with errors:
testConnectionRetry(org.apache.oozie.service.TestJMSAccessorService)
testAddRegistrationDBFAilure(org.apache.oozie.sla.TestSLACalculatorMemory)
testUpdateRegistrationDBFailure(org.apache.oozie.sla.TestSLACalculatorMemory)
testParallelFsAndShellWorkflowCompletesSuccessfully(org.apache.oozie.test.TestWorkflowRetries)
testWorkflowWithStartAndEndCompletesSuccessfully(org.apache.oozie.test.TestWorkflowRetries)
testFsDecisionWorkflowCompletesSuccessfully(org.apache.oozie.test.TestWorkflowRetries)
testParallelFsAndShellWorkflowCompletesSuccessfully(org.apache.oozie.test.TestWorkflow)
testWorkflowWithStartAndEndCompletesSuccessfully(org.apache.oozie.test.TestWorkflow)
testFsDecisionWorkflowCompletesSuccessfully(org.apache.oozie.test.TestWorkflow)
testParallelJPAOperationsOnWorkflowBeansRetryAndSucceed(org.apache.oozie.test.TestParallelJPAOperationRetries)

.{color:orange}Tests failed at first run:{color}
TestJavaActionExecutor#testCredentialsSkip
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/380/



> Potential inconsistency between the in-memory SLA map and the Oozie database
> 
>
> Key: OOZIE-3134
> URL: https://issues.apache.org/jira/browse/OOZIE-3134
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
>

Failed: OOZIE-3134 PreCommit Build #380

2018-02-06 Thread Apache Jenkins Server

Jira: https://issues.apache.org/jira/browse/OOZIE-3134
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/380/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.62 MB...]
  Running test-patch task TESTS
  Running test-patch task DISTRO


Testing JIRA OOZIE-3134

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 1 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 100 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [server].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS
Tests run: 2093
Tests failed: 4
Tests errors: 10

The patch failed the following testcases:

testPartitionDependency(org.apache.oozie.service.TestPartitionDependencyManagerService)
testCoordActionRecoveryServiceForWaitingRegisterPartition(org.apache.oozie.service.TestRecoveryService)
testConnectionDrop(org.apache.oozie.jms.TestJMSJobEventListener)
testPartitionDependency(org.apache.oozie.service.TestPartitionDependencyManagerEhcache)

Tests failing with errors:
testConnectionRetry(org.apache.oozie.service.TestJMSAccessorService)
testAddRegistrationDBFAilure(org.apache.oozie.sla.TestSLACalculatorMemory)
testUpdateRegistrationDBFailure(org.apache.oozie.sla.TestSLACalculatorMemory)
testParallelFsAndShellWorkflowCompletesSuccessfully(org.apache.oozie.test.TestWorkflowRetries)
testWorkflowWithStartAndEndCompletesSuccessfully(org.apache.oozie.test.TestWorkflowRetries)
testFsDecisionWorkflowCompletesSuccessfully(org.apache.oozie.test.TestWorkflowRetries)
testParallelFsAndShellWorkflowCompletesSuccessfully(org.apache.oozie.test.TestWorkflow)
testWorkflowWithStartAndEndCompletesSuccessfully(org.apache.oozie.test.TestWorkflow)
testFsDecisionWorkflowCompletesSuccessfully(org.apache.oozie.test.TestWorkflow)
testParallelJPAOperationsOnWorkflowBeansRetryAndSucceed(org.apache.oozie.test.TestParallelJPAOperationRetries)

Tests failed at first run:
TestJavaActionExecutor#testCredentialsSkip
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/380/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100  44230 0  100  4423  0   5168 --:--:-- --:--:-- --:--:--  
5167{"self":"https://issues.apache.org/jira/rest/api/2/issue/13120523/comment/16354175","id":"16354175","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","emailAddress":"blackhole
 at hadoop dot apache dot 
org","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop
 QA","active":true,"timeZone":"Etc/UTC"},"body":"\nTesting JIRA 
OOZIE-3134\n\nCleaning local git 
workspace\n\n\n\n{color:green}+1

Re: Review Request 65481: OOZIE-3134 - Potential inconsistency between the in-memory SLA map and the Oozie database

2018-02-06 Thread Peter Bacsko via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65481/#review196901
---




core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java
Line 58 (original), 56 (patched)


Using Predicate like that creates a raw-type warning



core/src/main/java/org/apache/oozie/util/db/FailingHSQLDBDriverWrapper.java
Lines 34 (patched)


super() is implicit, not needed to call directly



core/src/main/java/org/apache/oozie/util/db/FailingHSQLDBDriverWrapper.java
Lines 40 (patched)


Predicate is generic interface. I believe this code creates an unchecked 
raw-type warning.



core/src/main/java/org/apache/oozie/util/db/ThreadVariables.java
Lines 23-37 (patched)


Agree with Andras. Using ThreadLocal is very confusing here. If it's not 
possible to reload/re-instantiate the class that uses this predicate, we're 
still better off with a simple helper class that has a static Predicate field 
which can be set/unset.



core/src/main/java/org/apache/oozie/util/db/ThreadVariables.java
Lines 24 (patched)


Same here about Predicate, it should be Predicate or Predicate 
depending on your needs.



core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java
Lines 1176 (patched)


Try conf.setInt()


- Peter Bacsko


On febr. 6, 2018, 2:43 du, Kinga Marton wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65481/
> ---
> 
> (Updated febr. 6, 2018, 2:43 du)
> 
> 
> Review request for oozie, András Piros and Attila Sasvari.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Upon SLACalculatorMemory.addRegistration, Oozie puts an entry into an 
> in-memory concurrent hashmap ("slaMap"), and later persists it in the Oozie 
> database.
> However, if there is a failure during the database operation, a 
> JPAExecutorException is thrown, and the entry is not removed from the SLA map.
> It may introduce inconsistency between the Oozie database and the SLA map.
> To prevent this, a rollback mechanism (with proper logging) should be 
> implemented. It would also make sense to do more sanity/consistency check in 
> the Oozie server.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java ef019e73 
>   
> core/src/main/java/org/apache/oozie/util/db/AlwaysFailingHSQLDriverMapper.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java 
> 0e310253 
>   core/src/main/java/org/apache/oozie/util/db/FailingHSQLDBDriverWrapper.java 
> fe9f08b1 
>   core/src/main/java/org/apache/oozie/util/db/FailingMySQLDriverWrapper.java 
> f0e2b181 
>   core/src/main/java/org/apache/oozie/util/db/ThreadVariables.java 
> PRE-CREATION 
>   core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java 
> ee906f45 
> 
> 
> Diff: https://reviews.apache.org/r/65481/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Kinga Marton
> 
>

Re: Review Request 65481: OOZIE-3134 - Potential inconsistency between the in-memory SLA map and the Oozie database

2018-02-06 Thread András Piros via Review Board

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65481/#review196897
---

Several comments addressing mostly the original scope of 
[OOZIE-3134](https://issues.apache.org/jira/browse/OOZIE-3134). As Rohini 
pointed out, this JIRA could be extended by 
[OOZIE-1980](https://issues.apache.org/jira/browse/OOZIE-1980) functionality. 
Please wait for her response before going on with the fix.

core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java
Line 58 (original), 56 (patched)

`@Nullable final Predicate predicate`

core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java
Lines 59 (patched)

Could go to `else` path.

core/src/main/java/org/apache/oozie/util/db/FailingHSQLDBDriverWrapper.java
Lines 40 (patched)

I don't see calling `setDbPredicate()` or `unsetDbPredicate()` from 
anywhere else than test code.

When using `ThreadLocal` we have to be very careful to unset after every 
use to avoid resource / memory leaks.

core/src/main/java/org/apache/oozie/util/db/ThreadVariables.java
Lines 23-37 (patched)

I don't see calling `setDbPredicate()` or `unsetDbPredicate()` from 
anywhere else than test code.

When using `ThreadLocal` we have to be very careful to unset after every 
use to avoid resource / memory leaks.

Generally, I think it would be a better idea to inject a new DB predicate 
by other means - I would only go for `ThreadLocal` usage when most of the time 
I need different `dbPredicate` to every `Thread` - which is not the case here.

Let's think a bit more on `Predicate` injecting scenarios apart from using 
`ThreadLocal`.

core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java
Lines 1109-1110 (patched)

Inside `try`.

core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java
Lines  (patched)

Not needed when `fail(...)` is called from `catch`.

core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java
Lines 1115 (patched)

Not needed when `fail(...)` is called from `catch`.

core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java
Lines 1120 (patched)

Not needed when `fail(...)` is called from `catch`.

core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java
Lines 1154-1155 (patched)

Both should be part of `try`.

core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java
Lines 1156 (patched)

Not needed when `fail(...)` is called from `catch`.

core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java
Lines 1161 (patched)

Better call `fail(...)` here.

core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java
Lines 1166 (patched)

Not needed when `fail(...)` is called from `catch`.

core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java
Lines 1173 (patched)

`Boolean.TRUE.toString()`

core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java
Lines 1175 (patched)

Use `AlwaysFailingHSQLDBDriverMapper.class.getCanonicalName()` and we don't 
have to modify string values when renaming / moving.

core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java
Lines 1183 (patched)

`OPERATIONS`

- András Piros

On Feb. 6, 2018, 2:43 p.m., Kinga Marton wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65481/
> ---
> 
> (Updated Feb. 6, 2018, 2:43 p.m.)
> 
> 
> Review request for oozie, András Piros and Attila Sasvari.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Upon SLACalculatorMemory.addRegistration, Oozie puts an entry into an 
> in-memory concurrent hashmap ("slaMap"), and later persists it in the Oozie 
> database.
> However, if there is a failure during the database operation, a 
> JPAExecutorException is thrown, and the entry is not removed from the SLA map.
> It may introduce inconsistency between the

[jira] [Commented] (OOZIE-3134) Potential inconsistency between the in-memory SLA map and the Oozie database

2018-02-06 Thread Andras Piros (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354078#comment-16354078
 ] 

Andras Piros commented on OOZIE-3134:
-

[~rohini] while I see there is added value in addressing also 1), which, to my 
understanding, would be covered by OOZIE-1980, we've handled transient database 
hickups in OOZIE-2854 (part of {{5.0.0-beta1}}). So the main reason behind 
OOZIE-1980 would be resolved in {{5.0.0}} but not in {{4.3.1}}.

This issue was originally targeted to fix only scenario 2). What is your view 
upon that? E.g. in subsequent {{4.3}} minor versions, would we still need to 
address OOZIE-1980?

> Potential inconsistency between the in-memory SLA map and the Oozie database
> 
>
> Key: OOZIE-3134
> URL: https://issues.apache.org/jira/browse/OOZIE-3134
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: OOZIE-3134-001.patch, OOZIE-3134-002.patch
>
>
> Upon {{SLACalculatorMemory.addRegistration}}, Oozie puts an entry into an 
> in-memory concurrent hashmap ("slaMap"), and later [persists 
> it|https://github.com/apache/oozie/blob/50f4b5984832941f1341586e43fd832c293b3275/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L380]
>  in the Oozie database. 
> However, if there is a failure during the database operation, a 
> {{JPAExecutorException}} is thrown, and the entry [is not removed from the 
> SLA map| 
> https://github.com/apache/oozie/blob/50f4b5984832941f1341586e43fd832c293b3275/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L393].
> It may introduce inconsistency between the Oozie database and the SLA map.
> To prevent this, a rollback mechanism (with proper logging) should be 
> implemented.  It would also make sense to do more sanity/consistency check in 
> the Oozie server.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3134) Potential inconsistency between the in-memory SLA map and the Oozie database

2018-02-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354039#comment-16354039
 ] 

Hadoop QA commented on OOZIE-3134:
--

PreCommit-OOZIE-Build started


> Potential inconsistency between the in-memory SLA map and the Oozie database
> 
>
> Key: OOZIE-3134
> URL: https://issues.apache.org/jira/browse/OOZIE-3134
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: OOZIE-3134-001.patch, OOZIE-3134-002.patch
>
>
> Upon {{SLACalculatorMemory.addRegistration}}, Oozie puts an entry into an 
> in-memory concurrent hashmap ("slaMap"), and later [persists 
> it|https://github.com/apache/oozie/blob/50f4b5984832941f1341586e43fd832c293b3275/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L380]
>  in the Oozie database. 
> However, if there is a failure during the database operation, a 
> {{JPAExecutorException}} is thrown, and the entry [is not removed from the 
> SLA map| 
> https://github.com/apache/oozie/blob/50f4b5984832941f1341586e43fd832c293b3275/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L393].
> It may introduce inconsistency between the Oozie database and the SLA map.
> To prevent this, a rollback mechanism (with proper logging) should be 
> implemented.  It would also make sense to do more sanity/consistency check in 
> the Oozie server.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3121) bump all maven plugins to latest versions

2018-02-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354036#comment-16354036
 ] 

Hadoop QA commented on OOZIE-3121:
--


Testing JIRA OOZIE-3121

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:red}-1{color} There are [128] new bugs found below threshold in total 
that must be fixed.
. {color:red}-1{color} There are [67] new bugs found below threshold in [core] 
that must be fixed, listing only the first [5] ones.
. You can find the FindBugs diff here (look for the red and orange ones): 
core/findbugs-new.html
. The top [5] most important FindBugs errors are:
. At EmailActionExecutor.java:[line 268]: 
java/io/File.init(Ljava/lang/String;)V reads a file whose location 
might be specified by user input
. At EmailActionExecutor.java:[line 163]: At EmailActionExecutor.java:[line 161]
. At EmailActionExecutor.java:[line 177]: At EmailActionExecutor.java:[line 172]
. At SshActionExecutor.java:[line 135]: This usage of 
java/lang/Runtime.exec([Ljava/lang/String;)Ljava/lang/Process; can be 
vulnerable to Command Injection
. At SshActionExecutor.java:[line 132]: At SshActionExecutor.java:[line 131]
. {color:red}-1{color} There are [31] new bugs found below threshold in [tools] 
that must be fixed, listing only the first [5] ones.
. You can find the FindBugs diff here (look for the red and orange ones): 
tools/findbugs-new.html
. The top [5] most important FindBugs errors are:
. At OozieDBCLI.java:[line 561]: 
java/io/FileWriter.init(Ljava/lang/String;Z)V writes to a file whose 
location might be specified by user input
. At OozieDBCLI.java:[line 127]: At OozieDBCLI.java:[line 123]
. At OozieDBCLI.java:[line 232]: At OozieDBCLI.java:[line 134]
. At OozieDBCLI.java:[line 716]: At OozieDBCLI.java:[line 257]
. At OozieDBCLI.java:[line 590]: 
java/io/FileWriter.init(Ljava/lang/String;Z)V writes to a file whose 
location might be specified by user input
. {color:red}-1{color} There are [3] new bugs found below threshold in 
[sharelib/hive2] that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones): 
sharelib/hive2/findbugs-new.html
. The most important FindBugs errors are:
. At Hive2Main.java:[line 279]: 
java/io/FileInputStream.init(Ljava/lang/String;)V reads a file whose 
location might be specified by user input
. At Hive2Main.java:[line 165]: At Hive2Main.java:[line 146]
. At Hive2Main.java:[line 149]: java/io/File.init(Ljava/lang/String;)V 
reads a file whose location might be specified by user input
. java/io/FileOutputStream.init(Ljava/lang/String;)V writes to a file 
whose location might be specified by user input: At Hive2Main.java:[line 146]
. At Hive2Main.java:[line 118]: At Hive2Main.java:[line 267]
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:red}-1{color} There are [9] new bugs found below threshold in 
[sharelib/oozie] that must be fixed, listing only the first [5] ones.
. You can find the FindBugs diff here (look for the red and orange ones): 
sharelib/oozie/findbugs-new.html
. The top [5] most important FindBugs errors are:
. At LauncherMain.java:[line 473]: 
java/io/File.init(Ljava/lang/String;)V reads a file whose location 
might be specified by user input
. At LauncherMain.java:[line 463]: 
java/io/File.init(Ljava/lang/String;)V reads a file whose location 
might be specified by user input
. At LauncherMain.java:[line 140]: 
java/io/File.init(Ljava/lang/String;)V reads a file whose location 
might be specified by user input
. java/io/FileReader.init(Ljava/lang/String;)V reads a file whose 
location might be specified by user input: At LauncherMain.java:[line 176]
. At LauncherMain.java:[line 176]: At

[jira] [Commented] (OOZIE-3163) Improve documentation rendering: use fluido skin and better config

2018-02-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354038#comment-16354038
 ] 

Hadoop QA commented on OOZIE-3163:
--


Testing JIRA OOZIE-3163

Cleaning local git workspace



> Improve documentation rendering: use fluido skin and better config
> --
>
> Key: OOZIE-3163
> URL: https://issues.apache.org/jira/browse/OOZIE-3163
> Project: Oozie
>  Issue Type: Task
>  Components: docs
>Affects Versions: 4.3.0
>Reporter: Hervé Boutemy
>Assignee: Hervé Boutemy
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: 34.patch, oozie-docs-coordinator-fs-fluido.png, 
> oozie-docs-coordinator-fs-old.png, oozie-docs-index-fluido.png, 
> oozie-docs-index-old.png
>
>
> Current output is really ugly: as Maven maintainer, I can't let Oozie with 
> such awful result :)
> Even with old maven-site-plugin 2.0-beta-6 (we'll see later how to upgrade), 
> using Fluido skin would improve output a lot
> Pull Request coming



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3163) Improve documentation rendering: use fluido skin and better config

2018-02-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354037#comment-16354037
 ] 

Hadoop QA commented on OOZIE-3163:
--

PreCommit-OOZIE-Build started


> Improve documentation rendering: use fluido skin and better config
> --
>
> Key: OOZIE-3163
> URL: https://issues.apache.org/jira/browse/OOZIE-3163
> Project: Oozie
>  Issue Type: Task
>  Components: docs
>Affects Versions: 4.3.0
>Reporter: Hervé Boutemy
>Assignee: Hervé Boutemy
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: 34.patch, oozie-docs-coordinator-fs-fluido.png, 
> oozie-docs-coordinator-fs-old.png, oozie-docs-index-fluido.png, 
> oozie-docs-index-old.png
>
>
> Current output is really ugly: as Maven maintainer, I can't let Oozie with 
> such awful result :)
> Even with old maven-site-plugin 2.0-beta-6 (we'll see later how to upgrade), 
> using Fluido skin would improve output a lot
> Pull Request coming



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Failed: OOZIE-3163 PreCommit Build #379

2018-02-06 Thread Apache Jenkins Server

Jira: https://issues.apache.org/jira/browse/OOZIE-3163
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/379/

###
## LAST 100 LINES OF THE CONSOLE 
###
Started by remote host 127.0.0.1
[EnvInject] - Loading node environment variables.
Building remotely on H17 (Hadoop xenial) in workspace 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/oozie.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/oozie.git
 > git --version # timeout=10
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/oozie.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 33d60996ab07a65f2948f7e7c34ba3ad1c6582af 
(refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 33d60996ab07a65f2948f7e7c34ba3ad1c6582af
Commit message: "OOZIE-3163 amend (reword) Improve documentation rendering: use 
fluido skin and better config (hboutemy via andras.piros)"
 > git rev-list 33d60996ab07a65f2948f7e7c34ba3ad1c6582af # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
No emails were triggered.
[PreCommit-OOZIE-Build] $ /bin/bash /tmp/jenkins2126494143885055877.sh
OOZIE-3163
rm: cannot remove 'distro/downloads/*': No such file or directory
rm: invalid option -- '.'
Try 'rm --help' for more information.

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0100 
 15740  1531  10043   1868 52 --:--:-- --:--:-- --:--:--  1867
{"self":"https://issues.apache.org/jira/rest/api/2/issue/13132441/comment/16354037","id":"16354037","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","emailAddress":"blackhole
 at hadoop dot apache dot 
org","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop
 QA","active":true,"timeZone":"Etc/UTC"},"body":"PreCommit-OOZIE-Build 
started\n","updateAuthor":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","emailAddress":"blackhole
 at hadoop dot apache dot 
org","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop
 
QA","active":true,"timeZone":"Etc/UTC"},"created":"2018-02-06T15:46:50.259+","updated":"2018-02-06T15:46:50.259+"}
Testing JIRA OOZIE-3163

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0100 
 150k0  150k0 0   169k  0 --:--:-- --:--:-- --:--:--  169k
OOZIE-3163 is not "Patch Available".  Exiting.

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0100 
 16340  1561  10073   1390 65  0:00:01  0:00:01 --:--:--  1391
{"self":"https://issues.apache.org/jira/rest/api/2/issue/13132441/comment/16354038","id":"16354038","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","emailAddress":"blackhole
 at hadoop dot apache dot

Failed: OOZIE-3121 PreCommit Build #378

2018-02-06 Thread Apache Jenkins Server

Jira: https://issues.apache.org/jira/browse/OOZIE-3121
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/378/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.69 MB...]
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/sqoop].
 -1 There are [9] new bugs found below threshold in [sharelib/oozie] that must 
be fixed, listing only the first [5] ones.
 You can find the FindBugs diff here (look for the red and orange ones): 
sharelib/oozie/findbugs-new.html
 The top [5] most important FindBugs errors are:
 At LauncherMain.java:[line 473]: 
java/io/File.init(Ljava/lang/String;)V reads a file whose location 
might be specified by user input
 At LauncherMain.java:[line 463]: 
java/io/File.init(Ljava/lang/String;)V reads a file whose location 
might be specified by user input
 At LauncherMain.java:[line 140]: 
java/io/File.init(Ljava/lang/String;)V reads a file whose location 
might be specified by user input
 java/io/FileReader.init(Ljava/lang/String;)V reads a file whose 
location might be specified by user input: At LauncherMain.java:[line 176]
 At LauncherMain.java:[line 176]: At LauncherMain.java:[line 144]
 -1 There are [3] new bugs found below threshold in [sharelib/pig] that must be 
fixed.
 You can find the FindBugs diff here (look for the red and orange ones): 
sharelib/pig/findbugs-new.html
 The most important FindBugs errors are:
 At PigMain.java:[line 258]: java/io/File.init(Ljava/lang/String;)V 
reads a file whose location might be specified by user input
 At PigMain.java:[line 141]: At PigMain.java:[line 131]
 At PigMain.java:[line 245]: At PigMain.java:[line 199]
 At PigMain.java:[line 137]: java/io/File.init(Ljava/lang/String;)V 
reads a file whose location might be specified by user input
 java/io/File.init(Ljava/lang/String;)V reads a file whose location 
might be specified by user input: At PigMain.java:[line 131]
 -1 There are [2] new bugs found below threshold in [sharelib/hive] that must 
be fixed.
 You can find the FindBugs diff here (look for the red and orange ones): 
sharelib/hive/findbugs-new.html
 The most important FindBugs errors are:
 At HiveMain.java:[line 317]: 
java/io/FileInputStream.init(Ljava/lang/String;)V reads a file whose 
location might be specified by user input
 At HiveMain.java:[line 226]: At HiveMain.java:[line 207]
 At HiveMain.java:[line 210]: java/io/File.init(Ljava/lang/String;)V 
reads a file whose location might be specified by user input
 At HiveMain.java:[line 207]
 -1 There are [2] new bugs found below threshold in [sharelib/spark] that must 
be fixed.
 You can find the FindBugs diff here (look for the red and orange ones): 
sharelib/spark/findbugs-new.html
 The most important FindBugs errors are:
 At SparkMain.java:[line 128]: 
java/io/File.init(Ljava/io/File;Ljava/lang/String;)V reads a file whose 
location might be specified by user input
 java/io/File.init(Ljava/lang/String;)V reads a file whose location 
might be specified by user input: At SparkMain.java:[line 128]
 At SparkMain.java:[line 161]: At SparkMain.java:[line 168]
 At SparkMain.java:[line 167]
 -1 There are [8] new bugs found below threshold in [client] that must be 
fixed, listing only the first [5] ones.
 You can find the FindBugs diff here (look for the red and orange ones): 
client/findbugs-new.html
 The top [5] most important FindBugs errors are:
 At OozieCLI.java:[line 841]: java/io/File.init(Ljava/lang/String;)V 
reads a file whose location might be specified by user input
 java/io/FileInputStream.init(Ljava/lang/String;)V reads a file whose 
location might be specified by user input: At OozieCLI.java:[line 839]
 At OozieCLI.java:[line 839]: At OozieCLI.java:[line 849]
 At OozieCLI.java:[line 876]: java/io/File.init(Ljava/lang/String;)V 
reads a file whose location might be specified by user input
 java/io/File.init(Ljava/lang/String;)V reads a file whose location 
might be specified by user input: At OozieCLI.java:[line 871]
 -1 There are [2] new bugs found below threshold in [examples] that must be 
fixed.
 You can find the FindBugs diff here (look for the red and orange ones): 
examples/findbugs-new.html
 The most important FindBugs errors are:
 At LocalOozieExample.java:[line 47]: 
java/io/File.init(Ljava/lang/String;)V reads a file whose location 
might be specified by user input
 java/io/FileInputStream.init(Ljava/lang/String;)V reads a file whose 
location might be specified by user input: At LocalOozieExample.java:[line 35]
 At LocalOozieExample.java:[line 35]: At LocalOozieExample.java:[line 72]
 +1 There are no new bugs found in [docs].
 -1 There are [1] new bugs found below threshold in [server] that must be fixed.
 You can find the FindBugs diff here (look for the red and

[jira] [Commented] (OOZIE-3134) Potential inconsistency between the in-memory SLA map and the Oozie database

2018-02-06 Thread Rohini Palaniswamy (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354017#comment-16354017
 ] 

Rohini Palaniswamy commented on OOZIE-3134:
---

There is more to this issue. Please refer OOZIE-1980. 

There are two issues to be addressed.

1) Include the SLA inserts as part of the coord action table inserts and make 
them one transaction which JPA can rollback on database failure. Currently they 
are two separate JPA transactions.
2) Remove the SLA entry from the in memory map on JPA rollback.

> Potential inconsistency between the in-memory SLA map and the Oozie database
> 
>
> Key: OOZIE-3134
> URL: https://issues.apache.org/jira/browse/OOZIE-3134
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: OOZIE-3134-001.patch, OOZIE-3134-002.patch
>
>
> Upon {{SLACalculatorMemory.addRegistration}}, Oozie puts an entry into an 
> in-memory concurrent hashmap ("slaMap"), and later [persists 
> it|https://github.com/apache/oozie/blob/50f4b5984832941f1341586e43fd832c293b3275/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L380]
>  in the Oozie database. 
> However, if there is a failure during the database operation, a 
> {{JPAExecutorException}} is thrown, and the entry [is not removed from the 
> SLA map| 
> https://github.com/apache/oozie/blob/50f4b5984832941f1341586e43fd832c293b3275/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L393].
> It may introduce inconsistency between the Oozie database and the SLA map.
> To prevent this, a rollback mechanism (with proper logging) should be 
> implemented.  It would also make sense to do more sanity/consistency check in 
> the Oozie server.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3134) Potential inconsistency between the in-memory SLA map and the Oozie database

2018-02-06 Thread Julia Kinga Marton (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julia Kinga Marton updated OOZIE-3134:
--
Attachment: OOZIE-3134-002.patch

> Potential inconsistency between the in-memory SLA map and the Oozie database
> 
>
> Key: OOZIE-3134
> URL: https://issues.apache.org/jira/browse/OOZIE-3134
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: OOZIE-3134-001.patch, OOZIE-3134-002.patch
>
>
> Upon {{SLACalculatorMemory.addRegistration}}, Oozie puts an entry into an 
> in-memory concurrent hashmap ("slaMap"), and later [persists 
> it|https://github.com/apache/oozie/blob/50f4b5984832941f1341586e43fd832c293b3275/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L380]
>  in the Oozie database. 
> However, if there is a failure during the database operation, a 
> {{JPAExecutorException}} is thrown, and the entry [is not removed from the 
> SLA map| 
> https://github.com/apache/oozie/blob/50f4b5984832941f1341586e43fd832c293b3275/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L393].
> It may introduce inconsistency between the Oozie database and the SLA map.
> To prevent this, a rollback mechanism (with proper logging) should be 
> implemented.  It would also make sense to do more sanity/consistency check in 
> the Oozie server.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65481: OOZIE-3134 - Potential inconsistency between the in-memory SLA map and the Oozie database

2018-02-06 Thread Kinga Marton via Review Board



> On Feb. 2, 2018, 2:09 p.m., András Piros wrote:
> > core/src/main/java/org/apache/oozie/util/db/FailingHSQLDBDriverWrapper.java
> > Lines 36-39 (patched)
> > 
> >
> > Usage of System#getProperty(String, String) would be much cleaner:
> > 
> > 
> > https://docs.oracle.com/javase/8/docs/api/java/lang/System.html#getProperty-java.lang.String-java.lang.String-

With the AlwaysFailingdriverMapper, I will not need this system property anymore


> On Feb. 2, 2018, 2:09 p.m., András Piros wrote:
> > core/src/main/java/org/apache/oozie/util/db/RuntimeExceptionInjector.java
> > Lines 46-53 (patched)
> > 
> >
> > Would extract method refreshFailurePercent(), as well use 
> > https://docs.oracle.com/javase/8/docs/api/java/lang/System.html#getProperty-java.lang.String-java.lang.String-
> > 
> > But even better, I'd rather create three of this:
> > - FailingHSQLDBDriverWrapper
> > - NeverFailingHSQLDBDriverWrapper
> > - AlwaysFailingHSQLDBDriverWrapper
> > 
> > and reset this Configuration property for each of the unit tests.

I have created the AlwaysFailingHSQLDBDriverWrapper, but I skipped the creation 
of NeverFailingHSQLDBDriverWrapper, because the defaut jdbdDriver is actually a 
"Never failing" one.


- Kinga


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65481/#review196707
---


On Feb. 6, 2018, 2:43 p.m., Kinga Marton wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65481/
> ---
> 
> (Updated Feb. 6, 2018, 2:43 p.m.)
> 
> 
> Review request for oozie, András Piros and Attila Sasvari.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Upon SLACalculatorMemory.addRegistration, Oozie puts an entry into an 
> in-memory concurrent hashmap ("slaMap"), and later persists it in the Oozie 
> database.
> However, if there is a failure during the database operation, a 
> JPAExecutorException is thrown, and the entry is not removed from the SLA map.
> It may introduce inconsistency between the Oozie database and the SLA map.
> To prevent this, a rollback mechanism (with proper logging) should be 
> implemented. It would also make sense to do more sanity/consistency check in 
> the Oozie server.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java ef019e73 
>   
> core/src/main/java/org/apache/oozie/util/db/AlwaysFailingHSQLDriverMapper.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java 
> 0e310253 
>   core/src/main/java/org/apache/oozie/util/db/FailingHSQLDBDriverWrapper.java 
> fe9f08b1 
>   core/src/main/java/org/apache/oozie/util/db/FailingMySQLDriverWrapper.java 
> f0e2b181 
>   core/src/main/java/org/apache/oozie/util/db/ThreadVariables.java 
> PRE-CREATION 
>   core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java 
> ee906f45 
> 
> 
> Diff: https://reviews.apache.org/r/65481/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Kinga Marton
> 
>

Re: Review Request 65481: OOZIE-3134 - Potential inconsistency between the in-memory SLA map and the Oozie database

2018-02-06 Thread Kinga Marton via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65481/
---

(Updated Feb. 6, 2018, 2:43 p.m.)


Review request for oozie, András Piros and Attila Sasvari.


Repository: oozie-git


Description
---

Upon SLACalculatorMemory.addRegistration, Oozie puts an entry into an in-memory 
concurrent hashmap ("slaMap"), and later persists it in the Oozie database.
However, if there is a failure during the database operation, a 
JPAExecutorException is thrown, and the entry is not removed from the SLA map.
It may introduce inconsistency between the Oozie database and the SLA map.
To prevent this, a rollback mechanism (with proper logging) should be 
implemented. It would also make sense to do more sanity/consistency check in 
the Oozie server.


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java ef019e73 
  
core/src/main/java/org/apache/oozie/util/db/AlwaysFailingHSQLDriverMapper.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java 
0e310253 
  core/src/main/java/org/apache/oozie/util/db/FailingHSQLDBDriverWrapper.java 
fe9f08b1 
  core/src/main/java/org/apache/oozie/util/db/FailingMySQLDriverWrapper.java 
f0e2b181 
  core/src/main/java/org/apache/oozie/util/db/ThreadVariables.java PRE-CREATION 
  core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java ee906f45 


Diff: https://reviews.apache.org/r/65481/diff/2/

Changes: https://reviews.apache.org/r/65481/diff/1-2/


Testing
---


Thanks,

Kinga Marton

[jira] [Commented] (OOZIE-3121) bump all maven plugins to latest versions

2018-02-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353911#comment-16353911
 ] 

Hadoop QA commented on OOZIE-3121:
--

PreCommit-OOZIE-Build started


> bump all maven plugins to latest versions
> -
>
> Key: OOZIE-3121
> URL: https://issues.apache.org/jira/browse/OOZIE-3121
> Project: Oozie
>  Issue Type: Improvement
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: OOZIE-3121-02.patch, OOZIE-3121.patch
>
>
> bump all maven plugins to latest



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3173) Coordinator job with frequency using cron syntax creates only one action in catchup mode

2018-02-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353910#comment-16353910
 ] 

Hadoop QA commented on OOZIE-3173:
--


Testing JIRA OOZIE-3173

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 1 line(s) longer than 132 
characters
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2092
.Tests failed: 1
.Tests errors: 0

.The patch failed the following testcases:

testHiveAction(org.apache.oozie.action.hadoop.TestHiveActionExecutor)

.Tests failing with errors:


.{color:orange}Tests failed at first run:{color}
TestJavaActionExecutor#testCredentialsSkip
TestCoordActionsKillXCommand#testActionKillCommandActionNumbers
TestStatusTransitService#testBundleStatusTransitWithLock
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/377/



> Coordinator job with frequency using cron syntax creates only one action in 
> catchup mode
> 
>
> Key: OOZIE-3173
> URL: https://issues.apache.org/jira/browse/OOZIE-3173
> Project: Oozie
>  Issue Type: Bug
>  Components: coordinator
>Affects Versions: 5.0.0b1
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: OOZIE-3173.001.patch
>
>
> Using a coordinator with cron based {{frequency}}, when in catchup mode (job 
> is still in {{PREP}}), {{startTime}} lies in the past, next materialize time 
> lies also in the past), would create only one action, independent of 
> {{throttle}} value.
> As many actions should be created in catchup mode, as many times 
> {{frequency}} would fit in between {{startTime}}, {{Math.min(Math.min(now, 
> endTime), startTime + frequency * throttle)}}.
> Some examples, {{frequency="0 * * * *"}}:
> * when between {{startTime}} and {{endTime}} there are 10 hours, both lie in 
> the past, and {{throttle}} is {{1}} -> one action should be created
> * when between {{startTime}} and {{endTime}} there are 10 hours, both lie in 
> the past, and {{throttle}} is {{3}} -> three actions should be created
> * when between {{startTime}} and {{endTime}} there are

Failed: OOZIE-3173 PreCommit Build #377

2018-02-06 Thread Apache Jenkins Server

Jira: https://issues.apache.org/jira/browse/OOZIE-3173
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/377/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.58 MB...]
[DEBUG] There are no new bugs found in [docs].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [server].
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2184 bytes
[TRACE] Full summary file size is 1248 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO


Testing JIRA OOZIE-3173

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
-1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
-1 the patch contains 1 line(s) longer than 132 characters
+1 the patch adds/modifies 1 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 100 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [server].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS
Tests run: 2092
Tests failed: 1
Tests errors: 0

The patch failed the following testcases:

testHiveAction(org.apache.oozie.action.hadoop.TestHiveActionExecutor)

Tests failing with errors:


Tests failed at first run:
TestJavaActionExecutor#testCredentialsSkip
TestCoordActionsKillXCommand#testActionKillCommandActionNumbers
TestStatusTransitService#testBundleStatusTransitWithLock
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/377/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100  33970 0  100  3397  0   4042 --:--:-- --:--:-- --:--:--  
4039{"self":"https://issues.apache.org/jira/rest/api/2/issue/13136381/comment/16353910","id":"16353910","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","emailAddress":"blackhole
 at hadoop dot apache dot 
org","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop
 QA","active":true,"timeZone":"Etc/UTC"},"body":"\nTesting JIRA 
OOZIE-3173\n\nCleaning local git 
workspace\n\n\n\n{color:green}+1 
PATCH_APPLIES{color}\n{color:green}+1 CLEAN{color}\n{color:red}-1 
RAW_PATCH_ANALYSIS{color}\n.{color:green}+1{color} the patch does not 
introduce any @author tags\n.{color:green}+1{color} the patch does not 
introduce any tabs\n.{color:green}+1{color} the patch does not introduce 
any trailing

[jira] [Commented] (OOZIE-3163) Improve documentation rendering: use fluido skin and better config

2018-02-06 Thread Peter Cseh (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353906#comment-16353906
 ] 

Peter Cseh commented on OOZIE-3163:
---

I don't think I'll create a new RC to include this patch only, but f there is a 
need for a new RC for 4.3.1 I'll still try to generate the docs for 4.3.1 using 
the fluido skin and check whit other PMC members if that's ok

> Improve documentation rendering: use fluido skin and better config
> --
>
> Key: OOZIE-3163
> URL: https://issues.apache.org/jira/browse/OOZIE-3163
> Project: Oozie
>  Issue Type: Task
>  Components: docs
>Affects Versions: 4.3.0
>Reporter: Hervé Boutemy
>Assignee: Hervé Boutemy
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: 34.patch, oozie-docs-coordinator-fs-fluido.png, 
> oozie-docs-coordinator-fs-old.png, oozie-docs-index-fluido.png, 
> oozie-docs-index-old.png
>
>
> Current output is really ugly: as Maven maintainer, I can't let Oozie with 
> such awful result :)
> Even with old maven-site-plugin 2.0-beta-6 (we'll see later how to upgrade), 
> using Fluido skin would improve output a lot
> Pull Request coming



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3163) Improve documentation rendering: use fluido skin and better config

2018-02-06 Thread Andras Piros (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353900#comment-16353900
 ] 

Andras Piros commented on OOZIE-3163:
-

Committed to {{master}}. The actual cherry-pick to {{branch-4.3}} I leave up to 
[~gezapeti]'s decision, he's the release manager for {{4.3.1}}.

> Improve documentation rendering: use fluido skin and better config
> --
>
> Key: OOZIE-3163
> URL: https://issues.apache.org/jira/browse/OOZIE-3163
> Project: Oozie
>  Issue Type: Task
>  Components: docs
>Affects Versions: 4.3.0
>Reporter: Hervé Boutemy
>Assignee: Hervé Boutemy
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: 34.patch, oozie-docs-coordinator-fs-fluido.png, 
> oozie-docs-coordinator-fs-old.png, oozie-docs-index-fluido.png, 
> oozie-docs-index-old.png
>
>
> Current output is really ugly: as Maven maintainer, I can't let Oozie with 
> such awful result :)
> Even with old maven-site-plugin 2.0-beta-6 (we'll see later how to upgrade), 
> using Fluido skin would improve output a lot
> Pull Request coming



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2847) Oozie Ha timing issue

2018-02-06 Thread Peter Cseh (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353895#comment-16353895
 ] 

Peter Cseh commented on OOZIE-2847:
---

4.3.1 rc4 is already out, so this issue won't be part of that.
If there will be an rc5 I'll make sure this will be part of it. If not, it 
still can be part of 4.3.2 if there will be one.



> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Affects Versions: 4.3.0, 5.0.0b1, 5.0.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Fix For: trunk, 5.0.0
>
> Attachments: OOZIE-2847-4.3.patch, OOZIE-2847-5.0.patch, 
> OOZIE-2847.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on 
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a 
> second try, during recovery. On the second try, Oozie is trying to see if the 
> previously written job id on hdfs matches the current job id. In most 
> occasion, this will match. However, in the event when Oozie launcher is 
> killed right in the middle when Oozie is in the process of writing id in the 
> file, the Oozie file in hdfs is created, but the id has yet to be written to 
> the file. During the next recovery, Oozie will mistakenly think the id exists 
> in the file while the file is actually empty, therefore throwing this 
> exception: 
> {noformat}
> 2015-07-10 
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL  
>  : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code   
>  : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error 
> Message : Hadoop job Id mismatch, action file 
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
>  declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID  
>  : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External 
> Status   : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name 
>  : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries  
>  : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI  
>  : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type 
>  : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started  
>  : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status   
>  : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
>  : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External 
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External 
> ChildIDs : null
> 2015-07-10 
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main] 
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file 
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at 
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job 
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
>   at 
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
>   at 
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1039)
>   at 
>

[jira] [Updated] (OOZIE-3163) Improve documentation rendering: use fluido skin and better config

2018-02-06 Thread Andras Piros (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-3163:

Summary: Improve documentation rendering: use fluido skin and better config 
 (was: improve documentation rendering: use fluido skin and better config)

> Improve documentation rendering: use fluido skin and better config
> --
>
> Key: OOZIE-3163
> URL: https://issues.apache.org/jira/browse/OOZIE-3163
> Project: Oozie
>  Issue Type: Task
>  Components: docs
>Affects Versions: 4.3.0
>Reporter: Hervé Boutemy
>Assignee: Hervé Boutemy
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: 34.patch, oozie-docs-coordinator-fs-fluido.png, 
> oozie-docs-coordinator-fs-old.png, oozie-docs-index-fluido.png, 
> oozie-docs-index-old.png
>
>
> Current output is really ugly: as Maven maintainer, I can't let Oozie with 
> such awful result :)
> Even with old maven-site-plugin 2.0-beta-6 (we'll see later how to upgrade), 
> using Fluido skin would improve output a lot
> Pull Request coming



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3163) improve documentation rendering: use fluido skin and better config

2018-02-06 Thread Andras Piros (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-3163:

Attachment: oozie-docs-index-fluido.png
oozie-docs-index-old.png
oozie-docs-coordinator-fs-fluido.png
oozie-docs-coordinator-fs-old.png

> improve documentation rendering: use fluido skin and better config
> --
>
> Key: OOZIE-3163
> URL: https://issues.apache.org/jira/browse/OOZIE-3163
> Project: Oozie
>  Issue Type: Task
>  Components: docs
>Affects Versions: 4.3.0
>Reporter: Hervé Boutemy
>Assignee: Hervé Boutemy
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: 34.patch, oozie-docs-coordinator-fs-fluido.png, 
> oozie-docs-coordinator-fs-old.png, oozie-docs-index-fluido.png, 
> oozie-docs-index-old.png
>
>
> Current output is really ugly: as Maven maintainer, I can't let Oozie with 
> such awful result :)
> Even with old maven-site-plugin 2.0-beta-6 (we'll see later how to upgrade), 
> using Fluido skin would improve output a lot
> Pull Request coming



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3163) improve documentation rendering: use fluido skin and better config

2018-02-06 Thread Andras Piros (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353889#comment-16353889
 ] 

Andras Piros commented on OOZIE-3163:
-

[~hboutemy] I've generated the docs using the patch, looks OK to me. +1

> improve documentation rendering: use fluido skin and better config
> --
>
> Key: OOZIE-3163
> URL: https://issues.apache.org/jira/browse/OOZIE-3163
> Project: Oozie
>  Issue Type: Task
>  Components: docs
>Affects Versions: 4.3.0
>Reporter: Hervé Boutemy
>Assignee: Hervé Boutemy
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: 34.patch
>
>
> Current output is really ugly: as Maven maintainer, I can't let Oozie with 
> such awful result :)
> Even with old maven-site-plugin 2.0-beta-6 (we'll see later how to upgrade), 
> using Fluido skin would improve output a lot
> Pull Request coming



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2847) Oozie Ha timing issue

2018-02-06 Thread Denes Bodo (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353884#comment-16353884
 ] 

Denes Bodo commented on OOZIE-2847:
---

Thanks for your help, [~asasvari]!

> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Affects Versions: 4.3.0, 5.0.0b1, 5.0.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Fix For: trunk, 5.0.0
>
> Attachments: OOZIE-2847-4.3.patch, OOZIE-2847-5.0.patch, 
> OOZIE-2847.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on 
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a 
> second try, during recovery. On the second try, Oozie is trying to see if the 
> previously written job id on hdfs matches the current job id. In most 
> occasion, this will match. However, in the event when Oozie launcher is 
> killed right in the middle when Oozie is in the process of writing id in the 
> file, the Oozie file in hdfs is created, but the id has yet to be written to 
> the file. During the next recovery, Oozie will mistakenly think the id exists 
> in the file while the file is actually empty, therefore throwing this 
> exception: 
> {noformat}
> 2015-07-10 
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL  
>  : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code   
>  : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error 
> Message : Hadoop job Id mismatch, action file 
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
>  declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID  
>  : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External 
> Status   : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name 
>  : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries  
>  : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI  
>  : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type 
>  : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started  
>  : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status   
>  : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
>  : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External 
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External 
> ChildIDs : null
> 2015-07-10 
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main] 
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file 
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at 
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job 
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
>   at 
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
>   at 
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1039)
>   at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1519)
>   at

[VOTE] Release Oozie 4.3.1(candidate 4)

2018-02-06 Thread Gézapeti Cseh

Hi,

I have created a build for Oozie 4.3.1, candidate 4.

It includes all the changes in rc3 plus OOZIE-2888 to fix sharelib
creation with Hadoop 2.7.5. and possibly other newer versions.

Keys to verify the signature of the release artifact are available at
  http://www.apache.org/dist/oozie/KEYS

Please download, test, and try it out:

http://people.apache.org/~gezapeti/oozie-4.3.1-rc4/

The release, md5 signature, gpg signature, and rat report can be found at
the above address.

Vote closes on 02/12/2018 11:59 PM CET.

[jira] [Updated] (OOZIE-2847) Oozie Ha timing issue

2018-02-06 Thread Attila Sasvari (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2847:
--
Affects Version/s: 5.0.0
   5.0.0b1

> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Affects Versions: 4.3.0, 5.0.0b1, 5.0.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Fix For: trunk, 5.0.0
>
> Attachments: OOZIE-2847-4.3.patch, OOZIE-2847-5.0.patch, 
> OOZIE-2847.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on 
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a 
> second try, during recovery. On the second try, Oozie is trying to see if the 
> previously written job id on hdfs matches the current job id. In most 
> occasion, this will match. However, in the event when Oozie launcher is 
> killed right in the middle when Oozie is in the process of writing id in the 
> file, the Oozie file in hdfs is created, but the id has yet to be written to 
> the file. During the next recovery, Oozie will mistakenly think the id exists 
> in the file while the file is actually empty, therefore throwing this 
> exception: 
> {noformat}
> 2015-07-10 
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL  
>  : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code   
>  : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error 
> Message : Hadoop job Id mismatch, action file 
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
>  declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID  
>  : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External 
> Status   : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name 
>  : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries  
>  : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI  
>  : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type 
>  : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started  
>  : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status   
>  : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
>  : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External 
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External 
> ChildIDs : null
> 2015-07-10 
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main] 
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file 
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at 
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job 
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
>   at 
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
>   at 
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1039)
>   at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1519)
>   at java.security.AccessController.doPrivileged(Native

[jira] [Commented] (OOZIE-2847) Oozie Ha timing issue

2018-02-06 Thread Attila Sasvari (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353807#comment-16353807
 ] 

Attila Sasvari commented on OOZIE-2847:
---

I decided to only push it to master. [~gezapeti] is the release manager of 
4.3.1, that is in progress. He will commit OOZIE-2847-4.3.patch to branch-4.3 
later and update fix version.

> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Fix For: trunk, 5.0.0
>
> Attachments: OOZIE-2847-4.3.patch, OOZIE-2847-5.0.patch, 
> OOZIE-2847.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on 
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a 
> second try, during recovery. On the second try, Oozie is trying to see if the 
> previously written job id on hdfs matches the current job id. In most 
> occasion, this will match. However, in the event when Oozie launcher is 
> killed right in the middle when Oozie is in the process of writing id in the 
> file, the Oozie file in hdfs is created, but the id has yet to be written to 
> the file. During the next recovery, Oozie will mistakenly think the id exists 
> in the file while the file is actually empty, therefore throwing this 
> exception: 
> {noformat}
> 2015-07-10 
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL  
>  : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code   
>  : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error 
> Message : Hadoop job Id mismatch, action file 
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
>  declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID  
>  : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External 
> Status   : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name 
>  : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries  
>  : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI  
>  : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type 
>  : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started  
>  : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status   
>  : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
>  : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External 
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External 
> ChildIDs : null
> 2015-07-10 
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main] 
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file 
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at 
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job 
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
>   at 
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
>   at 
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1039)
>   at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>

[jira] [Updated] (OOZIE-2847) Oozie Ha timing issue

2018-02-06 Thread Attila Sasvari (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2847:
--
Fix Version/s: (was: 4.3.1)

> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Fix For: trunk, 5.0.0
>
> Attachments: OOZIE-2847-4.3.patch, OOZIE-2847-5.0.patch, 
> OOZIE-2847.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on 
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a 
> second try, during recovery. On the second try, Oozie is trying to see if the 
> previously written job id on hdfs matches the current job id. In most 
> occasion, this will match. However, in the event when Oozie launcher is 
> killed right in the middle when Oozie is in the process of writing id in the 
> file, the Oozie file in hdfs is created, but the id has yet to be written to 
> the file. During the next recovery, Oozie will mistakenly think the id exists 
> in the file while the file is actually empty, therefore throwing this 
> exception: 
> {noformat}
> 2015-07-10 
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL  
>  : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code   
>  : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error 
> Message : Hadoop job Id mismatch, action file 
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
>  declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID  
>  : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External 
> Status   : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name 
>  : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries  
>  : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI  
>  : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type 
>  : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started  
>  : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status   
>  : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
>  : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External 
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External 
> ChildIDs : null
> 2015-07-10 
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main] 
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file 
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at 
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job 
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
>   at 
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
>   at 
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1039)
>   at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1519)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at

[jira] [Commented] (OOZIE-3173) Coordinator job with frequency using cron syntax creates only one action in catchup mode

2018-02-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353796#comment-16353796
 ] 

Hadoop QA commented on OOZIE-3173:
--

PreCommit-OOZIE-Build started


> Coordinator job with frequency using cron syntax creates only one action in 
> catchup mode
> 
>
> Key: OOZIE-3173
> URL: https://issues.apache.org/jira/browse/OOZIE-3173
> Project: Oozie
>  Issue Type: Bug
>  Components: coordinator
>Affects Versions: 5.0.0b1
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: OOZIE-3173.001.patch
>
>
> Using a coordinator with cron based {{frequency}}, when in catchup mode (job 
> is still in {{PREP}}), {{startTime}} lies in the past, next materialize time 
> lies also in the past), would create only one action, independent of 
> {{throttle}} value.
> As many actions should be created in catchup mode, as many times 
> {{frequency}} would fit in between {{startTime}}, {{Math.min(Math.min(now, 
> endTime), startTime + frequency * throttle)}}.
> Some examples, {{frequency="0 * * * *"}}:
> * when between {{startTime}} and {{endTime}} there are 10 hours, both lie in 
> the past, and {{throttle}} is {{1}} -> one action should be created
> * when between {{startTime}} and {{endTime}} there are 10 hours, both lie in 
> the past, and {{throttle}} is {{3}} -> three actions should be created
> * when between {{startTime}} and {{endTime}} there are 10 hours, both lie in 
> the past, and {{throttle}} is {{15}} -> ten actions should be created



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2847) Oozie Ha timing issue

2018-02-06 Thread Attila Sasvari (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353797#comment-16353797
 ] 

Attila Sasvari commented on OOZIE-2847:
---

[~dionusos] thanks for your contribution, +1. I will commit the patches to 
branch-4.3 and master soon.

> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Fix For: trunk, 5.0.0, 4.3.1
>
> Attachments: OOZIE-2847-4.3.patch, OOZIE-2847-5.0.patch, 
> OOZIE-2847.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on 
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a 
> second try, during recovery. On the second try, Oozie is trying to see if the 
> previously written job id on hdfs matches the current job id. In most 
> occasion, this will match. However, in the event when Oozie launcher is 
> killed right in the middle when Oozie is in the process of writing id in the 
> file, the Oozie file in hdfs is created, but the id has yet to be written to 
> the file. During the next recovery, Oozie will mistakenly think the id exists 
> in the file while the file is actually empty, therefore throwing this 
> exception: 
> {noformat}
> 2015-07-10 
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL  
>  : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code   
>  : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error 
> Message : Hadoop job Id mismatch, action file 
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
>  declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID  
>  : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External 
> Status   : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name 
>  : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries  
>  : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI  
>  : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type 
>  : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started  
>  : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status   
>  : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
>  : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External 
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External 
> ChildIDs : null
> 2015-07-10 
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main] 
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file 
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at 
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job 
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
>   at 
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
>   at 
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1039)
>   at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>   at 
>

[jira] [Created] (OOZIE-3175) PurgeService should detect and delete orphaned jobs/actions

2018-02-06 Thread Attila Sasvari (JIRA)

Attila Sasvari created OOZIE-3175:
-

 Summary:  PurgeService should detect and delete orphaned 
jobs/actions
 Key: OOZIE-3175
 URL: https://issues.apache.org/jira/browse/OOZIE-3175
 Project: Oozie
  Issue Type: Improvement
  Components: core
Affects Versions: trunk
Reporter: Attila Sasvari


Improvement idea: the purge service could look for orphaned jobs/actions and 
delete those. It would help the cases where the user went in and only partially 
removed things. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OOZIE-3174) Allow users to retain SQL file generated by OozieDBCLI

2018-02-06 Thread Attila Sasvari (JIRA)

Attila Sasvari created OOZIE-3174:
-

 Summary: Allow users to retain SQL file generated by OozieDBCLI
 Key: OOZIE-3174
 URL: https://issues.apache.org/jira/browse/OOZIE-3174
 Project: Oozie
  Issue Type: Improvement
  Components: tools
Affects Versions: trunk
Reporter: Attila Sasvari


OozieDBCLI writes SQL statements to a temporary file. It is deleted without 
question at the end of the execution (see OOZIE-2727). We should allow the user 
to decide if they want to delete it or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-1717) Add indexes to speed up db queries

2018-02-06 Thread Attila Sasvari (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353729#comment-16353729
 ] 

Attila Sasvari commented on OOZIE-1717:
---

TestHiveActionExecutor locally passes, committed to master

> Add indexes to speed up db queries
> --
>
> Key: OOZIE-1717
> URL: https://issues.apache.org/jira/browse/OOZIE-1717
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Attila Sasvari
>Priority: Minor
> Fix For: trunk
>
> Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, 
> OOZIE-1717-003.patch, OOZIE-1717-004.patch, OOZIE-1717-01.patch, 
> amend-OOZIE-1717-00.patch, amend-OOZIE-1717-01.patch
>
>
> User_name is one of the frequently used filter. Adding index should avoid  
> full db scan.
> Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-2847) Oozie Ha timing issue

2018-02-06 Thread Attila Sasvari (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2847:
--
Fix Version/s: (was: 5.0.0b1)
   5.0.0
   trunk

> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Fix For: trunk, 5.0.0, 4.3.1
>
> Attachments: OOZIE-2847-4.3.patch, OOZIE-2847-5.0.patch, 
> OOZIE-2847.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on 
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a 
> second try, during recovery. On the second try, Oozie is trying to see if the 
> previously written job id on hdfs matches the current job id. In most 
> occasion, this will match. However, in the event when Oozie launcher is 
> killed right in the middle when Oozie is in the process of writing id in the 
> file, the Oozie file in hdfs is created, but the id has yet to be written to 
> the file. During the next recovery, Oozie will mistakenly think the id exists 
> in the file while the file is actually empty, therefore throwing this 
> exception: 
> {noformat}
> 2015-07-10 
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL  
>  : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code   
>  : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error 
> Message : Hadoop job Id mismatch, action file 
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
>  declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID  
>  : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External 
> Status   : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name 
>  : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries  
>  : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI  
>  : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type 
>  : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started  
>  : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status   
>  : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
>  : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External 
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External 
> ChildIDs : null
> 2015-07-10 
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main] 
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file 
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at 
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job 
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
>   at 
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
>   at 
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1039)
>   at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1519)
>   at

[jira] [Commented] (OOZIE-1717) Add indexes to speed up db queries

2018-02-06 Thread Peter Cseh (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353710#comment-16353710
 ] 

Peter Cseh commented on OOZIE-1717:
---

Test case seems unrelated
thanks for adding the tests! +1 for the amed-01 patch!

> Add indexes to speed up db queries
> --
>
> Key: OOZIE-1717
> URL: https://issues.apache.org/jira/browse/OOZIE-1717
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Attila Sasvari
>Priority: Minor
> Fix For: trunk
>
> Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, 
> OOZIE-1717-003.patch, OOZIE-1717-004.patch, OOZIE-1717-01.patch, 
> amend-OOZIE-1717-00.patch, amend-OOZIE-1717-01.patch
>
>
> User_name is one of the frequently used filter. Adding index should avoid  
> full db scan.
> Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2847) Oozie Ha timing issue

2018-02-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353704#comment-16353704
 ] 

Hadoop QA commented on OOZIE-2847:
--


Testing JIRA OOZIE-2847

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:orange}0{color} There are [1] new bugs found in total that would be nice 
to have fixed.
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:orange}0{color} There are [1] new bugs found in [sharelib/oozie] that 
would be nice to have fixed.
. You can find the FindBugs diff here: sharelib/oozie/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [tools].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2089
.{color:orange}Tests failed at first run:{color}
TestJavaActionExecutor#testCredentialsSkip
TestMetricsInstrumentation#testJMXInstrumentation
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/376/



> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Fix For: 5.0.0b1, 4.3.1
>
> Attachments: OOZIE-2847-4.3.patch, OOZIE-2847-5.0.patch, 
> OOZIE-2847.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on 
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a 
> second try, during recovery. On the second try, Oozie is trying to see if the 
> previously written job id on hdfs matches the current job id. In most 
> occasion, this will match. However, in the event when Oozie launcher is 
> killed right in the middle when Oozie is in the process of writing id in the 
> file, the Oozie file in hdfs is created, but the id has yet to be written to 
> the file. During the next recovery, Oozie will mistakenly think the id exists 
> in the file while the file is actually empty, therefore throwing this 
> exception: 
> {noformat}
> 2015-07-10 
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL

Success: OOZIE-2847 PreCommit Build #376

2018-02-06 Thread Apache Jenkins Server

Jira: https://issues.apache.org/jira/browse/OOZIE-2847
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/376/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.58 MB...]
[DEBUG] There are no new bugs found in [sharelib/streaming].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/spark].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hcatalog].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/sqoop].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hive2].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[WARN] There are [1] new bugs found in [sharelib/oozie].
[DEBUG] You can find the FindBugs diff here: sharelib/oozie/findbugs-new.html
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [tools].
[WARN] There are [1] new bugs found in total that would be nice to have fixed.
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2326 bytes
[TRACE] Full summary file size is 1388 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO


Testing JIRA OOZIE-2847

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 1 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 100 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
0 There are [1] new bugs found in total that would be nice to have fixed.
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/hive2].
 0 There are [1] new bugs found in [sharelib/oozie] that would be nice to have 
fixed.
 You can find the FindBugs diff here: sharelib/oozie/findbugs-new.html
 +1 There are no new bugs found in [tools].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2089
Tests failed at first run:
TestJavaActionExecutor#testCredentialsSkip
TestMetricsInstrumentation#testJMXInstrumentation
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch 


+1 Overall result, good!, no -1s

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/376/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0{"self":"https://issues.apache.org/jira/rest/api/2/issue/13059726/comment/16353704","id":"16353704","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","emailAddress":"blackhole
 at hadoop dot apache dot

[jira] [Commented] (OOZIE-2847) Oozie Ha timing issue

2018-02-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353601#comment-16353601
 ] 

Hadoop QA commented on OOZIE-2847:
--

PreCommit-OOZIE-Build started


> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Fix For: 5.0.0b1, 4.3.1
>
> Attachments: OOZIE-2847-4.3.patch, OOZIE-2847-5.0.patch, 
> OOZIE-2847.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on 
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a 
> second try, during recovery. On the second try, Oozie is trying to see if the 
> previously written job id on hdfs matches the current job id. In most 
> occasion, this will match. However, in the event when Oozie launcher is 
> killed right in the middle when Oozie is in the process of writing id in the 
> file, the Oozie file in hdfs is created, but the id has yet to be written to 
> the file. During the next recovery, Oozie will mistakenly think the id exists 
> in the file while the file is actually empty, therefore throwing this 
> exception: 
> {noformat}
> 2015-07-10 
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL  
>  : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code   
>  : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error 
> Message : Hadoop job Id mismatch, action file 
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
>  declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID  
>  : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External 
> Status   : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name 
>  : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries  
>  : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI  
>  : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type 
>  : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started  
>  : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status   
>  : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
>  : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External 
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External 
> ChildIDs : null
> 2015-07-10 
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main] 
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file 
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at 
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job 
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
>   at 
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
>   at 
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1039)
>   at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1519)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at

[jira] [Updated] (OOZIE-2847) Oozie Ha timing issue

2018-02-06 Thread Denes Bodo (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denes Bodo updated OOZIE-2847:
--
Attachment: OOZIE-2847-5.0.patch
OOZIE-2847-4.3.patch

> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Attachments: OOZIE-2847-4.3.patch, OOZIE-2847-5.0.patch, 
> OOZIE-2847.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on 
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a 
> second try, during recovery. On the second try, Oozie is trying to see if the 
> previously written job id on hdfs matches the current job id. In most 
> occasion, this will match. However, in the event when Oozie launcher is 
> killed right in the middle when Oozie is in the process of writing id in the 
> file, the Oozie file in hdfs is created, but the id has yet to be written to 
> the file. During the next recovery, Oozie will mistakenly think the id exists 
> in the file while the file is actually empty, therefore throwing this 
> exception: 
> {noformat}
> 2015-07-10 
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL  
>  : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code   
>  : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error 
> Message : Hadoop job Id mismatch, action file 
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
>  declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID  
>  : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External 
> Status   : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name 
>  : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries  
>  : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI  
>  : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type 
>  : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started  
>  : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status   
>  : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
>  : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External 
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External 
> ChildIDs : null
> 2015-07-10 
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main] 
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file 
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at 
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job 
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
>   at 
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
>   at 
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1039)
>   at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1519)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at

54 matches

Mail list logo