[jira] [Updated] (OOZIE-3529) Oozie not supported for s3 as filesystem
[ https://issues.apache.org/jira/browse/OOZIE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-3529: -- Fix Version/s: 5.2.0 > Oozie not supported for s3 as filesystem > > > Key: OOZIE-3529 > URL: https://issues.apache.org/jira/browse/OOZIE-3529 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 4.3.1, 5.1.0 >Reporter: Denes Bodo >Assignee: Denes Bodo >Priority: Critical > Labels: S3 > Fix For: 5.2.0 > > Attachments: OOZIE-3529.001.patch, id.pig, job.properties, > workflow.xml > > > Many customer who uses s3 file system as secondary one experiences the > following error when Oozie tries to submit the Yarn application: > {noformat} > 2019-04-29 13:02:53,770 WARN ForkedActionStartXCommand:523 - > SERVER[hwnode1.puretec.purestorage.com] USER[hrt_qa] GROUP[-] TOKEN[] > APP[demo-wf] JOB[001-190423141707256-oozie-oozi-W] > ACTION[001-190423141707256-oozie-oozi-W@streaming-node] Error starting > action [streaming-node]. ErrorType [ERROR], ErrorCode > [UnsupportedOperationException], Message [UnsupportedOperationException: > Accessing local file system is not allowed] > org.apache.oozie.action.ActionExecutorException: > UnsupportedOperationException: Accessing local file system is not allowed > at > org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446) > at > org.apache.oozie.action.hadoop.JavaActionExecutor.createLauncherConf(JavaActionExecutor.java:1092) > at > org.apache.oozie.action.hadoop.MapReduceActionExecutor.createLauncherConf(MapReduceActionExecutor.java:309) > at > org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1197) > at > org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472) > at > org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234) > at > org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:41) > at > org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:30) > at org.apache.oozie.command.XCommand.call(XCommand.java:287) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.UnsupportedOperationException: Accessing local file > system is not allowed > at > org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:48) > at > org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSystem.java:47) > at > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354) > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) > at > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477) > at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:433) > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:301) > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:378) > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForWrite(LocalDirAllocator.java:461) > at > org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocator.java:200) > at > org.apache.hadoop.fs.s3a.S3AFileSystem.createTmpFileForWrite(S3AFileSystem.java:572) > at > org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory.create(S3ADataBlocks.java:811) > at > org.apache.hadoop.fs.s3a.S3ABlockOutputStream.createBlockIfNeeded(S3ABlockOutputStream.java:190) > at > org.apache.hadoop.fs.s3a.S3ABlockOutputStream.(S3ABlockOutputStream.java:168) > at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:778) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1038) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1026) > at > org.apache.oozie.action.hadoop.LauncherMapperHelper.setupLauncherInfo(LauncherMapperHelper.java:156) > at > org.apache.oozie.action.hadoo
[jira] [Updated] (OOZIE-3484) [Java 11] Get rid of com.sun.tools. package usage
[ https://issues.apache.org/jira/browse/OOZIE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-3484: -- Fix Version/s: (was: OOZIE-3490) 5.2.0 > [Java 11] Get rid of com.sun.tools. package usage > - > > Key: OOZIE-3484 > URL: https://issues.apache.org/jira/browse/OOZIE-3484 > Project: Oozie > Issue Type: Sub-task >Reporter: Julia Kinga Marton >Priority: Major > Fix For: 5.2.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-3359) Check for Process#waitFor() usage and fix it in order to avoid indefinite waiting
[ https://issues.apache.org/jira/browse/OOZIE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-3359: -- Fix Version/s: (was: 5.2.0) 5.3.0 > Check for Process#waitFor() usage and fix it in order to avoid indefinite > waiting > - > > Key: OOZIE-3359 > URL: https://issues.apache.org/jira/browse/OOZIE-3359 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Julia Kinga Marton >Priority: Major > Fix For: 5.3.0 > > > {{Process#waitFor()}} will block until the process finishes. There are > situations where this will wait indefinitely. A similar case was in > SshActionExecutor, fixed in OOZIE-3354. > Let's check the code for other usages, and fix it. > Maybe we should check if we can use somehow the Process#waitFor(long timeout, > TimeUnit timeUnit) instead of it. The tricky part is that this method is > available only in Java 8. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-3392) Refactor ShellMain to use BufferDrainer class
[ https://issues.apache.org/jira/browse/OOZIE-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-3392: -- Fix Version/s: (was: 5.2.0) 5.3.0 > Refactor ShellMain to use BufferDrainer class > - > > Key: OOZIE-3392 > URL: https://issues.apache.org/jira/browse/OOZIE-3392 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Priority: Major > Fix For: 5.3.0 > > > OOZIE-3382 introduced a new {{BufferDrainer}} class to drain the standard > output and standard error stream of an ssh process. {{ShellMain}} should also > use this class for reading the standard output and standard error of a shell > process to avoid code duplication. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-3336) [persistence] Refactor entity classes to feature PK, FK, and UQ constraints
[ https://issues.apache.org/jira/browse/OOZIE-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-3336: -- Fix Version/s: (was: 5.2.0) 5.3.0 > [persistence] Refactor entity classes to feature PK, FK, and UQ constraints > --- > > Key: OOZIE-3336 > URL: https://issues.apache.org/jira/browse/OOZIE-3336 > Project: Oozie > Issue Type: Improvement > Components: core >Affects Versions: 5.0.0 >Reporter: Andras Piros >Priority: Major > Fix For: 5.3.0 > > > When an Oozie database grows substantial in size, let's say, over a few > hundred thousands of {{WorkflowActionBean}}, {{CoordinatorActionBean}} > instances, we face a couple of performance issues. Here is an analysis why. > Current Oozie JPA {{@Entity}} usage, and the resulting database DDL, suffers > from a couple of drawback from a performance point of view: > * {{@Id}} fields are {{String}}: > ** leaving no space for database primary key indices to work effectively > ** those values are calculated in case of {{WorkflowActionBean}}, > {{CoordinatorActionBean}}, and {{BundleActionBean}} instances > * no foreign constraint is set from {{WorkflowActionBean}} to > {{WorkflowJobBean}}, from {{CoordinatorActionBean}} to > {{CoordinatorJobBean}}, or from {{BundleActionBean}} to {{BundleJobBean}} > instances: > ** have to assess JPA queries discovering parent-child relationships by hand > ** no database indices are created, and hence, those queries that contain any > {{JOIN}} instances are slower > * no use of unique constraints whatsoever > * JPA queries are created by hand instead of relying on OpenJPA > * JPA entities are filled by hand instead of relying on OpenJPA > Following enhancements are necessary: > # keeping the existing {{String compositeId}} fields, let's break down the > contents to following new fields: > ## {{@Id long id}} - an auto-increment value that is unique across Oozie > database > ## {{long currentSequence}} - the sequence number of the current run since > last Oozie server restart. The first part of the {{compositeId}} > ## {{Timestamp serverStartupTimestamp}} - the timestamp when the Oozie server > was last started. The second part of the {{compositeId}} > ## {{String serverName}} - the third part of the {{compositeId}} > ## {{String name}} - the fourth and last part of the {{compositeId}} > ## {{compositeId}} might be calculated when an entity is loaded / persisted, > and then stored > # FK constraints: > ## {{@OneToMany}} fields where we have a list of child references inside > parent > ## {{@ManyToOne}} fields where we have a parent reference inside child > ## pay attention to {{FetchType}}, most of the times {{LAZY}} will be needed > ## the containment fields should not be {{@Transient}} anymore > # UQ constraints: > ## on {{currentSequence}} and {{serverStartupTimestamp}} > ## on {{currentSequence}} and {{name}} > # new JPQL queries: > ## to cover changed parent-child relationships > ## to get use of each disassembled part of {{originalId}} when doing e.g. > filtering > # let JPA fill entities instead performing this by hand > Following enhancements can be considered as nice-to-have: > * upgrade to an OpenJPA version that features JPA 2.1's composite indexing > capability > * see whether to have an optimistic locking field using {{@Version}} instead > of ZooKeeper based pessimistic locking would increase High Availability > characteristics > * refactor also SLA related entity classes > It's necessary to have performance benchmarks with some database types like > MySQL/MariaDB, and PostgreSQL before and after the changes for following use > cases: > * {{CoordinatorJobBean}} and {{WorkflowJobBean}} instances up to millions > * {{CoordinatorActionBean}} and {{WorkflowActionBean}} instances up to tens > of millions > * performance for JPQLs that get a list of entities > * performance of persisting a new entity > * performance of querying lists of entities based on popular / possible > filters like the ones used by {{VxJobsServlet}} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-3538) oozie-main(pom.xml) plugin maven-surefire-plugin configuration the Tag value is wrong
[ https://issues.apache.org/jira/browse/OOZIE-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-3538: -- Fix Version/s: (was: 5.2.0) 5.3.0 > oozie-main(pom.xml) plugin maven-surefire-plugin configuration the Tag > value is wrong > -- > > Key: OOZIE-3538 > URL: https://issues.apache.org/jira/browse/OOZIE-3538 > Project: Oozie > Issue Type: Bug > Components: build >Affects Versions: 5.2.0 >Reporter: duan xiong >Assignee: duan xiong >Priority: Major > Fix For: 5.3.0 > > > oozie-main(pom.xml) plugin maven-surefire-plugin configuration the Tag > value is wrong. That value should be boolean not number. > " the parameter {{threadCount}} can be used with the optional parameter > {{perCoreThreadCount=true}} (true by default)" can find this in > [http://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-1987) Fix existing findbugs warnings
[ https://issues.apache.org/jira/browse/OOZIE-1987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-1987: -- Fix Version/s: (was: 5.2.0) 5.3.0 > Fix existing findbugs warnings > -- > > Key: OOZIE-1987 > URL: https://issues.apache.org/jira/browse/OOZIE-1987 > Project: Oozie > Issue Type: Bug >Reporter: Robert Kanter >Priority: Major > Fix For: 5.3.0 > > > Now that we have OOZIE-1793, we should actually fix the warnings that > findbugs finds in the existing codebase. It may be a good idea to split this > into subtasks or multiple patches as there are a lot of warnings: over 500 in > oozie-core alone! > If we think something is ok, but findbugs complains, we should exclude it. > There's a way to create a file that lists exclusions (Hadoop does this). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-3105) testJMXInstrumentation from the org.apache.oozie.util.TestMetricsInstrumentation class is flaky
[ https://issues.apache.org/jira/browse/OOZIE-3105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-3105: -- Fix Version/s: (was: 5.2.0) 5.3.0 > testJMXInstrumentation from the > org.apache.oozie.util.TestMetricsInstrumentation class is flaky > --- > > Key: OOZIE-3105 > URL: https://issues.apache.org/jira/browse/OOZIE-3105 > Project: Oozie > Issue Type: Sub-task > Components: tests >Affects Versions: 4.3.0 > Environment: mvn: 3.5.0 > java: 1.8.0.144 >Reporter: Artem Ervits >Assignee: Peter Bacsko >Priority: Major > Fix For: 5.3.0 > > Attachments: OOZIE-3105-00.patch > > > I'm able to reproduce the test failure consistently by running test in > isolation with the following command > {code} > mvn clean > -Dtest=org.apache.oozie.util.TestMetricsInstrumentation#testJMXInstrumentation > test > {code} > the test succeeds consistently with the following > {code} > mvn clean -Dtest=org.apache.oozie.util.TestMetricsInstrumentation test > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-3428) [docs] Update Setting Up Oozie with an Alternate Tomcat documentation
[ https://issues.apache.org/jira/browse/OOZIE-3428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-3428: -- Fix Version/s: (was: 5.2.0) 5.3.0 > [docs] Update Setting Up Oozie with an Alternate Tomcat documentation > - > > Key: OOZIE-3428 > URL: https://issues.apache.org/jira/browse/OOZIE-3428 > Project: Oozie > Issue Type: Improvement > Components: docs >Affects Versions: trunk >Reporter: Andras Salamon >Priority: Major > Fix For: 5.3.0 > > > Although Oozie uses embedded Jetty now, during compilation it still creates a > {{war}} file which can be run in a different servlet container. > The [current > documentation|https://oozie.apache.org/docs/5.1.0/AG_Install.html#Setting_Up_Oozie_with_an_Alternate_Tomcat] > does not describe the full process, we need to enhance it: > * A full sample of the {{addtowar}} script usage. > * Necessary {{CATALINA_OPTS}} settings. > * A simple script to cleanup the {{webapps}}, copy the war, restart Tomcat. > * Some info about the supported Tomcat versions would be also useful. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-3198) Extend remaining job tracker references in code and documentation
[ https://issues.apache.org/jira/browse/OOZIE-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-3198: -- Fix Version/s: (was: 5.2.0) 5.3.0 > Extend remaining job tracker references in code and documentation > - > > Key: OOZIE-3198 > URL: https://issues.apache.org/jira/browse/OOZIE-3198 > Project: Oozie > Issue Type: Sub-task > Components: action, core, docs >Affects Versions: 5.0.0b1 >Reporter: Andras Piros >Priority: Major > Fix For: 5.3.0 > > > Extend the places in configuration, code and documentation where > {{jobTracker}} and {{job-tracker}} is still referenced without a proper > {{resourceManager}}, {{resource-manager}} reference that precedes it. > Example: {{oozie.service.HadoopAccessorService.jobTracker.whitelist}} in > {{AG_HadoopConfiguration.twiki}}, {{HadoopAccessorService.java}}, and > {{oozie-default.xml}}. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-3357) [core] More graceful CallableQueueService#destroy()
[ https://issues.apache.org/jira/browse/OOZIE-3357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-3357: -- Fix Version/s: (was: 5.2.0) 5.3.0 > [core] More graceful CallableQueueService#destroy() > --- > > Key: OOZIE-3357 > URL: https://issues.apache.org/jira/browse/OOZIE-3357 > Project: Oozie > Issue Type: Improvement > Components: core >Affects Versions: 5.1.0 >Reporter: Andras Piros >Priority: Major > Fix For: 5.3.0 > > > In OOZIE-3160 {{CallableQueueService}} has been refactored. It seems like not > all the {{XCallable}} elements are processed when one of the > {{ExecutorService}} instances are shut down. > Following issues to investigate: > # {{ThreadPoolExecutor#awaitTermination()}} gets stuck even if no > {{XCallable}} elements are present anymore > # await termination only conditionally, that is, when there are still some > {{XCallable}} elements in one of the {{ThreadPoolExecutor}} instances -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-3345) forkjoin validation error when multiple(>=3) actions with "ok to""error to" same action
[ https://issues.apache.org/jira/browse/OOZIE-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-3345: -- Fix Version/s: (was: 5.2.0) 5.3.0 > forkjoin validation error when multiple(>=3) actions with "ok to""error to" > same action > --- > > Key: OOZIE-3345 > URL: https://issues.apache.org/jira/browse/OOZIE-3345 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 4.3.0 >Reporter: wang jinyin >Priority: Major > Fix For: 5.3.0 > > Original Estimate: 120h > Remaining Estimate: 120h > > when multiple (n >= 3) actions with "ok to" "error to" transition to same > action, forkjoin validation will throw WorkflowException(ErrorCode.E0743) > {quote}org.apache.oozie.workflow.WorkflowException: E0743: Multiple "ok to" > transitions to the same node, [C], are not allowed > at > org.apache.oozie.workflow.lite.LiteWorkflowValidator.validateForkJoin(LiteWorkflowValidator.java:186) > at > org.apache.oozie.workflow.lite.LiteWorkflowValidator.validateForkJoin(LiteWorkflowValidator.java:267) > at > org.apache.oozie.workflow.lite.LiteWorkflowValidator.validateForkJoin(LiteWorkflowValidator.java:271) > at > org.apache.oozie.workflow.lite.LiteWorkflowValidator.validateForkJoin(LiteWorkflowValidator.java:275) > at > org.apache.oozie.workflow.lite.LiteWorkflowValidator.validateWorkflow(LiteWorkflowValidator.java:62) > at > org.apache.oozie.workflow.lite.LiteWorkflowAppParser.validateAndParse(LiteWorkflowAppParser.java:185) > at > org.apache.oozie.workflow.lite.LiteWorkflowLib.parseDef(LiteWorkflowLib.java:58) > at > org.apache.oozie.service.LiteWorkflowAppService.parseDef(LiteWorkflowAppService.java:57) > at > org.apache.oozie.service.LiteWorkflowAppService.parseDef(LiteWorkflowAppService.java:46) > at > org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:167) > {quote} > > workflow definition: > {quote} > > > > > > > > > > > > {quote} > > [reason] > When validate A's "ok to" path -> B's "ok to" path, action node C has already > been added into nodeAndDecisionParents. > When validate A's "error to" path -> B's "ok to" path, C's decisionParent == > null and topDecisionParent == null. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-2364) Remove deprecated SLAEventBean and related code
[ https://issues.apache.org/jira/browse/OOZIE-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-2364: -- Fix Version/s: (was: 5.2.0) 5.3.0 > Remove deprecated SLAEventBean and related code > --- > > Key: OOZIE-2364 > URL: https://issues.apache.org/jira/browse/OOZIE-2364 > Project: Oozie > Issue Type: Task >Reporter: Rohini Palaniswamy >Assignee: Satish Subhashrao Saley >Priority: Major > Labels: newbie > Fix For: 5.3.0 > > Attachments: OOZIE-2364-1.patch, OOZIE-2364-2.patch, > OOZIE-2364-3.patch, OOZIE-2364-4.patch > > > With new SLAService feature which uses different tables, the old SLA_EVENTS > table is not used anymore. Need to remove all code related to it and drop > that table as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-2809) Oozie HA with secure zookeeper access fails because ZKUtils#setJaasConfiguration does not replace "_HOST" in principal string
[ https://issues.apache.org/jira/browse/OOZIE-2809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-2809: -- Fix Version/s: (was: 5.2.0) 5.3.0 > Oozie HA with secure zookeeper access fails because > ZKUtils#setJaasConfiguration does not replace "_HOST" in principal string > - > > Key: OOZIE-2809 > URL: https://issues.apache.org/jira/browse/OOZIE-2809 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: trunk, 4.2.0, 4.3.0 >Reporter: Thomas Larsson >Priority: Major > Fix For: 5.3.0 > > > Setting "oozie.zookeeper.secure" = true causes the oozie server not to start > if "oozie.service.HadoopAccessorService.kerberos.principal" is defined with a > "_HOST" token instead of an actual hostname. > The symptom when this happens is that the oozie log shows something like this: > {noformat} > org.apache.oozie.service.ServiceException: E1700: Issue communicating with > ZooKeeper: KeeperErrorCode = NoAuth for /oozie/services/servers > at > org.apache.oozie.service.ZKLocksService.init(ZKLocksService.java:76) > at > org.apache.oozie.service.Services.setServiceInternal(Services.java:386) > at org.apache.oozie.service.Services.setService(Services.java:372) > at org.apache.oozie.service.Services.loadServices(Services.java:305) > at org.apache.oozie.service.Services.init(Services.java:213) > at > org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46) > at > org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210) > at > org.apache.catalina.core.StandardContext.start(StandardContext.java:4709) > at > org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:802) > at > org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779) > at > org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583) > at > org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:676) > at > org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:602) > at > org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:503) > at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322) > at > org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325) > at > org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142) > at > org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1068) > at org.apache.catalina.core.StandardHost.start(StandardHost.java:822) > at > org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1060) > at > org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463) > at > org.apache.catalina.core.StandardService.start(StandardService.java:525) > at > org.apache.catalina.core.StandardServer.start(StandardServer.java:759) > at org.apache.catalina.startup.Catalina.start(Catalina.java:595) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) > at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) > Caused by: org.apache.zookeeper.KeeperException$NoAuthException: > KeeperErrorCode = NoAuth for /oozie/services/servers > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:113) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) > at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:199) > at > org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:682) > at > org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:660) > at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) > at > org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:656) > at > org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:441) > at > org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:431) > at > org.apache.curator.framework.imps.CreateBui
[jira] [Updated] (OOZIE-3211) Warning: Deprecated property found in configuration in SparkConfigurationService is ambiguous
[ https://issues.apache.org/jira/browse/OOZIE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-3211: -- Fix Version/s: (was: 5.2.0) 5.3.0 > Warning: Deprecated property found in configuration in > SparkConfigurationService is ambiguous > - > > Key: OOZIE-3211 > URL: https://issues.apache.org/jira/browse/OOZIE-3211 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 5.0.0 > Environment: Apache Maven 3.5.3 > (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) > Maven home: /opt/maven/apache-maven-3.5.3 > Java version: 1.8.0_161, vendor: Oracle Corporation > Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre > Default locale: en_US, platform encoding: UTF-8 > OS name: "linux", version: "3.10.0-693.17.1.el7.x86_64", arch: "amd64", > family: "unix" > Oozie 5.0 RC0 compiled from clean Maven repo > Hadoop 2.7.5 pseudo distributed >Reporter: Artem Ervits >Priority: Major > Fix For: 5.3.0 > > > Getting > {code:java} > Deprecated property found in configuration: > oozie.service.SparkConfigurationService.spark.configurations.ignore.spark.yarn.jarUse > oozie.service.SparkConfigurationService.spark.configurations.blacklist > instead{code} > even though property > {code:java} > oozie.service.SparkConfigurationService.spark.configurations.blacklist{code} > is in oozie-site.xml. Looked at it with [~andras.piros] oozie-default.xml is > part of core and is on the classpath. Need to handle > [https://github.com/apache/oozie/blob/branch-5.0.0-beta1/core/src/main/java/org/apache/oozie/service/SparkConfigurationService.java#L74] > condition better -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-2973) Make sure Oozie works with Hadoop 3
[ https://issues.apache.org/jira/browse/OOZIE-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-2973: -- Fix Version/s: (was: 5.2.0) 5.3.0 > Make sure Oozie works with Hadoop 3 > > > Key: OOZIE-2973 > URL: https://issues.apache.org/jira/browse/OOZIE-2973 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Attila Sasvari >Priority: Blocker > Fix For: 5.3.0 > > Attachments: OOZIE-2973.001.patch, OOZIE-2973.002.patch > > > This JIRA is to track that Oozie works with Hadoop 3. > - Build Oozie with latest hadoop 3 (e.g. assuming Apache Hadoop 3.0.0-alpha3 > pass {{-D-Dhadoop.version=3.0.0-alpha3}} to maven) and execute tests > - It would be nice to create an Apache jenkins job that runs Oozie tests with > hadoop 3 (also with other component) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-2683) Rewrite the Oozie Web UI
[ https://issues.apache.org/jira/browse/OOZIE-2683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-2683: -- Fix Version/s: (was: 5.2.0) 5.3.0 > Rewrite the Oozie Web UI > > > Key: OOZIE-2683 > URL: https://issues.apache.org/jira/browse/OOZIE-2683 > Project: Oozie > Issue Type: New Feature >Affects Versions: 5.0.0 >Reporter: Robert Kanter >Assignee: Andras Piros >Priority: Major > Fix For: 5.3.0 > > > We're currently relying on a version of ExtJS that's so old, it's not linked > on their official website, and is often temporarily deleted. Looks like it's > now been deleted for good (OOZIE-2622). It also has a GPL license, so we > can't include it with Oozie, and users are forced to go and download it > themselves. And finally, it's a really outdated UI that's not very good. > We should invest in a new Web UI that's more modern and compatibly licensed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-2231) Upgrade curator to latest version 2.13.0
[ https://issues.apache.org/jira/browse/OOZIE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-2231: -- Fix Version/s: (was: 5.2.0) 5.3.0 > Upgrade curator to latest version 2.13.0 > > > Key: OOZIE-2231 > URL: https://issues.apache.org/jira/browse/OOZIE-2231 > Project: Oozie > Issue Type: Bug > Components: HA >Reporter: Purshotam Shah >Priority: Blocker > Fix For: 5.3.0 > > Attachments: OOZIE-2231-00.patch, OOZIE-2231-01.patch, > OOZIE-2231-02.patch, OOZIE-2231-02.patch, OOZIE-2231-03.patch, > OOZIE-2231-04.patch, OOZIE-2231-05.patch, OOZIE-2231-06.patch > > > It have some fix related to InterProcessReadWriteLock, ChildReaper, > LeaderSelector which we use. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-3533) Flaky test TestXLogService.testLog4jReload
[ https://issues.apache.org/jira/browse/OOZIE-3533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934304#comment-16934304 ] Andras Salamon commented on OOZIE-3533: --- [~kmarton] What do you think, which option should we choose? > Flaky test TestXLogService.testLog4jReload > -- > > Key: OOZIE-3533 > URL: https://issues.apache.org/jira/browse/OOZIE-3533 > Project: Oozie > Issue Type: Sub-task >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Andras Salamon >Priority: Minor > > Sometimes this test fails with the following error message: > {noformat} > junit.framework.AssertionFailedError > at junit.framework.Assert.fail(Assert.java:55) > at junit.framework.Assert.assertTrue(Assert.java:22) > at junit.framework.Assert.assertFalse(Assert.java:39) > at junit.framework.Assert.assertFalse(Assert.java:47) > at junit.framework.TestCase.assertFalse(TestCase.java:219) > at > org.apache.oozie.service.TestXLogService.testLog4jReload(TestXLogService.java:111) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at junit.framework.TestCase.runTest(TestCase.java:176) > at junit.framework.TestCase.runBare(TestCase.java:141) > at junit.framework.TestResult$1.protect(TestResult.java:122) > at junit.framework.TestResult.runProtected(TestResult.java:142) > at junit.framework.TestResult.run(TestResult.java:125) > at junit.framework.TestCase.run(TestCase.java:129) > at junit.framework.TestSuite.runTest(TestSuite.java:255) > at junit.framework.TestSuite.run(TestSuite.java:250) > at > org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84) > at org.junit.runners.Suite.runChild(Suite.java:127) > at org.junit.runners.Suite.runChild(Suite.java:26) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at org.junit.runners.Suite.runChild(Suite.java:127) > at org.junit.runners.Suite.runChild(Suite.java:26) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at > org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:410) > at > org.apache.maven.surefire.junitcore.pc.InvokerStrategy.schedule(InvokerStrategy.java:54) > at > org.apache.maven.surefire.junitcore.pc.Scheduler.schedule(Scheduler.java:367) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at > org.apache.maven.surefire.junitcore.pc.ParallelComputerBuilder$PC$1.run(ParallelComputerBuilder.java:593) > at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55) > at > org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137) > at > org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107) > at > org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83) > at > org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75) > at > org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:159) > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) > at > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) > {noformat} > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-2364) Remove deprecated SLAEventBean and related code
[ https://issues.apache.org/jira/browse/OOZIE-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934312#comment-16934312 ] Hadoop QA commented on OOZIE-2364: -- PreCommit-OOZIE-Build started > Remove deprecated SLAEventBean and related code > --- > > Key: OOZIE-2364 > URL: https://issues.apache.org/jira/browse/OOZIE-2364 > Project: Oozie > Issue Type: Task >Reporter: Rohini Palaniswamy >Assignee: Satish Subhashrao Saley >Priority: Major > Labels: newbie > Fix For: 5.3.0 > > Attachments: OOZIE-2364-1.patch, OOZIE-2364-2.patch, > OOZIE-2364-3.patch, OOZIE-2364-4.patch > > > With new SLAService feature which uses different tables, the old SLA_EVENTS > table is not used anymore. Need to remove all code related to it and drop > that table as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Failed: OOZIE-2364 PreCommit Build #1220
Jira: https://issues.apache.org/jira/browse/OOZIE-2364 Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/1220/ ### ## LAST 100 LINES OF THE CONSOLE ### [...truncated 955.53 KB...] + ""; Element eSla = XmlUtils.parseXml(slaXML); SLAEventBean regEvent = SLADbOperations.createSlaRegistrationEvent(eSla, null, "id1", SLAEvent.SlaAppType.WORKFLOW_JOB, "user1", "group1"); assertEquals(SLAEvent.SlaAppType.WORKFLOW_JOB, regEvent.getAppType()); assertEquals(new Date(nomDate.getTime() + 5 * 60 * 1000), regEvent.getExpectedEnd()); assertEquals("group1", regEvent.getGroupName()); assertEquals("id1", regEvent.getSlaId()); assertEquals("user1", regEvent.getUser()); regEvent = SLADbOperations.createSlaRegistrationEvent(eSla, "id1", SLAEvent.SlaAppType.WORKFLOW_JOB, "user1", "group1", null); assertEquals(SLAEvent.SlaAppType.WORKFLOW_JOB, regEvent.getAppType()); assertEquals(new Date(nomDate.getTime() + 5 * 60 * 1000), regEvent.getExpectedEnd()); assertEquals("group1", regEvent.getGroupName()); assertEquals("id1", regEvent.getSlaId()); assertEquals("user1", regEvent.getUser()); } } error: patch failed: core/src/test/java/org/apache/oozie/util/db/TestSLADbOperations.java:1 error: core/src/test/java/org/apache/oozie/util/db/TestSLADbOperations.java: patch does not apply Checking patch core/src/test/java/org/apache/oozie/util/db/TestSLADbXOperations.java... error: while searching for: /** * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package org.apache.oozie.util.db; import java.util.Date; import org.apache.oozie.SLAEventBean; import org.apache.oozie.client.SLAEvent; import org.apache.oozie.test.XTestCase; import org.apache.oozie.util.DateUtils; import org.apache.oozie.util.XmlUtils; import org.jdom.Element; public class TestSLADbXOperations extends XTestCase { @SuppressWarnings("deprecation") public void testCreateSlaRegistrationEventMinReqFields() throws Exception { Date nomDate = DateUtils.parseDateOozieTZ("2014-01-01T01:01Z"); String slaXML = " " + " " + DateUtils.formatDateOozieTZ(nomDate) + "" + " 5" + ""; Element eSla = XmlUtils.parseXml(slaXML); SLAEventBean regEvent = SLADbXOperations.createSlaRegistrationEvent(eSla, "id1", SLAEvent.SlaAppType.WORKFLOW_JOB, "user1", "group1"); assertEquals(SLAEvent.SlaAppType.WORKFLOW_JOB, regEvent.getAppType()); assertEquals(new Date(nomDate.getTime() + 5 * 60 * 1000), regEvent.getExpectedEnd()); assertEquals("group1", regEvent.getGroupName()); assertEquals("id1", regEvent.getSlaId()); assertEquals("user1", regEvent.getUser()); } } error: patch failed: core/src/test/java/org/apache/oozie/util/db/TestSLADbXOperations.java:1 error: core/src/test/java/org/apache/oozie/util/db/TestSLADbXOperations.java: patch does not apply Checking patch docs/src/site/twiki/CoordinatorFunctionalSpec.twiki... error: docs/src/site/twiki/CoordinatorFunctionalSpec.twiki: No such file or directory Checking patch docs/src/site/twiki/DG_CommandLineTool.twiki... error: docs/src/site/twiki/DG_CommandLineTool.twiki: No such file or directory Checking patch tools/src/main/java/org/apache/oozie/tools/OozieDBCLI.java... Hunk #1 succeeded at 1138 (offset 53 lines). Checking patch webapp/src/main/webapp/WEB-INF/web.xml... Patch failed to apply to head of branch Adding comment to JIRA % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0100 1990 0 100 199 0133 0:00:01 0:00:01 --:--:-- 133100 17740 1575 100 199 1001126 0:00:01 0:00:01 --:--:-- 1127 {"self":"https://issues.apache.org/jira/rest/api/2/issue/12888618/comment/16934319
[jira] [Commented] (OOZIE-2364) Remove deprecated SLAEventBean and related code
[ https://issues.apache.org/jira/browse/OOZIE-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934319#comment-16934319 ] Hadoop QA commented on OOZIE-2364: -- Testing JIRA OOZIE-2364 Cleaning local git workspace {color:red}-1{color} Patch failed to apply to head of branch > Remove deprecated SLAEventBean and related code > --- > > Key: OOZIE-2364 > URL: https://issues.apache.org/jira/browse/OOZIE-2364 > Project: Oozie > Issue Type: Task >Reporter: Rohini Palaniswamy >Assignee: Satish Subhashrao Saley >Priority: Major > Labels: newbie > Fix For: 5.3.0 > > Attachments: OOZIE-2364-1.patch, OOZIE-2364-2.patch, > OOZIE-2364-3.patch, OOZIE-2364-4.patch > > > With new SLAService feature which uses different tables, the old SLA_EVENTS > table is not used anymore. Need to remove all code related to it and drop > that table as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 65636: OOZIE-3179 - Adding a configurable config-default.xml location to a workflow
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65636/#review217883 --- Ship it! Ship It! - Andras Salamon On Aug. 30, 2019, 8:53 p.m., Jason Phelps wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65636/ > --- > > (Updated Aug. 30, 2019, 8:53 p.m.) > > > Review request for oozie. > > > Repository: oozie-git > > > Description > --- > > Added another client property: `oozie.default.configuration.path` which > allows for a comma seperated list of HDFS XML property files to be applied to > the job. This is useful if multiple workflows have shared properties (i.e. > database names, table names for Sqoop or Hive, spark arguments, etc). This is > similar to the `oozie.libpath` property which allows user's to define common > paths workflows to load jars. > > > Diffs > - > > client/src/main/java/org/apache/oozie/client/OozieClient.java 1073cb924 > core/src/main/java/org/apache/oozie/command/wf/SubmitXCommand.java > 1352db91b > core/src/test/java/org/apache/oozie/command/wf/TestSubmitXCommand.java > 2cd263f60 > > > Diff: https://reviews.apache.org/r/65636/diff/4/ > > > Testing > --- > > > Thanks, > > Jason Phelps > >
[jira] [Commented] (OOZIE-3179) Adding a configurable config-default.xml location to a workflow
[ https://issues.apache.org/jira/browse/OOZIE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934326#comment-16934326 ] ASF subversion and git services commented on OOZIE-3179: Commit 6bdca1d1c043d6ad688f53602667ca5201af5a28 in oozie's branch refs/heads/master from Andras Salamon [ https://gitbox.apache.org/repos/asf?p=oozie.git;h=6bdca1d ] OOZIE-3179 Adding a configurable config-default.xml location to a workflow (jphelps via asalamon74) > Adding a configurable config-default.xml location to a workflow > --- > > Key: OOZIE-3179 > URL: https://issues.apache.org/jira/browse/OOZIE-3179 > Project: Oozie > Issue Type: Improvement >Reporter: Jason Phelps >Assignee: Jason Phelps >Priority: Major > Attachments: OOZIE-3179-1.patch, OOZIE-3179-2.patch, > OOZIE-3179-3.patch, OOZIE-3179-4.patch, OOZIE-3179-5.patch > > > Recently a customer requested a way to include several job.properties > variables to every workflow in their cluster. > Normally the config-default.xml file would be the best way to include a set > of variables, but it does not scale very well if that file needs to be > updated, as it would require it to be updated in every workflow application > path. > What I am requesting is another job.properties variable that can point to a > config-default.xml file, or a comma separated list of config-default.xml > files that will all be applied to the workflow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-3179) Adding a configurable config-default.xml location to a workflow
[ https://issues.apache.org/jira/browse/OOZIE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934328#comment-16934328 ] Andras Salamon commented on OOZIE-3179: --- Thanks [~jphelps], +1, committed to master. > Adding a configurable config-default.xml location to a workflow > --- > > Key: OOZIE-3179 > URL: https://issues.apache.org/jira/browse/OOZIE-3179 > Project: Oozie > Issue Type: Improvement >Reporter: Jason Phelps >Assignee: Jason Phelps >Priority: Major > Attachments: OOZIE-3179-1.patch, OOZIE-3179-2.patch, > OOZIE-3179-3.patch, OOZIE-3179-4.patch, OOZIE-3179-5.patch > > > Recently a customer requested a way to include several job.properties > variables to every workflow in their cluster. > Normally the config-default.xml file would be the best way to include a set > of variables, but it does not scale very well if that file needs to be > updated, as it would require it to be updated in every workflow application > path. > What I am requesting is another job.properties variable that can point to a > config-default.xml file, or a comma separated list of config-default.xml > files that will all be applied to the workflow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-2231) Upgrade curator to latest version 2.13.0
[ https://issues.apache.org/jira/browse/OOZIE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934331#comment-16934331 ] Hadoop QA commented on OOZIE-2231: -- PreCommit-OOZIE-Build started > Upgrade curator to latest version 2.13.0 > > > Key: OOZIE-2231 > URL: https://issues.apache.org/jira/browse/OOZIE-2231 > Project: Oozie > Issue Type: Bug > Components: HA >Reporter: Purshotam Shah >Priority: Blocker > Fix For: 5.3.0 > > Attachments: OOZIE-2231-00.patch, OOZIE-2231-01.patch, > OOZIE-2231-02.patch, OOZIE-2231-02.patch, OOZIE-2231-03.patch, > OOZIE-2231-04.patch, OOZIE-2231-05.patch, OOZIE-2231-06.patch > > > It have some fix related to InterProcessReadWriteLock, ChildReaper, > LeaderSelector which we use. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Failed: OOZIE-2231 PreCommit Build #1221
Jira: https://issues.apache.org/jira/browse/OOZIE-2231 Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/1221/ ### ## LAST 100 LINES OF THE CONSOLE ### [...truncated 1.23 MB...] [INFO] Apache Oozie Fluent Job Client . SUCCESS [ 0.481 s] [INFO] Apache Oozie Server SUCCESS [ 3.512 s] [INFO] Apache Oozie Distro SUCCESS [ 1.062 s] [INFO] Apache Oozie ZooKeeper Security Tests .. SUCCESS [ 1.878 s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 01:01 min [INFO] Finished at: 2019-09-20T12:12:25Z [INFO] EXITVALUE 0 [TRACE] Modernizer output in HEAD verified and saved Applying patch Checking patch core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java... error: while searching for: package org.apache.oozie.action.hadoop; import com.google.common.annotations.VisibleForTesting; import com.google.common.base.Strings; import com.google.common.collect.ImmutableList; import com.google.common.io.Closeables; import com.google.common.primitives.Ints; import java.io.File; import java.io.FileNotFoundException; import java.io.IOException; import java.io.StringReader; import java.net.ConnectException; import java.net.URI; import java.net.URISyntaxException; import java.net.UnknownHostException; import java.nio.ByteBuffer; import java.text.MessageFormat; import java.util.ArrayList; import java.util.Arrays; import java.util.Collection; import java.util.HashMap; import java.util.HashSet; import java.util.Iterator; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; import java.util.Map.Entry; import org.apache.commons.io.IOUtils; import org.apache.hadoop.conf.Configuration; error: patch failed: core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java:19 error: core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java: patch does not apply Checking patch core/src/main/java/org/apache/oozie/service/JvmPauseMonitorService.java... error: while searching for: import java.util.List; import java.util.Map; import java.util.Set; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.util.Daemon; import org.apache.oozie.util.ConfigUtils; import org.apache.oozie.util.Instrumentation; import org.apache.oozie.util.XLog; error: patch failed: core/src/main/java/org/apache/oozie/service/JvmPauseMonitorService.java:29 error: core/src/main/java/org/apache/oozie/service/JvmPauseMonitorService.java: patch does not apply Checking patch docs/src/site/twiki/DG_QuickStart.twiki... error: docs/src/site/twiki/DG_QuickStart.twiki: No such file or directory Checking patch docs/src/site/twiki/ENG_Building.twiki... error: docs/src/site/twiki/ENG_Building.twiki: No such file or directory Checking patch pom.xml... error: while searching for: 2.12 0.16.0 h2 core 1.4.3 1.6.1 error: patch failed: pom.xml:94 error: pom.xml: patch does not apply Checking patch sharelib/pig/pom.xml... Hunk #1 succeeded at 72 (offset 7 lines). error: git diff header lacks filename information when removing 1 leading pathname component (line 255) Patch failed to apply to head of branch Adding comment to JIRA % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0100 1990 0 100 199 0134 0:00:01 0:00:01 --:--:-- 134100 17740 1575 100 199803101 0:00:01 0:00:01 --:--:-- 904 {"self":"https://issues.apache.org/jira/rest/api/2/issue/12828994/comment/16934337","id":"16934337","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa&avatarId=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small&ownerId=hadoopqa&avatarId=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall&ownerId=hadoopqa&avatarId=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium&ownerId=hadoopqa&avatarId=10393"},"displayName":"Hadoop QA","active":true,"timeZone":"Etc/UTC"},"body":"\nTesting JIRA OOZIE-2231\n\nCleaning local git workspace\n\n\n\n{color:red}-1{color} Patch failed to apply to head of branch\n\n\n","updateAuthor"
[jira] [Commented] (OOZIE-2231) Upgrade curator to latest version 2.13.0
[ https://issues.apache.org/jira/browse/OOZIE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934337#comment-16934337 ] Hadoop QA commented on OOZIE-2231: -- Testing JIRA OOZIE-2231 Cleaning local git workspace {color:red}-1{color} Patch failed to apply to head of branch > Upgrade curator to latest version 2.13.0 > > > Key: OOZIE-2231 > URL: https://issues.apache.org/jira/browse/OOZIE-2231 > Project: Oozie > Issue Type: Bug > Components: HA >Reporter: Purshotam Shah >Priority: Blocker > Fix For: 5.3.0 > > Attachments: OOZIE-2231-00.patch, OOZIE-2231-01.patch, > OOZIE-2231-02.patch, OOZIE-2231-02.patch, OOZIE-2231-03.patch, > OOZIE-2231-04.patch, OOZIE-2231-05.patch, OOZIE-2231-06.patch > > > It have some fix related to InterProcessReadWriteLock, ChildReaper, > LeaderSelector which we use. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] Subscription: Oozie Patch Available
Issue Subscription Filter: Oozie Patch Available (90 issues) Subscriber: ooziedaily Key Summary OOZIE-3536 oozie-main(pom.xml) plugin maven-javadoc-plugin upgrade version caused configuration can't find the Tag https://issues.apache.org/jira/browse/OOZIE-3536 OOZIE-3529 Oozie not supported for s3 as filesystem https://issues.apache.org/jira/browse/OOZIE-3529 OOZIE-3482 Fix bug in CoordSubmitXCommand#validateCoordinatorJob https://issues.apache.org/jira/browse/OOZIE-3482 OOZIE-3480 Add windowactionstatus metrics in DBLiteWorkflowStoreService https://issues.apache.org/jira/browse/OOZIE-3480 OOZIE-3461 CoordMaterializeTriggerService code cleanup https://issues.apache.org/jira/browse/OOZIE-3461 OOZIE-3449 Make spark-2 as the default profile https://issues.apache.org/jira/browse/OOZIE-3449 OOZIE-3447 Run test case in local : It shows oozie-hsqldb-orm.xml exception https://issues.apache.org/jira/browse/OOZIE-3447 OOZIE-3418 Upgrade to Guava 27 https://issues.apache.org/jira/browse/OOZIE-3418 OOZIE-3404 The env variable of SPARK_HOME needs to be set when running pySpark https://issues.apache.org/jira/browse/OOZIE-3404 OOZIE-3375 Can't use empty in coordinator https://issues.apache.org/jira/browse/OOZIE-3375 OOZIE-3367 Using && in EL expressions in oozie bundle.xml files generates parse errors https://issues.apache.org/jira/browse/OOZIE-3367 OOZIE-3366 Update workflow status and subworkflow status on suspend command https://issues.apache.org/jira/browse/OOZIE-3366 OOZIE-3364 Rerunning Oozie bundle jobs starts the coordinators in indeterminate order https://issues.apache.org/jira/browse/OOZIE-3364 OOZIE-3362 When killed, SSH action should kill the spawned processes on target host https://issues.apache.org/jira/browse/OOZIE-3362 OOZIE-3335 Cleanup parseFilter methods https://issues.apache.org/jira/browse/OOZIE-3335 OOZIE-3328 Create Hive compatibility action executor to run hive actions using beeline https://issues.apache.org/jira/browse/OOZIE-3328 OOZIE-3320 Oozie ShellAction should support absolute bash file path https://issues.apache.org/jira/browse/OOZIE-3320 OOZIE-3319 Log SSH action callback error output https://issues.apache.org/jira/browse/OOZIE-3319 OOZIE-3301 Update NOTICE file https://issues.apache.org/jira/browse/OOZIE-3301 OOZIE-3274 Remove slf4j https://issues.apache.org/jira/browse/OOZIE-3274 OOZIE-3266 Coord action rerun support RERUN_SKIP_NODES option https://issues.apache.org/jira/browse/OOZIE-3266 OOZIE-3256 refactor OozieCLI class https://issues.apache.org/jira/browse/OOZIE-3256 OOZIE-3254 [coordinator] LAST_ONLY and NONE execution modes: possible OutOfMemoryError when there are too many coordinator actions to materialize https://issues.apache.org/jira/browse/OOZIE-3254 OOZIE-3199 Let system property restriction configurable https://issues.apache.org/jira/browse/OOZIE-3199 OOZIE-3196 Authorization: restrict world readability by user https://issues.apache.org/jira/browse/OOZIE-3196 OOZIE-3170 Oozie Diagnostic Bundle tool fails with NPE due to missing service class https://issues.apache.org/jira/browse/OOZIE-3170 OOZIE-3137 Add support for log4j2 in HiveMain https://issues.apache.org/jira/browse/OOZIE-3137 OOZIE-3135 Configure log4j2 in SqoopMain https://issues.apache.org/jira/browse/OOZIE-3135 OOZIE-3091 Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper" https://issues.apache.org/jira/browse/OOZIE-3091 OOZIE-3071 Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0 https://issues.apache.org/jira/browse/OOZIE-3071 OOZIE-3063 Sanitizing variables that are part of openjpa.ConnectionProperties https://issues.apache.org/jira/browse/OOZIE-3063 OOZIE-3062 Set HADOOP_CONF_DIR for spark action https://issues.apache.org/jira/browse/OOZIE-3062 OOZIE-2952 Fix Findbugs warnings in oozie-sharelib-oozie https://issues.apache.org/jira/browse/OOZIE-2952 OOZIE-2834 ParameterVerifier logging non-useful warning for workflow definition https://issues.apache.org/jira/browse/OOZIE-2834 OOZIE-2812 SparkConfigurationService should support loading configurations from multiple Spark versions https://issues.apache.org/jira/browse/OOZIE-2812 OOZIE-2795 Create lib directory or symlink for Oozie CLI during packaging https://issues.apache.org/jira/browse/OOZIE-2795 OOZIE-2784 Include WEEK as a parameter in the Coordinator Expression Language Evaulator https://issues.apache.org/jira/browse/OOZIE-2784 OOZIE-2779 Mask Hive2 action Beeline JDBC password