[jira] [Updated] (OOZIE-3529) Oozie not supported for s3 as filesystem
[
https://issues.apache.org/jira/browse/OOZIE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon updated OOZIE-3529:
--
Fix Version/s: 5.2.0
> Oozie not supported for s3 as filesystem
>
>
> Key: OOZIE-3529
> URL: https://issues.apache.org/jira/browse/OOZIE-3529
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.1, 5.1.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Labels: S3
> Fix For: 5.2.0
>
> Attachments: OOZIE-3529.001.patch, id.pig, job.properties,
> workflow.xml
>
>
> Many customer who uses s3 file system as secondary one experiences the
> following error when Oozie tries to submit the Yarn application:
> {noformat}
> 2019-04-29 13:02:53,770 WARN ForkedActionStartXCommand:523 -
> SERVER[hwnode1.puretec.purestorage.com] USER[hrt_qa] GROUP[-] TOKEN[]
> APP[demo-wf] JOB[001-190423141707256-oozie-oozi-W]
> ACTION[001-190423141707256-oozie-oozi-W@streaming-node] Error starting
> action [streaming-node]. ErrorType [ERROR], ErrorCode
> [UnsupportedOperationException], Message [UnsupportedOperationException:
> Accessing local file system is not allowed]
> org.apache.oozie.action.ActionExecutorException:
> UnsupportedOperationException: Accessing local file system is not allowed
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.createLauncherConf(JavaActionExecutor.java:1092)
> at
> org.apache.oozie.action.hadoop.MapReduceActionExecutor.createLauncherConf(MapReduceActionExecutor.java:309)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1197)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:41)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:30)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.UnsupportedOperationException: Accessing local file
> system is not allowed
> at
> org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:48)
> at
> org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSystem.java:47)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
> at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:433)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:301)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:378)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForWrite(LocalDirAllocator.java:461)
> at
> org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocator.java:200)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.createTmpFileForWrite(S3AFileSystem.java:572)
> at
> org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory.create(S3ADataBlocks.java:811)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.createBlockIfNeeded(S3ABlockOutputStream.java:190)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.(S3ABlockOutputStream.java:168)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:778)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1038)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1026)
> at
> org.apache.oozie.action.hadoop.LauncherMapperHelper.setupLauncherInfo(LauncherMapperHelper.java:156)
> at
> org.apache.oozie.action.hadoo
[jira] [Updated] (OOZIE-3484) [Java 11] Get rid of com.sun.tools. package usage
[ https://issues.apache.org/jira/browse/OOZIE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-3484: -- Fix Version/s: (was: OOZIE-3490) 5.2.0 > [Java 11] Get rid of com.sun.tools. package usage > - > > Key: OOZIE-3484 > URL: https://issues.apache.org/jira/browse/OOZIE-3484 > Project: Oozie > Issue Type: Sub-task >Reporter: Julia Kinga Marton >Priority: Major > Fix For: 5.2.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-3359) Check for Process#waitFor() usage and fix it in order to avoid indefinite waiting
[
https://issues.apache.org/jira/browse/OOZIE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon updated OOZIE-3359:
--
Fix Version/s: (was: 5.2.0)
5.3.0
> Check for Process#waitFor() usage and fix it in order to avoid indefinite
> waiting
> -
>
> Key: OOZIE-3359
> URL: https://issues.apache.org/jira/browse/OOZIE-3359
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Julia Kinga Marton
>Priority: Major
> Fix For: 5.3.0
>
>
> {{Process#waitFor()}} will block until the process finishes. There are
> situations where this will wait indefinitely. A similar case was in
> SshActionExecutor, fixed in OOZIE-3354.
> Let's check the code for other usages, and fix it.
> Maybe we should check if we can use somehow the Process#waitFor(long timeout,
> TimeUnit timeUnit) instead of it. The tricky part is that this method is
> available only in Java 8.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3392) Refactor ShellMain to use BufferDrainer class
[
https://issues.apache.org/jira/browse/OOZIE-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon updated OOZIE-3392:
--
Fix Version/s: (was: 5.2.0)
5.3.0
> Refactor ShellMain to use BufferDrainer class
> -
>
> Key: OOZIE-3392
> URL: https://issues.apache.org/jira/browse/OOZIE-3392
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Priority: Major
> Fix For: 5.3.0
>
>
> OOZIE-3382 introduced a new {{BufferDrainer}} class to drain the standard
> output and standard error stream of an ssh process. {{ShellMain}} should also
> use this class for reading the standard output and standard error of a shell
> process to avoid code duplication.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3336) [persistence] Refactor entity classes to feature PK, FK, and UQ constraints
[
https://issues.apache.org/jira/browse/OOZIE-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon updated OOZIE-3336:
--
Fix Version/s: (was: 5.2.0)
5.3.0
> [persistence] Refactor entity classes to feature PK, FK, and UQ constraints
> ---
>
> Key: OOZIE-3336
> URL: https://issues.apache.org/jira/browse/OOZIE-3336
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Priority: Major
> Fix For: 5.3.0
>
>
> When an Oozie database grows substantial in size, let's say, over a few
> hundred thousands of {{WorkflowActionBean}}, {{CoordinatorActionBean}}
> instances, we face a couple of performance issues. Here is an analysis why.
> Current Oozie JPA {{@Entity}} usage, and the resulting database DDL, suffers
> from a couple of drawback from a performance point of view:
> * {{@Id}} fields are {{String}}:
> ** leaving no space for database primary key indices to work effectively
> ** those values are calculated in case of {{WorkflowActionBean}},
> {{CoordinatorActionBean}}, and {{BundleActionBean}} instances
> * no foreign constraint is set from {{WorkflowActionBean}} to
> {{WorkflowJobBean}}, from {{CoordinatorActionBean}} to
> {{CoordinatorJobBean}}, or from {{BundleActionBean}} to {{BundleJobBean}}
> instances:
> ** have to assess JPA queries discovering parent-child relationships by hand
> ** no database indices are created, and hence, those queries that contain any
> {{JOIN}} instances are slower
> * no use of unique constraints whatsoever
> * JPA queries are created by hand instead of relying on OpenJPA
> * JPA entities are filled by hand instead of relying on OpenJPA
> Following enhancements are necessary:
> # keeping the existing {{String compositeId}} fields, let's break down the
> contents to following new fields:
> ## {{@Id long id}} - an auto-increment value that is unique across Oozie
> database
> ## {{long currentSequence}} - the sequence number of the current run since
> last Oozie server restart. The first part of the {{compositeId}}
> ## {{Timestamp serverStartupTimestamp}} - the timestamp when the Oozie server
> was last started. The second part of the {{compositeId}}
> ## {{String serverName}} - the third part of the {{compositeId}}
> ## {{String name}} - the fourth and last part of the {{compositeId}}
> ## {{compositeId}} might be calculated when an entity is loaded / persisted,
> and then stored
> # FK constraints:
> ## {{@OneToMany}} fields where we have a list of child references inside
> parent
> ## {{@ManyToOne}} fields where we have a parent reference inside child
> ## pay attention to {{FetchType}}, most of the times {{LAZY}} will be needed
> ## the containment fields should not be {{@Transient}} anymore
> # UQ constraints:
> ## on {{currentSequence}} and {{serverStartupTimestamp}}
> ## on {{currentSequence}} and {{name}}
> # new JPQL queries:
> ## to cover changed parent-child relationships
> ## to get use of each disassembled part of {{originalId}} when doing e.g.
> filtering
> # let JPA fill entities instead performing this by hand
> Following enhancements can be considered as nice-to-have:
> * upgrade to an OpenJPA version that features JPA 2.1's composite indexing
> capability
> * see whether to have an optimistic locking field using {{@Version}} instead
> of ZooKeeper based pessimistic locking would increase High Availability
> characteristics
> * refactor also SLA related entity classes
> It's necessary to have performance benchmarks with some database types like
> MySQL/MariaDB, and PostgreSQL before and after the changes for following use
> cases:
> * {{CoordinatorJobBean}} and {{WorkflowJobBean}} instances up to millions
> * {{CoordinatorActionBean}} and {{WorkflowActionBean}} instances up to tens
> of millions
> * performance for JPQLs that get a list of entities
> * performance of persisting a new entity
> * performance of querying lists of entities based on popular / possible
> filters like the ones used by {{VxJobsServlet}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3538) oozie-main(pom.xml) plugin maven-surefire-plugin configuration the Tag value is wrong
[
https://issues.apache.org/jira/browse/OOZIE-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon updated OOZIE-3538:
--
Fix Version/s: (was: 5.2.0)
5.3.0
> oozie-main(pom.xml) plugin maven-surefire-plugin configuration the Tag
> value is wrong
> --
>
> Key: OOZIE-3538
> URL: https://issues.apache.org/jira/browse/OOZIE-3538
> Project: Oozie
> Issue Type: Bug
> Components: build
>Affects Versions: 5.2.0
>Reporter: duan xiong
>Assignee: duan xiong
>Priority: Major
> Fix For: 5.3.0
>
>
> oozie-main(pom.xml) plugin maven-surefire-plugin configuration the Tag
> value is wrong. That value should be boolean not number.
> " the parameter {{threadCount}} can be used with the optional parameter
> {{perCoreThreadCount=true}} (true by default)" can find this in
> [http://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-1987) Fix existing findbugs warnings
[ https://issues.apache.org/jira/browse/OOZIE-1987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-1987: -- Fix Version/s: (was: 5.2.0) 5.3.0 > Fix existing findbugs warnings > -- > > Key: OOZIE-1987 > URL: https://issues.apache.org/jira/browse/OOZIE-1987 > Project: Oozie > Issue Type: Bug >Reporter: Robert Kanter >Priority: Major > Fix For: 5.3.0 > > > Now that we have OOZIE-1793, we should actually fix the warnings that > findbugs finds in the existing codebase. It may be a good idea to split this > into subtasks or multiple patches as there are a lot of warnings: over 500 in > oozie-core alone! > If we think something is ok, but findbugs complains, we should exclude it. > There's a way to create a file that lists exclusions (Hadoop does this). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-3105) testJMXInstrumentation from the org.apache.oozie.util.TestMetricsInstrumentation class is flaky
[
https://issues.apache.org/jira/browse/OOZIE-3105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon updated OOZIE-3105:
--
Fix Version/s: (was: 5.2.0)
5.3.0
> testJMXInstrumentation from the
> org.apache.oozie.util.TestMetricsInstrumentation class is flaky
> ---
>
> Key: OOZIE-3105
> URL: https://issues.apache.org/jira/browse/OOZIE-3105
> Project: Oozie
> Issue Type: Sub-task
> Components: tests
>Affects Versions: 4.3.0
> Environment: mvn: 3.5.0
> java: 1.8.0.144
>Reporter: Artem Ervits
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 5.3.0
>
> Attachments: OOZIE-3105-00.patch
>
>
> I'm able to reproduce the test failure consistently by running test in
> isolation with the following command
> {code}
> mvn clean
> -Dtest=org.apache.oozie.util.TestMetricsInstrumentation#testJMXInstrumentation
> test
> {code}
> the test succeeds consistently with the following
> {code}
> mvn clean -Dtest=org.apache.oozie.util.TestMetricsInstrumentation test
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3428) [docs] Update Setting Up Oozie with an Alternate Tomcat documentation
[
https://issues.apache.org/jira/browse/OOZIE-3428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon updated OOZIE-3428:
--
Fix Version/s: (was: 5.2.0)
5.3.0
> [docs] Update Setting Up Oozie with an Alternate Tomcat documentation
> -
>
> Key: OOZIE-3428
> URL: https://issues.apache.org/jira/browse/OOZIE-3428
> Project: Oozie
> Issue Type: Improvement
> Components: docs
>Affects Versions: trunk
>Reporter: Andras Salamon
>Priority: Major
> Fix For: 5.3.0
>
>
> Although Oozie uses embedded Jetty now, during compilation it still creates a
> {{war}} file which can be run in a different servlet container.
> The [current
> documentation|https://oozie.apache.org/docs/5.1.0/AG_Install.html#Setting_Up_Oozie_with_an_Alternate_Tomcat]
> does not describe the full process, we need to enhance it:
> * A full sample of the {{addtowar}} script usage.
> * Necessary {{CATALINA_OPTS}} settings.
> * A simple script to cleanup the {{webapps}}, copy the war, restart Tomcat.
> * Some info about the supported Tomcat versions would be also useful.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3198) Extend remaining job tracker references in code and documentation
[
https://issues.apache.org/jira/browse/OOZIE-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon updated OOZIE-3198:
--
Fix Version/s: (was: 5.2.0)
5.3.0
> Extend remaining job tracker references in code and documentation
> -
>
> Key: OOZIE-3198
> URL: https://issues.apache.org/jira/browse/OOZIE-3198
> Project: Oozie
> Issue Type: Sub-task
> Components: action, core, docs
>Affects Versions: 5.0.0b1
>Reporter: Andras Piros
>Priority: Major
> Fix For: 5.3.0
>
>
> Extend the places in configuration, code and documentation where
> {{jobTracker}} and {{job-tracker}} is still referenced without a proper
> {{resourceManager}}, {{resource-manager}} reference that precedes it.
> Example: {{oozie.service.HadoopAccessorService.jobTracker.whitelist}} in
> {{AG_HadoopConfiguration.twiki}}, {{HadoopAccessorService.java}}, and
> {{oozie-default.xml}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3357) [core] More graceful CallableQueueService#destroy()
[
https://issues.apache.org/jira/browse/OOZIE-3357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon updated OOZIE-3357:
--
Fix Version/s: (was: 5.2.0)
5.3.0
> [core] More graceful CallableQueueService#destroy()
> ---
>
> Key: OOZIE-3357
> URL: https://issues.apache.org/jira/browse/OOZIE-3357
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Priority: Major
> Fix For: 5.3.0
>
>
> In OOZIE-3160 {{CallableQueueService}} has been refactored. It seems like not
> all the {{XCallable}} elements are processed when one of the
> {{ExecutorService}} instances are shut down.
> Following issues to investigate:
> # {{ThreadPoolExecutor#awaitTermination()}} gets stuck even if no
> {{XCallable}} elements are present anymore
> # await termination only conditionally, that is, when there are still some
> {{XCallable}} elements in one of the {{ThreadPoolExecutor}} instances
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3345) forkjoin validation error when multiple(>=3) actions with "ok to""error to" same action
[
https://issues.apache.org/jira/browse/OOZIE-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon updated OOZIE-3345:
--
Fix Version/s: (was: 5.2.0)
5.3.0
> forkjoin validation error when multiple(>=3) actions with "ok to""error to"
> same action
> ---
>
> Key: OOZIE-3345
> URL: https://issues.apache.org/jira/browse/OOZIE-3345
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: wang jinyin
>Priority: Major
> Fix For: 5.3.0
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> when multiple (n >= 3) actions with "ok to" "error to" transition to same
> action, forkjoin validation will throw WorkflowException(ErrorCode.E0743)
> {quote}org.apache.oozie.workflow.WorkflowException: E0743: Multiple "ok to"
> transitions to the same node, [C], are not allowed
> at
> org.apache.oozie.workflow.lite.LiteWorkflowValidator.validateForkJoin(LiteWorkflowValidator.java:186)
> at
> org.apache.oozie.workflow.lite.LiteWorkflowValidator.validateForkJoin(LiteWorkflowValidator.java:267)
> at
> org.apache.oozie.workflow.lite.LiteWorkflowValidator.validateForkJoin(LiteWorkflowValidator.java:271)
> at
> org.apache.oozie.workflow.lite.LiteWorkflowValidator.validateForkJoin(LiteWorkflowValidator.java:275)
> at
> org.apache.oozie.workflow.lite.LiteWorkflowValidator.validateWorkflow(LiteWorkflowValidator.java:62)
> at
> org.apache.oozie.workflow.lite.LiteWorkflowAppParser.validateAndParse(LiteWorkflowAppParser.java:185)
> at
> org.apache.oozie.workflow.lite.LiteWorkflowLib.parseDef(LiteWorkflowLib.java:58)
> at
> org.apache.oozie.service.LiteWorkflowAppService.parseDef(LiteWorkflowAppService.java:57)
> at
> org.apache.oozie.service.LiteWorkflowAppService.parseDef(LiteWorkflowAppService.java:46)
> at
> org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:167)
> {quote}
>
> workflow definition:
> {quote}
>
>
>
>
>
>
>
>
>
>
>
> {quote}
>
> [reason]
> When validate A's "ok to" path -> B's "ok to" path, action node C has already
> been added into nodeAndDecisionParents.
> When validate A's "error to" path -> B's "ok to" path, C's decisionParent ==
> null and topDecisionParent == null.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-2364) Remove deprecated SLAEventBean and related code
[ https://issues.apache.org/jira/browse/OOZIE-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-2364: -- Fix Version/s: (was: 5.2.0) 5.3.0 > Remove deprecated SLAEventBean and related code > --- > > Key: OOZIE-2364 > URL: https://issues.apache.org/jira/browse/OOZIE-2364 > Project: Oozie > Issue Type: Task >Reporter: Rohini Palaniswamy >Assignee: Satish Subhashrao Saley >Priority: Major > Labels: newbie > Fix For: 5.3.0 > > Attachments: OOZIE-2364-1.patch, OOZIE-2364-2.patch, > OOZIE-2364-3.patch, OOZIE-2364-4.patch > > > With new SLAService feature which uses different tables, the old SLA_EVENTS > table is not used anymore. Need to remove all code related to it and drop > that table as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-2809) Oozie HA with secure zookeeper access fails because ZKUtils#setJaasConfiguration does not replace "_HOST" in principal string
[
https://issues.apache.org/jira/browse/OOZIE-2809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon updated OOZIE-2809:
--
Fix Version/s: (was: 5.2.0)
5.3.0
> Oozie HA with secure zookeeper access fails because
> ZKUtils#setJaasConfiguration does not replace "_HOST" in principal string
> -
>
> Key: OOZIE-2809
> URL: https://issues.apache.org/jira/browse/OOZIE-2809
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: trunk, 4.2.0, 4.3.0
>Reporter: Thomas Larsson
>Priority: Major
> Fix For: 5.3.0
>
>
> Setting "oozie.zookeeper.secure" = true causes the oozie server not to start
> if "oozie.service.HadoopAccessorService.kerberos.principal" is defined with a
> "_HOST" token instead of an actual hostname.
> The symptom when this happens is that the oozie log shows something like this:
> {noformat}
> org.apache.oozie.service.ServiceException: E1700: Issue communicating with
> ZooKeeper: KeeperErrorCode = NoAuth for /oozie/services/servers
> at
> org.apache.oozie.service.ZKLocksService.init(ZKLocksService.java:76)
> at
> org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:305)
> at org.apache.oozie.service.Services.init(Services.java:213)
> at
> org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46)
> at
> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210)
> at
> org.apache.catalina.core.StandardContext.start(StandardContext.java:4709)
> at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:802)
> at
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
> at
> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)
> at
> org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:676)
> at
> org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:602)
> at
> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:503)
> at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322)
> at
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325)
> at
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
> at
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1068)
> at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)
> at
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1060)
> at
> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
> at
> org.apache.catalina.core.StandardService.start(StandardService.java:525)
> at
> org.apache.catalina.core.StandardServer.start(StandardServer.java:759)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
> Caused by: org.apache.zookeeper.KeeperException$NoAuthException:
> KeeperErrorCode = NoAuth for /oozie/services/servers
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
> at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:199)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:682)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:660)
> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:656)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:441)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:431)
> at
> org.apache.curator.framework.imps.CreateBui
[jira] [Updated] (OOZIE-3211) Warning: Deprecated property found in configuration in SparkConfigurationService is ambiguous
[
https://issues.apache.org/jira/browse/OOZIE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon updated OOZIE-3211:
--
Fix Version/s: (was: 5.2.0)
5.3.0
> Warning: Deprecated property found in configuration in
> SparkConfigurationService is ambiguous
> -
>
> Key: OOZIE-3211
> URL: https://issues.apache.org/jira/browse/OOZIE-3211
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.0.0
> Environment: Apache Maven 3.5.3
> (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z)
> Maven home: /opt/maven/apache-maven-3.5.3
> Java version: 1.8.0_161, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.10.0-693.17.1.el7.x86_64", arch: "amd64",
> family: "unix"
> Oozie 5.0 RC0 compiled from clean Maven repo
> Hadoop 2.7.5 pseudo distributed
>Reporter: Artem Ervits
>Priority: Major
> Fix For: 5.3.0
>
>
> Getting
> {code:java}
> Deprecated property found in configuration:
> oozie.service.SparkConfigurationService.spark.configurations.ignore.spark.yarn.jarUse
> oozie.service.SparkConfigurationService.spark.configurations.blacklist
> instead{code}
> even though property
> {code:java}
> oozie.service.SparkConfigurationService.spark.configurations.blacklist{code}
> is in oozie-site.xml. Looked at it with [~andras.piros] oozie-default.xml is
> part of core and is on the classpath. Need to handle
> [https://github.com/apache/oozie/blob/branch-5.0.0-beta1/core/src/main/java/org/apache/oozie/service/SparkConfigurationService.java#L74]
> condition better
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-2973) Make sure Oozie works with Hadoop 3
[
https://issues.apache.org/jira/browse/OOZIE-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon updated OOZIE-2973:
--
Fix Version/s: (was: 5.2.0)
5.3.0
> Make sure Oozie works with Hadoop 3
>
>
> Key: OOZIE-2973
> URL: https://issues.apache.org/jira/browse/OOZIE-2973
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 4.3.0
>Reporter: Attila Sasvari
>Priority: Blocker
> Fix For: 5.3.0
>
> Attachments: OOZIE-2973.001.patch, OOZIE-2973.002.patch
>
>
> This JIRA is to track that Oozie works with Hadoop 3.
> - Build Oozie with latest hadoop 3 (e.g. assuming Apache Hadoop 3.0.0-alpha3
> pass {{-D-Dhadoop.version=3.0.0-alpha3}} to maven) and execute tests
> - It would be nice to create an Apache jenkins job that runs Oozie tests with
> hadoop 3 (also with other component)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-2683) Rewrite the Oozie Web UI
[ https://issues.apache.org/jira/browse/OOZIE-2683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-2683: -- Fix Version/s: (was: 5.2.0) 5.3.0 > Rewrite the Oozie Web UI > > > Key: OOZIE-2683 > URL: https://issues.apache.org/jira/browse/OOZIE-2683 > Project: Oozie > Issue Type: New Feature >Affects Versions: 5.0.0 >Reporter: Robert Kanter >Assignee: Andras Piros >Priority: Major > Fix For: 5.3.0 > > > We're currently relying on a version of ExtJS that's so old, it's not linked > on their official website, and is often temporarily deleted. Looks like it's > now been deleted for good (OOZIE-2622). It also has a GPL license, so we > can't include it with Oozie, and users are forced to go and download it > themselves. And finally, it's a really outdated UI that's not very good. > We should invest in a new Web UI that's more modern and compatibly licensed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-2231) Upgrade curator to latest version 2.13.0
[ https://issues.apache.org/jira/browse/OOZIE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated OOZIE-2231: -- Fix Version/s: (was: 5.2.0) 5.3.0 > Upgrade curator to latest version 2.13.0 > > > Key: OOZIE-2231 > URL: https://issues.apache.org/jira/browse/OOZIE-2231 > Project: Oozie > Issue Type: Bug > Components: HA >Reporter: Purshotam Shah >Priority: Blocker > Fix For: 5.3.0 > > Attachments: OOZIE-2231-00.patch, OOZIE-2231-01.patch, > OOZIE-2231-02.patch, OOZIE-2231-02.patch, OOZIE-2231-03.patch, > OOZIE-2231-04.patch, OOZIE-2231-05.patch, OOZIE-2231-06.patch > > > It have some fix related to InterProcessReadWriteLock, ChildReaper, > LeaderSelector which we use. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-3533) Flaky test TestXLogService.testLog4jReload
[
https://issues.apache.org/jira/browse/OOZIE-3533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934304#comment-16934304
]
Andras Salamon commented on OOZIE-3533:
---
[~kmarton] What do you think, which option should we choose?
> Flaky test TestXLogService.testLog4jReload
> --
>
> Key: OOZIE-3533
> URL: https://issues.apache.org/jira/browse/OOZIE-3533
> Project: Oozie
> Issue Type: Sub-task
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Minor
>
> Sometimes this test fails with the following error message:
> {noformat}
> junit.framework.AssertionFailedError
> at junit.framework.Assert.fail(Assert.java:55)
> at junit.framework.Assert.assertTrue(Assert.java:22)
> at junit.framework.Assert.assertFalse(Assert.java:39)
> at junit.framework.Assert.assertFalse(Assert.java:47)
> at junit.framework.TestCase.assertFalse(TestCase.java:219)
> at
> org.apache.oozie.service.TestXLogService.testLog4jReload(TestXLogService.java:111)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at junit.framework.TestCase.runBare(TestCase.java:141)
> at junit.framework.TestResult$1.protect(TestResult.java:122)
> at junit.framework.TestResult.runProtected(TestResult.java:142)
> at junit.framework.TestResult.run(TestResult.java:125)
> at junit.framework.TestCase.run(TestCase.java:129)
> at junit.framework.TestSuite.runTest(TestSuite.java:255)
> at junit.framework.TestSuite.run(TestSuite.java:250)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
> at org.junit.runners.Suite.runChild(Suite.java:127)
> at org.junit.runners.Suite.runChild(Suite.java:26)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runners.Suite.runChild(Suite.java:127)
> at org.junit.runners.Suite.runChild(Suite.java:26)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:410)
> at
> org.apache.maven.surefire.junitcore.pc.InvokerStrategy.schedule(InvokerStrategy.java:54)
> at
> org.apache.maven.surefire.junitcore.pc.Scheduler.schedule(Scheduler.java:367)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junitcore.pc.ParallelComputerBuilder$PC$1.run(ParallelComputerBuilder.java:593)
> at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407)
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-2364) Remove deprecated SLAEventBean and related code
[ https://issues.apache.org/jira/browse/OOZIE-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934312#comment-16934312 ] Hadoop QA commented on OOZIE-2364: -- PreCommit-OOZIE-Build started > Remove deprecated SLAEventBean and related code > --- > > Key: OOZIE-2364 > URL: https://issues.apache.org/jira/browse/OOZIE-2364 > Project: Oozie > Issue Type: Task >Reporter: Rohini Palaniswamy >Assignee: Satish Subhashrao Saley >Priority: Major > Labels: newbie > Fix For: 5.3.0 > > Attachments: OOZIE-2364-1.patch, OOZIE-2364-2.patch, > OOZIE-2364-3.patch, OOZIE-2364-4.patch > > > With new SLAService feature which uses different tables, the old SLA_EVENTS > table is not used anymore. Need to remove all code related to it and drop > that table as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Failed: OOZIE-2364 PreCommit Build #1220
Jira: https://issues.apache.org/jira/browse/OOZIE-2364
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/1220/
###
## LAST 100 LINES OF THE CONSOLE
###
[...truncated 955.53 KB...]
+ "";
Element eSla = XmlUtils.parseXml(slaXML);
SLAEventBean regEvent =
SLADbOperations.createSlaRegistrationEvent(eSla, null, "id1",
SLAEvent.SlaAppType.WORKFLOW_JOB, "user1", "group1");
assertEquals(SLAEvent.SlaAppType.WORKFLOW_JOB, regEvent.getAppType());
assertEquals(new Date(nomDate.getTime() + 5 * 60 * 1000),
regEvent.getExpectedEnd());
assertEquals("group1", regEvent.getGroupName());
assertEquals("id1", regEvent.getSlaId());
assertEquals("user1", regEvent.getUser());
regEvent =
SLADbOperations.createSlaRegistrationEvent(eSla, "id1",
SLAEvent.SlaAppType.WORKFLOW_JOB, "user1", "group1", null);
assertEquals(SLAEvent.SlaAppType.WORKFLOW_JOB, regEvent.getAppType());
assertEquals(new Date(nomDate.getTime() + 5 * 60 * 1000),
regEvent.getExpectedEnd());
assertEquals("group1", regEvent.getGroupName());
assertEquals("id1", regEvent.getSlaId());
assertEquals("user1", regEvent.getUser());
}
}
error: patch failed:
core/src/test/java/org/apache/oozie/util/db/TestSLADbOperations.java:1
error: core/src/test/java/org/apache/oozie/util/db/TestSLADbOperations.java:
patch does not apply
Checking patch
core/src/test/java/org/apache/oozie/util/db/TestSLADbXOperations.java...
error: while searching for:
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.oozie.util.db;
import java.util.Date;
import org.apache.oozie.SLAEventBean;
import org.apache.oozie.client.SLAEvent;
import org.apache.oozie.test.XTestCase;
import org.apache.oozie.util.DateUtils;
import org.apache.oozie.util.XmlUtils;
import org.jdom.Element;
public class TestSLADbXOperations extends XTestCase {
@SuppressWarnings("deprecation")
public void testCreateSlaRegistrationEventMinReqFields() throws Exception {
Date nomDate = DateUtils.parseDateOozieTZ("2014-01-01T01:01Z");
String slaXML = " "
+ " " + DateUtils.formatDateOozieTZ(nomDate)
+ ""
+ " 5"
+ "";
Element eSla = XmlUtils.parseXml(slaXML);
SLAEventBean regEvent =
SLADbXOperations.createSlaRegistrationEvent(eSla, "id1",
SLAEvent.SlaAppType.WORKFLOW_JOB, "user1", "group1");
assertEquals(SLAEvent.SlaAppType.WORKFLOW_JOB, regEvent.getAppType());
assertEquals(new Date(nomDate.getTime() + 5 * 60 * 1000),
regEvent.getExpectedEnd());
assertEquals("group1", regEvent.getGroupName());
assertEquals("id1", regEvent.getSlaId());
assertEquals("user1", regEvent.getUser());
}
}
error: patch failed:
core/src/test/java/org/apache/oozie/util/db/TestSLADbXOperations.java:1
error: core/src/test/java/org/apache/oozie/util/db/TestSLADbXOperations.java:
patch does not apply
Checking patch docs/src/site/twiki/CoordinatorFunctionalSpec.twiki...
error: docs/src/site/twiki/CoordinatorFunctionalSpec.twiki: No such file or
directory
Checking patch docs/src/site/twiki/DG_CommandLineTool.twiki...
error: docs/src/site/twiki/DG_CommandLineTool.twiki: No such file or directory
Checking patch tools/src/main/java/org/apache/oozie/tools/OozieDBCLI.java...
Hunk #1 succeeded at 1138 (offset 53 lines).
Checking patch webapp/src/main/webapp/WEB-INF/web.xml...
Patch failed to apply to head of branch
Adding comment to JIRA
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0100
1990 0 100 199 0133 0:00:01 0:00:01 --:--:-- 133100
17740 1575 100 199 1001126 0:00:01 0:00:01 --:--:-- 1127
{"self":"https://issues.apache.org/jira/rest/api/2/issue/12888618/comment/16934319
[jira] [Commented] (OOZIE-2364) Remove deprecated SLAEventBean and related code
[
https://issues.apache.org/jira/browse/OOZIE-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934319#comment-16934319
]
Hadoop QA commented on OOZIE-2364:
--
Testing JIRA OOZIE-2364
Cleaning local git workspace
{color:red}-1{color} Patch failed to apply to head of branch
> Remove deprecated SLAEventBean and related code
> ---
>
> Key: OOZIE-2364
> URL: https://issues.apache.org/jira/browse/OOZIE-2364
> Project: Oozie
> Issue Type: Task
>Reporter: Rohini Palaniswamy
>Assignee: Satish Subhashrao Saley
>Priority: Major
> Labels: newbie
> Fix For: 5.3.0
>
> Attachments: OOZIE-2364-1.patch, OOZIE-2364-2.patch,
> OOZIE-2364-3.patch, OOZIE-2364-4.patch
>
>
> With new SLAService feature which uses different tables, the old SLA_EVENTS
> table is not used anymore. Need to remove all code related to it and drop
> that table as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Review Request 65636: OOZIE-3179 - Adding a configurable config-default.xml location to a workflow
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65636/#review217883 --- Ship it! Ship It! - Andras Salamon On Aug. 30, 2019, 8:53 p.m., Jason Phelps wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65636/ > --- > > (Updated Aug. 30, 2019, 8:53 p.m.) > > > Review request for oozie. > > > Repository: oozie-git > > > Description > --- > > Added another client property: `oozie.default.configuration.path` which > allows for a comma seperated list of HDFS XML property files to be applied to > the job. This is useful if multiple workflows have shared properties (i.e. > database names, table names for Sqoop or Hive, spark arguments, etc). This is > similar to the `oozie.libpath` property which allows user's to define common > paths workflows to load jars. > > > Diffs > - > > client/src/main/java/org/apache/oozie/client/OozieClient.java 1073cb924 > core/src/main/java/org/apache/oozie/command/wf/SubmitXCommand.java > 1352db91b > core/src/test/java/org/apache/oozie/command/wf/TestSubmitXCommand.java > 2cd263f60 > > > Diff: https://reviews.apache.org/r/65636/diff/4/ > > > Testing > --- > > > Thanks, > > Jason Phelps > >
[jira] [Commented] (OOZIE-3179) Adding a configurable config-default.xml location to a workflow
[ https://issues.apache.org/jira/browse/OOZIE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934326#comment-16934326 ] ASF subversion and git services commented on OOZIE-3179: Commit 6bdca1d1c043d6ad688f53602667ca5201af5a28 in oozie's branch refs/heads/master from Andras Salamon [ https://gitbox.apache.org/repos/asf?p=oozie.git;h=6bdca1d ] OOZIE-3179 Adding a configurable config-default.xml location to a workflow (jphelps via asalamon74) > Adding a configurable config-default.xml location to a workflow > --- > > Key: OOZIE-3179 > URL: https://issues.apache.org/jira/browse/OOZIE-3179 > Project: Oozie > Issue Type: Improvement >Reporter: Jason Phelps >Assignee: Jason Phelps >Priority: Major > Attachments: OOZIE-3179-1.patch, OOZIE-3179-2.patch, > OOZIE-3179-3.patch, OOZIE-3179-4.patch, OOZIE-3179-5.patch > > > Recently a customer requested a way to include several job.properties > variables to every workflow in their cluster. > Normally the config-default.xml file would be the best way to include a set > of variables, but it does not scale very well if that file needs to be > updated, as it would require it to be updated in every workflow application > path. > What I am requesting is another job.properties variable that can point to a > config-default.xml file, or a comma separated list of config-default.xml > files that will all be applied to the workflow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-3179) Adding a configurable config-default.xml location to a workflow
[ https://issues.apache.org/jira/browse/OOZIE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934328#comment-16934328 ] Andras Salamon commented on OOZIE-3179: --- Thanks [~jphelps], +1, committed to master. > Adding a configurable config-default.xml location to a workflow > --- > > Key: OOZIE-3179 > URL: https://issues.apache.org/jira/browse/OOZIE-3179 > Project: Oozie > Issue Type: Improvement >Reporter: Jason Phelps >Assignee: Jason Phelps >Priority: Major > Attachments: OOZIE-3179-1.patch, OOZIE-3179-2.patch, > OOZIE-3179-3.patch, OOZIE-3179-4.patch, OOZIE-3179-5.patch > > > Recently a customer requested a way to include several job.properties > variables to every workflow in their cluster. > Normally the config-default.xml file would be the best way to include a set > of variables, but it does not scale very well if that file needs to be > updated, as it would require it to be updated in every workflow application > path. > What I am requesting is another job.properties variable that can point to a > config-default.xml file, or a comma separated list of config-default.xml > files that will all be applied to the workflow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-2231) Upgrade curator to latest version 2.13.0
[ https://issues.apache.org/jira/browse/OOZIE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934331#comment-16934331 ] Hadoop QA commented on OOZIE-2231: -- PreCommit-OOZIE-Build started > Upgrade curator to latest version 2.13.0 > > > Key: OOZIE-2231 > URL: https://issues.apache.org/jira/browse/OOZIE-2231 > Project: Oozie > Issue Type: Bug > Components: HA >Reporter: Purshotam Shah >Priority: Blocker > Fix For: 5.3.0 > > Attachments: OOZIE-2231-00.patch, OOZIE-2231-01.patch, > OOZIE-2231-02.patch, OOZIE-2231-02.patch, OOZIE-2231-03.patch, > OOZIE-2231-04.patch, OOZIE-2231-05.patch, OOZIE-2231-06.patch > > > It have some fix related to InterProcessReadWriteLock, ChildReaper, > LeaderSelector which we use. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Failed: OOZIE-2231 PreCommit Build #1221
Jira: https://issues.apache.org/jira/browse/OOZIE-2231
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/1221/
###
## LAST 100 LINES OF THE CONSOLE
###
[...truncated 1.23 MB...]
[INFO] Apache Oozie Fluent Job Client . SUCCESS [ 0.481 s]
[INFO] Apache Oozie Server SUCCESS [ 3.512 s]
[INFO] Apache Oozie Distro SUCCESS [ 1.062 s]
[INFO] Apache Oozie ZooKeeper Security Tests .. SUCCESS [ 1.878 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 01:01 min
[INFO] Finished at: 2019-09-20T12:12:25Z
[INFO]
EXITVALUE 0
[TRACE] Modernizer output in HEAD verified and saved
Applying patch
Checking patch
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java...
error: while searching for:
package org.apache.oozie.action.hadoop;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Strings;
import com.google.common.collect.ImmutableList;
import com.google.common.io.Closeables;
import com.google.common.primitives.Ints;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.StringReader;
import java.net.ConnectException;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.UnknownHostException;
import java.nio.ByteBuffer;
import java.text.MessageFormat;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import org.apache.commons.io.IOUtils;
import org.apache.hadoop.conf.Configuration;
error: patch failed:
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java:19
error:
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java:
patch does not apply
Checking patch
core/src/main/java/org/apache/oozie/service/JvmPauseMonitorService.java...
error: while searching for:
import java.util.List;
import java.util.Map;
import java.util.Set;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.util.Daemon;
import org.apache.oozie.util.ConfigUtils;
import org.apache.oozie.util.Instrumentation;
import org.apache.oozie.util.XLog;
error: patch failed:
core/src/main/java/org/apache/oozie/service/JvmPauseMonitorService.java:29
error: core/src/main/java/org/apache/oozie/service/JvmPauseMonitorService.java:
patch does not apply
Checking patch docs/src/site/twiki/DG_QuickStart.twiki...
error: docs/src/site/twiki/DG_QuickStart.twiki: No such file or directory
Checking patch docs/src/site/twiki/ENG_Building.twiki...
error: docs/src/site/twiki/ENG_Building.twiki: No such file or directory
Checking patch pom.xml...
error: while searching for:
2.12
0.16.0
h2
core
1.4.3
1.6.1
error: patch failed: pom.xml:94
error: pom.xml: patch does not apply
Checking patch sharelib/pig/pom.xml...
Hunk #1 succeeded at 72 (offset 7 lines).
error: git diff header lacks filename information when removing 1 leading
pathname component (line 255)
Patch failed to apply to head of branch
Adding comment to JIRA
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0100
1990 0 100 199 0134 0:00:01 0:00:01 --:--:-- 134100
17740 1575 100 199803101 0:00:01 0:00:01 --:--:-- 904
{"self":"https://issues.apache.org/jira/rest/api/2/issue/12828994/comment/16934337","id":"16934337","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa&avatarId=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small&ownerId=hadoopqa&avatarId=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall&ownerId=hadoopqa&avatarId=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium&ownerId=hadoopqa&avatarId=10393"},"displayName":"Hadoop
QA","active":true,"timeZone":"Etc/UTC"},"body":"\nTesting JIRA
OOZIE-2231\n\nCleaning local git
workspace\n\n\n\n{color:red}-1{color} Patch failed
to apply to head of
branch\n\n\n","updateAuthor"
[jira] [Commented] (OOZIE-2231) Upgrade curator to latest version 2.13.0
[
https://issues.apache.org/jira/browse/OOZIE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934337#comment-16934337
]
Hadoop QA commented on OOZIE-2231:
--
Testing JIRA OOZIE-2231
Cleaning local git workspace
{color:red}-1{color} Patch failed to apply to head of branch
> Upgrade curator to latest version 2.13.0
>
>
> Key: OOZIE-2231
> URL: https://issues.apache.org/jira/browse/OOZIE-2231
> Project: Oozie
> Issue Type: Bug
> Components: HA
>Reporter: Purshotam Shah
>Priority: Blocker
> Fix For: 5.3.0
>
> Attachments: OOZIE-2231-00.patch, OOZIE-2231-01.patch,
> OOZIE-2231-02.patch, OOZIE-2231-02.patch, OOZIE-2231-03.patch,
> OOZIE-2231-04.patch, OOZIE-2231-05.patch, OOZIE-2231-06.patch
>
>
> It have some fix related to InterProcessReadWriteLock, ChildReaper,
> LeaderSelector which we use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] Subscription: Oozie Patch Available
Issue Subscription Filter: Oozie Patch Available (90 issues) Subscriber: ooziedaily Key Summary OOZIE-3536 oozie-main(pom.xml) plugin maven-javadoc-plugin upgrade version caused configuration can't find the Tag https://issues.apache.org/jira/browse/OOZIE-3536 OOZIE-3529 Oozie not supported for s3 as filesystem https://issues.apache.org/jira/browse/OOZIE-3529 OOZIE-3482 Fix bug in CoordSubmitXCommand#validateCoordinatorJob https://issues.apache.org/jira/browse/OOZIE-3482 OOZIE-3480 Add windowactionstatus metrics in DBLiteWorkflowStoreService https://issues.apache.org/jira/browse/OOZIE-3480 OOZIE-3461 CoordMaterializeTriggerService code cleanup https://issues.apache.org/jira/browse/OOZIE-3461 OOZIE-3449 Make spark-2 as the default profile https://issues.apache.org/jira/browse/OOZIE-3449 OOZIE-3447 Run test case in local : It shows oozie-hsqldb-orm.xml exception https://issues.apache.org/jira/browse/OOZIE-3447 OOZIE-3418 Upgrade to Guava 27 https://issues.apache.org/jira/browse/OOZIE-3418 OOZIE-3404 The env variable of SPARK_HOME needs to be set when running pySpark https://issues.apache.org/jira/browse/OOZIE-3404 OOZIE-3375 Can't use empty in coordinator https://issues.apache.org/jira/browse/OOZIE-3375 OOZIE-3367 Using && in EL expressions in oozie bundle.xml files generates parse errors https://issues.apache.org/jira/browse/OOZIE-3367 OOZIE-3366 Update workflow status and subworkflow status on suspend command https://issues.apache.org/jira/browse/OOZIE-3366 OOZIE-3364 Rerunning Oozie bundle jobs starts the coordinators in indeterminate order https://issues.apache.org/jira/browse/OOZIE-3364 OOZIE-3362 When killed, SSH action should kill the spawned processes on target host https://issues.apache.org/jira/browse/OOZIE-3362 OOZIE-3335 Cleanup parseFilter methods https://issues.apache.org/jira/browse/OOZIE-3335 OOZIE-3328 Create Hive compatibility action executor to run hive actions using beeline https://issues.apache.org/jira/browse/OOZIE-3328 OOZIE-3320 Oozie ShellAction should support absolute bash file path https://issues.apache.org/jira/browse/OOZIE-3320 OOZIE-3319 Log SSH action callback error output https://issues.apache.org/jira/browse/OOZIE-3319 OOZIE-3301 Update NOTICE file https://issues.apache.org/jira/browse/OOZIE-3301 OOZIE-3274 Remove slf4j https://issues.apache.org/jira/browse/OOZIE-3274 OOZIE-3266 Coord action rerun support RERUN_SKIP_NODES option https://issues.apache.org/jira/browse/OOZIE-3266 OOZIE-3256 refactor OozieCLI class https://issues.apache.org/jira/browse/OOZIE-3256 OOZIE-3254 [coordinator] LAST_ONLY and NONE execution modes: possible OutOfMemoryError when there are too many coordinator actions to materialize https://issues.apache.org/jira/browse/OOZIE-3254 OOZIE-3199 Let system property restriction configurable https://issues.apache.org/jira/browse/OOZIE-3199 OOZIE-3196 Authorization: restrict world readability by user https://issues.apache.org/jira/browse/OOZIE-3196 OOZIE-3170 Oozie Diagnostic Bundle tool fails with NPE due to missing service class https://issues.apache.org/jira/browse/OOZIE-3170 OOZIE-3137 Add support for log4j2 in HiveMain https://issues.apache.org/jira/browse/OOZIE-3137 OOZIE-3135 Configure log4j2 in SqoopMain https://issues.apache.org/jira/browse/OOZIE-3135 OOZIE-3091 Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper" https://issues.apache.org/jira/browse/OOZIE-3091 OOZIE-3071 Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0 https://issues.apache.org/jira/browse/OOZIE-3071 OOZIE-3063 Sanitizing variables that are part of openjpa.ConnectionProperties https://issues.apache.org/jira/browse/OOZIE-3063 OOZIE-3062 Set HADOOP_CONF_DIR for spark action https://issues.apache.org/jira/browse/OOZIE-3062 OOZIE-2952 Fix Findbugs warnings in oozie-sharelib-oozie https://issues.apache.org/jira/browse/OOZIE-2952 OOZIE-2834 ParameterVerifier logging non-useful warning for workflow definition https://issues.apache.org/jira/browse/OOZIE-2834 OOZIE-2812 SparkConfigurationService should support loading configurations from multiple Spark versions https://issues.apache.org/jira/browse/OOZIE-2812 OOZIE-2795 Create lib directory or symlink for Oozie CLI during packaging https://issues.apache.org/jira/browse/OOZIE-2795 OOZIE-2784 Include WEEK as a parameter in the Coordinator Expression Language Evaulator https://issues.apache.org/jira/browse/OOZIE-2784 OOZIE-2779 Mask Hive2 action Beeline JDBC password
