[jira] [Commented] (OOZIE-1319) LAST_ONLY in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013352#comment-14013352 ] Robert Kanter commented on OOZIE-1319: -- Test failures are unrelated. Lines that are too long are Named Queries. LAST_ONLY in execution control for coordinator job still runs all the actions --- Key: OOZIE-1319 URL: https://issues.apache.org/jira/browse/OOZIE-1319 Project: Oozie Issue Type: Bug Reporter: Bowen Zhang Assignee: Robert Kanter Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the top item from a LIFO query result, we do not discard or delete the remaining items from the result list. As a result, the next time execute() is invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY strategy will also execute all ready actions for a given coordinator job, making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (OOZIE-1860) Oozie job mapper launch fails due to null value returned from action file
[ https://issues.apache.org/jira/browse/OOZIE-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Osadchiy Artem updated OOZIE-1860: -- Attachment: OOZIE-1860.patch Oozie job mapper launch fails due to null value returned from action file - Key: OOZIE-1860 URL: https://issues.apache.org/jira/browse/OOZIE-1860 Project: Oozie Issue Type: Bug Affects Versions: 4.0.1 Reporter: Osadchiy Artem Fix For: trunk Attachments: OOZIE-1860.patch, action.log, oozie.log Certain oozie workflows fail at launch mapper stage when the id retrieved from the recovery action file returns a null value. Logs attached below -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (OOZIE-1860) Oozie job mapper launch fails due to null value returned from action file
[ https://issues.apache.org/jira/browse/OOZIE-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Osadchiy Artem updated OOZIE-1860: -- Attachment: (was: OOZIE-1860.patch) Oozie job mapper launch fails due to null value returned from action file - Key: OOZIE-1860 URL: https://issues.apache.org/jira/browse/OOZIE-1860 Project: Oozie Issue Type: Bug Affects Versions: 4.0.1 Reporter: Osadchiy Artem Fix For: trunk Attachments: OOZIE-1860.patch, action.log, oozie.log Certain oozie workflows fail at launch mapper stage when the id retrieved from the recovery action file returns a null value. Logs attached below -- This message was sent by Atlassian JIRA (v6.2#6252)
Build failed in Jenkins: oozie-trunk-precommit-build #1276
See https://builds.apache.org/job/oozie-trunk-precommit-build/1276/ -- [...truncated 9031 lines...] [INFO] share/lib already added, skipping [INFO] already added, skipping [INFO] share already added, skipping [INFO] share/lib already added, skipping [INFO] already added, skipping [INFO] share already added, skipping [INFO] share/lib already added, skipping [INFO] Building tar : https://builds.apache.org/job/oozie-trunk-precommit-build/ws/sharelib/target/oozie-sharelib-4.1.0-SNAPSHOT.tar.gz [INFO] already added, skipping [INFO] share already added, skipping [INFO] share/lib already added, skipping [INFO] already added, skipping [INFO] share already added, skipping [INFO] share/lib already added, skipping [INFO] already added, skipping [INFO] share already added, skipping [INFO] share/lib already added, skipping [INFO] already added, skipping [INFO] share already added, skipping [INFO] share/lib already added, skipping [INFO] already added, skipping [INFO] share already added, skipping [INFO] share/lib already added, skipping [INFO] already added, skipping [INFO] share already added, skipping [INFO] share/lib already added, skipping [INFO] [INFO] [INFO] Building Apache Oozie Tools 4.1.0-SNAPSHOT [INFO] [WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to local.repository (file:../../local.repository/trunk): No connector available to access repository local.repository (file:../../local.repository/trunk) of type legacy using the available factories WagonRepositoryConnectorFactory [WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to local.repository (file:../../local.repository/trunk): No connector available to access repository local.repository (file:../../local.repository/trunk) of type legacy using the available factories WagonRepositoryConnectorFactory [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ oozie-tools --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory https://builds.apache.org/job/oozie-trunk-precommit-build/ws/tools/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-tools --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ oozie-tools --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ oozie-tools --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-tools --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools --- [INFO] Building jar: https://builds.apache.org/job/oozie-trunk-precommit-build/ws/tools/target/oozie-tools-4.1.0-SNAPSHOT.jar [INFO] [INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools --- [INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml [WARNING] The following patterns were never triggered in this artifact exclusion filter: o '*:*:pom:*' [INFO] Copying files to https://builds.apache.org/job/oozie-trunk-precommit-build/ws/tools/target/oozie-tools-4.1.0-SNAPSHOT-tools [WARNING] Assembly file: https://builds.apache.org/job/oozie-trunk-precommit-build/ws/tools/target/oozie-tools-4.1.0-SNAPSHOT-tools is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment. [INFO] [INFO] [INFO] Building Apache Oozie MiniOozie 4.1.0-SNAPSHOT [INFO] [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ oozie-mini --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory https://builds.apache.org/job/oozie-trunk-precommit-build/ws/minitest/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini --- [INFO] No sources to compile [INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ oozie-mini --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ oozie-mini --- [INFO] Nothing to compile - all classes are up to date
[jira] [Commented] (OOZIE-1860) Oozie job mapper launch fails due to null value returned from action file
[
https://issues.apache.org/jira/browse/OOZIE-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013413#comment-14013413
]
Hadoop QA commented on OOZIE-1860:
--
Testing JIRA OOZIE-1860
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/1276/
Oozie job mapper launch fails due to null value returned from action file
-
Key: OOZIE-1860
URL: https://issues.apache.org/jira/browse/OOZIE-1860
Project: Oozie
Issue Type: Bug
Affects Versions: 4.0.1
Reporter: Osadchiy Artem
Fix For: trunk
Attachments: OOZIE-1860.patch, action.log, oozie.log
Certain oozie workflows fail at launch mapper stage when the id retrieved
from the recovery action file returns a null value.
Logs attached below
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: Review Request 19929: OOZIE-1685: Oozie doesn’t process correctly workflows with a non-default name node
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19929/ --- (Updated May 30, 2014, 2:32 p.m.) Review request for oozie. Changes --- Fixed according to CR comments Repository: oozie-git Description --- When name-node element in Oozie workflow specifies a name node different from the default one (specified in core-site.xml), the following functionality doesn’t work properly: ?Location of libraries specified via oozie.service.WorkflowAppService.system.libpath. Oozie first (during launcher configuration) tries to locate them using name node specified by the name-node element, but later during job submission it expects this path to be under the default Oozie name node ?Processing of the job-xml element if job xml is specified via absolute path. Oozie tries locate it under the default Oozie name node instead of the name-node specified in action. Specifying non-default name node makes a lot of sense in Azure environment, because it allows to submit the same job to different Hadoop clusters. Diffs (updated) - core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 40add2c core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java bb68b0e core/src/main/java/org/apache/oozie/service/ShareLibService.java 353b382 core/src/main/java/org/apache/oozie/util/JobUtils.java 135b096 core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java bc2c1b6 core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 390ad3f core/src/test/java/org/apache/oozie/test/XFsTestCase.java 18cb742 core/src/test/java/org/apache/oozie/test/XTestCase.java 1536927 docs/src/site/twiki/WorkflowFunctionalSpec.twiki f7590d0 Diff: https://reviews.apache.org/r/19929/diff/ Testing --- On deployed Hadoop cluster. Two tests were added. Thanks, Benjamin Zhitomirsky
[jira] [Updated] (OOZIE-1685) Oozie doesn’t process correctly workflows with a non-default name node
[ https://issues.apache.org/jira/browse/OOZIE-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Zhitomirsky updated OOZIE-1685: Attachment: oozie-1685.5.patch CR comments applied Oozie doesn’t process correctly workflows with a non-default name node -- Key: OOZIE-1685 URL: https://issues.apache.org/jira/browse/OOZIE-1685 Project: Oozie Issue Type: Bug Components: core Affects Versions: trunk, 3.3.2, 4.0.0 Environment: Any Reporter: Benjamin Zhitomirsky Fix For: trunk Attachments: Design of the fix OOZIE-1685-rev1.docx, oozie-1685-trunk.patch, oozie-1685-trunk.patch, oozie-1685-trunk.patch, oozie-1685.1.patch, oozie-1685.2.patch, oozie-1685.3.patch, oozie-1685.4.patch, oozie-1685.5.patch Original Estimate: 168h Remaining Estimate: 168h When name-node element in Oozie workflow specifies a name node different from the default one (specified in core-site.xml), the following functionality doesn’t work properly: - Location of libraries specified via oozie.service.WorkflowAppService.system.libpath. Oozie first (during launcher configuration) tries to locate them using name node specified by the name-node element, but later during job submission it expects this path to be under the default Oozie name node - Processing of the job-xml element if job xml is specified via absolute path. Oozie tries locate it under the default Oozie name node instead of the name-node specified in action. Specifying non-default name node makes a lot of sense in Azure environment, because it allows to submit the same job to different Hadoop clusters. I will submit a fix for CR soon. Please refer attached short document for more information. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Reopened] (OOZIE-1703) User should be able to set coord end-time before start time
[ https://issues.apache.org/jira/browse/OOZIE-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah reopened OOZIE-1703: --- User should be able to set coord end-time before start time --- Key: OOZIE-1703 URL: https://issues.apache.org/jira/browse/OOZIE-1703 Project: Oozie Issue Type: Bug Reporter: Purshotam Shah Assignee: Purshotam Shah Fix For: trunk Attachments: OOZIE-1703-V5.patch, OOZIE-1703-V6.patch, OOZIE-1703-V7.patch, OOZIE-1703.patch This is one of the important use-case in case of versioning. User can set end date of old coord before start date( if coord is supposed to run in future), so that it doesn't run and user can safely upload new version. We should also lookahead at created actions that become invalid because of the new end time. These actions should be deleted from DB. We should also update the status of bundle. If it's in PREP and new end-date is before kick off time, then job status will be set to SUCCEEDED. User can again change the end date to future date, the status will be changed to RUNNING. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1703) User should be able to set coord end-time before start time
[ https://issues.apache.org/jira/browse/OOZIE-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013931#comment-14013931 ] Purshotam Shah commented on OOZIE-1703: --- We noticed some issue while testing internally. If we set endtime multiple times, coord remains in running state. Attaching amendment patch. User should be able to set coord end-time before start time --- Key: OOZIE-1703 URL: https://issues.apache.org/jira/browse/OOZIE-1703 Project: Oozie Issue Type: Bug Reporter: Purshotam Shah Assignee: Purshotam Shah Fix For: trunk Attachments: OOZIE-1703-V5.patch, OOZIE-1703-V6.patch, OOZIE-1703-V7.patch, OOZIE-1703.patch This is one of the important use-case in case of versioning. User can set end date of old coord before start date( if coord is supposed to run in future), so that it doesn't run and user can safely upload new version. We should also lookahead at created actions that become invalid because of the new end time. These actions should be deleted from DB. We should also update the status of bundle. If it's in PREP and new end-date is before kick off time, then job status will be set to SUCCEEDED. User can again change the end date to future date, the status will be changed to RUNNING. -- This message was sent by Atlassian JIRA (v6.2#6252)
Build failed in Jenkins: oozie-trunk-precommit-build #1277
See https://builds.apache.org/job/oozie-trunk-precommit-build/1277/ -- [...truncated 9694 lines...] [INFO] share/lib already added, skipping [INFO] already added, skipping [INFO] share already added, skipping [INFO] share/lib already added, skipping [INFO] already added, skipping [INFO] share already added, skipping [INFO] share/lib already added, skipping [INFO] already added, skipping [INFO] share already added, skipping [INFO] share/lib already added, skipping [INFO] already added, skipping [INFO] share already added, skipping [INFO] share/lib already added, skipping [INFO] already added, skipping [INFO] share already added, skipping [INFO] share/lib already added, skipping [INFO] [INFO] [INFO] Building Apache Oozie Tools 4.1.0-SNAPSHOT [INFO] [WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to local.repository (file:../../local.repository/trunk): No connector available to access repository local.repository (file:../../local.repository/trunk) of type legacy using the available factories WagonRepositoryConnectorFactory [WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to local.repository (file:../../local.repository/trunk): No connector available to access repository local.repository (file:../../local.repository/trunk) of type legacy using the available factories WagonRepositoryConnectorFactory [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ oozie-tools --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory https://builds.apache.org/job/oozie-trunk-precommit-build/ws/tools/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-tools --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ oozie-tools --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ oozie-tools --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-tools --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools --- [INFO] Building jar: https://builds.apache.org/job/oozie-trunk-precommit-build/ws/tools/target/oozie-tools-4.1.0-SNAPSHOT.jar [INFO] [INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools --- [INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml [WARNING] The following patterns were never triggered in this artifact exclusion filter: o '*:*:pom:*' [INFO] Copying files to https://builds.apache.org/job/oozie-trunk-precommit-build/ws/tools/target/oozie-tools-4.1.0-SNAPSHOT-tools [WARNING] Assembly file: https://builds.apache.org/job/oozie-trunk-precommit-build/ws/tools/target/oozie-tools-4.1.0-SNAPSHOT-tools is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment. [INFO] [INFO] [INFO] Building Apache Oozie MiniOozie 4.1.0-SNAPSHOT [INFO] [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ oozie-mini --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory https://builds.apache.org/job/oozie-trunk-precommit-build/ws/minitest/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini --- [INFO] No sources to compile [INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ oozie-mini --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ oozie-mini --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-mini --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-mini --- [WARNING] JAR will be empty - no content was marked for inclusion! [INFO] Building jar: https://builds.apache.org/job/oozie-trunk-precommit-build/ws/minitest/target/oozie-mini-4.1.0-SNAPSHOT.jar [INFO] [INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-mini ---
[jira] [Commented] (OOZIE-1685) Oozie doesn’t process correctly workflows with a non-default name node
[
https://issues.apache.org/jira/browse/OOZIE-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013983#comment-14013983
]
Hadoop QA commented on OOZIE-1685:
--
Testing JIRA OOZIE-1685
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 4 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1452
.Tests failed: 3
.Tests errors: 2
.The patch failed the following testcases:
.
testMaxMatThrottleNotPickedMultipleJobs(org.apache.oozie.service.TestCoordMaterializeTriggerService)
. testSingleRecord(org.apache.oozie.servlet.TestBulkMonitorWebServiceAPI)
.
testCoordDefUnsupportedChange(org.apache.oozie.command.coord.TestCoordUpdateXCommand)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/1277/
Oozie doesn’t process correctly workflows with a non-default name node
--
Key: OOZIE-1685
URL: https://issues.apache.org/jira/browse/OOZIE-1685
Project: Oozie
Issue Type: Bug
Components: core
Affects Versions: trunk
Environment: Any
Reporter: Benjamin Zhitomirsky
Fix For: trunk
Attachments: Design of the fix OOZIE-1685-rev1.docx,
oozie-1685-trunk.patch, oozie-1685-trunk.patch, oozie-1685-trunk.patch,
oozie-1685.1.patch, oozie-1685.2.patch, oozie-1685.3.patch,
oozie-1685.4.patch, oozie-1685.5.patch
Original Estimate: 168h
Remaining Estimate: 168h
When name-node element in Oozie workflow specifies a name node different
from the default one (specified in core-site.xml), the following
functionality doesn’t work properly:
- Location of libraries specified via
oozie.service.WorkflowAppService.system.libpath. Oozie first (during launcher
configuration) tries to locate them using name node specified by the
name-node element, but later during job submission it expects this path to
be under the default Oozie name node
- Processing of the job-xml element if job xml is specified via absolute
path. Oozie tries locate it under the default Oozie name node instead of the
name-node specified in action.
Specifying non-default name node makes a lot of sense in Azure environment,
because it allows to submit the same job to different Hadoop clusters.
I will submit a fix for CR soon. Please refer attached short document for
more information.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1862) Add hadoop token file location for Hive/Tez jobs
[ https://issues.apache.org/jira/browse/OOZIE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014052#comment-14014052 ] Bowen Zhang commented on OOZIE-1862: committed to trunk. Thanks Venkat. Add hadoop token file location for Hive/Tez jobs Key: OOZIE-1862 URL: https://issues.apache.org/jira/browse/OOZIE-1862 Project: Oozie Issue Type: Bug Reporter: Venkat Ranganathan Assignee: Venkat Ranganathan Fix For: 4.1.0 Attachments: OOZIE-1862.diff We need to set the configuration parameter tez.credentials.path to the same hadoop token file location as we set for mapreduce.job.credentials.binary This is needed to run Hive/Tez jobs -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1817) Oozie timers are not biased
[ https://issues.apache.org/jira/browse/OOZIE-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014093#comment-14014093 ] Mona Chitnis commented on OOZIE-1817: - I agree we can get rid of the meters. they are not useful at the moment since our logging levels are not quite perfect, and also errors/warns here can be due to myriad reasons - user definitions etc, and do not necessarily pinpoint to system issues. the JSON attachment show useful metrics around locks and times they were held. That will be useful. taking a second look at code now Oozie timers are not biased --- Key: OOZIE-1817 URL: https://issues.apache.org/jira/browse/OOZIE-1817 Project: Oozie Issue Type: Improvement Components: monitoring Affects Versions: 4.0.0 Reporter: Gilad Wolff Assignee: Robert Kanter Attachments: OOZIE-1817.patch, OOZIE-1817.patch, OOZIE-1817.patch, OOZIE-1817.patch, OOZIE-1817.patch, OOZIE-1817.patch, OOZIE-1817.patch, counters.jpg, gauges.jpg, histograms.jpg, json.txt, meters.jpg, timers.jpg Oozie timers are not biased, that is, the statistical metrics they expose are over the run-time of the Oozie server instead of a window of time. This makes them not very useful especially after the server has been running for a while (codehale has very efficient and easy to use biased histograms that can be used instead). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (OOZIE-1863) Bundle submit should fail without submitting any coord if one of coord has issue
Purshotam Shah created OOZIE-1863: - Summary: Bundle submit should fail without submitting any coord if one of coord has issue Key: OOZIE-1863 URL: https://issues.apache.org/jira/browse/OOZIE-1863 Project: Oozie Issue Type: Bug Reporter: Purshotam Shah -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (OOZIE-1813) Add service to report/kill rogue bundles and coordinator jobs
[ https://issues.apache.org/jira/browse/OOZIE-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-1813: -- Attachment: OOZIE-1813-V7.patch Add service to report/kill rogue bundles and coordinator jobs - Key: OOZIE-1813 URL: https://issues.apache.org/jira/browse/OOZIE-1813 Project: Oozie Issue Type: Bug Reporter: Purshotam Shah Assignee: Purshotam Shah Attachments: OOZIE-1813-V2.patch, OOZIE-1813-V3.patch, OOZIE-1813-V4.patch, OOZIE-1813-V5.patch, OOZIE-1813-V6.patch, OOZIE-1813-V7.patch People leave their test coordinator and bundle jobs without ever killing them and they just eat up resources heavily. We should have a service which periodically check for abandoned coords and report/kill them. We can add multiple logic to this like ( number of consecutive failed/timedout action, total number of failed/timedout action). To start with if number of coord action with failed/timedout status defined value, then coord is considered to be rogue. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (OOZIE-1863) Bundle submit should fail without submitting any coord if one of coord has issue
[ https://issues.apache.org/jira/browse/OOZIE-1863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-1863: -- Description: Currently, bundle submit command queues coord submit for all coords. StatusTransitService, which normally runs after 5 min check the status of each submitted coord and if any of them has failed, it kills running bundle. This approach has two issue a. Bundle is status is shown as killed ( few submitted coord is also killed). It will be difficult for user to find out the root cause. b. Few of the coord will be in running state for sometime and might corrupt or produce stale data. Bundle submit should dryrun coord before queuing submit command. We can further improve the bundle job submission logic. We can call CoordSubmitXCommand synchronously and if any coord submit fails ( bcz of DB or other evn issues), kill other submitted coord jobs. Bundle submit should fail without submitting any coord if one of coord has issue Key: OOZIE-1863 URL: https://issues.apache.org/jira/browse/OOZIE-1863 Project: Oozie Issue Type: Bug Reporter: Purshotam Shah Currently, bundle submit command queues coord submit for all coords. StatusTransitService, which normally runs after 5 min check the status of each submitted coord and if any of them has failed, it kills running bundle. This approach has two issue a. Bundle is status is shown as killed ( few submitted coord is also killed). It will be difficult for user to find out the root cause. b. Few of the coord will be in running state for sometime and might corrupt or produce stale data. Bundle submit should dryrun coord before queuing submit command. We can further improve the bundle job submission logic. We can call CoordSubmitXCommand synchronously and if any coord submit fails ( bcz of DB or other evn issues), kill other submitted coord jobs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1862) Add hadoop token file location for Hive/Tez jobs
[ https://issues.apache.org/jira/browse/OOZIE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014191#comment-14014191 ] Rohini Palaniswamy commented on OOZIE-1862: --- PigMain? Add hadoop token file location for Hive/Tez jobs Key: OOZIE-1862 URL: https://issues.apache.org/jira/browse/OOZIE-1862 Project: Oozie Issue Type: Bug Reporter: Venkat Ranganathan Assignee: Venkat Ranganathan Fix For: 4.1.0 Attachments: OOZIE-1862.diff We need to set the configuration parameter tez.credentials.path to the same hadoop token file location as we set for mapreduce.job.credentials.binary This is needed to run Hive/Tez jobs -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1862) Add hadoop token file location for Hive/Tez jobs
[ https://issues.apache.org/jira/browse/OOZIE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014210#comment-14014210 ] Venkat Ranganathan commented on OOZIE-1862: --- Isn't Pig/Tez targeted for Pig 0.14 and not available? Yes, we have to do the same when Pig with Tez is available. Add hadoop token file location for Hive/Tez jobs Key: OOZIE-1862 URL: https://issues.apache.org/jira/browse/OOZIE-1862 Project: Oozie Issue Type: Bug Reporter: Venkat Ranganathan Assignee: Venkat Ranganathan Fix For: 4.1.0 Attachments: OOZIE-1862.diff We need to set the configuration parameter tez.credentials.path to the same hadoop token file location as we set for mapreduce.job.credentials.binary This is needed to run Hive/Tez jobs -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 21278: OOZIE-1817: Oozie timers are not biased
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21278/#review44419 --- some more comments in addition to the couple before core/src/main/java/org/apache/oozie/servlet/V2AdminServlet.java https://reviews.apache.org/r/21278/#comment78796 where is the disabling of instrumentation endpoint taking place? core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java https://reviews.apache.org/r/21278/#comment78797 nitpick- typo retrieve core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java https://reviews.apache.org/r/21278/#comment78798 why use lock when guages is a concurrenthashmap? core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java https://reviews.apache.org/r/21278/#comment78799 same as above - Mona Chitnis On May 22, 2014, 11:27 p.m., Robert Kanter wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21278/ --- (Updated May 22, 2014, 11:27 p.m.) Review request for oozie. Bugs: OOZIE-1817 https://issues.apache.org/jira/browse/OOZIE-1817 Repository: oozie-git Description --- See https://issues.apache.org/jira/browse/OOZIE-1817?focusedCommentId=13993838page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13993838 Diffs - client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 97fd465 core/pom.xml c935dd7 core/src/main/java/org/apache/oozie/command/Command.java d3f1011 core/src/main/java/org/apache/oozie/command/XCommand.java b712ce0 core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java e8667c1 core/src/main/java/org/apache/oozie/command/coord/CoordActionMaterializeCommand.java 6962fb2 core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 57cbb34 core/src/main/java/org/apache/oozie/service/InstrumentationService.java 80437b1 core/src/main/java/org/apache/oozie/service/MetricsInstrumentationService.java PRE-CREATION core/src/main/java/org/apache/oozie/service/Services.java 5feac7b core/src/main/java/org/apache/oozie/service/XLogService.java 403a089 core/src/main/java/org/apache/oozie/servlet/BaseAdminServlet.java 29d7bd6 core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 5c05acd core/src/main/java/org/apache/oozie/servlet/V0AdminServlet.java 97bd81c core/src/main/java/org/apache/oozie/servlet/V1AdminServlet.java a47f737 core/src/main/java/org/apache/oozie/servlet/V2AdminServlet.java 002a367 core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java PRE-CREATION core/src/main/java/org/apache/oozie/util/XLog.java 21e00c0 core/src/main/resources/oozie-default.xml 9788daf core/src/test/java/org/apache/oozie/service/TestInstrumentationService.java b5206cf core/src/test/java/org/apache/oozie/service/TestMetricsInstrumentationService.java PRE-CREATION core/src/test/java/org/apache/oozie/util/TestInstrumentation.java 6040b47 core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java PRE-CREATION docs/src/site/twiki/AG_Install.twiki e343d7e docs/src/site/twiki/WebServicesAPI.twiki 5769924 pom.xml b5e0e4e webapp/src/main/webapp/index.jsp 3c7ffe5 webapp/src/main/webapp/oozie-console.js 764d888 Diff: https://reviews.apache.org/r/21278/diff/ Testing --- Unit tests, also verified in a cluster Thanks, Robert Kanter
[jira] [Created] (OOZIE-1864) Improve chid job id aggregation logic
Purshotam Shah created OOZIE-1864: - Summary: Improve chid job id aggregation logic Key: OOZIE-1864 URL: https://issues.apache.org/jira/browse/OOZIE-1864 Project: Oozie Issue Type: Bug Reporter: Purshotam Shah -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (OOZIE-1864) Improve chid job id aggregation logic
[ https://issues.apache.org/jira/browse/OOZIE-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-1864: -- Description: Current chid job id aggregation logic Once launcher job complete submitting child job (jobs in case on pig), it writes jobID to file. From Oozie server side, we collect childID in two ways 1. As soon as we submit launcher jobs, we check if launcher job terminated or not. If it's terminated, we read child-id from file and populated to DB. And once kill command is issued we kill all child jobs. 2. We have a timer task (ActionCheckerService) which keeps on checking the status of all running actions and if launcher job is terminated, it's update the DB with childIDs. Jobend notification is rejected if action is not running. Assume that launcher is killed after it has submitted child job. Child job will never be killed. To fix this, we should do following things. 1. If oozie receives job end notification and if launcher job is killed, collect all child job and kill them if they are not killed. 2. Have a better way logic to collect child job id. Launcher job can call callbackServlet ( may be periodically) to update child job ids. This could be useful in pig jobs. In current scenario we report child jobs job only when launcher job completes. Improve chid job id aggregation logic - Key: OOZIE-1864 URL: https://issues.apache.org/jira/browse/OOZIE-1864 Project: Oozie Issue Type: Bug Reporter: Purshotam Shah Current chid job id aggregation logic Once launcher job complete submitting child job (jobs in case on pig), it writes jobID to file. From Oozie server side, we collect childID in two ways 1. As soon as we submit launcher jobs, we check if launcher job terminated or not. If it's terminated, we read child-id from file and populated to DB. And once kill command is issued we kill all child jobs. 2. We have a timer task (ActionCheckerService) which keeps on checking the status of all running actions and if launcher job is terminated, it's update the DB with childIDs. Jobend notification is rejected if action is not running. Assume that launcher is killed after it has submitted child job. Child job will never be killed. To fix this, we should do following things. 1. If oozie receives job end notification and if launcher job is killed, collect all child job and kill them if they are not killed. 2. Have a better way logic to collect child job id. Launcher job can call callbackServlet ( may be periodically) to update child job ids. This could be useful in pig jobs. In current scenario we report child jobs job only when launcher job completes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) LAST_ONLY in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014224#comment-14014224 ] Bowen Zhang commented on OOZIE-1319: one last issue. When computing nextNominalTime, you need to make sure nextNominalTime is end time of the coord job. Otherwise, you may wrongly skip the last action of the coord job. After that, +1. LAST_ONLY in execution control for coordinator job still runs all the actions --- Key: OOZIE-1319 URL: https://issues.apache.org/jira/browse/OOZIE-1319 Project: Oozie Issue Type: Bug Reporter: Bowen Zhang Assignee: Robert Kanter Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the top item from a LIFO query result, we do not discard or delete the remaining items from the result list. As a result, the next time execute() is invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY strategy will also execute all ready actions for a given coordinator job, making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) LAST_ONLY in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014226#comment-14014226
]
Robert Kanter commented on OOZIE-1319:
--
It already checks that:
{code:java}
// If the next nominal time is after the job's end time, then this is the last
action, so return null
if (nextNominalTime.after(coordJob.getEndTime())) {
nextNominalTime = null;
}
return nextNominalTime;
{code}
LAST_ONLY in execution control for coordinator job still runs all the
actions
---
Key: OOZIE-1319
URL: https://issues.apache.org/jira/browse/OOZIE-1319
Project: Oozie
Issue Type: Bug
Reporter: Bowen Zhang
Assignee: Robert Kanter
Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
oozie-1319.patch
In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
top item from a LIFO query result, we do not discard or delete the
remaining items from the result list. As a result, the next time execute() is
invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
strategy will also execute all ready actions for a given coordinator job,
making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (OOZIE-1864) Improve chid job id aggregation logic
[ https://issues.apache.org/jira/browse/OOZIE-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-1864: -- Description: Improve chid job id aggregation logic Current chid job id aggregation logic Once launcher job complete submitting child job (jobs in case on pig), it writes jobID to file. From Oozie server side, we collect childID in two ways 1. As soon as we submit launcher jobs, we check if launcher job terminated or not. If it's terminated, we read child-id from file and populated to DB. And once kill command is issued we kill all child jobs. 2. We have a timer task (ActionCheckerService) which keeps on checking the status of all running actions and if launcher job is terminated, it's update the DB with childIDs. Jobend notification is rejected if action is not running. Assume that launcher is killed after it has submitted child job. Child job will never be killed. To fix this, we should do following things. 1. If oozie receives job end notification and if launcher job is killed, collect all child job and kill them if they are not killed. 2. Have a better way logic to collect child job id. Launcher job can call callbackServlet ( may be periodically) to update child job ids. This could be useful in pig jobs. In current scenario we report child jobs job only when launcher job completes. was: Current chid job id aggregation logic Once launcher job complete submitting child job (jobs in case on pig), it writes jobID to file. From Oozie server side, we collect childID in two ways 1. As soon as we submit launcher jobs, we check if launcher job terminated or not. If it's terminated, we read child-id from file and populated to DB. And once kill command is issued we kill all child jobs. 2. We have a timer task (ActionCheckerService) which keeps on checking the status of all running actions and if launcher job is terminated, it's update the DB with childIDs. Jobend notification is rejected if action is not running. Assume that launcher is killed after it has submitted child job. Child job will never be killed. To fix this, we should do following things. 1. If oozie receives job end notification and if launcher job is killed, collect all child job and kill them if they are not killed. 2. Have a better way logic to collect child job id. Launcher job can call callbackServlet ( may be periodically) to update child job ids. This could be useful in pig jobs. In current scenario we report child jobs job only when launcher job completes. Improve chid job id aggregation logic - Key: OOZIE-1864 URL: https://issues.apache.org/jira/browse/OOZIE-1864 Project: Oozie Issue Type: Bug Reporter: Purshotam Shah Improve chid job id aggregation logic Current chid job id aggregation logic Once launcher job complete submitting child job (jobs in case on pig), it writes jobID to file. From Oozie server side, we collect childID in two ways 1. As soon as we submit launcher jobs, we check if launcher job terminated or not. If it's terminated, we read child-id from file and populated to DB. And once kill command is issued we kill all child jobs. 2. We have a timer task (ActionCheckerService) which keeps on checking the status of all running actions and if launcher job is terminated, it's update the DB with childIDs. Jobend notification is rejected if action is not running. Assume that launcher is killed after it has submitted child job. Child job will never be killed. To fix this, we should do following things. 1. If oozie receives job end notification and if launcher job is killed, collect all child job and kill them if they are not killed. 2. Have a better way logic to collect child job id. Launcher job can call callbackServlet ( may be periodically) to update child job ids. This could be useful in pig jobs. In current scenario we report child jobs job only when launcher job completes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1813) Add service to report/kill rogue bundles and coordinator jobs
[
https://issues.apache.org/jira/browse/OOZIE-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014236#comment-14014236
]
Hadoop QA commented on OOZIE-1813:
--
Testing JIRA OOZIE-1813
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 2 line(s) longer than 132
characters
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1454
.Tests failed: 4
.Tests errors: 0
.The patch failed the following testcases:
.
testConcurrencyReachedAndChooseNextEligible(org.apache.oozie.service.TestCallableQueueService)
.
testBundleEngineKill(org.apache.oozie.servlet.TestV1JobServletBundleEngine)
.
testActionInputCheckLatestActionCreationTime(org.apache.oozie.command.coord.TestCoordActionInputCheckXCommand)
.
testTimeOutWithUnresolvedMissingDependencies(org.apache.oozie.command.coord.TestCoordPushDependencyCheckXCommand)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/1278/
Add service to report/kill rogue bundles and coordinator jobs
-
Key: OOZIE-1813
URL: https://issues.apache.org/jira/browse/OOZIE-1813
Project: Oozie
Issue Type: Bug
Reporter: Purshotam Shah
Assignee: Purshotam Shah
Attachments: OOZIE-1813-V2.patch, OOZIE-1813-V3.patch,
OOZIE-1813-V4.patch, OOZIE-1813-V5.patch, OOZIE-1813-V6.patch,
OOZIE-1813-V7.patch
People leave their test coordinator and bundle jobs without ever killing them
and they just eat up resources heavily. We should have a service which
periodically check for abandoned coords and report/kill them.
We can add multiple logic to this like ( number of consecutive
failed/timedout action, total number of failed/timedout action).
To start with if number of coord action with failed/timedout status defined
value, then coord is considered to be rogue.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Build failed in Jenkins: oozie-trunk-precommit-build #1278
See https://builds.apache.org/job/oozie-trunk-precommit-build/1278/changes Changes: [bowenzhangusa] OOZIE-1862 Add hadoop token file location for Hive/Tez jobs (venkatnrangan via bzhang) -- [...truncated 9612 lines...] [INFO] already added, skipping [INFO] share already added, skipping [INFO] share/lib already added, skipping [INFO] already added, skipping [INFO] share already added, skipping [INFO] share/lib already added, skipping [INFO] already added, skipping [INFO] share already added, skipping [INFO] share/lib already added, skipping [INFO] already added, skipping [INFO] share already added, skipping [INFO] share/lib already added, skipping [INFO] already added, skipping [INFO] share already added, skipping [INFO] share/lib already added, skipping [INFO] [INFO] [INFO] Building Apache Oozie Tools 4.1.0-SNAPSHOT [INFO] [WARNING] Failure to transfer asm:asm/maven-metadata.xml from file:../../local.repository/trunk was cached in the local repository, resolution will not be reattempted until the update interval of local.repository has elapsed or updates are forced. Original error: Could not transfer metadata asm:asm/maven-metadata.xml from/to local.repository (file:../../local.repository/trunk): No connector available to access repository local.repository (file:../../local.repository/trunk) of type legacy using the available factories WagonRepositoryConnectorFactory [WARNING] Failure to transfer asm:asm/maven-metadata.xml from file:../../local.repository/trunk was cached in the local repository, resolution will not be reattempted until the update interval of local.repository has elapsed or updates are forced. Original error: Could not transfer metadata asm:asm/maven-metadata.xml from/to local.repository (file:../../local.repository/trunk): No connector available to access repository local.repository (file:../../local.repository/trunk) of type legacy using the available factories WagonRepositoryConnectorFactory [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ oozie-tools --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory https://builds.apache.org/job/oozie-trunk-precommit-build/ws/tools/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-tools --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ oozie-tools --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ oozie-tools --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-tools --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools --- [INFO] Building jar: https://builds.apache.org/job/oozie-trunk-precommit-build/ws/tools/target/oozie-tools-4.1.0-SNAPSHOT.jar [INFO] [INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools --- [INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml [WARNING] The following patterns were never triggered in this artifact exclusion filter: o '*:*:pom:*' [INFO] Copying files to https://builds.apache.org/job/oozie-trunk-precommit-build/ws/tools/target/oozie-tools-4.1.0-SNAPSHOT-tools [WARNING] Assembly file: https://builds.apache.org/job/oozie-trunk-precommit-build/ws/tools/target/oozie-tools-4.1.0-SNAPSHOT-tools is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment. [INFO] [INFO] [INFO] Building Apache Oozie MiniOozie 4.1.0-SNAPSHOT [INFO] [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ oozie-mini --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory https://builds.apache.org/job/oozie-trunk-precommit-build/ws/minitest/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini --- [INFO] No sources to compile [INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ oozie-mini --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 3 resources [INFO] [INFO] ---
[jira] [Commented] (OOZIE-1319) LAST_ONLY in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014238#comment-14014238 ] Robert Kanter commented on OOZIE-1319: -- Thanks for the reviews and catching all of those edge cases. I'm glad we can finally get this in :) LAST_ONLY in execution control for coordinator job still runs all the actions --- Key: OOZIE-1319 URL: https://issues.apache.org/jira/browse/OOZIE-1319 Project: Oozie Issue Type: Bug Reporter: Bowen Zhang Assignee: Robert Kanter Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the top item from a LIFO query result, we do not discard or delete the remaining items from the result list. As a result, the next time execute() is invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY strategy will also execute all ready actions for a given coordinator job, making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) LAST_ONLY in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014248#comment-14014248 ] Bowen Zhang commented on OOZIE-1319: my bad, didn't see it. LAST_ONLY in execution control for coordinator job still runs all the actions --- Key: OOZIE-1319 URL: https://issues.apache.org/jira/browse/OOZIE-1319 Project: Oozie Issue Type: Bug Reporter: Bowen Zhang Assignee: Robert Kanter Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the top item from a LIFO query result, we do not discard or delete the remaining items from the result list. As a result, the next time execute() is invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY strategy will also execute all ready actions for a given coordinator job, making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (OOZIE-1812) Bundle status is always in RUNNING if one of the action status is in PREP
[ https://issues.apache.org/jira/browse/OOZIE-1812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-1812: -- Attachment: OOZIE-1812-V1.patch Bundle status is always in RUNNING if one of the action status is in PREP - Key: OOZIE-1812 URL: https://issues.apache.org/jira/browse/OOZIE-1812 Project: Oozie Issue Type: Bug Affects Versions: 4.0.1 Reporter: Rohini Palaniswamy Assignee: Rohini Palaniswamy Attachments: OOZIE-1812-V1.patch Even if there are FAILED or KILLED coordinator jobs, bundle action status remains in RUNNING because the checkPrepStatus either does PREP or RUNNING and does not take terminal states into account. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 21278: OOZIE-1817: Oozie timers are not biased
On May 30, 2014, 9 p.m., Mona Chitnis wrote: core/src/main/java/org/apache/oozie/servlet/V2AdminServlet.java, line 53 https://reviews.apache.org/r/21278/diff/6/?file=588573#file588573line53 where is the disabling of instrumentation endpoint taking place? In V2AdminServlet, if (metricsInstrumentation == null) sendMetricsResponse(...) will response when a SC_SERVICE_UNAVAILABLE; sentInstrumentationReponse(...) will do the opposite. On May 30, 2014, 9 p.m., Mona Chitnis wrote: core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java, line 165 https://reviews.apache.org/r/21278/diff/6/?file=588574#file588574line165 why use lock when guages is a concurrenthashmap? Besides adding to the gauges hashmap, addVariable(...) also adds and/or removes the gauge to/from the metricsRegistry so that entire set of operations should be thread safe On May 30, 2014, 9 p.m., Mona Chitnis wrote: core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java, line 221 https://reviews.apache.org/r/21278/diff/6/?file=588574#file588574line221 same as above Same as before, but for histograms instead of gauges. - Robert --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21278/#review44419 --- On May 22, 2014, 11:27 p.m., Robert Kanter wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21278/ --- (Updated May 22, 2014, 11:27 p.m.) Review request for oozie. Bugs: OOZIE-1817 https://issues.apache.org/jira/browse/OOZIE-1817 Repository: oozie-git Description --- See https://issues.apache.org/jira/browse/OOZIE-1817?focusedCommentId=13993838page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13993838 Diffs - client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 97fd465 core/pom.xml c935dd7 core/src/main/java/org/apache/oozie/command/Command.java d3f1011 core/src/main/java/org/apache/oozie/command/XCommand.java b712ce0 core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java e8667c1 core/src/main/java/org/apache/oozie/command/coord/CoordActionMaterializeCommand.java 6962fb2 core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 57cbb34 core/src/main/java/org/apache/oozie/service/InstrumentationService.java 80437b1 core/src/main/java/org/apache/oozie/service/MetricsInstrumentationService.java PRE-CREATION core/src/main/java/org/apache/oozie/service/Services.java 5feac7b core/src/main/java/org/apache/oozie/service/XLogService.java 403a089 core/src/main/java/org/apache/oozie/servlet/BaseAdminServlet.java 29d7bd6 core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 5c05acd core/src/main/java/org/apache/oozie/servlet/V0AdminServlet.java 97bd81c core/src/main/java/org/apache/oozie/servlet/V1AdminServlet.java a47f737 core/src/main/java/org/apache/oozie/servlet/V2AdminServlet.java 002a367 core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java PRE-CREATION core/src/main/java/org/apache/oozie/util/XLog.java 21e00c0 core/src/main/resources/oozie-default.xml 9788daf core/src/test/java/org/apache/oozie/service/TestInstrumentationService.java b5206cf core/src/test/java/org/apache/oozie/service/TestMetricsInstrumentationService.java PRE-CREATION core/src/test/java/org/apache/oozie/util/TestInstrumentation.java 6040b47 core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java PRE-CREATION docs/src/site/twiki/AG_Install.twiki e343d7e docs/src/site/twiki/WebServicesAPI.twiki 5769924 pom.xml b5e0e4e webapp/src/main/webapp/index.jsp 3c7ffe5 webapp/src/main/webapp/oozie-console.js 764d888 Diff: https://reviews.apache.org/r/21278/diff/ Testing --- Unit tests, also verified in a cluster Thanks, Robert Kanter
[jira] Subscription: Oozie Patch Available
Issue Subscription Filter: Oozie Patch Available (39 issues) Subscriber: ooziedaily Key Summary OOZIE-1860 Oozie job mapper launch fails due to null value returned from action file https://issues.apache.org/jira/browse/OOZIE-1860 OOZIE-1855 TestPriorityDelayQueue#testPoll failed intermittently in Jenkins https://issues.apache.org/jira/browse/OOZIE-1855 OOZIE-1830 Change hadoop-1 profile to use 1.2.1 https://issues.apache.org/jira/browse/OOZIE-1830 OOZIE-1829 URIHandlerService doesn't support URI schemes with query strings but no path segment https://issues.apache.org/jira/browse/OOZIE-1829 OOZIE-1828 Introduce counters JobStatus terminal states metrics https://issues.apache.org/jira/browse/OOZIE-1828 OOZIE-1821 Oozie java action fails due to AlreadyBeingCreatedException https://issues.apache.org/jira/browse/OOZIE-1821 OOZIE-1817 Oozie timers are not biased https://issues.apache.org/jira/browse/OOZIE-1817 OOZIE-1816 LogInfo uses action name instead of id https://issues.apache.org/jira/browse/OOZIE-1816 OOZIE-1813 Add service to report/kill rogue bundles and coordinator jobs https://issues.apache.org/jira/browse/OOZIE-1813 OOZIE-1812 Bundle status is always in RUNNING if one of the action status is in PREP https://issues.apache.org/jira/browse/OOZIE-1812 OOZIE-1807 Make bundle change command synchronous https://issues.apache.org/jira/browse/OOZIE-1807 OOZIE-1804 Improve documentation for Coordinator Specification https://issues.apache.org/jira/browse/OOZIE-1804 OOZIE-1803 Improvement in Purge service https://issues.apache.org/jira/browse/OOZIE-1803 OOZIE-1802 Support workflow action log https://issues.apache.org/jira/browse/OOZIE-1802 OOZIE-1793 Improve find bugs reporting for Oozie https://issues.apache.org/jira/browse/OOZIE-1793 OOZIE-1782 Workflow path not found is thrown as SC_UNAUTHORIZED https://issues.apache.org/jira/browse/OOZIE-1782 OOZIE-1741 Add new coord EL function to get input partitions value string https://issues.apache.org/jira/browse/OOZIE-1741 OOZIE-1724 Make it easier to specify the HCat hive-site.xml for the Oozie Server https://issues.apache.org/jira/browse/OOZIE-1724 OOZIE-1705 Enable gc logs and print thread id in logs https://issues.apache.org/jira/browse/OOZIE-1705 OOZIE-1688 New configuration to specify server-server authentication type. https://issues.apache.org/jira/browse/OOZIE-1688 OOZIE-1686 Typo in DG_CommandLineTool https://issues.apache.org/jira/browse/OOZIE-1686 OOZIE-1685 Oozie doesn’t process correctly workflows with a non-default name node https://issues.apache.org/jira/browse/OOZIE-1685 OOZIE-1659 oozie-site is missing email-action-0.2 schema https://issues.apache.org/jira/browse/OOZIE-1659 OOZIE-1654 Fix typo (inteval to interval) https://issues.apache.org/jira/browse/OOZIE-1654 OOZIE-1638 Action retry does not use default retry max count. https://issues.apache.org/jira/browse/OOZIE-1638 OOZIE-1636 OOZIE_SYS table engine should be innodb https://issues.apache.org/jira/browse/OOZIE-1636 OOZIE-1624 Exclusion pattern for sharelib. https://issues.apache.org/jira/browse/OOZIE-1624 OOZIE-1602 pom.xml should take use of the profiles when building hadooplibs https://issues.apache.org/jira/browse/OOZIE-1602 OOZIE-1586 upgrade oozie to hive 13.1 (including hcatalog) https://issues.apache.org/jira/browse/OOZIE-1586 OOZIE-1579 Add basic HTTP auth to Oozie CLI https://issues.apache.org/jira/browse/OOZIE-1579 OOZIE-1567 Provide a wait tool in Oozie https://issues.apache.org/jira/browse/OOZIE-1567 OOZIE-1533 Coordinator action materialization is too slow due to coarse job level locks https://issues.apache.org/jira/browse/OOZIE-1533 OOZIE-1532 Purging should remove completed children job for long running coordinator jobs https://issues.apache.org/jira/browse/OOZIE-1532 OOZIE-1376 Extending Oozie ACLs like admin groups and proxy users to support both groups and users https://issues.apache.org/jira/browse/OOZIE-1376 OOZIE-1369 OozieDBCLI code should not hardcode the Oozie table filenames https://issues.apache.org/jira/browse/OOZIE-1369 OOZIE-1232 GroupsService should be able to reference Hadoop configurations in Hadoop configuration folder (such as /etc/hadoop/conf) https://issues.apache.org/jira/browse/OOZIE-1232 OOZIE-1074 workflowgenerator does not package a tar.gz file though it seems creating one is the intention https://issues.apache.org/jira/browse/OOZIE-1074 OOZIE-891 Add pagination for all popup panels in Oozie Web UI https://issues.apache.org/jira/browse/OOZIE-891
[jira] [Created] (OOZIE-1865) Oozie servers can't talk to each other with Oozie HA and Kerberos
Robert Kanter created OOZIE-1865:
Summary: Oozie servers can't talk to each other with Oozie HA and
Kerberos
Key: OOZIE-1865
URL: https://issues.apache.org/jira/browse/OOZIE-1865
Project: Oozie
Issue Type: Bug
Components: HA
Affects Versions: trunk
Reporter: Robert Kanter
Assignee: Robert Kanter
When you use Oozie HA with Kerberos, you have to set
{{oozie.authentication.kerberos.principal}} to {{HTTP/load-balancer-host}}
instead of {{HTTP/oozie-server-host}}. This allows clients to connect to any
of the Oozie servers through the load balancer. However, it also blocks
clients from directly talking to any of the Oozie servers. In and of itself,
that's okay, but it turns out that in most cases, it also blocks the Oozie
servers from talking to each other, namely for log streaming, the
sharelibupdate command, and collating instrumentation/metrics (OOZIE-1676).
Ultimately, what we need to do is allow Oozie to use both
{{HTTP/load-balancer-host}} instead of {{HTTP/oozie-server-host}} at the
same time so that clients (including Oozie servers, users, Web UI, etc) can
talk to Oozie both through the load balancer and directly. If my understanding
of HADOOP-10158 is correct, HADOOP-10158 adds this ability. For this JIRA, we
should update Oozie to take advantage of HADOOP-10158.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
