[jira] [Created] (OOZIE-3640) Add support for recursive directories fs:dirSize
Purshotam Shah created OOZIE-3640: - Summary: Add support for recursive directories fs:dirSize Key: OOZIE-3640 URL: https://issues.apache.org/jira/browse/OOZIE-3640 Project: Oozie Issue Type: Improvement Reporter: Purshotam Shah There are three ways to do that. # Use getContentSummary: This can be dangerous for name nodes. getContentSummary on a large HDFS dirs can take minutes. During that time, an oozie thread will be blocked, waiting for the RPC response. A 5 min workflow doing a content summary on a directory that takes > 5 min may lead to oozie thread exhaustion. or if oozie times out and retries, the NN has no support for aborting a call being processed, so now there will be multiple concurrent content summaries which may also exhaust the NN's handler threads. # Use getQuotaUsage: If quote is not enabled, it will fall back on getContentSummary. So this is as bad as getContentSummary. # Use recursive listing to compute size. Enforce system-level dir size(or file count) and recursive level if it reached the max-level or max-size throw exception. Considering all three options. Option 3 is the best option. A system admin can configure max-level based on system load and user use-cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (OOZIE-2150) Shell launcher should print shell script
[ https://issues.apache.org/jira/browse/OOZIE-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah reassigned OOZIE-2150: - Assignee: Jacob Tolar (was: Purshotam Shah) > Shell launcher should print shell script > > > Key: OOZIE-2150 > URL: https://issues.apache.org/jira/browse/OOZIE-2150 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah >Assignee: Jacob Tolar >Priority: Minor > > Pig and Hive does this. It will be helpful in debugging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (OOZIE-2150) Shell launcher should print shell script
[ https://issues.apache.org/jira/browse/OOZIE-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah reassigned OOZIE-2150: - Assignee: Purshotam Shah (was: Jacob Tolar) > Shell launcher should print shell script > > > Key: OOZIE-2150 > URL: https://issues.apache.org/jira/browse/OOZIE-2150 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah >Priority: Minor > > Pig and Hive does this. It will be helpful in debugging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-3095) Job submit command is not logged in audit log
[ https://issues.apache.org/jira/browse/OOZIE-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-3095: -- Fix Version/s: 5.0.0 > Job submit command is not logged in audit log > - > > Key: OOZIE-3095 > URL: https://issues.apache.org/jira/browse/OOZIE-3095 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah > Fix For: 5.0.0 > > Attachments: OOZIE-3095-V1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OOZIE-3095) Job submit command is not logged in audit log
[ https://issues.apache.org/jira/browse/OOZIE-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16219712#comment-16219712 ] Purshotam Shah commented on OOZIE-3095: --- Thanks Rohini for review. Committed to trunk. > Job submit command is not logged in audit log > - > > Key: OOZIE-3095 > URL: https://issues.apache.org/jira/browse/OOZIE-3095 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah > Fix For: 5.0.0 > > Attachments: OOZIE-3095-V1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OOZIE-3095) Job submit command is not logged in audit log
[
https://issues.apache.org/jira/browse/OOZIE-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216046#comment-16216046
]
Purshotam Shah commented on OOZIE-3095:
---
Submit command creates a jobid, but the job will in prep state. it will have
jodId.
It will look something like,
{code}
2017-10-23 15:04:15,281 INFO oozieaudit:520 - IP [127.0.0.1], USER [puru],
GROUP [null], APP [pig-wf], JOBID [000-171023150228072-oozie-puru-W],
OPERATION [submit], PARAMETER [null], STATUS [SUCCESS], HTTPCODE [200],
ERRORCODE [null], ERRORMESSAGE [null]
{code}
> Job submit command is not logged in audit log
> -
>
> Key: OOZIE-3095
> URL: https://issues.apache.org/jira/browse/OOZIE-3095
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Attachments: OOZIE-3095-V1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-3095) Job submit command is not logged in audit log
[ https://issues.apache.org/jira/browse/OOZIE-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-3095: -- Attachment: OOZIE-3095-V1.patch > Job submit command is not logged in audit log > - > > Key: OOZIE-3095 > URL: https://issues.apache.org/jira/browse/OOZIE-3095 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah > Attachments: OOZIE-3095-V1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-3095) Job submit command is not logged in audit log
[ https://issues.apache.org/jira/browse/OOZIE-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-3095: -- Attachment: (was: OOZIE-3095-V1.patch) > Job submit command is not logged in audit log > - > > Key: OOZIE-3095 > URL: https://issues.apache.org/jira/browse/OOZIE-3095 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-3095) Job submit command is not logged in audit log
[ https://issues.apache.org/jira/browse/OOZIE-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-3095: -- Attachment: OOZIE-3095-V1.patch > Job submit command is not logged in audit log > - > > Key: OOZIE-3095 > URL: https://issues.apache.org/jira/browse/OOZIE-3095 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Attachments: OOZIE-3095-V1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (OOZIE-3095) Job submit command is not logged in audit log
[ https://issues.apache.org/jira/browse/OOZIE-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah reassigned OOZIE-3095: - Assignee: Purshotam Shah > Job submit command is not logged in audit log > - > > Key: OOZIE-3095 > URL: https://issues.apache.org/jira/browse/OOZIE-3095 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah > Attachments: OOZIE-3095-V1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OOZIE-3095) Job submit command is not logged in audit log
Purshotam Shah created OOZIE-3095: - Summary: Job submit command is not logged in audit log Key: OOZIE-3095 URL: https://issues.apache.org/jira/browse/OOZIE-3095 Project: Oozie Issue Type: Bug Reporter: Purshotam Shah -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-3031) Coord job with only unresolved dependencies doesn't timeout
[
https://issues.apache.org/jira/browse/OOZIE-3031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-3031:
--
Attachment: OOZIE-3031-V2.patch
> Coord job with only unresolved dependencies doesn't timeout
> ---
>
> Key: OOZIE-3031
> URL: https://issues.apache.org/jira/browse/OOZIE-3031
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
>Assignee: Purshotam Shah
>Priority: Critical
> Attachments: OOZIE-3031-V1.patch, OOZIE-3031-V2.patch
>
>
> {code:title=CoordActionInputCheckXCommand.java}
> else if (!isTimeout(currentTime)) {
> if (!status) {
> long addtionalDelay = isChangeInDependency ? 0
> :
> ConfigurationService.getInt(CONF_COORD_INPUT_CHECK_REQUEUE_INTERVAL_ADDITIONAL_DELAY)
> * 1000L;
> queue(new
> CoordActionInputCheckXCommand(coordAction.getId(), coordAction.getJobId()),
> addtionalDelay +
> getCoordInputCheckRequeueInterval());
> }
> updateCoordAction(coordAction, isChangeInDependency);
> }
> else {
> if (!nonExistListStr.isEmpty() && isPushDependenciesMet) {
> queue(new CoordActionTimeOutXCommand(coordAction,
> coordJob.getUser(), coordJob.getAppName()));
> }
> else {
> // Let CoordPushDependencyCheckXCommand queue the timeout
> queue(new
> CoordPushDependencyCheckXCommand(coordAction.getId()));
> }
> updateCoordAction(coordAction, isChangeInDependency);
> }
> }
> {code}
> {{!nonExistList Str.isEmpty()}} check is unnecessary, once the action has
> passed the timeout, it should be marked as timedout.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-3031) Coord job with only unresolved dependencies doesn't timeout
[
https://issues.apache.org/jira/browse/OOZIE-3031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-3031:
--
Attachment: OOZIE-3031-V1.patch
> Coord job with only unresolved dependencies doesn't timeout
> ---
>
> Key: OOZIE-3031
> URL: https://issues.apache.org/jira/browse/OOZIE-3031
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
>Assignee: Purshotam Shah
>Priority: Critical
> Attachments: OOZIE-3031-V1.patch
>
>
> {code:title=CoordActionInputCheckXCommand.java}
> else if (!isTimeout(currentTime)) {
> if (!status) {
> long addtionalDelay = isChangeInDependency ? 0
> :
> ConfigurationService.getInt(CONF_COORD_INPUT_CHECK_REQUEUE_INTERVAL_ADDITIONAL_DELAY)
> * 1000L;
> queue(new
> CoordActionInputCheckXCommand(coordAction.getId(), coordAction.getJobId()),
> addtionalDelay +
> getCoordInputCheckRequeueInterval());
> }
> updateCoordAction(coordAction, isChangeInDependency);
> }
> else {
> if (!nonExistListStr.isEmpty() && isPushDependenciesMet) {
> queue(new CoordActionTimeOutXCommand(coordAction,
> coordJob.getUser(), coordJob.getAppName()));
> }
> else {
> // Let CoordPushDependencyCheckXCommand queue the timeout
> queue(new
> CoordPushDependencyCheckXCommand(coordAction.getId()));
> }
> updateCoordAction(coordAction, isChangeInDependency);
> }
> }
> {code}
> {{!nonExistList Str.isEmpty()}} check is unnecessary, once the action has
> passed the timeout, it should be marked as timedout.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-3031) Coord job with only unresolved dependencies doesn't timeout
[
https://issues.apache.org/jira/browse/OOZIE-3031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-3031:
--
Description:
{code}
else if (!isTimeout(currentTime)) {
if (!status) {
long addtionalDelay = isChangeInDependency ? 0
:
ConfigurationService.getInt(CONF_COORD_INPUT_CHECK_REQUEUE_INTERVAL_ADDITIONAL_DELAY)
* 1000L;
queue(new
CoordActionInputCheckXCommand(coordAction.getId(), coordAction.getJobId()),
addtionalDelay +
getCoordInputCheckRequeueInterval());
}
updateCoordAction(coordAction, isChangeInDependency);
}
else {
if (!nonExistListStr.isEmpty() && isPushDependenciesMet) {
queue(new CoordActionTimeOutXCommand(coordAction,
coordJob.getUser(), coordJob.getAppName()));
}
else {
// Let CoordPushDependencyCheckXCommand queue the timeout
queue(new
CoordPushDependencyCheckXCommand(coordAction.getId()));
}
updateCoordAction(coordAction, isChangeInDependency);
}
}
{code}
{{!nonExistList Str.isEmpty()}} check is unnecessary, once the action has
passed the timeout, it should be marked as timedout.
> Coord job with only unresolved dependencies doesn't timeout
> ---
>
> Key: OOZIE-3031
> URL: https://issues.apache.org/jira/browse/OOZIE-3031
> Project: Oozie
> Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
>Priority: Critical
>
> {code}
> else if (!isTimeout(currentTime)) {
> if (!status) {
> long addtionalDelay = isChangeInDependency ? 0
> :
> ConfigurationService.getInt(CONF_COORD_INPUT_CHECK_REQUEUE_INTERVAL_ADDITIONAL_DELAY)
> * 1000L;
> queue(new
> CoordActionInputCheckXCommand(coordAction.getId(), coordAction.getJobId()),
> addtionalDelay +
> getCoordInputCheckRequeueInterval());
> }
> updateCoordAction(coordAction, isChangeInDependency);
> }
> else {
> if (!nonExistListStr.isEmpty() && isPushDependenciesMet) {
> queue(new CoordActionTimeOutXCommand(coordAction,
> coordJob.getUser(), coordJob.getAppName()));
> }
> else {
> // Let CoordPushDependencyCheckXCommand queue the timeout
> queue(new
> CoordPushDependencyCheckXCommand(coordAction.getId()));
> }
> updateCoordAction(coordAction, isChangeInDependency);
> }
> }
> {code}
> {{!nonExistList Str.isEmpty()}} check is unnecessary, once the action has
> passed the timeout, it should be marked as timedout.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-3031) Coord job with only unresolved dependencies doesn't timeout
[
https://issues.apache.org/jira/browse/OOZIE-3031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-3031:
--
Description:
{code:title=CoordActionInputCheckXCommand.java}
else if (!isTimeout(currentTime)) {
if (!status) {
long addtionalDelay = isChangeInDependency ? 0
:
ConfigurationService.getInt(CONF_COORD_INPUT_CHECK_REQUEUE_INTERVAL_ADDITIONAL_DELAY)
* 1000L;
queue(new
CoordActionInputCheckXCommand(coordAction.getId(), coordAction.getJobId()),
addtionalDelay +
getCoordInputCheckRequeueInterval());
}
updateCoordAction(coordAction, isChangeInDependency);
}
else {
if (!nonExistListStr.isEmpty() && isPushDependenciesMet) {
queue(new CoordActionTimeOutXCommand(coordAction,
coordJob.getUser(), coordJob.getAppName()));
}
else {
// Let CoordPushDependencyCheckXCommand queue the timeout
queue(new
CoordPushDependencyCheckXCommand(coordAction.getId()));
}
updateCoordAction(coordAction, isChangeInDependency);
}
}
{code}
{{!nonExistList Str.isEmpty()}} check is unnecessary, once the action has
passed the timeout, it should be marked as timedout.
was:
{code}
else if (!isTimeout(currentTime)) {
if (!status) {
long addtionalDelay = isChangeInDependency ? 0
:
ConfigurationService.getInt(CONF_COORD_INPUT_CHECK_REQUEUE_INTERVAL_ADDITIONAL_DELAY)
* 1000L;
queue(new
CoordActionInputCheckXCommand(coordAction.getId(), coordAction.getJobId()),
addtionalDelay +
getCoordInputCheckRequeueInterval());
}
updateCoordAction(coordAction, isChangeInDependency);
}
else {
if (!nonExistListStr.isEmpty() && isPushDependenciesMet) {
queue(new CoordActionTimeOutXCommand(coordAction,
coordJob.getUser(), coordJob.getAppName()));
}
else {
// Let CoordPushDependencyCheckXCommand queue the timeout
queue(new
CoordPushDependencyCheckXCommand(coordAction.getId()));
}
updateCoordAction(coordAction, isChangeInDependency);
}
}
{code}
{{!nonExistList Str.isEmpty()}} check is unnecessary, once the action has
passed the timeout, it should be marked as timedout.
> Coord job with only unresolved dependencies doesn't timeout
> ---
>
> Key: OOZIE-3031
> URL: https://issues.apache.org/jira/browse/OOZIE-3031
> Project: Oozie
> Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
>Priority: Critical
>
> {code:title=CoordActionInputCheckXCommand.java}
> else if (!isTimeout(currentTime)) {
> if (!status) {
> long addtionalDelay = isChangeInDependency ? 0
> :
> ConfigurationService.getInt(CONF_COORD_INPUT_CHECK_REQUEUE_INTERVAL_ADDITIONAL_DELAY)
> * 1000L;
> queue(new
> CoordActionInputCheckXCommand(coordAction.getId(), coordAction.getJobId()),
> addtionalDelay +
> getCoordInputCheckRequeueInterval());
> }
> updateCoordAction(coordAction, isChangeInDependency);
> }
> else {
> if (!nonExistListStr.isEmpty() && isPushDependenciesMet) {
> queue(new CoordActionTimeOutXCommand(coordAction,
> coordJob.getUser(), coordJob.getAppName()));
> }
> else {
> // Let CoordPushDependencyCheckXCommand queue the timeout
> queue(new
> CoordPushDependencyCheckXCommand(coordAction.getId()));
> }
> updateCoordAction(coordAction, isChangeInDependency);
> }
> }
> {code}
> {{!nonExistList Str.isEmpty()}} check is unnecessary, once the action has
> passed the timeout, it should be marked as timedout.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (OOZIE-3031) Coord job with only unresolved dependencies doesn't timeout
[ https://issues.apache.org/jira/browse/OOZIE-3031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah reassigned OOZIE-3031: - Assignee: Purshotam Shah > Coord job with only unresolved dependencies doesn't timeout > --- > > Key: OOZIE-3031 > URL: https://issues.apache.org/jira/browse/OOZIE-3031 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah >Assignee: Purshotam Shah >Priority: Critical > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OOZIE-3031) Coord job with only unresolved dependencies doesn't timeout
Purshotam Shah created OOZIE-3031: - Summary: Coord job with only unresolved dependencies doesn't timeout Key: OOZIE-3031 URL: https://issues.apache.org/jira/browse/OOZIE-3031 Project: Oozie Issue Type: Bug Reporter: Purshotam Shah Priority: Critical -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: Review Request 52373: OOZIE-2688 YAML representation of Oozie unit testcase
kipWorkflow.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/main/java/org/apache/oozie/oozieyamlunit/config/skip/SkipWorkflowAction.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/main/java/org/apache/oozie/oozieyamlunit/config/skip/SkipWorkflowDeserializer.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/main/java/org/apache/oozie/oozieyamlunit/config/skip/YamlStringDeserializer.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/main/java/org/apache/oozie/oozieyamlunit/config/validate/JobCheck.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/main/java/org/apache/oozie/oozieyamlunit/config/validate/JobConfValidation.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/main/java/org/apache/oozie/oozieyamlunit/config/validate/NotStatusValidation.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/main/java/org/apache/oozie/oozieyamlunit/config/validate/Sleep.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/main/java/org/apache/oozie/oozieyamlunit/config/validate/StatusValidation.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/main/java/org/apache/oozie/oozieyamlunit/config/validate/ValidateBundle.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/main/java/org/apache/oozie/oozieyamlunit/config/validate/ValidateCoodJob.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/main/java/org/apache/oozie/oozieyamlunit/config/validate/ValidateCoordAction.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/main/java/org/apache/oozie/oozieyamlunit/config/validate/ValidateJob.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/main/java/org/apache/oozie/oozieyamlunit/config/validate/ValidateWorkflow.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/main/java/org/apache/oozie/oozieyamlunit/config/validate/ValidateWorkflowAction.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/main/java/org/apache/oozie/oozieyamlunit/config/validate/ValidationOutPut.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/main/java/org/apache/oozie/oozieyamlunit/config/validate/Validations.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/main/java/org/apache/oozie/oozieyamlunit/config/validate/WaitForJobStatus.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/main/resources/META-INF/m2e/lifecycle-mapping-metadata.xml e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/test/java/org/apache/oozie/oozieyamlunit/TestHdfs.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/test/java/org/apache/oozie/oozieyamlunit/TestJobValidation.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/test/java/org/apache/oozie/oozieyamlunit/TestSetup.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/test/java/org/apache/oozie/oozieyamlunit/TestSkip.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 yaml-unit/src/test/java/org/apache/oozie/oozieyamlunit/YamlTestUtil.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 Diff: https://reviews.apache.org/r/52373/diff/2/ Changes: https://reviews.apache.org/r/52373/diff/1-2/ Testing --- Thanks, Purshotam Shah
[jira] [Created] (OOZIE-3006) Oozie submit duplicate job if action has retry and oozie is not able to lookup hadoop job
Purshotam Shah created OOZIE-3006:
-
Summary: Oozie submit duplicate job if action has retry and oozie
is not able to lookup hadoop job
Key: OOZIE-3006
URL: https://issues.apache.org/jira/browse/OOZIE-3006
Project: Oozie
Issue Type: Bug
Reporter: Purshotam Shah
{code:title=JavaActionExecutor.java}
Element actionXml = XmlUtils.parseXml(action.getConf());
FileSystem actionFs = context.getAppFileSystem();
JobConf jobConf = createBaseHadoopConf(context, actionXml);
jobClient = createJobClient(context, jobConf);
RunningJob runningJob = getRunningJob(context, action, jobClient);
if (runningJob == null) {
context.setExecutionData(FAILED, null);
throw new
ActionExecutorException(ActionExecutorException.ErrorType.FAILED, "JA017",
"Could not lookup launched hadoop Job ID [{0}] which
was associated with " +
" action [{1}]. Failing this action!",
getActualExternalId(action), action.getId());
}
{code}
Oozie fails the action if it's not able to look up Hadoop job. If the action
has retry, oozie will submit a new job, and it could end up running multiple
jobs for the same action.
Not able to look up Hadoop job != job failure.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-3004) Forked action retry info is not working
[
https://issues.apache.org/jira/browse/OOZIE-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16097463#comment-16097463
]
Purshotam Shah commented on OOZIE-3004:
---
Thanks, Satish. Committed to master.
> Forked action retry info is not working
> ---
>
> Key: OOZIE-3004
> URL: https://issues.apache.org/jira/browse/OOZIE-3004
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Fix For: 5.0.0
>
> Attachments: OOZIE-3004-V1.patch, OOZIE-3004-V2.patch,
> screenshot-1.png
>
>
> UI shows N/A for start time and console URL
> We do parallel job submission for forked actions, and each job submission
> needs to update workflow instance.
> To avoid data race, ForkedActionStartXCommand write workflow instance to a
> temporary map.
> {code:title=ForkedActionExecutorContext.java}
> private Map contextVariableMap = new HashMap String>();
> public ForkedActionExecutorContext(WorkflowJobBean workflow,
> WorkflowActionBean action, boolean isRetry,
> boolean isUserRetry) {
> super(workflow, action, isRetry, isUserRetry);
> }
> public void setVar(String name, String value) {
> if (value != null) {
> contextVariableMap.remove(name);
> }
> else {
> contextVariableMap.put(name, value);
> }
> }
> {code}
> And when forked submission completes, the temporary map is written to the
> workflow instance.
> {code:title=SignalXCommand.java}
> List> futures =
> Services.get().get(CallableQueueService.class)
> .invokeAll(tasks);
> for (Future result : futures) {
> if (result == null) {
> submitJobByQueuing = true;
> continue;
> }
> ActionExecutorContext context = result.get();
> Map contextVariableMap =
> ((ForkedActionExecutorContext) context).getContextMap();
> LOG.debug("contextVariableMap size of action " +
> context.getAction().getId() + " is " + contextVariableMap.size());
> for (String key : contextVariableMap.keySet()) {
> context.setVarToWorkflow(key,
> contextVariableMap.get(key));
> }
> {code}
> The logic for writing to the temporary map is incorrect. It should be
> {code}
> public void setVar(String name, String value) {
> if (value == null) {
> contextVariableMap.remove(name);
> }
> else {
> contextVariableMap.put(name, value);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-3004) Forked action retry info is not working
[
https://issues.apache.org/jira/browse/OOZIE-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-3004:
--
Fix Version/s: 5.0.0
> Forked action retry info is not working
> ---
>
> Key: OOZIE-3004
> URL: https://issues.apache.org/jira/browse/OOZIE-3004
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Fix For: 5.0.0
>
> Attachments: OOZIE-3004-V1.patch, OOZIE-3004-V2.patch,
> screenshot-1.png
>
>
> UI shows N/A for start time and console URL
> We do parallel job submission for forked actions, and each job submission
> needs to update workflow instance.
> To avoid data race, ForkedActionStartXCommand write workflow instance to a
> temporary map.
> {code:title=ForkedActionExecutorContext.java}
> private Map contextVariableMap = new HashMap String>();
> public ForkedActionExecutorContext(WorkflowJobBean workflow,
> WorkflowActionBean action, boolean isRetry,
> boolean isUserRetry) {
> super(workflow, action, isRetry, isUserRetry);
> }
> public void setVar(String name, String value) {
> if (value != null) {
> contextVariableMap.remove(name);
> }
> else {
> contextVariableMap.put(name, value);
> }
> }
> {code}
> And when forked submission completes, the temporary map is written to the
> workflow instance.
> {code:title=SignalXCommand.java}
> List> futures =
> Services.get().get(CallableQueueService.class)
> .invokeAll(tasks);
> for (Future result : futures) {
> if (result == null) {
> submitJobByQueuing = true;
> continue;
> }
> ActionExecutorContext context = result.get();
> Map contextVariableMap =
> ((ForkedActionExecutorContext) context).getContextMap();
> LOG.debug("contextVariableMap size of action " +
> context.getAction().getId() + " is " + contextVariableMap.size());
> for (String key : contextVariableMap.keySet()) {
> context.setVarToWorkflow(key,
> contextVariableMap.get(key));
> }
> {code}
> The logic for writing to the temporary map is incorrect. It should be
> {code}
> public void setVar(String name, String value) {
> if (value == null) {
> contextVariableMap.remove(name);
> }
> else {
> contextVariableMap.put(name, value);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-3004) Forked action retry info is not working
[
https://issues.apache.org/jira/browse/OOZIE-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096780#comment-16096780
]
Purshotam Shah commented on OOZIE-3004:
---
bq. Purshotam Shah nice catch. Could you please add technical cause and
solution in brief?
Updated description.
> Forked action retry info is not working
> ---
>
> Key: OOZIE-3004
> URL: https://issues.apache.org/jira/browse/OOZIE-3004
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Attachments: OOZIE-3004-V1.patch, OOZIE-3004-V2.patch,
> screenshot-1.png
>
>
> UI shows N/A for start time and console URL
> We do parallel job submission for forked actions, and each job submission
> needs to update workflow instance.
> To avoid data race, ForkedActionStartXCommand write workflow instance to a
> temporary map.
> {code:title=ForkedActionExecutorContext.java}
> private Map contextVariableMap = new HashMap String>();
> public ForkedActionExecutorContext(WorkflowJobBean workflow,
> WorkflowActionBean action, boolean isRetry,
> boolean isUserRetry) {
> super(workflow, action, isRetry, isUserRetry);
> }
> public void setVar(String name, String value) {
> if (value != null) {
> contextVariableMap.remove(name);
> }
> else {
> contextVariableMap.put(name, value);
> }
> }
> {code}
> And when forked submission completes, the temporary map is written to the
> workflow instance.
> {code:title=SignalXCommand.java}
> List> futures =
> Services.get().get(CallableQueueService.class)
> .invokeAll(tasks);
> for (Future result : futures) {
> if (result == null) {
> submitJobByQueuing = true;
> continue;
> }
> ActionExecutorContext context = result.get();
> Map contextVariableMap =
> ((ForkedActionExecutorContext) context).getContextMap();
> LOG.debug("contextVariableMap size of action " +
> context.getAction().getId() + " is " + contextVariableMap.size());
> for (String key : contextVariableMap.keySet()) {
> context.setVarToWorkflow(key,
> contextVariableMap.get(key));
> }
> {code}
> The logic for writing to the temporary map is incorrect. It should be
> {code}
> public void setVar(String name, String value) {
> if (value == null) {
> contextVariableMap.remove(name);
> }
> else {
> contextVariableMap.put(name, value);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-3004) Forked action retry info is not working
[
https://issues.apache.org/jira/browse/OOZIE-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-3004:
--
Description:
UI shows N/A for start time and console URL
We do parallel job submission for forked actions, and each job submission needs
to update workflow instance.
To avoid data race, ForkedActionStartXCommand write workflow instance to a
temporary map.
{code:title=ForkedActionExecutorContext.java}
private Map contextVariableMap = new HashMap();
public ForkedActionExecutorContext(WorkflowJobBean workflow,
WorkflowActionBean action, boolean isRetry,
boolean isUserRetry) {
super(workflow, action, isRetry, isUserRetry);
}
public void setVar(String name, String value) {
if (value != null) {
contextVariableMap.remove(name);
}
else {
contextVariableMap.put(name, value);
}
}
{code}
And when forked submission completes, the temporary map is written to the
workflow instance.
{code:title=SignalXCommand.java}
List> futures =
Services.get().get(CallableQueueService.class)
.invokeAll(tasks);
for (Future result : futures) {
if (result == null) {
submitJobByQueuing = true;
continue;
}
ActionExecutorContext context = result.get();
Map contextVariableMap =
((ForkedActionExecutorContext) context).getContextMap();
LOG.debug("contextVariableMap size of action " +
context.getAction().getId() + " is " + contextVariableMap.size());
for (String key : contextVariableMap.keySet()) {
context.setVarToWorkflow(key, contextVariableMap.get(key));
}
{code}
The logic for writing to the temporary map is incorrect. It should be
{code}
public void setVar(String name, String value) {
if (value == null) {
contextVariableMap.remove(name);
}
else {
contextVariableMap.put(name, value);
}
}
{code}
was:UI shows N/A for start time and console URL
> Forked action retry info is not working
> ---
>
> Key: OOZIE-3004
> URL: https://issues.apache.org/jira/browse/OOZIE-3004
> Project: Oozie
> Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-3004-V1.patch, OOZIE-3004-V2.patch,
> screenshot-1.png
>
>
> UI shows N/A for start time and console URL
> We do parallel job submission for forked actions, and each job submission
> needs to update workflow instance.
> To avoid data race, ForkedActionStartXCommand write workflow instance to a
> temporary map.
> {code:title=ForkedActionExecutorContext.java}
> private Map contextVariableMap = new HashMap String>();
> public ForkedActionExecutorContext(WorkflowJobBean workflow,
> WorkflowActionBean action, boolean isRetry,
> boolean isUserRetry) {
> super(workflow, action, isRetry, isUserRetry);
> }
> public void setVar(String name, String value) {
> if (value != null) {
> contextVariableMap.remove(name);
> }
> else {
> contextVariableMap.put(name, value);
> }
> }
> {code}
> And when forked submission completes, the temporary map is written to the
> workflow instance.
> {code:title=SignalXCommand.java}
> List> futures =
> Services.get().get(CallableQueueService.class)
> .invokeAll(tasks);
> for (Future result : futures) {
> if (result == null) {
> submitJobByQueuing = true;
> continue;
> }
> ActionExecutorContext context = result.get();
> Map contextVariableMap =
> ((ForkedActionExecutorContext) context).getContextMap();
> LOG.debug("contextVariableMap size of action " +
> context.getAction().getId() + " is " + contextVariableMap.size());
> for (String key : contextVariableMap.keySet()) {
> context.setVarToWorkflow(key,
> contextVariableMap.get(key));
> }
> {code}
> The logic for writing to the temporary map is incorrect. It should be
> {code}
> public void setVar(String name, String value) {
> if (value == null) {
> contextVariableMap.remove(name);
> }
> else {
> contextVariableMap.put(name, value);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-3004) Forked action retry info is not working
[ https://issues.apache.org/jira/browse/OOZIE-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-3004: -- Attachment: OOZIE-3004-V2.patch > Forked action retry info is not working > --- > > Key: OOZIE-3004 > URL: https://issues.apache.org/jira/browse/OOZIE-3004 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah > Attachments: OOZIE-3004-V1.patch, OOZIE-3004-V2.patch, > screenshot-1.png > > > UI shows N/A for start time and console URL -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (OOZIE-3004) Forked action retry info is not working
[ https://issues.apache.org/jira/browse/OOZIE-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah reassigned OOZIE-3004: - Assignee: Purshotam Shah > Forked action retry info is not working > --- > > Key: OOZIE-3004 > URL: https://issues.apache.org/jira/browse/OOZIE-3004 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah > Attachments: OOZIE-3004-V1.patch, screenshot-1.png > > > UI shows N/A for start time and console URL -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-3004) Forked action retry info is not working
[ https://issues.apache.org/jira/browse/OOZIE-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-3004: -- Attachment: OOZIE-3004-V1.patch > Forked action retry info is not working > --- > > Key: OOZIE-3004 > URL: https://issues.apache.org/jira/browse/OOZIE-3004 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah > Attachments: OOZIE-3004-V1.patch, screenshot-1.png > > > UI shows N/A for start time and console URL -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-3004) Forked action retry info is not working
[ https://issues.apache.org/jira/browse/OOZIE-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-3004: -- Attachment: screenshot-1.png > Forked action retry info is not working > --- > > Key: OOZIE-3004 > URL: https://issues.apache.org/jira/browse/OOZIE-3004 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Attachments: screenshot-1.png > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-3004) Forked action retry info is not working
[ https://issues.apache.org/jira/browse/OOZIE-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-3004: -- Description: UI shows N/A for start time and console URL > Forked action retry info is not working > --- > > Key: OOZIE-3004 > URL: https://issues.apache.org/jira/browse/OOZIE-3004 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Attachments: screenshot-1.png > > > UI shows N/A for start time and console URL -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OOZIE-3004) Forked action retry info is not working
Purshotam Shah created OOZIE-3004: - Summary: Forked action retry info is not working Key: OOZIE-3004 URL: https://issues.apache.org/jira/browse/OOZIE-3004 Project: Oozie Issue Type: Bug Reporter: Purshotam Shah -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OOZIE-2987) Coord action missing dependencies should show URI template with unresolved dependencies
[
https://issues.apache.org/jira/browse/OOZIE-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090857#comment-16090857
]
Purshotam Shah commented on OOZIE-2987:
---
Thank you, Satish for review. Committed to trunk.
> Coord action missing dependencies should show URI template with unresolved
> dependencies
> ---
>
> Key: OOZIE-2987
> URL: https://issues.apache.org/jira/browse/OOZIE-2987
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Fix For: 5.0.0
>
> Attachments: OOZIE-2987-V1.patch
>
>
> Patch also fixes and an issue where if there are unresolved dependencies and
> hcat dependencies, {{blocked on}} show unresolved dependency.
> Missing dependency for unresolved will show as
> {code}${coord:latestRange(-5,0)} ->
> hcat://localhost:30083/db_c/table1/dt=${YEAR}${DAY};country=usa{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-2987) Coord action missing dependencies should show URI template with unresolved dependencies
[
https://issues.apache.org/jira/browse/OOZIE-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-2987:
--
Fix Version/s: (was: trunk)
5.0.0
> Coord action missing dependencies should show URI template with unresolved
> dependencies
> ---
>
> Key: OOZIE-2987
> URL: https://issues.apache.org/jira/browse/OOZIE-2987
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Fix For: 5.0.0
>
> Attachments: OOZIE-2987-V1.patch
>
>
> Patch also fixes and an issue where if there are unresolved dependencies and
> hcat dependencies, {{blocked on}} show unresolved dependency.
> Missing dependency for unresolved will show as
> {code}${coord:latestRange(-5,0)} ->
> hcat://localhost:30083/db_c/table1/dt=${YEAR}${DAY};country=usa{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2987) Coord action missing dependencies should show URI template with unresolved dependencies
[
https://issues.apache.org/jira/browse/OOZIE-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16086046#comment-16086046
]
Purshotam Shah commented on OOZIE-2987:
---
[~andras.piros] I have linked the parent JIRA. You will get more details about
feature @OOZIE-2770.
I have modified test case to validate the new features.
> Coord action missing dependencies should show URI template with unresolved
> dependencies
> ---
>
> Key: OOZIE-2987
> URL: https://issues.apache.org/jira/browse/OOZIE-2987
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Attachments: OOZIE-2987-V1.patch
>
>
> Patch also fixes and an issue where if there are unresolved dependencies and
> hcat dependencies, {{blocked on}} show unresolved dependency.
> Missing dependency for unresolved will show as
> {code}${coord:latestRange(-5,0)} ->
> hcat://localhost:30083/db_c/table1/dt=${YEAR}${DAY};country=usa{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (OOZIE-2987) Coord action missing dependencies should show URI template with unresolved dependencies
[ https://issues.apache.org/jira/browse/OOZIE-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah reassigned OOZIE-2987: - Assignee: Purshotam Shah > Coord action missing dependencies should show URI template with unresolved > dependencies > --- > > Key: OOZIE-2987 > URL: https://issues.apache.org/jira/browse/OOZIE-2987 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah > Attachments: OOZIE-2987-V1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-2987) Coord action missing dependencies should show URI template with unresolved dependencies
[
https://issues.apache.org/jira/browse/OOZIE-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-2987:
--
Description:
Patch also fixes and an issue where if there are unresolved dependencies and
hcat dependencies, {{blocked on}} show unresolved dependency.
Missing dependency for unresolved will show as
{{${coord:latestRange(-5,0)} ->
hcat://localhost:30083/db_c/table1/dt=${YEAR}${DAY};country=usa}}
> Coord action missing dependencies should show URI template with unresolved
> dependencies
> ---
>
> Key: OOZIE-2987
> URL: https://issues.apache.org/jira/browse/OOZIE-2987
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Attachments: OOZIE-2987-V1.patch
>
>
> Patch also fixes and an issue where if there are unresolved dependencies and
> hcat dependencies, {{blocked on}} show unresolved dependency.
> Missing dependency for unresolved will show as
> {{${coord:latestRange(-5,0)} ->
> hcat://localhost:30083/db_c/table1/dt=${YEAR}${DAY};country=usa}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-2987) Coord action missing dependencies should show URI template with unresolved dependencies
[
https://issues.apache.org/jira/browse/OOZIE-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-2987:
--
Description:
Patch also fixes and an issue where if there are unresolved dependencies and
hcat dependencies, {{blocked on}} show unresolved dependency.
Missing dependency for unresolved will show as
{code}${coord:latestRange(-5,0)} ->
hcat://localhost:30083/db_c/table1/dt=${YEAR}${DAY};country=usa{code}
was:
Patch also fixes and an issue where if there are unresolved dependencies and
hcat dependencies, {{blocked on}} show unresolved dependency.
Missing dependency for unresolved will show as
{{${coord:latestRange(-5,0)} ->
hcat://localhost:30083/db_c/table1/dt=${YEAR}${DAY};country=usa}}
> Coord action missing dependencies should show URI template with unresolved
> dependencies
> ---
>
> Key: OOZIE-2987
> URL: https://issues.apache.org/jira/browse/OOZIE-2987
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-2987-V1.patch
>
>
> Patch also fixes and an issue where if there are unresolved dependencies and
> hcat dependencies, {{blocked on}} show unresolved dependency.
> Missing dependency for unresolved will show as
> {code}${coord:latestRange(-5,0)} ->
> hcat://localhost:30083/db_c/table1/dt=${YEAR}${DAY};country=usa{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-2987) Coord action missing dependencies should show URI template with unresolved dependencies
[ https://issues.apache.org/jira/browse/OOZIE-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-2987: -- Attachment: OOZIE-2987-V1.patch > Coord action missing dependencies should show URI template with unresolved > dependencies > --- > > Key: OOZIE-2987 > URL: https://issues.apache.org/jira/browse/OOZIE-2987 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Attachments: OOZIE-2987-V1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OOZIE-2987) Coord action missing dependencies should show URI template with unresolved dependencies
Purshotam Shah created OOZIE-2987: - Summary: Coord action missing dependencies should show URI template with unresolved dependencies Key: OOZIE-2987 URL: https://issues.apache.org/jira/browse/OOZIE-2987 Project: Oozie Issue Type: Bug Reporter: Purshotam Shah -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OOZIE-2873) Check El Functions before submitting the coordinator
[
https://issues.apache.org/jira/browse/OOZIE-2873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999170#comment-15999170
]
Purshotam Shah commented on OOZIE-2873:
---
+1. Yes, I missed that while reviewing. You can't evaluate the coord action
function at coord submit time. Coord submit does do phase one evaluation of
coord action functions. So we are good.
> Check El Functions before submitting the coordinator
>
>
> Key: OOZIE-2873
> URL: https://issues.apache.org/jira/browse/OOZIE-2873
> Project: Oozie
> Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Minor
> Attachments: OOZIE-2873-1.patch
>
>
> Oozie doesn't check for el functions while submitting the coordinator job.
> Later on the coordinator action(s) can remain in WAITING state if there user
> has messed up the el functions.
> For Example.
> {code}
>
> frequency="${coord:hours(1)}" start="${startTime}" end="${endTime}"
> timezone="UTC">
>
> 1
> FIFO
>
>
> initial-instance="${initInstanceTime}" timezone="UTC">
>
> hcat://${HCAT_SERVER}/${HCAT_DB_NAME}/${TABLE_NAME}/dt=${YEAR}${MONTH}${DAY};hr=${HOUR}
>
>
>
>
> ${coord:current(0)}
>
>
>
>
>
> ${oozieAppWorkflowPath}/my_workflow.xml
>
>
> mmdd
> ${coord:formatTime(coord:nominalTime(),
> 'DAY')}
>
>
> hh
>
> ${coord:formatTime(coord:nominalTime(),'HH')}
>
>
>
>
>
> {code}
> After Oozie finds out the dependency.
> {code}
> 2017-04-25 16:51:53,503 DEBUG DependencyChecker:526 [pool-11-thread-66] -
> SERVER[localhost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0100010100101-saley-C]
> ACTION[0100010100101-saley-C@1] Dependency
> [hcat://localhost:9098/my_database/my_table/dt=20170411;hr=02] is available
> {code}
> The issue is with el function
> {code}
> 2017-04-25 16:51:53,506 ERROR CoordPushDependencyCheckXCommand:517
> [pool-11-thread-66] - SERVER[localhost] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[0100010100101-saley-C] ACTION[0100010100101-saley-C@1] XException,
> org.apache.oozie.command.CommandException: E1021: Coord Action Input Check
> Error: E1021: Coord Action Input Check Error: Unable to evaluate
> :${coord:formatTime(coord:nominalTime(), 'DAY')}:
>
>
> mmdd
> ${coord:formatTime(coord:nominalTime(),
> 'DAY')}
>
> {code}
> The coord action remained in WAITING state.
> Solution:
> We should error out at the time of job submission. Currently users are
> supposed to run dry run on the coordinator before actually running it. But
> everybody wants to run directly. We should run dry run by default to catch
> such errors. While working the fix, I have found some buggy test cases which
> would have been caught if we run dry run first. Fixing those cases as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2873) Check El Functions before submitting the coordinator
[
https://issues.apache.org/jira/browse/OOZIE-2873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995926#comment-15995926
]
Purshotam Shah commented on OOZIE-2873:
---
+1
> Check El Functions before submitting the coordinator
>
>
> Key: OOZIE-2873
> URL: https://issues.apache.org/jira/browse/OOZIE-2873
> Project: Oozie
> Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Minor
> Attachments: OOZIE-2873-1.patch
>
>
> Oozie doesn't check for el functions while submitting the coordinator job.
> Later on the coordinator action(s) can remain in WAITING state if there user
> has messed up the el functions.
> For Example.
> {code}
>
> frequency="${coord:hours(1)}" start="${startTime}" end="${endTime}"
> timezone="UTC">
>
> 1
> FIFO
>
>
> initial-instance="${initInstanceTime}" timezone="UTC">
>
> hcat://${HCAT_SERVER}/${HCAT_DB_NAME}/${TABLE_NAME}/dt=${YEAR}${MONTH}${DAY};hr=${HOUR}
>
>
>
>
> ${coord:current(0)}
>
>
>
>
>
> ${oozieAppWorkflowPath}/my_workflow.xml
>
>
> mmdd
> ${coord:formatTime(coord:nominalTime(),
> 'DAY')}
>
>
> hh
>
> ${coord:formatTime(coord:nominalTime(),'HH')}
>
>
>
>
>
> {code}
> After Oozie finds out the dependency.
> {code}
> 2017-04-25 16:51:53,503 DEBUG DependencyChecker:526 [pool-11-thread-66] -
> SERVER[localhost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0100010100101-saley-C]
> ACTION[0100010100101-saley-C@1] Dependency
> [hcat://localhost:9098/my_database/my_table/dt=20170411;hr=02] is available
> {code}
> The issue is with el function
> {code}
> 2017-04-25 16:51:53,506 ERROR CoordPushDependencyCheckXCommand:517
> [pool-11-thread-66] - SERVER[localhost] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[0100010100101-saley-C] ACTION[0100010100101-saley-C@1] XException,
> org.apache.oozie.command.CommandException: E1021: Coord Action Input Check
> Error: E1021: Coord Action Input Check Error: Unable to evaluate
> :${coord:formatTime(coord:nominalTime(), 'DAY')}:
>
>
> mmdd
> ${coord:formatTime(coord:nominalTime(),
> 'DAY')}
>
> {code}
> The coord action remained in WAITING state.
> Solution:
> We should error out at the time of job submission. Currently users are
> supposed to run dry run on the coordinator before actually running it. But
> everybody wants to run directly. We should run dry run by default to catch
> such errors. While working the fix, I have found some buggy test cases which
> would have been caught if we run dry run first. Fixing those cases as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2863) SLACalculatorMemory.loadOnRestart causing delay in server start
[ https://issues.apache.org/jira/browse/OOZIE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975862#comment-15975862 ] Purshotam Shah commented on OOZIE-2863: --- Test failures are related to OOZIE-2457. > SLACalculatorMemory.loadOnRestart causing delay in server start > --- > > Key: OOZIE-2863 > URL: https://issues.apache.org/jira/browse/OOZIE-2863 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah > Attachments: OOZIE-2863-V2.patch > > > LoadOnRestart load all job and tries to calculate SLA status. Based on job > and SLA status, it moves job into historyMap or slaMap. > Mostly, the slowness is caused by calculating SLA status (since it needs to > acquire ZK lock and make DB calls for each SLA job modified in last seven > days). > LoadOnRestart should only populate slaMap entry, and periodic SLA check > should take care of calculating SLA status. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OOZIE-2863) SLACalculatorMemory.loadOnRestart causing delay in server start
[ https://issues.apache.org/jira/browse/OOZIE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-2863: -- Attachment: OOZIE-2863-V2.patch > SLACalculatorMemory.loadOnRestart causing delay in server start > --- > > Key: OOZIE-2863 > URL: https://issues.apache.org/jira/browse/OOZIE-2863 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah > Attachments: OOZIE-2863-V2.patch > > > LoadOnRestart load all job and tries to calculate SLA status. Based on job > and SLA status, it moves job into historyMap or slaMap. > Mostly, the slowness is caused by calculating SLA status (since it needs to > acquire ZK lock and make DB calls for each SLA job modified in last seven > days). > LoadOnRestart should only populate slaMap entry, and periodic SLA check > should take care of calculating SLA status. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OOZIE-2863) SLACalculatorMemory.loadOnRestart causing delay in server start
[ https://issues.apache.org/jira/browse/OOZIE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-2863: -- Attachment: (was: OOZIE-2863-V2.patch) > SLACalculatorMemory.loadOnRestart causing delay in server start > --- > > Key: OOZIE-2863 > URL: https://issues.apache.org/jira/browse/OOZIE-2863 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah > Attachments: OOZIE-2863-V2.patch > > > LoadOnRestart load all job and tries to calculate SLA status. Based on job > and SLA status, it moves job into historyMap or slaMap. > Mostly, the slowness is caused by calculating SLA status (since it needs to > acquire ZK lock and make DB calls for each SLA job modified in last seven > days). > LoadOnRestart should only populate slaMap entry, and periodic SLA check > should take care of calculating SLA status. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Re: Review Request 58406: OOZIE-2863 SLACalculatorMemory.loadOnRestart causing delay in server start
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58406/ --- (Updated April 19, 2017, 1:23 a.m.) Review request for oozie. Bugs: OOZIE-2863 https://issues.apache.org/jira/browse/OOZIE-2863 Repository: oozie-git Description --- SLACalculatorMemory.loadOnRestart causing delay in server start Diffs (updated) - core/src/main/java/org/apache/oozie/sla/SLACalcStatus.java 3a76dfe6 core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java 347f8537 core/src/main/java/org/apache/oozie/sla/service/SLAService.java 08cd07e7 core/src/test/java/org/apache/oozie/service/TestHASLAService.java 3af263e1 core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java 559e2b31 Diff: https://reviews.apache.org/r/58406/diff/4/ Changes: https://reviews.apache.org/r/58406/diff/3-4/ Testing --- Thanks, Purshotam Shah
Re: Review Request 58406: OOZIE-2863 SLACalculatorMemory.loadOnRestart causing delay in server start
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58406/ --- (Updated April 18, 2017, 7 p.m.) Review request for oozie. Bugs: OOZIE-2863 https://issues.apache.org/jira/browse/OOZIE-2863 Repository: oozie-git Description --- SLACalculatorMemory.loadOnRestart causing delay in server start Diffs (updated) - core/src/main/java/org/apache/oozie/sla/SLACalcStatus.java 3a76dfe6 core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java 347f8537 core/src/test/java/org/apache/oozie/service/TestHASLAService.java 3af263e1 core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java 559e2b31 Diff: https://reviews.apache.org/r/58406/diff/3/ Changes: https://reviews.apache.org/r/58406/diff/2-3/ Testing --- Thanks, Purshotam Shah
[jira] [Updated] (OOZIE-2863) SLACalculatorMemory.loadOnRestart causing delay in server start
[ https://issues.apache.org/jira/browse/OOZIE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-2863: -- Attachment: OOZIE-2863-V2.patch > SLACalculatorMemory.loadOnRestart causing delay in server start > --- > > Key: OOZIE-2863 > URL: https://issues.apache.org/jira/browse/OOZIE-2863 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah > Attachments: OOZIE-2863-V2.patch > > > LoadOnRestart load all job and tries to calculate SLA status. Based on job > and SLA status, it moves job into historyMap or slaMap. > Mostly, the slowness is caused by calculating SLA status (since it needs to > acquire ZK lock and make DB calls for each SLA job modified in last seven > days). > LoadOnRestart should only populate slaMap entry, and periodic SLA check > should take care of calculating SLA status. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Re: Review Request 58406: OOZIE-2863 SLACalculatorMemory.loadOnRestart causing delay in server start
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58406/ --- (Updated April 18, 2017, 5:34 p.m.) Review request for oozie. Bugs: OOZIE-2863 https://issues.apache.org/jira/browse/OOZIE-2863 Repository: oozie-git Description --- SLACalculatorMemory.loadOnRestart causing delay in server start Diffs (updated) - core/src/main/java/org/apache/oozie/sla/SLACalcStatus.java 3a76dfe6 core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java 347f8537 core/src/test/java/org/apache/oozie/service/TestHASLAService.java 3af263e1 core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java 559e2b31 Diff: https://reviews.apache.org/r/58406/diff/2/ Changes: https://reviews.apache.org/r/58406/diff/1-2/ Testing --- Thanks, Purshotam Shah
Re: Review Request 58406: OOZIE-2863 SLACalculatorMemory.loadOnRestart causing delay in server start
> On April 17, 2017, 11:42 p.m., Satish Saley wrote:
> > core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java
> > Lines 221 (patched)
> > <https://reviews.apache.org/r/58406/diff/1/?file=1692867#file1692867line249>
> >
> > Please move this inside synchronized block on line below to get
> > consistent view of slaCalc object across threads. Adjust the code inside
> > checkAndUpdateSLACalcAfterRestart-->updateSLARegistartion accordingly. We
> > don't need the synchronized block in updateSLARegistartion then.
We need to synchronize block so that only one thread update the slaCalc.
> On April 17, 2017, 11:42 p.m., Satish Saley wrote:
> > core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java
> > Line 250 (original), 234 (patched)
> > <https://reviews.apache.org/r/58406/diff/1/?file=1692867#file1692867line262>
> >
> > Do we need else condition similar to that line 265?
> >
> > ```
> > else{
> > if (firstCheckAfterRetstart) {
> >slaCalc.setSLARegistrationBean(null);
> > }
> > }
> > ```
Not sure what do you mean by this? Are you saying that we need to check
following condition?
if (e.getErrorCode().equals(ErrorCode.E0604) ||
e.getErrorCode().equals(ErrorCode.E0605)) {
LOG.debug("job [{0}] is is not in DB, removing from
Memory", slaCalc.getId());
slaMap.remove(jobId);
}
If yes, No we don't have to check if an object exists or not. addJobStatus will
get called when there is a change in job status which means the object is in DB.
> On April 17, 2017, 11:42 p.m., Satish Saley wrote:
> > core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java
> > Line 476 (original), 464 (patched)
> > <https://reviews.apache.org/r/58406/diff/1/?file=1692867#file1692867line493>
> >
> > Need to wrap the call to checkAndUpdateSLACalcAfterRestart in
> > synchronized(slaCalc) {} with the changes I suggested regarding
> > synchronized block on line 221.
If I understand correctly, Do you mean to say that we should synchronize
checkAndUpdateSLACalcAfterRestart.
We don't want to synchronized check. Only update need synchronization.
> On April 17, 2017, 11:42 p.m., Satish Saley wrote:
> > core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java
> > Lines 657 (patched)
> > <https://reviews.apache.org/r/58406/diff/1/?file=1692867#file1692867line686>
> >
> > No need to check slaCalc.getSLARegistrationBean() == null here, inside
> > synchronized(slaCalc) we are doing it.
Yes. If the call fails, we need to make sure that next run will execute SLA
check.
- Purshotam
-------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58406/#review172149
---
On April 14, 2017, 10:41 p.m., Purshotam Shah wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58406/
> ---
>
> (Updated April 14, 2017, 10:41 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-2863
> https://issues.apache.org/jira/browse/OOZIE-2863
>
>
> Repository: oozie-git
>
>
> Description
> ---
>
> SLACalculatorMemory.loadOnRestart causing delay in server start
>
>
> Diffs
> -
>
> core/src/main/java/org/apache/oozie/sla/SLACalcStatus.java 3a76dfe6
> core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java 347f8537
> core/src/test/java/org/apache/oozie/service/TestHASLAService.java 3af263e1
> core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java
> 559e2b31
>
>
> Diff: https://reviews.apache.org/r/58406/diff/1/
>
>
> Testing
> ---
>
>
> Thanks,
>
> Purshotam Shah
>
>
[jira] [Assigned] (OOZIE-2863) SLACalculatorMemory.loadOnRestart causing delay in server start
[ https://issues.apache.org/jira/browse/OOZIE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah reassigned OOZIE-2863: - Assignee: Purshotam Shah > SLACalculatorMemory.loadOnRestart causing delay in server start > --- > > Key: OOZIE-2863 > URL: https://issues.apache.org/jira/browse/OOZIE-2863 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah > > LoadOnRestart load all job and tries to calculate SLA status. Based on job > and SLA status, it moves job into historyMap or slaMap. > Mostly, the slowness is caused by calculating SLA status (since it needs to > acquire ZK lock and make DB calls for each SLA job modified in last seven > days). > LoadOnRestart should only populate slaMap entry, and periodic SLA check > should take care of calculating SLA status. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Review Request 58406: OOZIE-2863 SLACalculatorMemory.loadOnRestart causing delay in server start
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58406/ --- Review request for oozie. Bugs: OOZIE-2863 https://issues.apache.org/jira/browse/OOZIE-2863 Repository: oozie-git Description --- SLACalculatorMemory.loadOnRestart causing delay in server start Diffs - core/src/main/java/org/apache/oozie/sla/SLACalcStatus.java 3a76dfe6 core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java 347f8537 core/src/test/java/org/apache/oozie/service/TestHASLAService.java 3af263e1 core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java 559e2b31 Diff: https://reviews.apache.org/r/58406/diff/1/ Testing --- Thanks, Purshotam Shah
[jira] [Updated] (OOZIE-2815) Oozie not always display job log
[ https://issues.apache.org/jira/browse/OOZIE-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-2815: -- Fix Version/s: 5.0.0 > Oozie not always display job log > > > Key: OOZIE-2815 > URL: https://issues.apache.org/jira/browse/OOZIE-2815 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah > Fix For: 5.0.0 > > Attachments: OOZIE-2815-V2.patch, OOZIE-2815-V3.patch, > OOZIE-2815-V4.patch, OOZIE-2815-V5.patch > > > There are few issues with log streaming. We will fix following things. > 1. UI has an issue with displaying "loading... done". It used to show loading > done even if logs were not loaded entirely. > 2. Server-server communication time out. The default time is 1 min. Setting > to 3 min same as UI. > 3. We use to throw an error message if log scan duration is more than > configured duration. We will be truncating log. > 4. No log scan duration check for audit and error log since it doesn't > support log filter. > 5. Different buffer.len setting for error and audit log, so that partial > loading can be faster. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OOZIE-2862) Coord change command doesn't change job to running if job was killed without creating any actions
[
https://issues.apache.org/jira/browse/OOZIE-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-2862:
--
Fix Version/s: 5.0.0
> Coord change command doesn't change job to running if job was killed without
> creating any actions
> -
>
> Key: OOZIE-2862
> URL: https://issues.apache.org/jira/browse/OOZIE-2862
> Project: Oozie
> Issue Type: Bug
>Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Fix For: 5.0.0
>
> Attachments: OOZIE-2862-V1.patch, OOZIE-2862-V2.patch
>
>
> {code:title=CoordChangeXCommand.java}
> if (jobStatus != null) {
> coordJob.setStatus(jobStatus);
> LOG.info("Coord status is changed to " + jobStatus + " from " +
> prevStatus);
> if (jobStatus.equals(CoordinatorJob.Status.RUNNING)) {
> coordJob.setPending();
> if (coordJob.getNextMaterializedTime() != null
> &&
> coordJob.getEndTime().after(coordJob.getNextMaterializedTime())) {
> coordJob.resetDoneMaterialization();
> }
> } else if (jobStatus.equals(CoordinatorJob.Status.IGNORED)) {
> coordJob.resetPending();
> coordJob.setDoneMaterialization();
> }
> }
> {code}
> If the job was killed without creating any action, then
> {{coordJob.getNextMaterializedTime()}} will be null and change command will
> not reset done materialization.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2815) Oozie not always display job log
[ https://issues.apache.org/jira/browse/OOZIE-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966646#comment-15966646 ] Purshotam Shah commented on OOZIE-2815: --- Thanks, Satish for review. Committed to master. > Oozie not always display job log > > > Key: OOZIE-2815 > URL: https://issues.apache.org/jira/browse/OOZIE-2815 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah > Attachments: OOZIE-2815-V2.patch, OOZIE-2815-V3.patch, > OOZIE-2815-V4.patch, OOZIE-2815-V5.patch > > > There are few issues with log streaming. We will fix following things. > 1. UI has an issue with displaying "loading... done". It used to show loading > done even if logs were not loaded entirely. > 2. Server-server communication time out. The default time is 1 min. Setting > to 3 min same as UI. > 3. We use to throw an error message if log scan duration is more than > configured duration. We will be truncating log. > 4. No log scan duration check for audit and error log since it doesn't > support log filter. > 5. Different buffer.len setting for error and audit log, so that partial > loading can be faster. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2862) Coord change command doesn't change job to running if job was killed without creating any actions
[
https://issues.apache.org/jira/browse/OOZIE-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966644#comment-15966644
]
Purshotam Shah commented on OOZIE-2862:
---
Thanks Satish. Committed to master.
> Coord change command doesn't change job to running if job was killed without
> creating any actions
> -
>
> Key: OOZIE-2862
> URL: https://issues.apache.org/jira/browse/OOZIE-2862
> Project: Oozie
> Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-2862-V1.patch, OOZIE-2862-V2.patch
>
>
> {code:title=CoordChangeXCommand.java}
> if (jobStatus != null) {
> coordJob.setStatus(jobStatus);
> LOG.info("Coord status is changed to " + jobStatus + " from " +
> prevStatus);
> if (jobStatus.equals(CoordinatorJob.Status.RUNNING)) {
> coordJob.setPending();
> if (coordJob.getNextMaterializedTime() != null
> &&
> coordJob.getEndTime().after(coordJob.getNextMaterializedTime())) {
> coordJob.resetDoneMaterialization();
> }
> } else if (jobStatus.equals(CoordinatorJob.Status.IGNORED)) {
> coordJob.resetPending();
> coordJob.setDoneMaterialization();
> }
> }
> {code}
> If the job was killed without creating any action, then
> {{coordJob.getNextMaterializedTime()}} will be null and change command will
> not reset done materialization.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2862) Coord change command doesn't change job to running if job was killed without creating any actions
[
https://issues.apache.org/jira/browse/OOZIE-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-2862:
--
Attachment: (was: OOZIE-2862-V2.patch)
> Coord change command doesn't change job to running if job was killed without
> creating any actions
> -
>
> Key: OOZIE-2862
> URL: https://issues.apache.org/jira/browse/OOZIE-2862
> Project: Oozie
> Issue Type: Bug
>Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Attachments: OOZIE-2862-V1.patch, OOZIE-2862-V2.patch
>
>
> {code:title=CoordChangeXCommand.java}
> if (jobStatus != null) {
> coordJob.setStatus(jobStatus);
> LOG.info("Coord status is changed to " + jobStatus + " from " +
> prevStatus);
> if (jobStatus.equals(CoordinatorJob.Status.RUNNING)) {
> coordJob.setPending();
> if (coordJob.getNextMaterializedTime() != null
> &&
> coordJob.getEndTime().after(coordJob.getNextMaterializedTime())) {
> coordJob.resetDoneMaterialization();
> }
> } else if (jobStatus.equals(CoordinatorJob.Status.IGNORED)) {
> coordJob.resetPending();
> coordJob.setDoneMaterialization();
> }
> }
> {code}
> If the job was killed without creating any action, then
> {{coordJob.getNextMaterializedTime()}} will be null and change command will
> not reset done materialization.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2862) Coord change command doesn't change job to running if job was killed without creating any actions
[
https://issues.apache.org/jira/browse/OOZIE-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-2862:
--
Attachment: OOZIE-2862-V2.patch
> Coord change command doesn't change job to running if job was killed without
> creating any actions
> -
>
> Key: OOZIE-2862
> URL: https://issues.apache.org/jira/browse/OOZIE-2862
> Project: Oozie
> Issue Type: Bug
>Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Attachments: OOZIE-2862-V1.patch, OOZIE-2862-V2.patch
>
>
> {code:title=CoordChangeXCommand.java}
> if (jobStatus != null) {
> coordJob.setStatus(jobStatus);
> LOG.info("Coord status is changed to " + jobStatus + " from " +
> prevStatus);
> if (jobStatus.equals(CoordinatorJob.Status.RUNNING)) {
> coordJob.setPending();
> if (coordJob.getNextMaterializedTime() != null
> &&
> coordJob.getEndTime().after(coordJob.getNextMaterializedTime())) {
> coordJob.resetDoneMaterialization();
> }
> } else if (jobStatus.equals(CoordinatorJob.Status.IGNORED)) {
> coordJob.resetPending();
> coordJob.setDoneMaterialization();
> }
> }
> {code}
> If the job was killed without creating any action, then
> {{coordJob.getNextMaterializedTime()}} will be null and change command will
> not reset done materialization.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2862) Coord change command doesn't change job to running if job was killed without creating any actions
[
https://issues.apache.org/jira/browse/OOZIE-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15964819#comment-15964819
]
Purshotam Shah commented on OOZIE-2862:
---
bq. Hi Purshotam Shah, thanks for your contribution! Can you please put the
changes to ReviewBoard?
[~andras.piros] this is just 1 line code change + some test cases. Anyway, I
have uploaded the patch to RB, if that make easy to review. Thanks.
> Coord change command doesn't change job to running if job was killed without
> creating any actions
> -
>
> Key: OOZIE-2862
> URL: https://issues.apache.org/jira/browse/OOZIE-2862
> Project: Oozie
> Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-2862-V1.patch, OOZIE-2862-V2.patch
>
>
> {code:title=CoordChangeXCommand.java}
> if (jobStatus != null) {
> coordJob.setStatus(jobStatus);
> LOG.info("Coord status is changed to " + jobStatus + " from " +
> prevStatus);
> if (jobStatus.equals(CoordinatorJob.Status.RUNNING)) {
> coordJob.setPending();
> if (coordJob.getNextMaterializedTime() != null
> &&
> coordJob.getEndTime().after(coordJob.getNextMaterializedTime())) {
> coordJob.resetDoneMaterialization();
> }
> } else if (jobStatus.equals(CoordinatorJob.Status.IGNORED)) {
> coordJob.resetPending();
> coordJob.setDoneMaterialization();
> }
> }
> {code}
> If the job was killed without creating any action, then
> {{coordJob.getNextMaterializedTime()}} will be null and change command will
> not reset done materialization.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
Review Request 58365: OOZIE-2862 Coord change command doesn't change job to running if job was killed without creating any actions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58365/ --- Review request for oozie. Bugs: OOZIE-2862 https://issues.apache.org/jira/browse/OOZIE-2862 Repository: oozie-git Description --- Coord change command doesn't change job to running if job was killed without creating any actions Diffs - core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java e65b74fc core/src/test/java/org/apache/oozie/command/coord/TestCoordChangeXCommand.java 8034bbe9 Diff: https://reviews.apache.org/r/58365/diff/1/ Testing --- Thanks, Purshotam Shah
[jira] [Updated] (OOZIE-2862) Coord change command doesn't change job to running if job was killed without creating any actions
[
https://issues.apache.org/jira/browse/OOZIE-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-2862:
--
Attachment: OOZIE-2862-V2.patch
> Coord change command doesn't change job to running if job was killed without
> creating any actions
> -
>
> Key: OOZIE-2862
> URL: https://issues.apache.org/jira/browse/OOZIE-2862
> Project: Oozie
> Issue Type: Bug
>Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Attachments: OOZIE-2862-V1.patch, OOZIE-2862-V2.patch
>
>
> {code:title=CoordChangeXCommand.java}
> if (jobStatus != null) {
> coordJob.setStatus(jobStatus);
> LOG.info("Coord status is changed to " + jobStatus + " from "
> + prevStatus);
> if (jobStatus.equals(CoordinatorJob.Status.RUNNING)) {
> coordJob.setPending();
> if (coordJob.getNextMaterializedTime() != null
> &&
> coordJob.getEndTime().after(coordJob.getNextMaterializedTime())) {
> coordJob.resetDoneMaterialization();
> }
> } else if (jobStatus.equals(CoordinatorJob.Status.IGNORED)) {
> coordJob.resetPending();
> coordJob.setDoneMaterialization();
> }
> }
> {code}
> If the job was killed without creating any action, then
> coordJob.getNextMaterializedTime() will be null and change command will not
> rest done materialization.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2862) Coord change command doesn't change job to running if job was killed without creating any actions
[
https://issues.apache.org/jira/browse/OOZIE-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-2862:
--
Attachment: OOZIE-2862-V1.patch
> Coord change command doesn't change job to running if job was killed without
> creating any actions
> -
>
> Key: OOZIE-2862
> URL: https://issues.apache.org/jira/browse/OOZIE-2862
> Project: Oozie
> Issue Type: Bug
>Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Attachments: OOZIE-2862-V1.patch
>
>
> {code:title=CoordChangeXCommand.java}
> if (jobStatus != null) {
> coordJob.setStatus(jobStatus);
> LOG.info("Coord status is changed to " + jobStatus + " from "
> + prevStatus);
> if (jobStatus.equals(CoordinatorJob.Status.RUNNING)) {
> coordJob.setPending();
> if (coordJob.getNextMaterializedTime() != null
> &&
> coordJob.getEndTime().after(coordJob.getNextMaterializedTime())) {
> coordJob.resetDoneMaterialization();
> }
> } else if (jobStatus.equals(CoordinatorJob.Status.IGNORED)) {
> coordJob.resetPending();
> coordJob.setDoneMaterialization();
> }
> }
> {code}
> If the job was killed without creating any action, then
> coordJob.getNextMaterializedTime() will be null and change command will not
> rest done materialization.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2815) Oozie not always display job log
[ https://issues.apache.org/jira/browse/OOZIE-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-2815: -- Attachment: OOZIE-2815-V5.patch > Oozie not always display job log > > > Key: OOZIE-2815 > URL: https://issues.apache.org/jira/browse/OOZIE-2815 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah > Attachments: OOZIE-2815-V2.patch, OOZIE-2815-V3.patch, > OOZIE-2815-V4.patch, OOZIE-2815-V5.patch > > > There are few issues with log streaming. We will fix following things. > 1. UI has an issue with displaying "loading... done". It used to show loading > done even if logs were not loaded entirely. > 2. Server-server communication time out. The default time is 1 min. Setting > to 3 min same as UI. > 3. We use to throw an error message if log scan duration is more than > configured duration. We will be truncating log. > 4. No log scan duration check for audit and error log since it doesn't > support log filter. > 5. Different buffer.len setting for error and audit log, so that partial > loading can be faster. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OOZIE-2863) SLACalculatorMemory.loadOnRestart causing delay in server start
[ https://issues.apache.org/jira/browse/OOZIE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-2863: -- Description: LoadOnRestart load all job and tries to calculate SLA status. Based on job and SLA status, it moves job into historyMap or slaMap. Mostly, the slowness is caused by calculating SLA status (since it needs to acquire ZK lock and make DB calls for each SLA job modified in last seven days). LoadOnRestart should only populate slaMap entry, and periodic SLA check should take care of calculating SLA status. > SLACalculatorMemory.loadOnRestart causing delay in server start > --- > > Key: OOZIE-2863 > URL: https://issues.apache.org/jira/browse/OOZIE-2863 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > > LoadOnRestart load all job and tries to calculate SLA status. Based on job > and SLA status, it moves job into historyMap or slaMap. > Mostly, the slowness is caused by calculating SLA status (since it needs to > acquire ZK lock and make DB calls for each SLA job modified in last seven > days). > LoadOnRestart should only populate slaMap entry, and periodic SLA check > should take care of calculating SLA status. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (OOZIE-2863) SLACalculatorMemory.loadOnRestart causing delay in server start
Purshotam Shah created OOZIE-2863: - Summary: SLACalculatorMemory.loadOnRestart causing delay in server start Key: OOZIE-2863 URL: https://issues.apache.org/jira/browse/OOZIE-2863 Project: Oozie Issue Type: Bug Reporter: Purshotam Shah -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (OOZIE-2862) Coord change command doesn't change job to running if job was killed without creating any actions
[
https://issues.apache.org/jira/browse/OOZIE-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah reassigned OOZIE-2862:
-
Assignee: Purshotam Shah
> Coord change command doesn't change job to running if job was killed without
> creating any actions
> -
>
> Key: OOZIE-2862
> URL: https://issues.apache.org/jira/browse/OOZIE-2862
> Project: Oozie
> Issue Type: Bug
>Reporter: Purshotam Shah
> Assignee: Purshotam Shah
>
> {code:title=CoordChangeXCommand.java}
> if (jobStatus != null) {
> coordJob.setStatus(jobStatus);
> LOG.info("Coord status is changed to " + jobStatus + " from "
> + prevStatus);
> if (jobStatus.equals(CoordinatorJob.Status.RUNNING)) {
> coordJob.setPending();
> if (coordJob.getNextMaterializedTime() != null
> &&
> coordJob.getEndTime().after(coordJob.getNextMaterializedTime())) {
> coordJob.resetDoneMaterialization();
> }
> } else if (jobStatus.equals(CoordinatorJob.Status.IGNORED)) {
> coordJob.resetPending();
> coordJob.setDoneMaterialization();
> }
> }
> {code}
> If the job was killed without creating any action, then
> coordJob.getNextMaterializedTime() will be null and change command will not
> rest done materialization.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2862) Coord change command doesn't change job to running if job was killed without creating any actions
[
https://issues.apache.org/jira/browse/OOZIE-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-2862:
--
Description:
{code:title=CoordChangeXCommand.java}
if (jobStatus != null) {
coordJob.setStatus(jobStatus);
LOG.info("Coord status is changed to " + jobStatus + " from " +
prevStatus);
if (jobStatus.equals(CoordinatorJob.Status.RUNNING)) {
coordJob.setPending();
if (coordJob.getNextMaterializedTime() != null
&&
coordJob.getEndTime().after(coordJob.getNextMaterializedTime())) {
coordJob.resetDoneMaterialization();
}
} else if (jobStatus.equals(CoordinatorJob.Status.IGNORED)) {
coordJob.resetPending();
coordJob.setDoneMaterialization();
}
}
{code}
If the job was killed without creating any action, then
coordJob.getNextMaterializedTime() will be null and change command will not
rest done materialization.
> Coord change command doesn't change job to running if job was killed without
> creating any actions
> -
>
> Key: OOZIE-2862
> URL: https://issues.apache.org/jira/browse/OOZIE-2862
> Project: Oozie
> Issue Type: Bug
>Reporter: Purshotam Shah
>
> {code:title=CoordChangeXCommand.java}
> if (jobStatus != null) {
> coordJob.setStatus(jobStatus);
> LOG.info("Coord status is changed to " + jobStatus + " from "
> + prevStatus);
> if (jobStatus.equals(CoordinatorJob.Status.RUNNING)) {
> coordJob.setPending();
> if (coordJob.getNextMaterializedTime() != null
> &&
> coordJob.getEndTime().after(coordJob.getNextMaterializedTime())) {
> coordJob.resetDoneMaterialization();
> }
> } else if (jobStatus.equals(CoordinatorJob.Status.IGNORED)) {
> coordJob.resetPending();
> coordJob.setDoneMaterialization();
> }
> }
> {code}
> If the job was killed without creating any action, then
> coordJob.getNextMaterializedTime() will be null and change command will not
> rest done materialization.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (OOZIE-2862) Coord change command doesn't change job to running if job was killed without creating any actions
Purshotam Shah created OOZIE-2862: - Summary: Coord change command doesn't change job to running if job was killed without creating any actions Key: OOZIE-2862 URL: https://issues.apache.org/jira/browse/OOZIE-2862 Project: Oozie Issue Type: Bug Reporter: Purshotam Shah -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OOZIE-2815) Oozie not always display job log
[ https://issues.apache.org/jira/browse/OOZIE-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-2815: -- Attachment: OOZIE-2815-V4.patch > Oozie not always display job log > > > Key: OOZIE-2815 > URL: https://issues.apache.org/jira/browse/OOZIE-2815 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah > Attachments: OOZIE-2815-V2.patch, OOZIE-2815-V3.patch, > OOZIE-2815-V4.patch > > > There are few issues with log streaming. We will fix following things. > 1. UI has an issue with displaying "loading... done". It used to show loading > done even if logs were not loaded entirely. > 2. Server-server communication time out. The default time is 1 min. Setting > to 3 min same as UI. > 3. We use to throw an error message if log scan duration is more than > configured duration. We will be truncating log. > 4. No log scan duration check for audit and error log since it doesn't > support log filter. > 5. Different buffer.len setting for error and audit log, so that partial > loading can be faster. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OOZIE-2815) Oozie not always display job log
[ https://issues.apache.org/jira/browse/OOZIE-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-2815: -- Attachment: OOZIE-2815-V3.patch > Oozie not always display job log > > > Key: OOZIE-2815 > URL: https://issues.apache.org/jira/browse/OOZIE-2815 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah > Attachments: OOZIE-2815-V2.patch, OOZIE-2815-V3.patch > > > There are few issues with log streaming. We will fix following things. > 1. UI has an issue with displaying "loading... done". It used to show loading > done even if logs were not loaded entirely. > 2. Server-server communication time out. The default time is 1 min. Setting > to 3 min same as UI. > 3. We use to throw an error message if log scan duration is more than > configured duration. We will be truncating log. > 4. No log scan duration check for audit and error log since it doesn't > support log filter. > 5. Different buffer.len setting for error and audit log, so that partial > loading can be faster. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OOZIE-2815) Oozie not always display job log
[ https://issues.apache.org/jira/browse/OOZIE-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-2815: -- Attachment: OOZIE-2815-V2.patch > Oozie not always display job log > > > Key: OOZIE-2815 > URL: https://issues.apache.org/jira/browse/OOZIE-2815 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah > Attachments: OOZIE-2815-V2.patch > > > There are few issues with log streaming. We will fix following things. > 1. UI has an issue with displaying "loading... done". It used to show loading > done even if logs were not loaded entirely. > 2. Server-server communication time out. The default time is 1 min. Setting > to 3 min same as UI. > 3. We use to throw an error message if log scan duration is more than > configured duration. We will be truncating log. > 4. No log scan duration check for audit and error log since it doesn't > support log filter. > 5. Different buffer.len setting for error and audit log, so that partial > loading can be faster. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Re: Review Request 57107: OOZIE-2815 oozie not always display job log
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57107/ --- (Updated April 4, 2017, 7:41 p.m.) Review request for oozie. Bugs: OOZIE-2815 https://issues.apache.org/jira/browse/OOZIE-2815 Repository: oozie-git Description --- OOZIE-2815 oozie not always display job log Diffs (updated) - core/src/main/java/org/apache/oozie/BaseEngine.java 50df8978773ed54fb5a39bb142d7235de1ed396f core/src/main/java/org/apache/oozie/BundleEngine.java d0099b48772069217375c032d171e8dc6b1d core/src/main/java/org/apache/oozie/CoordinatorEngine.java 2f9f8227f1b2bbb0074cfa19aac37ab9a3fe7c0f core/src/main/java/org/apache/oozie/DagEngine.java 57d276199de149b294f64570a8878b660e9a5a0c core/src/main/java/org/apache/oozie/service/XLogService.java 04f04f41403fc6446967524124b8cf86f816d71c core/src/main/java/org/apache/oozie/service/XLogStreamingService.java c15c4c1731838655d8f899f9c0405b226f8111b0 core/src/main/java/org/apache/oozie/service/ZKXLogStreamingService.java 97771ad22be84d1a8167c69a6b48b7012652cf8b core/src/main/java/org/apache/oozie/servlet/V0JobServlet.java d3b468965f33e300d7ca0f31efc596fd969b47a5 core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 9356768639e6e3ea548b25baa75d576351fa80e3 core/src/main/java/org/apache/oozie/util/AuthUrlClient.java b45a96acc397c26cdba8d5c62503013dd572f247 core/src/main/java/org/apache/oozie/util/TimestampedMessageParser.java a676f4d35a49147aadde73357397b211ab829c85 core/src/main/java/org/apache/oozie/util/XLogAuditFilter.java c377db5df5dd0bbeed77b4d5c88e1c4e2e64a51e core/src/main/java/org/apache/oozie/util/XLogAuditStreamer.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 core/src/main/java/org/apache/oozie/util/XLogErrorStreamer.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 core/src/main/java/org/apache/oozie/util/XLogFilter.java 3b49f77e016cbeaedfa03708dbc680e68f40ca16 core/src/main/java/org/apache/oozie/util/XLogStreamer.java 19f1fee55ba8d6c456e798f01e55cfe84cfb5803 core/src/main/resources/oozie-default.xml 95e0c36e1b2f5aef18637f6f956a990ba36166f2 core/src/test/java/org/apache/oozie/TestCoordinatorEngineStreamLog.java 3eb1016f3ffccaa9db077580b90cc1b3c53fd479 core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 42ffdbeb41e5829cdf814cf13ada331973fc33bd core/src/test/java/org/apache/oozie/service/TestXLogStreamingService.java bebb678798ec4a01c9710660d4b99fddd5167560 core/src/test/java/org/apache/oozie/service/TestZKXLogStreamingService.java fca8d844f1563da5856eff36f53f21d519630d31 core/src/test/java/org/apache/oozie/util/TestLogStreamer.java f90f784503e101ae2f96fee52977ee83f3098301 core/src/test/java/org/apache/oozie/util/TestSimplifiedTimestampedMessageParser.java ea899faf00aa2f98f43838987c9b716321a50dea core/src/test/java/org/apache/oozie/util/TestTimestampedMessageParser.java 9e28cbc77129bb2675b92ec9c18c77accfd78a55 core/src/test/java/org/apache/oozie/util/TestXLogUserFilterParam.java 46f273fde7f14b446463448c90f22519761997b6 webapp/src/main/webapp/oozie-console.js 76864a9a06e98357c2d17e35609e2a76afbd595c Diff: https://reviews.apache.org/r/57107/diff/3/ Changes: https://reviews.apache.org/r/57107/diff/2-3/ Testing --- Thanks, Purshotam Shah
Re: Review Request 57107: OOZIE-2815 oozie not always display job log
> On March 31, 2017, 5:48 p.m., Satish Saley wrote: > > core/src/main/java/org/apache/oozie/service/ZKXLogStreamingService.java > > Line 324 (original), 251 (patched) > > <https://reviews.apache.org/r/57107/diff/2/?file=1663060#file1663060line332> > > > > I see that we are truncating logs for XLogErrorStreamer and > > XLogAuditStreamer rather than throwing exception. > > But still there could be exeptions (may not be explicitly thrown) such > > as IOException from TimestampedMessageParser.increment() method. It would > > be safer to have a catch block, setting isException=true and flushing to > > the writer only if isException is false. That will take care of OOZIE-2636 We don't have to call flush on finally block. Writer is HttpServletResponse writer. In the end, the container will flush and close the HttpServletResponse stream. > On March 31, 2017, 5:48 p.m., Satish Saley wrote: > > core/src/main/java/org/apache/oozie/service/ZKXLogStreamingService.java > > Line 324 (original), 251 (patched) > > <https://reviews.apache.org/r/57107/diff/2/?file=1663060#file1663060line332> > > > > I see that we are truncating logs for XLogErrorStreamer and > > XLogAuditStreamer rather than throwing exception. > > But still there could be exeptions (may not be explicitly thrown) such > > as IOException from TimestampedMessageParser.increment() method. It would > > be safer to have a catch block, setting isException=true and flushing to > > the writer only if isException is false. That will take care of OOZIE-2636 We don't have to call flush on finally block. Writer is HttpServletResponse writer. In the end, the container will flush and close the HttpServletResponse stream. - Purshotam --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57107/#review170651 --- On March 14, 2017, 2:03 a.m., Purshotam Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/57107/ > --- > > (Updated March 14, 2017, 2:03 a.m.) > > > Review request for oozie. > > > Bugs: OOZIE-2815 > https://issues.apache.org/jira/browse/OOZIE-2815 > > > Repository: oozie-git > > > Description > --- > > OOZIE-2815 oozie not always display job log > > > Diffs > - > > core/src/main/java/org/apache/oozie/BaseEngine.java > 50df8978773ed54fb5a39bb142d7235de1ed396f > core/src/main/java/org/apache/oozie/BundleEngine.java > d0099b48772069217375c032d171e8dc6b1d > core/src/main/java/org/apache/oozie/CoordinatorEngine.java > 2f9f8227f1b2bbb0074cfa19aac37ab9a3fe7c0f > core/src/main/java/org/apache/oozie/DagEngine.java > 57d276199de149b294f64570a8878b660e9a5a0c > core/src/main/java/org/apache/oozie/service/XLogService.java > 04f04f41403fc6446967524124b8cf86f816d71c > core/src/main/java/org/apache/oozie/service/XLogStreamingService.java > c15c4c1731838655d8f899f9c0405b226f8111b0 > core/src/main/java/org/apache/oozie/service/ZKXLogStreamingService.java > 97771ad22be84d1a8167c69a6b48b7012652cf8b > core/src/main/java/org/apache/oozie/servlet/V0JobServlet.java > d3b468965f33e300d7ca0f31efc596fd969b47a5 > core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java > 9356768639e6e3ea548b25baa75d576351fa80e3 > core/src/main/java/org/apache/oozie/util/AuthUrlClient.java > b45a96acc397c26cdba8d5c62503013dd572f247 > core/src/main/java/org/apache/oozie/util/TimestampedMessageParser.java > a676f4d35a49147aadde73357397b211ab829c85 > core/src/main/java/org/apache/oozie/util/XLogAuditFilter.java > c377db5df5dd0bbeed77b4d5c88e1c4e2e64a51e > core/src/main/java/org/apache/oozie/util/XLogAuditStreamer.java > e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 > core/src/main/java/org/apache/oozie/util/XLogErrorStreamer.java > e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 > core/src/main/java/org/apache/oozie/util/XLogFilter.java > 3b49f77e016cbeaedfa03708dbc680e68f40ca16 > core/src/main/java/org/apache/oozie/util/XLogStreamer.java > 19f1fee55ba8d6c456e798f01e55cfe84cfb5803 > core/src/main/resources/oozie-default.xml > 95e0c36e1b2f5aef18637f6f956a990ba36166f2 > core/src/test/java/org/apache/oozie/TestCoordinatorEngineStreamLog.java > 3eb1016f3ffccaa9db077580b90cc1b3c53fd479 > core/src/test/java/org/apache/oozie/service/TestConfigurationService.java > 42ffdbeb41e5829cdf814cf13ada331973fc33bd > core
Re: Review Request 57107: OOZIE-2815 oozie not always display job log
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57107/ --- (Updated March 14, 2017, 2:03 a.m.) Review request for oozie. Bugs: OOZIE-2815 https://issues.apache.org/jira/browse/OOZIE-2815 Repository: oozie-git Description --- OOZIE-2815 oozie not always display job log Diffs (updated) - core/src/main/java/org/apache/oozie/BaseEngine.java 50df8978773ed54fb5a39bb142d7235de1ed396f core/src/main/java/org/apache/oozie/BundleEngine.java d0099b48772069217375c032d171e8dc6b1d core/src/main/java/org/apache/oozie/CoordinatorEngine.java 2f9f8227f1b2bbb0074cfa19aac37ab9a3fe7c0f core/src/main/java/org/apache/oozie/DagEngine.java 57d276199de149b294f64570a8878b660e9a5a0c core/src/main/java/org/apache/oozie/service/XLogService.java 04f04f41403fc6446967524124b8cf86f816d71c core/src/main/java/org/apache/oozie/service/XLogStreamingService.java c15c4c1731838655d8f899f9c0405b226f8111b0 core/src/main/java/org/apache/oozie/service/ZKXLogStreamingService.java 97771ad22be84d1a8167c69a6b48b7012652cf8b core/src/main/java/org/apache/oozie/servlet/V0JobServlet.java d3b468965f33e300d7ca0f31efc596fd969b47a5 core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 9356768639e6e3ea548b25baa75d576351fa80e3 core/src/main/java/org/apache/oozie/util/AuthUrlClient.java b45a96acc397c26cdba8d5c62503013dd572f247 core/src/main/java/org/apache/oozie/util/TimestampedMessageParser.java a676f4d35a49147aadde73357397b211ab829c85 core/src/main/java/org/apache/oozie/util/XLogAuditFilter.java c377db5df5dd0bbeed77b4d5c88e1c4e2e64a51e core/src/main/java/org/apache/oozie/util/XLogAuditStreamer.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 core/src/main/java/org/apache/oozie/util/XLogErrorStreamer.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 core/src/main/java/org/apache/oozie/util/XLogFilter.java 3b49f77e016cbeaedfa03708dbc680e68f40ca16 core/src/main/java/org/apache/oozie/util/XLogStreamer.java 19f1fee55ba8d6c456e798f01e55cfe84cfb5803 core/src/main/resources/oozie-default.xml 95e0c36e1b2f5aef18637f6f956a990ba36166f2 core/src/test/java/org/apache/oozie/TestCoordinatorEngineStreamLog.java 3eb1016f3ffccaa9db077580b90cc1b3c53fd479 core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 42ffdbeb41e5829cdf814cf13ada331973fc33bd core/src/test/java/org/apache/oozie/service/TestXLogStreamingService.java bebb678798ec4a01c9710660d4b99fddd5167560 core/src/test/java/org/apache/oozie/service/TestZKXLogStreamingService.java fca8d844f1563da5856eff36f53f21d519630d31 core/src/test/java/org/apache/oozie/util/TestLogStreamer.java f90f784503e101ae2f96fee52977ee83f3098301 core/src/test/java/org/apache/oozie/util/TestSimplifiedTimestampedMessageParser.java ea899faf00aa2f98f43838987c9b716321a50dea core/src/test/java/org/apache/oozie/util/TestTimestampedMessageParser.java 9e28cbc77129bb2675b92ec9c18c77accfd78a55 core/src/test/java/org/apache/oozie/util/TestXLogUserFilterParam.java 46f273fde7f14b446463448c90f22519761997b6 webapp/src/main/webapp/oozie-console.js 76864a9a06e98357c2d17e35609e2a76afbd595c Diff: https://reviews.apache.org/r/57107/diff/2/ Changes: https://reviews.apache.org/r/57107/diff/1-2/ Testing --- Thanks, Purshotam Shah
Re: Review Request 57107: OOZIE-2815 oozie not always display job log
> On March 9, 2017, 1:36 p.m., András Piros wrote: > > core/src/main/java/org/apache/oozie/BaseEngine.java > > Line 192 (original), 197-203 (patched) > > <https://reviews.apache.org/r/57107/diff/1/?file=1659376#file1659376line202> > > > > Code duplication: can you just please use template method pattern? It's not duplicate code. Each function takes different streamer. > On March 9, 2017, 1:36 p.m., András Piros wrote: > > core/src/main/java/org/apache/oozie/BaseEngine.java > > Line 204 (original), 216-221 (patched) > > <https://reviews.apache.org/r/57107/diff/1/?file=1659376#file1659376line221> > > > > Code duplication: can you just please use template method pattern? It's not duplicate code. Each function takes different streamer. > On March 9, 2017, 1:36 p.m., András Piros wrote: > > core/src/main/java/org/apache/oozie/util/XLogFilter.java > > Lines 302-305 (patched) > > <https://reviews.apache.org/r/57107/diff/1/?file=1659390#file1659390line304> > > > > This check should come just after `startDate` and `endDate` have both > > been set. Not sure what you mean by this? > On March 9, 2017, 1:36 p.m., András Piros wrote: > > core/src/main/java/org/apache/oozie/util/XLogStreamer.java > > Lines 62-64 (patched) > > <https://reviews.apache.org/r/57107/diff/1/?file=1659391#file1659391line65> > > > > Since the parameter `logFile` shadows the field `logFile`, and the > > field `logFile` has already been set, no effect is taken by that, breaking > > the functionality. > > > > Either move that check-and-set back to its original place or remove it > > completely. You are right. Intention was to set this.logFile = "oozie-app.log"; not logFile = "oozie-app.log". This is one of the reasons we should avoid the same name for global and local variables. > On March 9, 2017, 1:36 p.m., András Piros wrote: > > core/src/main/java/org/apache/oozie/util/XLogStreamer.java > > Lines 318-327 (patched) > > <https://reviews.apache.org/r/57107/diff/1/?file=1659391#file1659391line321> > > > > Parameter `writtenBytes` is not used. Is it intentional? If so, please > > rename it, like `byteCountIgnored`. > > > > Moreover, please rename the method to e.g. `checkAndFlush()` to better > > reflect what it actually does. writtenBytes is used. checkAndFlush may not be an appropriate name because the function doesn't call flush. It only says should caller flush or not. > On March 9, 2017, 1:36 p.m., András Piros wrote: > > core/src/main/java/org/apache/oozie/util/XLogStreamer.java > > Lines 334-335 (patched) > > <https://reviews.apache.org/r/57107/diff/1/?file=1659391#file1659391line337> > > > > Using `String.format()`, a `StringBuilder` instance, or Guava's > > `Objects.toStringHelper()` would be way sexier. Agree. > On March 9, 2017, 1:36 p.m., András Piros wrote: > > core/src/main/resources/oozie-default.xml > > Lines 207 (patched) > > <https://reviews.apache.org/r/57107/diff/1/?file=1659392#file1659392line207> > > > > `oozie.service.XLogStreamingService.audit.buffer.lineCount` would be a > > better name. This is debatbale. In most of the we just replace the type to change to value. > On March 9, 2017, 1:36 p.m., András Piros wrote: > > core/src/main/resources/oozie-default.xml > > Lines 2235 (patched) > > <https://reviews.apache.org/r/57107/diff/1/?file=1659392#file1659392line2235> > > > > `oozie.server.connection.timeout.seconds` would be a better name. hmmm... Most of are value connection timeouts are set in millisecond like oozie.command.default.lock.timeout, oozie.notification.url.connection.timeout. Setting in seconds is more favorable because we are never going to set it in ms. - Purshotam --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57107/#review168443 --- On March 14, 2017, 2:03 a.m., Purshotam Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/57107/ > --- > > (Updated March 14, 2017, 2:03 a.m.) > > > Review request for oozie. > > > Bugs: OOZIE-2815 > https://issues.apache.org/jira/browse/OOZIE-2815 > > > Repository: oozie-git > > > Description > --- > > OOZIE-2815 oozie not always display job log > &g
[jira] [Assigned] (OOZIE-2815) Oozie not always display job log
[ https://issues.apache.org/jira/browse/OOZIE-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah reassigned OOZIE-2815: - Assignee: Purshotam Shah > Oozie not always display job log > > > Key: OOZIE-2815 > URL: https://issues.apache.org/jira/browse/OOZIE-2815 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah > > There are few issues with log streaming. We will fix following things. > 1. UI has an issue with displaying "loading... done". It used to show loading > done even if logs were not loaded entirely. > 2. Server-server communication time out. The default time is 1 min. Setting > to 3 min same as UI. > 3. We use to throw an error message if log scan duration is more than > configured duration. We will be truncating log. > 4. No log scan duration check for audit and error log since it doesn't > support log filter. > 5. Different buffer.len setting for error and audit log, so that partial > loading can be faster. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OOZIE-2815) Oozie not always display job log
[ https://issues.apache.org/jira/browse/OOZIE-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-2815: -- Description: There are few issues with log streaming. We will fix following things. 1. UI has an issue with displaying "loading... done". It used to show loading done even if logs were not loaded entirely. 2. Server-server communication time out. The default time is 1 min. Setting to 3 min same as UI. 3. We use to throw an error message if log scan duration is more than configured duration. We will be truncating log. 4. No log scan duration check for audit and error log since it doesn't support log filter. 5. Different buffer.len setting for error and audit log, so that partial loading can be faster. > Oozie not always display job log > > > Key: OOZIE-2815 > URL: https://issues.apache.org/jira/browse/OOZIE-2815 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah > > There are few issues with log streaming. We will fix following things. > 1. UI has an issue with displaying "loading... done". It used to show loading > done even if logs were not loaded entirely. > 2. Server-server communication time out. The default time is 1 min. Setting > to 3 min same as UI. > 3. We use to throw an error message if log scan duration is more than > configured duration. We will be truncating log. > 4. No log scan duration check for audit and error log since it doesn't > support log filter. > 5. Different buffer.len setting for error and audit log, so that partial > loading can be faster. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Review Request 57107: OOZIE-2815 oozie not always display job log
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57107/ --- Review request for oozie. Bugs: OOZIE-2815 https://issues.apache.org/jira/browse/OOZIE-2815 Repository: oozie-git Description --- OOZIE-2815 oozie not always display job log Diffs - core/src/main/java/org/apache/oozie/BaseEngine.java 50df8978773ed54fb5a39bb142d7235de1ed396f core/src/main/java/org/apache/oozie/BundleEngine.java d0099b48772069217375c032d171e8dc6b1d core/src/main/java/org/apache/oozie/CoordinatorEngine.java 2f9f8227f1b2bbb0074cfa19aac37ab9a3fe7c0f core/src/main/java/org/apache/oozie/DagEngine.java 57d276199de149b294f64570a8878b660e9a5a0c core/src/main/java/org/apache/oozie/service/XLogService.java 04f04f41403fc6446967524124b8cf86f816d71c core/src/main/java/org/apache/oozie/service/XLogStreamingService.java c15c4c1731838655d8f899f9c0405b226f8111b0 core/src/main/java/org/apache/oozie/service/ZKXLogStreamingService.java 97771ad22be84d1a8167c69a6b48b7012652cf8b core/src/main/java/org/apache/oozie/servlet/V0JobServlet.java d3b468965f33e300d7ca0f31efc596fd969b47a5 core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 9356768639e6e3ea548b25baa75d576351fa80e3 core/src/main/java/org/apache/oozie/util/AuthUrlClient.java b45a96acc397c26cdba8d5c62503013dd572f247 core/src/main/java/org/apache/oozie/util/TimestampedMessageParser.java a676f4d35a49147aadde73357397b211ab829c85 core/src/main/java/org/apache/oozie/util/XLogAuditFilter.java c377db5df5dd0bbeed77b4d5c88e1c4e2e64a51e core/src/main/java/org/apache/oozie/util/XLogAuditStreamer.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 core/src/main/java/org/apache/oozie/util/XLogErrorStreamer.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 core/src/main/java/org/apache/oozie/util/XLogFilter.java 3b49f77e016cbeaedfa03708dbc680e68f40ca16 core/src/main/java/org/apache/oozie/util/XLogStreamer.java 19f1fee55ba8d6c456e798f01e55cfe84cfb5803 core/src/main/resources/oozie-default.xml 95e0c36e1b2f5aef18637f6f956a990ba36166f2 core/src/test/java/org/apache/oozie/TestCoordinatorEngineStreamLog.java 3eb1016f3ffccaa9db077580b90cc1b3c53fd479 core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 42ffdbeb41e5829cdf814cf13ada331973fc33bd core/src/test/java/org/apache/oozie/service/TestXLogStreamingService.java bebb678798ec4a01c9710660d4b99fddd5167560 core/src/test/java/org/apache/oozie/service/TestZKXLogStreamingService.java fca8d844f1563da5856eff36f53f21d519630d31 core/src/test/java/org/apache/oozie/util/TestLogStreamer.java f90f784503e101ae2f96fee52977ee83f3098301 core/src/test/java/org/apache/oozie/util/TestSimplifiedTimestampedMessageParser.java ea899faf00aa2f98f43838987c9b716321a50dea core/src/test/java/org/apache/oozie/util/TestTimestampedMessageParser.java 9e28cbc77129bb2675b92ec9c18c77accfd78a55 core/src/test/java/org/apache/oozie/util/TestXLogUserFilterParam.java 46f273fde7f14b446463448c90f22519761997b6 webapp/src/main/webapp/oozie-console.js 76864a9a06e98357c2d17e35609e2a76afbd595c Diff: https://reviews.apache.org/r/57107/diff/1/ Testing --- Thanks, Purshotam Shah
[jira] [Created] (OOZIE-2815) Oozie not always display job log
Purshotam Shah created OOZIE-2815: - Summary: Oozie not always display job log Key: OOZIE-2815 URL: https://issues.apache.org/jira/browse/OOZIE-2815 Project: Oozie Issue Type: Bug Reporter: Purshotam Shah -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2312) oozie doesn't purge audit and error log
[ https://issues.apache.org/jira/browse/OOZIE-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890886#comment-15890886 ] Purshotam Shah commented on OOZIE-2312: --- By default audit log and error log purge is set to 30 days. I think it's not a good idea to purge audit in 30 days. There can be a long running bundle and coord jobs. I guess by default it should be one year and disable it. > oozie doesn't purge audit and error log > --- > > Key: OOZIE-2312 > URL: https://issues.apache.org/jira/browse/OOZIE-2312 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah >Assignee: Purshotam Shah > Fix For: 4.3.0 > > Attachments: OOZIE-2312-V1.patch, OOZIE-2312-V3.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail
[
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860506#comment-15860506
]
Purshotam Shah commented on OOZIE-2787:
---
+1 for OOZIE-2787-amend-5.patch.
> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
> Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch,
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch,
> OOZIE-2787-amend-1.patch, OOZIE-2787-amend-2.patch, OOZIE-2787-amend-3.patch,
> OOZIE-2787-amend-4.patch, OOZIE-2787-amend-5.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to
> distributed cache. Since this gets added twice, the job fails. This is
> observed from spark 2.1.0 which introduces a check for same file and fails
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2788) Fix jobs API servlet mapping for EmbeddedOozieServer
[
https://issues.apache.org/jira/browse/OOZIE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860069#comment-15860069
]
Purshotam Shah commented on OOZIE-2788:
---
+1
> Fix jobs API servlet mapping for EmbeddedOozieServer
>
>
> Key: OOZIE-2788
> URL: https://issues.apache.org/jira/browse/OOZIE-2788
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.0.0
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2788-00.patch
>
>
> The {{/v1/jobs}} and {{/v2/jobs}} are mapping to {{/v0/jobs}} which does not
> allow Oozie to submit coordinator and bundle.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2790) log4j configuration is not passed to spark executors
[
https://issues.apache.org/jira/browse/OOZIE-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15855071#comment-15855071
]
Purshotam Shah commented on OOZIE-2790:
---
+1
> log4j configuration is not passed to spark executors
>
>
> Key: OOZIE-2790
> URL: https://issues.apache.org/jira/browse/OOZIE-2790
> Project: Oozie
> Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2790-1.patch
>
>
> We prepare the {{spark-log4j.properties}} properties file but do not pass it
> over the spark executors in --files option. Therefore the log settings do get
> applied to the executor logs.
> Solution- pass the {{spark-log4j.properties}} file with --files option.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
Re: Review Request 56158: [OOZIE-2630] Amend patch for OOZIE-2630
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56158/#review163796 --- core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java (line 384) <https://reviews.apache.org/r/56158/#comment235302> End of functions calculation should not include initial instance calculation. It's purely on nominal time. Endof functions at dataset frequency and at datasets instance should return the same value. - Purshotam Shah On Feb. 1, 2017, 8:44 a.m., Satish Saley wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56158/ > --- > > (Updated Feb. 1, 2017, 8:44 a.m.) > > > Review request for oozie. > > > Bugs: OOZIE-2630 > https://issues.apache.org/jira/browse/OOZIE-2630 > > > Repository: oozie-git > > > Description > --- > > [OOZIE-2630] Amend patch for OOZIE-2630 > > > Diffs > - > > core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java > 3a7a930 > core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java bc18f4d > > core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java > 22d1f61 > core/src/test/resources/coord-dataset-endOfDays.xml 6aa3f00 > core/src/test/resources/coord-dataset-endOfMonths.xml 0ea5cac > core/src/test/resources/coord-dataset-endOfWeeks.xml 7879bf4 > > Diff: https://reviews.apache.org/r/56158/diff/ > > > Testing > --- > > Tested locally > > > Thanks, > > Satish Saley > >
[jira] [Commented] (OOZIE-2770) Show missing dependencies for coord actions
[
https://issues.apache.org/jira/browse/OOZIE-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15848106#comment-15848106
]
Purshotam Shah commented on OOZIE-2770:
---
Thanks Satish. Committed to trunk.
> Show missing dependencies for coord actions
> ---
>
> Key: OOZIE-2770
> URL: https://issues.apache.org/jira/browse/OOZIE-2770
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Fix For: trunk
>
> Attachments: amend-OOZIE-2770-V1.patch, DG_CommandLineTool.html,
> OOZIE-2770-V1.patch, OOZIE-2770-V2.patch, OOZIE-2770-V4.patch,
> OOZIE-2770-V5.patch, OOZIE-2770-V6.patch, OOZIE-2770-V7.patch,
> OOZIE-2770-V8.patch, UI.png
>
>
> {code}
> $oozie job -oozie http://localhost:11000/oozie -missingDependencies
> 002-170106125510433-oozie-puru-C@6
> CoordAction : 6
> Blocked on :
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
> DataSet: input-1
> Pending Dependencies :
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/40/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/20/_SUCCESS
> Blocked on : input-2
> Pending Dependencies :
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/40/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/20/_SUCCESS
> $
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2770) Show missing dependencies for coord actions
[
https://issues.apache.org/jira/browse/OOZIE-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-2770:
--
Attachment: DG_CommandLineTool.html
> Show missing dependencies for coord actions
> ---
>
> Key: OOZIE-2770
> URL: https://issues.apache.org/jira/browse/OOZIE-2770
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Fix For: trunk
>
> Attachments: amend-OOZIE-2770-V1.patch, DG_CommandLineTool.html,
> OOZIE-2770-V1.patch, OOZIE-2770-V2.patch, OOZIE-2770-V4.patch,
> OOZIE-2770-V5.patch, OOZIE-2770-V6.patch, OOZIE-2770-V7.patch,
> OOZIE-2770-V8.patch, UI.png
>
>
> {code}
> $oozie job -oozie http://localhost:11000/oozie -missingDependencies
> 002-170106125510433-oozie-puru-C@6
> CoordAction : 6
> Blocked on :
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
> DataSet: input-1
> Pending Dependencies :
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/40/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/20/_SUCCESS
> Blocked on : input-2
> Pending Dependencies :
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/40/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/20/_SUCCESS
> $
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2770) Show missing dependencies for coord actions
[
https://issues.apache.org/jira/browse/OOZIE-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15848091#comment-15848091
]
Purshotam Shah commented on OOZIE-2770:
---
There was a minor issue with Doc. We changes the CLI command to {{missingdeps}}
from {{missingDependencies}}. Doc was still pointing to{{missingDependencies}}.
amend-OOZIE-2770-V1.patch fixed that. Also attached CLI TWiki for reference.
> Show missing dependencies for coord actions
> ---
>
> Key: OOZIE-2770
> URL: https://issues.apache.org/jira/browse/OOZIE-2770
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Fix For: trunk
>
> Attachments: amend-OOZIE-2770-V1.patch, OOZIE-2770-V1.patch,
> OOZIE-2770-V2.patch, OOZIE-2770-V4.patch, OOZIE-2770-V5.patch,
> OOZIE-2770-V6.patch, OOZIE-2770-V7.patch, OOZIE-2770-V8.patch, UI.png
>
>
> {code}
> $oozie job -oozie http://localhost:11000/oozie -missingDependencies
> 002-170106125510433-oozie-puru-C@6
> CoordAction : 6
> Blocked on :
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
> DataSet: input-1
> Pending Dependencies :
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/40/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/20/_SUCCESS
> Blocked on : input-2
> Pending Dependencies :
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/40/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/20/_SUCCESS
> $
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2770) Show missing dependencies for coord actions
[
https://issues.apache.org/jira/browse/OOZIE-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-2770:
--
Attachment: amend-OOZIE-2770-V1.patch
> Show missing dependencies for coord actions
> ---
>
> Key: OOZIE-2770
> URL: https://issues.apache.org/jira/browse/OOZIE-2770
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Fix For: trunk
>
> Attachments: amend-OOZIE-2770-V1.patch, OOZIE-2770-V1.patch,
> OOZIE-2770-V2.patch, OOZIE-2770-V4.patch, OOZIE-2770-V5.patch,
> OOZIE-2770-V6.patch, OOZIE-2770-V7.patch, OOZIE-2770-V8.patch, UI.png
>
>
> {code}
> $oozie job -oozie http://localhost:11000/oozie -missingDependencies
> 002-170106125510433-oozie-puru-C@6
> CoordAction : 6
> Blocked on :
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
> DataSet: input-1
> Pending Dependencies :
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/40/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/20/_SUCCESS
> Blocked on : input-2
> Pending Dependencies :
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/40/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/20/_SUCCESS
> $
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
Re: Review Request 56100: OOZIE-2782 Input logic wait documentation is confusing
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56100/ --- (Updated Jan. 30, 2017, 11:29 p.m.) Review request for oozie. Bugs: OOZIE-2782 https://issues.apache.org/jira/browse/OOZIE-2782 Repository: oozie-git Description --- OOZIE-2782 Input logic wait documentation is confusing Diffs (updated) - docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 744f5d9 Diff: https://reviews.apache.org/r/56100/diff/ Testing --- Thanks, Purshotam Shah
Re: Review Request 56100: OOZIE-2782 Input logic wait documentation is confusing
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56100/ --- (Updated Jan. 30, 2017, 11:25 p.m.) Review request for oozie. Bugs: OOZIE-2782 https://issues.apache.org/jira/browse/OOZIE-2782 Repository: oozie-git Description --- OOZIE-2782 Input logic wait documentation is confusing Diffs (updated) - docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 744f5d9 Diff: https://reviews.apache.org/r/56100/diff/ Testing --- Thanks, Purshotam Shah
Review Request 56100: OOZIE-2782 Input logic wait documentation is confusing
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56100/ --- Review request for oozie. Bugs: OOZIE-2782 https://issues.apache.org/jira/browse/OOZIE-2782 Repository: oozie-git Description --- OOZIE-2782 Input logic wait documentation is confusing Diffs - docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 744f5d9 Diff: https://reviews.apache.org/r/56100/diff/ Testing --- Thanks, Purshotam Shah
Re: Review Request 55214: OOZIE-2770 Show missing dependencies for coord actions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55214/ --- (Updated Jan. 29, 2017, 9:15 p.m.) Review request for oozie. Bugs: OOZIE-2770 https://issues.apache.org/jira/browse/OOZIE-2770 Repository: oozie-git Description --- OOZIE-2770 Show missing dependencies for coord actions Diffs (updated) - client/src/main/java/org/apache/oozie/cli/OozieCLI.java 6e30d7e client/src/main/java/org/apache/oozie/client/OozieClient.java a107c4a client/src/main/java/org/apache/oozie/client/rest/JsonTags.java d670142 client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 9a3be97 core/src/main/java/org/apache/oozie/CoordinatorEngine.java 91fe5a1 core/src/main/java/org/apache/oozie/ErrorCode.java 9a843bd core/src/main/java/org/apache/oozie/command/coord/CoordActionMissingDependenciesXCommand.java PRE-CREATION core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java 63287b9 core/src/main/java/org/apache/oozie/command/coord/CoordSLAAlertsXCommand.java b8affd6 core/src/main/java/org/apache/oozie/coord/CoordUtils.java 82f9bed core/src/main/java/org/apache/oozie/coord/input/dependency/AbstractCoordInputDependency.java 0da60ec core/src/main/java/org/apache/oozie/coord/input/dependency/CoordInputDependency.java 504bc3d core/src/main/java/org/apache/oozie/coord/input/dependency/CoordOldInputDependency.java aabd2bf core/src/main/java/org/apache/oozie/coord/input/dependency/CoordPullInputDependency.java f20dcae core/src/main/java/org/apache/oozie/dependency/ActionDependency.java fe7a327 core/src/main/java/org/apache/oozie/executor/jpa/CoordActionQueryExecutor.java c0e6c19 core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java 87a2b42 core/src/main/java/org/apache/oozie/servlet/V0JobServlet.java 0c42128 core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 95dcca6 core/src/main/java/org/apache/oozie/servlet/V2JobServlet.java 3a0ffb0 core/src/test/java/org/apache/oozie/client/TestOozieCLI.java 8ec38e4 core/src/test/java/org/apache/oozie/command/coord/TestCoordActionMissingDependenciesXCommand.java PRE-CREATION core/src/test/java/org/apache/oozie/coord/input/logic/TestCoordInputLogicPush.java 6684a1f core/src/test/java/org/apache/oozie/servlet/MockCoordinatorEngineService.java 0e74f2d core/src/test/resources/coord-multiple-output-instance5.xml PRE-CREATION docs/src/site/twiki/DG_CommandLineTool.twiki 2dbbd4c docs/src/site/twiki/WebServicesAPI.twiki 8406da6 webapp/src/main/webapp/oozie-console.js 7b20e91 Diff: https://reviews.apache.org/r/55214/diff/ Testing --- Thanks, Purshotam Shah
Re: Review Request 55214: OOZIE-2770 Show missing dependencies for coord actions
> On Jan. 28, 2017, 1:32 p.m., Attila Sasvari wrote: > > client/src/main/java/org/apache/oozie/cli/OozieCLI.java, line 1354 > > <https://reviews.apache.org/r/55214/diff/4/?file=1617968#file1617968line1354> > > > > If you close System.out via ps, you will no longer be able to write to > > the console. If there is a subsequent call, nothing will be printed to > > System.out. So, you can leave the task of closing System.out to the JVM. > > > > I would simply pass System.out getCoordActionMissingDependencies and > > remove the surrounding try catch and would fix other similar methods. I > > beleive it would make easier to read the code. (Or if you want to keep the > > current logic, you could use try-with-resources from java 7.) You are correct. But it may not matter to Oozie CLI because it just makes one command in one run. But I agree that it's better to not close System.out. We do it in multiple places. I guess I just copied it. You can create a different JIRA to fix them. - Purshotam --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55214/#review163400 ------- On Jan. 27, 2017, 8:14 p.m., Purshotam Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55214/ > --- > > (Updated Jan. 27, 2017, 8:14 p.m.) > > > Review request for oozie. > > > Bugs: OOZIE-2770 > https://issues.apache.org/jira/browse/OOZIE-2770 > > > Repository: oozie-git > > > Description > --- > > OOZIE-2770 Show missing dependencies for coord actions > > > Diffs > - > > client/src/main/java/org/apache/oozie/cli/OozieCLI.java > 6e30d7eb667fe74440281fab3ca06c9f4df7bcf7 > client/src/main/java/org/apache/oozie/client/OozieClient.java > a107c4a3224dba76a59b9eb7c1903c0022d553a4 > client/src/main/java/org/apache/oozie/client/rest/JsonTags.java > d670142f446798f39fbb1cd8c3aed738a0897b18 > client/src/main/java/org/apache/oozie/client/rest/RestConstants.java > 9a3be979e1acf6fce8dc13ccdf985ce47215cf49 > core/src/main/java/org/apache/oozie/CoordinatorEngine.java > 91fe5a1d709ba1a90e493d301030465aa35ca790 > core/src/main/java/org/apache/oozie/ErrorCode.java > 9a843bd0f60ef3b946748adcd45563863c291a60 > > core/src/main/java/org/apache/oozie/command/coord/CoordActionMissingDependenciesXCommand.java > e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 > core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java > 0af7edc9094aaf24a6636ccebea17493e457fcce > > core/src/main/java/org/apache/oozie/command/coord/CoordSLAAlertsXCommand.java > b8affd67a412fac7d4ec452d7e2efa02c62ce25a > core/src/main/java/org/apache/oozie/coord/CoordUtils.java > 82f9bede0878b361ffb47d865036573b15448c68 > > core/src/main/java/org/apache/oozie/coord/input/dependency/AbstractCoordInputDependency.java > 0da60ec70373399414a637c22753da7e2a7e6618 > > core/src/main/java/org/apache/oozie/coord/input/dependency/CoordInputDependency.java > 504bc3d5f8c48676736cb461462677ddac400d17 > > core/src/main/java/org/apache/oozie/coord/input/dependency/CoordOldInputDependency.java > aabd2bf25c99710ce0b025b6b66ae0fd0bb00abf > > core/src/main/java/org/apache/oozie/coord/input/dependency/CoordPullInputDependency.java > f20dcae0ff44d745ffbcb22bce2653d042a65122 > core/src/main/java/org/apache/oozie/dependency/ActionDependency.java > fe7a3272190718f21aa594188013fadfa9b58646 > > core/src/main/java/org/apache/oozie/executor/jpa/CoordActionQueryExecutor.java > c0e6c198f4978744016f35dd8940db345a71f92c > core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java > 87a2b426ad3f5eeb08e76e1be473fdcdf387d293 > core/src/main/java/org/apache/oozie/servlet/V0JobServlet.java > 0c4212877f3051ef85ca04bb8d2e075b5f9c4c72 > core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java > 95dcca63c459475b8d41a85e0db11ca2a530a2e4 > core/src/main/java/org/apache/oozie/servlet/V2JobServlet.java > 3a0ffb0181a0feec31d9aceeea2df5aba0681ae9 > core/src/test/java/org/apache/oozie/client/TestOozieCLI.java > 8ec38e498f31bb69923125e34cda5abfc8e72aa9 > > core/src/test/java/org/apache/oozie/command/coord/TestCoordActionMissingDependenciesXCommand.java > e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 > > core/src/test/java/org/apache/oozie/coord/input/logic/TestCoordInputLogicPush.java > 6684a1fc7e8d1ed85b9e309cd163ae92f005f281 > > cor
[jira] [Updated] (OOZIE-2770) Show missing dependencies for coord actions
[
https://issues.apache.org/jira/browse/OOZIE-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-2770:
--
Attachment: OOZIE-2770-V5.patch
> Show missing dependencies for coord actions
> ---
>
> Key: OOZIE-2770
> URL: https://issues.apache.org/jira/browse/OOZIE-2770
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Attachments: OOZIE-2770-V1.patch, OOZIE-2770-V2.patch,
> OOZIE-2770-V4.patch, OOZIE-2770-V5.patch, UI.png
>
>
> {code}
> $oozie job -oozie http://localhost:11000/oozie -missingDependencies
> 002-170106125510433-oozie-puru-C@6
> CoordAction : 6
> Blocked on :
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
> DataSet: input-1
> Pending Dependencies :
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/40/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/20/_SUCCESS
> Blocked on : input-2
> Pending Dependencies :
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/40/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/20/_SUCCESS
> $
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OOZIE-2770) Show missing dependencies for coord actions
[
https://issues.apache.org/jira/browse/OOZIE-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-2770:
--
Attachment: OOZIE-2770-V4.patch
> Show missing dependencies for coord actions
> ---
>
> Key: OOZIE-2770
> URL: https://issues.apache.org/jira/browse/OOZIE-2770
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Attachments: OOZIE-2770-V1.patch, OOZIE-2770-V2.patch,
> OOZIE-2770-V4.patch, UI.png
>
>
> {code}
> $oozie job -oozie http://localhost:11000/oozie -missingDependencies
> 002-170106125510433-oozie-puru-C@6
> CoordAction : 6
> Blocked on :
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
> DataSet: input-1
> Pending Dependencies :
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/40/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/20/_SUCCESS
> Blocked on : input-2
> Pending Dependencies :
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/40/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/20/_SUCCESS
> $
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OOZIE-2770) Show missing dependencies for coord actions
[
https://issues.apache.org/jira/browse/OOZIE-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-2770:
--
Attachment: (was: OOZIE-2770-V3.patch)
> Show missing dependencies for coord actions
> ---
>
> Key: OOZIE-2770
> URL: https://issues.apache.org/jira/browse/OOZIE-2770
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Attachments: OOZIE-2770-V1.patch, OOZIE-2770-V2.patch, UI.png
>
>
> {code}
> $oozie job -oozie http://localhost:11000/oozie -missingDependencies
> 002-170106125510433-oozie-puru-C@6
> CoordAction : 6
> Blocked on :
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
> DataSet: input-1
> Pending Dependencies :
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/40/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/20/_SUCCESS
> Blocked on : input-2
> Pending Dependencies :
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/40/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/20/_SUCCESS
> $
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OOZIE-2770) Show missing dependencies for coord actions
[
https://issues.apache.org/jira/browse/OOZIE-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-2770:
--
Attachment: OOZIE-2770-V3.patch
> Show missing dependencies for coord actions
> ---
>
> Key: OOZIE-2770
> URL: https://issues.apache.org/jira/browse/OOZIE-2770
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Attachments: OOZIE-2770-V1.patch, OOZIE-2770-V2.patch,
> OOZIE-2770-V3.patch, UI.png
>
>
> {code}
> $oozie job -oozie http://localhost:11000/oozie -missingDependencies
> 002-170106125510433-oozie-puru-C@6
> CoordAction : 6
> Blocked on :
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
> DataSet: input-1
> Pending Dependencies :
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/40/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/20/_SUCCESS
> Blocked on : input-2
> Pending Dependencies :
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/40/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/20/_SUCCESS
> $
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 55963: OOZIE-2781 HCat partition available notification is not sent to coordinator actions if coordinator job is using a different hostname (cname, IP address, etc. ) for HCat URL.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55963/ --- (Updated Jan. 27, 2017, 10:26 p.m.) Review request for oozie. Bugs: OOZIE-2781 https://issues.apache.org/jira/browse/OOZIE-2781 Repository: oozie-git Description --- OOZIE-2781 HCat partition available notification is not sent to coordinator actions if coordinator job is using a different hostname (cname, IP address, etc. ) for HCat URL. Diffs (updated) - core/src/main/java/org/apache/oozie/dependency/hcat/EhcacheHCatDependencyCache.java 3bc467535202e13387b1d29ac678573f4154c522 core/src/main/java/org/apache/oozie/dependency/hcat/SimpleHCatDependencyCache.java 9e24c9aa5f10821bf76029c51bb49e0728376caa core/src/main/resources/oozie-default.xml ad103864d7631e9d3678a828bf297ca1c80f09ca core/src/test/java/org/apache/oozie/service/TestPartitionDependencyManagerService.java a5d2ed92ffe9db77e2c8914b66c7806275dcc592 Diff: https://reviews.apache.org/r/55963/diff/ Testing --- Thanks, Purshotam Shah
[jira] [Updated] (OOZIE-2781) HCat partition available notification is not sent to coordinator actions if coordinator job is using a different hostname (cname, IP address, etc. ) for HCat URL.
[ https://issues.apache.org/jira/browse/OOZIE-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah updated OOZIE-2781: -- Attachment: OOZIE-2781-V2.patch > HCat partition available notification is not sent to coordinator actions if > coordinator job is using a different hostname (cname, IP address, etc. ) for > HCat URL. > --- > > Key: OOZIE-2781 > URL: https://issues.apache.org/jira/browse/OOZIE-2781 > Project: Oozie > Issue Type: Bug > Reporter: Purshotam Shah > Assignee: Purshotam Shah > Attachments: OOZIE-2781-V1.patch, OOZIE-2781-V2.patch > > > HCatDependencyCache maintains a map which is a list of HCat partition and > waiting for actions for HCat partition. > Key of Map is server;db;table. > If a user uses a different host name(IP Address, CNames, VIP, etc..) for > HCat in coordinator job and if JMS message from HCat doesn't match the used > hostname used by a user, then HCat partition available notification will be > not sent to waiting coordinator actions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2691) Show workflow action retry information in UI
[
https://issues.apache.org/jira/browse/OOZIE-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15843439#comment-15843439
]
Purshotam Shah commented on OOZIE-2691:
---
There was a minor issue where action retries Console URL was not clickable.
Also fixed to display retries count to CLI job info.
amend-OOZIE-2691.patch fixed both of them.
> Show workflow action retry information in UI
>
>
> Key: OOZIE-2691
> URL: https://issues.apache.org/jira/browse/OOZIE-2691
> Project: Oozie
> Issue Type: New Feature
> Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Fix For: trunk
>
> Attachments: Action Retries - webservice.png, amend-OOZIE-2691.patch,
> OOZIE-2691-V1.patch, OOZIE-2691-V2.patch, OOZIE-2691-V3.patch,
> OOZIE-2691-V5.patch, OOZIE-2691-V6.patch, OOZIE-2691-V6.patch,
> screenshot.png, UI.png
>
>
> CLI
> {code}
> $ oozie job -info 000-161212175234862-oozie-puru-W@:start: -oozie
> http://localhost:11000/oozie -retries
> ID : 000-161212175234862-oozie-puru-W@:start:
> No Retries
> ---
> $ oozie job -info 000-161212175234862-oozie-puru-W@pig-node -oozie
> http://localhost:11000/oozie -retries
> ID : 000-161212175234862-oozie-puru-W@pig-node
> ---
> Retry : 1
> Start Time : Tue, 13 Dec 2016 01:54:13 GMT
> End Time : Tue, 13 Dec 2016 01:54:20 GMT
> Console URL:
> http://localhost:50030/jobdetails.jsp?jobid=job_201612051339_2648
> ---
> Retry : 2
> Start Time : Tue, 13 Dec 2016 01:55:20 GMT
> End Time : Tue, 13 Dec 2016 01:55:24 GMT
> Console URL:
> http://localhost:50030/jobdetails.jsp?jobid=job_201612051339_2649
> ---
> Retry : 3
> Start Time : Tue, 13 Dec 2016 01:56:24 GMT
> End Time : Tue, 13 Dec 2016 01:56:27 GMT
> Console URL:
> http://localhost:50030/jobdetails.jsp?jobid=job_201612051339_2650
> ---
> $
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OOZIE-2691) Show workflow action retry information in UI
[
https://issues.apache.org/jira/browse/OOZIE-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-2691:
--
Attachment: amend-OOZIE-2691.patch
> Show workflow action retry information in UI
>
>
> Key: OOZIE-2691
> URL: https://issues.apache.org/jira/browse/OOZIE-2691
> Project: Oozie
> Issue Type: New Feature
> Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Fix For: trunk
>
> Attachments: Action Retries - webservice.png, amend-OOZIE-2691.patch,
> OOZIE-2691-V1.patch, OOZIE-2691-V2.patch, OOZIE-2691-V3.patch,
> OOZIE-2691-V5.patch, OOZIE-2691-V6.patch, OOZIE-2691-V6.patch,
> screenshot.png, UI.png
>
>
> CLI
> {code}
> $ oozie job -info 000-161212175234862-oozie-puru-W@:start: -oozie
> http://localhost:11000/oozie -retries
> ID : 000-161212175234862-oozie-puru-W@:start:
> No Retries
> ---
> $ oozie job -info 000-161212175234862-oozie-puru-W@pig-node -oozie
> http://localhost:11000/oozie -retries
> ID : 000-161212175234862-oozie-puru-W@pig-node
> ---
> Retry : 1
> Start Time : Tue, 13 Dec 2016 01:54:13 GMT
> End Time : Tue, 13 Dec 2016 01:54:20 GMT
> Console URL:
> http://localhost:50030/jobdetails.jsp?jobid=job_201612051339_2648
> ---
> Retry : 2
> Start Time : Tue, 13 Dec 2016 01:55:20 GMT
> End Time : Tue, 13 Dec 2016 01:55:24 GMT
> Console URL:
> http://localhost:50030/jobdetails.jsp?jobid=job_201612051339_2649
> ---
> Retry : 3
> Start Time : Tue, 13 Dec 2016 01:56:24 GMT
> End Time : Tue, 13 Dec 2016 01:56:27 GMT
> Console URL:
> http://localhost:50030/jobdetails.jsp?jobid=job_201612051339_2650
> ---
> $
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OOZIE-2770) Show missing dependencies for coord actions
[
https://issues.apache.org/jira/browse/OOZIE-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated OOZIE-2770:
--
Attachment: OOZIE-2770-V2.patch
> Show missing dependencies for coord actions
> ---
>
> Key: OOZIE-2770
> URL: https://issues.apache.org/jira/browse/OOZIE-2770
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
> Assignee: Purshotam Shah
> Attachments: OOZIE-2770-V1.patch, OOZIE-2770-V2.patch, UI.png
>
>
> {code}
> $oozie job -oozie http://localhost:11000/oozie -missingDependencies
> 002-170106125510433-oozie-puru-C@6
> CoordAction : 6
> Blocked on :
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
> DataSet: input-1
> Pending Dependencies :
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/40/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/20/_SUCCESS
> Blocked on : input-2
> Pending Dependencies :
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/06/00/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/40/_SUCCESS
>
> hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/05/20/_SUCCESS
> $
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 55214: OOZIE-2770 Show missing dependencies for coord actions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55214/ --- (Updated Jan. 27, 2017, 8:14 p.m.) Review request for oozie. Bugs: OOZIE-2770 https://issues.apache.org/jira/browse/OOZIE-2770 Repository: oozie-git Description --- OOZIE-2770 Show missing dependencies for coord actions Diffs (updated) - client/src/main/java/org/apache/oozie/cli/OozieCLI.java 6e30d7eb667fe74440281fab3ca06c9f4df7bcf7 client/src/main/java/org/apache/oozie/client/OozieClient.java a107c4a3224dba76a59b9eb7c1903c0022d553a4 client/src/main/java/org/apache/oozie/client/rest/JsonTags.java d670142f446798f39fbb1cd8c3aed738a0897b18 client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 9a3be979e1acf6fce8dc13ccdf985ce47215cf49 core/src/main/java/org/apache/oozie/CoordinatorEngine.java 91fe5a1d709ba1a90e493d301030465aa35ca790 core/src/main/java/org/apache/oozie/ErrorCode.java 9a843bd0f60ef3b946748adcd45563863c291a60 core/src/main/java/org/apache/oozie/command/coord/CoordActionMissingDependenciesXCommand.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java 0af7edc9094aaf24a6636ccebea17493e457fcce core/src/main/java/org/apache/oozie/command/coord/CoordSLAAlertsXCommand.java b8affd67a412fac7d4ec452d7e2efa02c62ce25a core/src/main/java/org/apache/oozie/coord/CoordUtils.java 82f9bede0878b361ffb47d865036573b15448c68 core/src/main/java/org/apache/oozie/coord/input/dependency/AbstractCoordInputDependency.java 0da60ec70373399414a637c22753da7e2a7e6618 core/src/main/java/org/apache/oozie/coord/input/dependency/CoordInputDependency.java 504bc3d5f8c48676736cb461462677ddac400d17 core/src/main/java/org/apache/oozie/coord/input/dependency/CoordOldInputDependency.java aabd2bf25c99710ce0b025b6b66ae0fd0bb00abf core/src/main/java/org/apache/oozie/coord/input/dependency/CoordPullInputDependency.java f20dcae0ff44d745ffbcb22bce2653d042a65122 core/src/main/java/org/apache/oozie/dependency/ActionDependency.java fe7a3272190718f21aa594188013fadfa9b58646 core/src/main/java/org/apache/oozie/executor/jpa/CoordActionQueryExecutor.java c0e6c198f4978744016f35dd8940db345a71f92c core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java 87a2b426ad3f5eeb08e76e1be473fdcdf387d293 core/src/main/java/org/apache/oozie/servlet/V0JobServlet.java 0c4212877f3051ef85ca04bb8d2e075b5f9c4c72 core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 95dcca63c459475b8d41a85e0db11ca2a530a2e4 core/src/main/java/org/apache/oozie/servlet/V2JobServlet.java 3a0ffb0181a0feec31d9aceeea2df5aba0681ae9 core/src/test/java/org/apache/oozie/client/TestOozieCLI.java 8ec38e498f31bb69923125e34cda5abfc8e72aa9 core/src/test/java/org/apache/oozie/command/coord/TestCoordActionMissingDependenciesXCommand.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 core/src/test/java/org/apache/oozie/coord/input/logic/TestCoordInputLogicPush.java 6684a1fc7e8d1ed85b9e309cd163ae92f005f281 core/src/test/java/org/apache/oozie/servlet/MockCoordinatorEngineService.java 0e74f2d1df67dd5d4aaccbdb66a3829f5894e751 core/src/test/resources/coord-multiple-output-instance5.xml e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 docs/src/site/twiki/DG_CommandLineTool.twiki 2dbbd4c820b9d8e11a51fdfe84185f5a1b3e6707 docs/src/site/twiki/WebServicesAPI.twiki 8406da6ab3bec859a9ef2b889e558f6c9ad1d652 webapp/src/main/webapp/oozie-console.js 7b20e9196cc6129b53fc610d34f6090311474eae Diff: https://reviews.apache.org/r/55214/diff/ Testing --- Thanks, Purshotam Shah
