http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_HiveActionExtension.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/DG_HiveActionExtension.md b/docs/src/site/markdown/DG_HiveActionExtension.md new file mode 100644 index 0000000..99a73c6 --- /dev/null +++ b/docs/src/site/markdown/DG_HiveActionExtension.md @@ -0,0 +1,476 @@ + + +[::Go back to Oozie Documentation Index::](index.html) + +----- + +# Oozie Hive Action Extension + +<!-- MACRO{toc|fromDepth=1|toDepth=4} --> + +<a name="HiveAction"></a> +## Hive Action + +The `hive` action runs a Hive job. + +The workflow job will wait until the Hive job completes before +continuing to the next action. + +To run the Hive job, you have to configure the `hive` action with the `resource-manager`, `name-node` and Hive `script` +(or Hive `query`) elements as well as the necessary parameters and configuration. + +A `hive` action can be configured to create or delete HDFS directories +before starting the Hive job. + +Hive configuration can be specified with a file, using the `job-xml` +element, and inline, using the `configuration` elements. + +Oozie EL expressions can be used in the inline configuration. Property +values specified in the `configuration` element override values specified +in the `job-xml` file. + +Note that YARN `yarn.resourcemanager.address` (`resource-manager`) and HDFS `fs.default.name` (`name-node`) properties +must not be present in the inline configuration. + +As with Hadoop `map-reduce` jobs, it is possible to add files and +archives in order to make them available to the Hive job. Refer to the +[WorkflowFunctionalSpec#FilesArchives][Adding Files and Archives for the Job] +section for more information about this feature. + +Oozie Hive action supports Hive scripts with parameter variables, their +syntax is `${VARIABLES}`. + +**Syntax:** + + +``` +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="[NODE-NAME]"> + <hive xmlns="uri:oozie:hive-action:1.0"> + <resource-manager>[RESOURCE-MANAGER]</resource-manager> + <name-node>[NAME-NODE]</name-node> + <prepare> + <delete path="[PATH]"/> + ... + <mkdir path="[PATH]"/> + ... + </prepare> + <job-xml>[HIVE SETTINGS FILE]</job-xml> + <configuration> + <property> + <name>[PROPERTY-NAME]</name> + <value>[PROPERTY-VALUE]</value> + </property> + ... + </configuration> + <script>[HIVE-SCRIPT]</script> + <param>[PARAM-VALUE]</param> + ... + <param>[PARAM-VALUE]</param> + <file>[FILE-PATH]</file> + ... + <archive>[FILE-PATH]</archive> + ... + </hive> + <ok to="[NODE-NAME]"/> + <error to="[NODE-NAME]"/> + </action> + ... +</workflow-app> +``` + +The `prepare` element, if present, indicates a list of paths to delete +or create before starting the job. Specified paths must start with `hdfs://HOST:PORT`. + +The `job-xml` element, if present, specifies a file containing configuration +for the Hive job. As of schema 0.3, multiple `job-xml` elements are allowed in order to +specify multiple `job.xml` files. + +The `configuration` element, if present, contains configuration +properties that are passed to the Hive job. + +The `script` element must contain the path of the Hive script to +execute. The Hive script can be templatized with variables of the form +`${VARIABLE}`. The values of these variables can then be specified +using the `params` element. + +The `query` element available from uri:oozie:hive-action:0.6, can be used instead of the +`script` element. It allows for embedding queries within the `worklfow.xml` directly. +Similar to the `script` element, it also allows for the templatization of variables in the +form `${VARIABLE}`. + +The `params` element, if present, contains parameters to be passed to +the Hive script. + +All the above elements can be parameterized (templatized) using EL +expressions. + +**Example:** + + +``` +<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="myfirsthivejob"> + <hive xmlns="uri:oozie:hive-action:1.0"> + <resource-manager>foo:8032</resource-manager> + <name-node>bar:8020</name-node> + <prepare> + <delete path="${jobOutput}"/> + </prepare> + <configuration> + <property> + <name>mapred.compress.map.output</name> + <value>true</value> + </property> + </configuration> + <script>myscript.q</script> + <param>InputDir=/home/tucu/input-data</param> + <param>OutputDir=${jobOutput}</param> + </hive> + <ok to="myotherjob"/> + <error to="errorcleanup"/> + </action> + ... +</workflow-app> +``` + +### Hive Default and Site Configuration Files + +Hive (as of Hive 0.8) ignores a `hive-default.xml` file. As a result, Oozie (as of Oozie 3.4) ignores the `oozie.hive.defaults` +property that was previously required by earlier versions of Oozie for the Hive action. + +### Hive Action Logging + +Hive action logs are redirected to the Oozie Launcher map-reduce job task STDOUT/STDERR that runs Hive. + +From Oozie web-console, from the Hive action pop up using the 'Console URL' link, it is possible +to navigate to the Oozie Launcher map-reduce job task logs via the Hadoop job-tracker web-console. + +The logging level of the Hive action can set in the Hive action configuration using the +property `oozie.hive.log.level`. The default value is `INFO`. + +## Appendix, Hive XML-Schema + +### AE.A Appendix A, Hive XML-Schema + +#### Hive Action Schema Version 1.0 + +``` +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; + xmlns:hive="uri:oozie:hive-action:1.0" + elementFormDefault="qualified" + targetNamespace="uri:oozie:hive-action:1.0"> +. + <xs:include schemaLocation="oozie-common-1.0.xsd"/> +. + <xs:element name="hive" type="hive:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:choice> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:choice> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="launcher" type="hive:LAUNCHER" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:choice minOccurs="1" maxOccurs="1"> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="query" type="xs:string" minOccurs="1" maxOccurs="1"/> + </xs:choice> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. +</xs:schema> +``` + +#### Hive Action Schema Version 0.6 + +``` +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; + xmlns:hive="uri:oozie:hive-action:0.6" elementFormDefault="qualified" + targetNamespace="uri:oozie:hive-action:0.6"> +. + <xs:element name="hive" type="hive:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:choice minOccurs="1" maxOccurs="1"> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="query" type="xs:string" minOccurs="1" maxOccurs="1"/> + </xs:choice> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. +</xs:schema> +``` +#### Hive Action Schema Version 0.5 + +``` +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; + xmlns:hive="uri:oozie:hive-action:0.5" elementFormDefault="qualified" + targetNamespace="uri:oozie:hive-action:0.5"> +. + <xs:element name="hive" type="hive:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. +</xs:schema> +``` + +#### Hive Action Schema Version 0.4 + +``` +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; + xmlns:hive="uri:oozie:hive-action:0.4" elementFormDefault="qualified" + targetNamespace="uri:oozie:hive-action:0.4"> +. + <xs:element name="hive" type="hive:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. +</xs:schema> +``` + +#### Hive Action Schema Version 0.3 + +``` +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; + xmlns:hive="uri:oozie:hive-action:0.3" elementFormDefault="qualified" + targetNamespace="uri:oozie:hive-action:0.3"> +. + <xs:element name="hive" type="hive:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. +</xs:schema> +``` + +#### Hive Action Schema Version 0.2 + +``` +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; + xmlns:hive="uri:oozie:hive-action:0.2" elementFormDefault="qualified" + targetNamespace="uri:oozie:hive-action:0.2"> +. + <xs:element name="hive" type="hive:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. +</xs:schema> +``` + +[::Go back to Oozie Documentation Index::](index.html) + +
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_JMSNotifications.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/DG_JMSNotifications.md b/docs/src/site/markdown/DG_JMSNotifications.md new file mode 100644 index 0000000..e8f8a76 --- /dev/null +++ b/docs/src/site/markdown/DG_JMSNotifications.md @@ -0,0 +1,236 @@ + + +[::Go back to Oozie Documentation Index::](index.html) + +# JMS Notifications + +<!-- MACRO{toc|fromDepth=1|toDepth=4} --> + +## Overview +Since Oozie 4.0, Oozie supports publishing notifications to a JMS Provider for job status changes and SLA met and miss +events. This provides an alternative to polling Oozie for Job or SLA related information and getting events as they +happen without any delay. Clients can be written to consume these notifications and integrate with different monitoring +and alerting systems. + +## Oozie Server Configuration +Refer to [Notifications Configuration](AG_Install.html#Notifications_Configuration) section of [Oozie Install](AG_Install.html) +documentation for the Oozie server side configuration required to support publishing notifications to a JMS Provider. +The JNDI properties for the JMS provider, the topics to publish to and the notification types to publish (Job and/or SLA) +need to be configured. + +## Consuming Notifications + +### Notification types +Job and SLA notifications are published to the configured JMS Provider on the configured topics. + +Job status change notifications include job start, success, failure, suspended, etc. Currently only workflow job and +coordinator action status change notifications are published. + +SLA notifications include START_MET, END_MET, DURATION_MET, START_MISS, END_MISS, DURATION_MISS events and are published +for a workflow job, workflow action or coordinator action for which SLA information is configured in the job xml. Refer +to [SLA Configuration](DG_SLAMonitoring.html#Configuring_SLA_in_Applications) for information on configuring SLA for a workflow or +coordinator. + +### JMS Topic +Consumers interested in notification on events will require to know the JNDI properties to connect to the JMS provider. +They will also need to know the JMS topic on which notifications for a particular job are published. + +Oozie Client provides the following APIs : + +``` +public JMSConnectionInfo getJMSConnectionInfo() +public String getJMSTopicName(String jobId) +``` + +The JMSConnectionInfo exposes 3 methods: + + +``` +Properties getJNDIProperties(); +String getTopicPattern(AppType appType); +String getTopicPrefix(); +``` + +The topic is obtained by concatenating topic prefix and the substituted value for topic pattern. The topic pattern +can be a constant value like workflow or coordinator which the administrator has configured or a variable (either ${username} +or ${jobId}). If ${username}, it has to be substituted with the name of the user who has submitted the job; and if ${jobId} it has +to be substituted with the job Id. Administrators can chose to publish messages to topics containing user names to avoid having one +topic containing all messages and all users having to apply selectors to filter the message they are interested in. + +The getJMSTopicName API can be used if the job id is already known and will give the exact topic name to which the +notifications for that job are published. + + +### JMS Message Format +JMS messages published are `javax.jms.TextMessage`. The body contains JSON and the header contains multiple properties +that can be used as selectors. The header properties are not repeated in the body of the message to keep the messages +small. + +<b>Message Header:</b> <br/> +The different header properties are: + + * msgType - Value can be JOB or SLA. + * user - The user who submitted the job. + * appName - Application name of the job. + * appType - Type of the job. One of WORKFLOW_JOB, WORKFLOW_ACTION, COORDINATOR_ACTION + * slaStatus - Applicable only to SLA messages. Value is one of NOT_STARTED, IN_PROCESS, MET, MISS + * eventStatus - It takes one of the following values- START_MET, START_MISS, DURATION_MET, DURATION_MISS, END_MET, END_MISS for SLA notifications. It can take any of the following values- WAITING, STARTED, SUCCESS, SUSPEND, FAILURE for job notifications. Note that event status is different from the Job status. It is included in the header to provide better filtering. Below is the mapping of event status to the actual status of workflow job or coordinator action. + +WAITING = When a Coordinator Action is waiting for input dependency + +STARTED = When the Workflow Job or Coordinator Action is in RUNNING state + +SUCCESS = When the Workflow Job or Coordinator Action is in SUCCEEDED state + +SUSPEND = When the Workflow Job or Coordinator Action is in SUSPENDED state + +FAILURE = When the Workflow Job or Coordinator Action is in terminal state other than SUCCEEDED. + +<b>Message Body for Job Notifications:</b> <br/> + Sample JSON response for different job and sla events as below. + + +``` +Workflow Job in RUNNING state: +{"status":"RUNNING","id":"0000042-130618221729631-oozie-oozi-W","startTime":1342915200000} +``` + + +``` +Workflow Job in FAILED state: +{"status":"FAILED","errorCode":"EL_ERROR","errorMessage":"variable [dummyvalue] cannot be resolved", +"id":"0000042-130618221729631-oozie-oozi-W","startTime":1342915200000,"endTime":1366672183543} +``` + + +``` +Workflow Job in SUCCEEDED state: +{"status":"SUCCEEDED","id":"0000039-130618221729631-oozie-oozi-W","startTime":1342915200000, +"parentId":"0000025-130618221729631-oozie-oozi-C@1","endTime":1366676224154} +``` + + +``` +Workflow Job in SUSPENDED state: +{"status":"SUSPENDED","id":"0000039-130618221729631-oozie-oozi-W","startTime":1342915200000, +"parentId":"0000025-130618221729631-oozie-oozi-C@1"} +``` + + +``` +Coordinator Action in WAITING state: +{"status":"WAITING","nominalTime":1310342400000,"missingDependency":"hdfs://gsbl90107.blue.com:8020/user/john/dir1/file1", +"id":"0000025-130618221729631-oozie-oozi-C@1","startTime":1342915200000,"parentId":"0000025-130618221729631-oozie-oozi-C"} +``` + + +``` +Coordinator Action in RUNNING state: +{"status":"RUNNING","nominalTime":1310342400000,"id":"0000025-130618221729631-oozie-oozi-C@1", +"startTime":1342915200000,"parentId":"0000025-130618221729631-oozie-oozi-C"} +``` + + +``` +Coordinator Action in SUCCEEDED state: +{"status":"SUCCEEDED","nominalTime":1310342400000,"id":"0000025-130618221729631-oozie-oozi-C@1", +"startTime":1342915200000,"parentId":"0000025-130618221729631-oozie-oozi-C","endTime":1366677082799} +``` + + +``` +Coordinator Action in FAILED state: +{"status":"FAILED","errorCode":"E0101","errorMessage":"dummyError","nominalTime":1310342400000, +"id":"0000025-130618221729631-oozie-oozi-C@1","startTime":1342915200000, +"parentId":"0000025-130618221729631-oozie-oozi-C","endTime":1366677140818} +``` + +<b>Message Body for SLA Notifications:</b> <br/> + + +``` +Workflow Job in sla END_MISS state: +{"id":"0000000-000000000000001-oozie-wrkf-C@1","parentId":"0000000-000000000000001-oozie-wrkf-C", +"expectedStartTime":1356998400000,"notificationMessage":"notification of start miss","actualStartTime":1357002000000, +"expectedDuration":-1, "actualDuration":3600,"expectedEndTime":1356998400000,"actualEndTime":1357002000000} +``` + +### JMS Client + +Oozie provides a helper class JMSMessagingUtils for consumers to deserialize the JMS messages back to Java objects. +The below method getEventMessage() expects a sub type of EventMessage. +There are different implementations of EventMessage - WorkflowJobMessage, CoordinatorActionMessage and SLAMessage. + + +``` +<T extends EventMessage> T JMSMessagingUtils.getEventMessage(Message jmsMessage) +``` +#### Example +Below is a sample code to consume notifications. + +First, create the Oozie client and retrieve the JNDI properties to make a connection to the JMS server. + +``` + OozieClient oc = new OozieClient("http://localhost:11000/oozie";); + JMSConnectionInfo jmsInfo = oc.getJMSConnectionInfo(); + Properties jndiProperties = jmsInfo.getJNDIProperties(); + Context jndiContext = new InitialContext(jndiProperties); + String connectionFactoryName = (String) jndiContext.getEnvironment().get("connectionFactoryNames"); + ConnectionFactory connectionFactory = (ConnectionFactory) jndiContext.lookup(connectionFactoryName); + Connection connection = connectionFactory.createConnection(); + Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE); + String topicPrefix = jmsInfo.getTopicPrefix(); + String topicPattern = jmsInfo.getTopicPattern(AppType.WORKFLOW_JOB); + // Following code checks if the topic pattern is + //'username', then the topic name is set to the actual user submitting the job + String topicName = null; + if (topicPattern.equals("${username}")) { + topicName = "john"; + // Following code checks if the topic pattern is + //'jobId', then the topic name is set to the job id + } else if (topicPattern.equals("${jobId}")) { + topicName = "0000004-140328125200198-oozie-oozi-W"; + } + Destination topic = session.createTopic(topicPrefix+topicName); + MessageConsumer consumer = session.createConsumer(topic); + consumer.setMessageListener(this); + connection.start(); +``` + +To start receiving messages, the JMS [MessageListener](http://docs.oracle.com/javaee/6/api/javax/jms/MessageListener.html) +interface needs to be implemented. Also, its onMessage() method needs to be implemented. +This method will be called whenever a message is available on the JMS bus. + + +``` + public void onMessage(Message message) { + if (message.getStringProperty(JMSHeaderConstants.MESSAGE_TYPE).equals(MessageType.SLA.name())){ + SLAMessage slaMessage = JMSMessagingUtils.getEventMessage(message); + // Further processing + } + else if (message.getStringProperty(JMSHeaderConstants.APP_TYPE).equals(AppType.WORKFLOW_JOB.name())){ + WorkflowJobMessage wfJobMessage = JMSMessagingUtils.getEventMessage(message); + // Further processing + } + } +``` + +#### Applying Selectors + +Below is a sample ActiveMQ text message header properties section. + +``` +ActiveMQTextMessage +{properties = {appName = map-reduce-wf, msgType=JOB, appType=WORKFLOW_JOB, user=john, msgFormat=json, eventStatus=STARTED} ...} +``` + +On the header properties, consumers can apply JMS selectors to filter messages from JMS provider. +They are listed at [JMSHeaderConstants](../docs/client/apidocs/org/apache/oozie/client/event/jms/JMSHeaderConstants.html) + +Sample use of selector to filter events related to Job which have failed and has a particular app-name + + +``` +String selector=JMSHeaderConstants.EVENT_STATUS + "='FAILURE' AND " + JMSHeaderConstants.APP_NAME + "='app-name'"; +MessageConsumer consumer = session.createConsumer(topic, selector); +``` http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_Overview.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/DG_Overview.md b/docs/src/site/markdown/DG_Overview.md new file mode 100644 index 0000000..6a2b9d2 --- /dev/null +++ b/docs/src/site/markdown/DG_Overview.md @@ -0,0 +1,86 @@ + + +[::Go back to Oozie Documentation Index::](index.html) + +# Oozie Workflow Overview + +Oozie is a server based _Workflow Engine_ specialized in running workflow jobs with actions that run Hadoop Map/Reduce +and Pig jobs. + +Oozie is a Java Web-Application that runs in a Java servlet-container. + +For the purposes of Oozie, a workflow is a collection of actions (i.e. Hadoop Map/Reduce jobs, Pig jobs) arranged in +a control dependency DAG (Directed Acyclic Graph). "control dependency" from one action to another means that the second +action can't run until the first action has completed. + +Oozie workflows definitions are written in hPDL (a XML Process Definition Language similar to +[JBOSS JBPM](http://www.jboss.org/jbossjbpm/) jPDL). + +Oozie workflow actions start jobs in remote systems (i.e. Hadoop, Pig). Upon action completion, the remote systems +callback Oozie to notify the action completion, at this point Oozie proceeds to the next action in the workflow. + +Oozie uses a custom SecurityManager inside it's launcher to catch exit() calls from the user code. Make sure to delegate checkExit() +calls to Oozie's SecurityManager if the user code uses its own SecurityManager. The Launcher also grants java.security.AllPermission +by default to the user code. + +Oozie workflows contain control flow nodes and action nodes. + +Control flow nodes define the beginning and the end of a workflow ( `start`, `end` and `fail` nodes) and provide a +mechanism to control the workflow execution path ( `decision`, `fork` and `join` nodes). + +Action nodes are the mechanism by which a workflow triggers the execution of a computation/processing task. Oozie +provides support for different types of actions: Hadoop map-reduce, Hadoop file system, Pig, SSH, HTTP, eMail and +Oozie sub-workflow. Oozie can be extended to support additional type of actions. + +Oozie workflows can be parameterized (using variables like `${inputDir}` within the workflow definition). When +submitting a workflow job values for the parameters must be provided. If properly parameterized (i.e. using different +output directories) several identical workflow jobs can concurrently. + +## WordCount Workflow Example + +**Workflow Diagram:** + +<img src="./DG_Overview.png"/> + +**hPDL Workflow Definition:** + + +``` +<workflow-app name='wordcount-wf' xmlns="uri:oozie:workflow:0.1"> + <start to='wordcount'/> + <action name='wordcount'> + <map-reduce> + <job-tracker>${jobTracker}</job-tracker> + <name-node>${nameNode}</name-node> + <configuration> + <property> + <name>mapred.mapper.class</name> + <value>org.myorg.WordCount.Map</value> + </property> + <property> + <name>mapred.reducer.class</name> + <value>org.myorg.WordCount.Reduce</value> + </property> + <property> + <name>mapred.input.dir</name> + <value>${inputDir}</value> + </property> + <property> + <name>mapred.output.dir</name> + <value>${outputDir}</value> + </property> + </configuration> + </map-reduce> + <ok to='end'/> + <error to='end'/> + </action> + <kill name='kill'> + <message>Something went wrong: ${wf:errorCode('wordcount')}</message> + </kill/> + <end name='end'/> +</workflow-app> +``` + +[::Go back to Oozie Documentation Index::](index.html) + + http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_SLAMonitoring.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/DG_SLAMonitoring.md b/docs/src/site/markdown/DG_SLAMonitoring.md new file mode 100644 index 0000000..0831b93 --- /dev/null +++ b/docs/src/site/markdown/DG_SLAMonitoring.md @@ -0,0 +1,459 @@ + + +[::Go back to Oozie Documentation Index::](index.html) + +# Oozie SLA Monitoring + +<!-- MACRO{toc|fromDepth=1|toDepth=4} --> + +## Overview + +Critical jobs can have certain SLA requirements associated with them. This SLA can be in terms of time +i.e. a maximum allowed time limit associated with when the job should start, by when should it end, +and its duration of run. Oozie workflows and coordinators allow defining such SLA limits in the application definition xml. + +With the addition of SLA Monitoring, Oozie can now actively monitor the state of these SLA-sensitive jobs +and send out notifications for SLA mets and misses. + +In versions earlier than 4.x, this was a passive feature where users needed to query the Oozie client SLA API +to fetch the records regarding job status changes, and use their own custom calculation engine to compute +whether SLA was met or missed, based on initial definition of time limits. + +Oozie now also has a SLA tab in the Oozie UI, where users can query for SLA information and have a summarized view +of how their jobs fared against their SLAs. + + +## Oozie Server Configuration + +Refer to [Notifications Configuration](AG_Install.html#Notifications_Configuration) for configuring Oozie server to track +SLA for jobs and send notifications. + +## SLA Tracking + +Oozie allows tracking SLA for meeting the following criteria: + + * Start time + * End time + * Job Duration + +### Event Status +Corresponding to each of these 3 criteria, your jobs are processed for whether Met or Miss i.e. + + * START_MET, START_MISS + * END_MET, END_MISS + * DURATION_MET, DURATION_MISS + +### SLA Status +Expected end-time is the most important criterion for majority of users while deciding overall SLA Met or Miss. +Hence the _"SLA_Status"_ for a job will transition through these four stages + + * Not_Started <-- Job not yet begun + * In_Process <-- Job started and is running, and SLAs are being tracked + * Met <-- caused by an END_MET + * Miss <-- caused by an END_MISS + +In addition to overshooting expected end-time, and END_MISS (and so an eventual SLA MISS) also occurs when the +job does not end successfully e.g. goes to error state - Failed/Killed/Error/Timedout. + +## Configuring SLA in Applications + +To make your jobs trackable for SLA, you simply need to add the `<sla:info>` tag to your workflow application definition. +If you were already using the existing SLA schema +in your workflows (Schema xmlns:sla="uri:oozie:sla:0.1"), you don't need to +do anything extra to receive SLA notifications via JMS messages. This new SLA monitoring framework is backward-compatible - +no need to change application XML for now and you can continue to fetch old records via the [command line API](DG_CommandLineTool.html#SLAOperations). +However, usage of old schema +and API is deprecated and we strongly recommend using new schema. + + * New SLA schema +is 'uri:oozie:sla:0.2' + * In order to use new SLA schema, + you will need to upgrade your workflow/coordinator schema +to 0.5 i.e. 'uri:oozie:workflow:0.5' + +### SLA Definition in Workflow +Example: + +``` +<workflow-app name="test-wf-job-sla" + xmlns="uri:oozie:workflow:0.5" + xmlns:sla="uri:oozie:sla:0.2"> + <start to="grouper"/> + <action name="grouper"> + <map-reduce> + <job-tracker>jt</job-tracker> + <name-node>nn</name-node> + <configuration> + <property> + <name>mapred.input.dir</name> + <value>input</value> + </property> + <property> + <name>mapred.output.dir</name> + <value>output</value> + </property> + </configuration> + </map-reduce> + <ok to="end"/> + <error to="end"/> + </action> + <end name="end"/> + <sla:info> + <sla:nominal-time>${nominal_time}</sla:nominal-time> + <sla:should-start>${10 * MINUTES}</sla:should-start> + <sla:should-end>${30 * MINUTES}</sla:should-end> + <sla:max-duration>${30 * MINUTES}</sla:max-duration> + <sla:alert-events>start_miss,end_miss,duration_miss</sla:alert-events> + <sla:alert-contact>j...@example.com</sla:alert-contact> + </sla:info> +</workflow-app> +``` + +For the list of tags usable under `<sla:info>`, refer to [Schemas Appendix](WorkflowFunctionalSpec.html#SLASchema). +This new schema +is much more compact and meaningful, getting rid of redundant and unused tags. + + * `nominal-time`: As the name suggests, this is the time relative to which your jobs' SLAs will be calculated. Generally since Oozie workflows are aligned with synchronous data dependencies, this nominal time can be parameterized to be passed the value of your coordinator nominal time. Nominal time is also required in case of independent workflows and you can specify the time in which you expect the workflow to be run if you don't have a synchronous dataset associated with it. + * `should-start`: Relative to `nominal-time` this is the amount of time (along with time-unit - MINUTES, HOURS, DAYS) within which your job should *start running* to meet SLA. This is optional. + * `should-end`: Relative to `nominal-time` this is the amount of time (along with time-unit - MINUTES, HOURS, DAYS) within which your job should *finish* to meet SLA. + * `max-duration`: This is the maximum amount of time (along with time-unit - MINUTES, HOURS, DAYS) your job is expected to run. This is optional. + * `alert-events`: Specify the types of events for which **Email** alerts should be sent. Allowable values in this comma-separated list are start_miss, end_miss and duration_miss. *_met events can generally be deemed low priority and hence email alerting for these is not necessary. However, note that this setting is only for alerts via *email* alerts and not via JMS messages, where all events send out notifications, and user can filter them using desired selectors. This is optional and only applicable when alert-contact is configured. + * `alert-contact`: Specify a comma separated list of email addresses where you wish your alerts to be sent. This is optional and need not be configured if you just want to view your job SLA history in the UI and do not want to receive email alerts. + +NOTE: All tags can be parameterized as a EL function or a fixed value. + +Same schema +can be applied to and embedded under Workflow-Action as well as Coordinator-Action XML. + +### SLA Definition in Workflow Action + + +``` +<workflow-app name="test-wf-action-sla" xmlns="uri:oozie:workflow:0.5" xmlns:sla="uri:oozie:sla:0.2"> + <start to="grouper"/> + <action name="grouper"> + ... + <ok to="end"/> + <error to="end"/> + <sla:info> + <sla:nominal-time>${nominal_time}</sla:nominal-time> + <sla:should-start>${10 * MINUTES}</sla:should-start> + ... + </sla:info> + </action> + <end name="end"/> +</workflow-app> +``` + +### SLA Definition in Coordinator Action + +``` +<coordinator-app name="test-coord-sla" frequency="${coord:days(1)}" freq_timeunit="DAY" + end_of_duration="NONE" start="2013-06-20T08:01Z" end="2013-12-01T08:01Z" + timezone="America/Los_Angeles" xmlns="uri:oozie:coordinator:0.4" xmlns:sla="uri:oozie:sla:0.2"> + <action> + <workflow> + <app-path>${wfAppPath}</app-path> + </workflow> + <sla:info> + <sla:nominal-time>${nominal_time}</sla:nominal-time> + ... + </sla:info> + </action> +</coordinator-app> +``` + +## Accessing SLA Information + +SLA information is accessible via the following ways: + + * Through the SLA tab of the Oozie Web UI. + * JMS messages sent to a configured JMS provider for instantaneous tracking. + * RESTful API to query for SLA summary. + * As an `Instrumentation.Counter` entry that is accessible via RESTful API and reflects to the number of all SLA tracked external + entities. Name of this counter is `sla-calculator.sla-map`. + +For JMS Notifications, you have to have a message broker in place, on which Oozie publishes messages and you can +hook on a subscriber to receive those messages. For more info on setting up and consuming JMS messages, refer +[JMS Notifications](DG_JMSNotifications.html) documentation. + +In the REST API, the following filters can be applied while fetching SLA information: + + * app_name - Application name + * id - id of the workflow job, workflow action or coordinator action + * parent_id - Parent id of the workflow job, workflow action or coordinator action + * nominal_start and nominal_end - Start and End range for nominal time of the workflow or coordinator. + * bundle - Bundle Job ID or Bundle App Name. Fetches SLA information for actions of all coordinators in that bundle. + * event_status - event status such as START_MET/START_MISS/DURATION_MET/DURATION_MISS/END_MET/END_MISS + * sla_status - sla status such as NOT_STARTED/IN_PROCESS/MET/MISS + +multiple event_status and sla_status can be specified with comma separation. When multiple statuses are specified, they are considered as OR. +For example, event_status=START_MET;END_MISS list the coordinator actions where event status is either START_MET OR END_MISS. + +When timezone query parameter is specified, the expected and actual start/end time returned is formatted. If not specified, +the number of milliseconds that have elapsed since January 1, 1970 00:00:00.000 GMT is returned. + +The examples below demonstrate the use of REST API and explains the JSON response. + +### Scenario 1: Workflow Job Start_Miss +**Request:** + +``` +GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=nominal_start=2013-06-18T00:01Z;nominal_end=2013-06-23T00:01Z;app_name=my-sla-app +``` + +**JSON Response** + +``` +{ + + id : "000056-1238791320234-oozie-joe-W" + parentId : "000001-1238791320234-oozie-joe-C@8" + appType : "WORKFLOW_JOB" + msgType : "SLA" + appName : "my-sla-app" + slaStatus : "IN_PROCESS" + jobStatus : "RUNNING" + user: "joe" + nominalTime: "2013-16-22T05:00Z" + expectedStartTime: "2013-16-22T05:10Z" <-- (should start by this time) + actualStartTime: "2013-16-22T05:30Z" <-- (20 min late relative to expected start) + expectedEndTime: "2013-16-22T05:40Z" <-- (should end by this time) + actualEndTime: null + expectedDuration: 900000 <-- (expected duration in milliseconds) + actualDuration: 120000 <-- (actual duration in milliseconds) + notificationMessage: "My Job has encountered an SLA event!" + upstreamApps: "dependent-app-1, dependent-app-2" + +} +``` + +### Scenario 2: Workflow Action End_Miss +**Request:** + +``` +GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=parent_id=000056-1238791320234-oozie-joe-W +``` + +**JSON Response** + +``` +{ + + id : "000056-1238791320234-oozie-joe-W@map-reduce-action" + parentId : "000056-1238791320234-oozie-joe-W" + appType : "WORKFLOW_ACTION" + msgType : "SLA" + appName : "map-reduce-action" + slaStatus : "MISS" + jobStatus : "SUCCEEDED" + user: "joe" + nominalTime: "2013-16-22T05:00Z" + expectedStartTime: "2013-16-22T05:10Z" + actualStartTime: "2013-16-22T05:05Z" + expectedEndTime: "2013-16-22T05:40Z" <-- (should end by this time) + actualEndTime: "2013-16-22T06:00Z" <-- (20 min late relative to expected end) + expectedDuration: 3600000 <-- (expected duration in milliseconds) + actualDuration: 3300000 <-- (actual duration in milliseconds) + notificationMessage: "My Job has encountered an SLA event!" + upstreamApps: "dependent-app-1, dependent-app-2" + +} +``` + +### Scenario 3: Coordinator Action Duration_Miss +**Request:** + +``` +GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=id=000001-1238791320234-oozie-joe-C +``` + +**JSON Response** + +``` +{ + + id : "000001-1238791320234-oozie-joe-C@2" + parentId : "000001-1238791320234-oozie-joe-C" + appType : "COORDINATOR_ACTION" + msgType : "SLA" + appName : "my-coord-app" + slaStatus : "MET" + jobStatus : "SUCCEEDED" + user: "joe" + nominalTime: "2013-16-22T05:00Z" + expectedStartTime: "2013-16-22T05:10Z" + actualStartTime: "2013-16-22T05:05Z" + expectedEndTime: "2013-16-22T05:40Z" + actualEndTime: "2013-16-22T05:30Z" + expectedDuration: 900000 <-- (expected duration in milliseconds) + actualDuration: 1500000 <- (actual duration in milliseconds) + notificationMessage: "My Job has encountered an SLA event!" + upstreamApps: "dependent-app-1, dependent-app-2" + +} +``` + +Scenario #3 is particularly interesting because it is an overall "MET" because it met its expected End-time, +but it is "Duration_Miss" because the actual run (between actual start and actual end) exceeded expected duration. + +### Scenario 4: All Coordinator actions in a Bundle +**Request:** + +``` +GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=bundle=1234567-150130225116604-oozie-B;event_status=END_MISS +``` + +**JSON Response** + +``` +{ + id : "000001-1238791320234-oozie-joe-C@1" + parentId : "000001-1238791320234-oozie-joe-C" + appType : "COORDINATOR_ACTION" + msgType : "SLA" + appName : "my-coord-app" + slaStatus : "MET" + eventStatus : "START_MET,DURATION_MISS,END_MISS" + user: "joe" + nominalTime: "2014-01-10T12:00Z" + expectedStartTime: "2014-01-10T12:00Z" + actualStartTime: "2014-01-10T11:59Z" + startDelay: -1 + expectedEndTime: "2014-01-10T13:00Z" + actualEndTime: "2014-01-10T13:05Z" + endDelay: 5 + expectedDuration: 3600000 <-- (expected duration in milliseconds) + actualDuration: 3960000 <-- (actual duration in milliseconds) + durationDelay: 6 <-- (duration delay in minutes) +} +{ + id : "000001-1238791320234-oozie-joe-C@2" + parentId : "000001-1238791320234-oozie-joe-C" + appType : "COORDINATOR_ACTION" + msgType : "SLA" + appName : "my-coord-app" + slaStatus : "MET" + eventStatus : "START_MISS,DURATION_MET,END_MISS" + user: "joe" + nominalTime: "2014-01-11T12:00Z" + expectedStartTime: "2014-01-11T12:00Z" + actualStartTime: "2014-01-11T12:05Z" + startDelay: 5 + expectedEndTime: "2014-01-11T13:00Z" + actualEndTime: "2014-01-11T13:01Z" + endDelay: 1 + expectedDuration: 3600000 <-- (expected duration in milliseconds) + actualDuration: 3360000 <-- (actual duration in milliseconds) + durationDelay: -4 <-- (duration delay in minutes) +} +``` + +Scenario #4 (All Coordinator actions in a Bundle) is to get SLA information of all coordinator actions under bundle job in one call. +startDelay/durationDelay/endDelay values returned indicate how much delay compared to expected time (positive values in case of MISS, and negative values in case of MET). + +### Sample Email Alert + +``` +Subject: OOZIE - SLA END_MISS (AppName=wf-sla-job, JobID=0000004-130610225200680-oozie-oozi-W) + + +Status: + SLA Status - END_MISS + Job Status - RUNNING + Notification Message - Missed SLA for Data Pipeline job +Job Details: + App Name - wf-sla-job + App Type - WORKFLOW_JOB + User - strat_ci + Job ID - 0000004-130610225200680-oozie-oozi-W + Job URL - http://host.domain.com:4080/oozie//?job=0000004-130610225200680-oozie-oozi-W + Parent Job ID - N/A + Parent Job URL - N/A + Upstream Apps - wf-sla-up-app +SLA Details: + Nominal Time - Mon Jun 10 23:33:00 UTC 2013 + Expected Start Time - Mon Jun 10 23:35:00 UTC 2013 + Actual Start Time - Mon Jun 10 23:34:04 UTC 2013 + Expected End Time - Mon Jun 10 23:38:00 UTC 2013 + Expected Duration (in mins) - 5 + Actual Duration (in mins) - -1 +``` + +### Changing job SLA definition and alerting +Following are ways to enable/disable SLA alerts for coordinator actions. + +#### 1. Specify in Bundle XML during submission. +Following properties can be specified in bundle xml as properties for coordinator. + +`oozie.sla.disable.alerts.older.than` this property can be specified in hours, the SLA notification for +coord actions will be disabled whose nominal is time older then this value. Default is 48 hours. + +``` +<property> + <name>oozie.sla.disable.alerts.older.than</name> + <value>12</value> +</property> +``` + +`oozie.sla.disable.alerts` List of coord actions to be disabled. Value can be specified as list of coord actions or date range. + +``` +<property> + <name>oozie.sla.disable.alerts</name> + <value>1,3-4,7-10</value> +</property> +``` +Will disable alert for coord actions 1,3,5,7,8,9,10 + +`oozie.sla.enable.alerts` List of coord actions to be enabled. Value can be specified as list of coord actions or date range. + +``` +<property> + <name>oozie.sla.enable.alerts</name> + <value>2009-01-01T01:00Z::2009-05-31T23:59Z</value> +</property> +``` +This will enable SLA alert for coord actions whose nominal time is in between (inclusive) 2009-01-01T01:00Z and 2009-05-31T23:59Z. + +ALL keyword can be specified to specify all actions. Below property will disable SLA notifications for all coord actions. + +``` +<property> + <name>oozie.sla.disable.alerts</name> + <value>ALL</value> +</property> +``` + +#### 2. Specify during Coordinator job submission or update +Above properties can be specified in job.properties in +[Coord job update command](DG_CommandLineTool.html#Updating_coordinator_definition_and_properties), +in [Coord job submit command](DG_CommandLineTool.html#Submitting_a_Workflow_Coordinator_or_Bundle_Job) +or in [Coord job run command](DG_CommandLineTool.html#Running_a_Workflow_Coordinator_or_Bundle_Job) + +#### 3. Change using command line +Refer [Changing job SLA definition and alerting](DG_CommandLineTool.html#Changing_job_SLA_definition_and_alerting) for commandline usage. + +#### 4. Change using REST API +Refer the REST API [Changing job SLA definition and alerting](WebServicesAPI.html#Changing_job_SLA_definition_and_alerting). + +## In-memory SLA entries and database content + +There are special circumstances when the in-memory `SLACalcStatus` entries can exist without the workflow or coordinator job or +action instances in database. For example: + + * SLA tracked database content may already have been deleted, and `SLA_SUMMARY` entry is not present anymore in database + * SLA tracked database content and `SLA_SUMMARY` entry aren't yet present in database + +By the time `SLAService` scheduled job will be running, SLA map contents are checked. When the `SLA_SUMMARY` entry for the in-memory +SLA entry is missing, a counter is increased. When this counter reaches the server-wide preconfigured value +`oozie.sla.service.SLAService.maximum.retry.count` (by default `3`), in-memory SLA entry will get purged. + +## Known issues +There are two known issues when you define SLA for a workflow action. + * If there are decision nodes and SLA is defined for a workflow action not in the execution path because of the decision node, you will still get an SLA_MISS notification. + * If you have dangling action nodes in your workflow definition and SLA is defined for it, you will still get an SLA_MISS notification. + +[::Go back to Oozie Documentation Index::](index.html) + + http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_ShellActionExtension.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/DG_ShellActionExtension.md b/docs/src/site/markdown/DG_ShellActionExtension.md new file mode 100644 index 0000000..eff4b08 --- /dev/null +++ b/docs/src/site/markdown/DG_ShellActionExtension.md @@ -0,0 +1,494 @@ + + +[::Go back to Oozie Documentation Index::](index.html) + +----- + +# Oozie Shell Action Extension + +<!-- MACRO{toc|fromDepth=1|toDepth=4} --> + +<a name="ShellAction"></a> +## Shell Action + +The `shell` action runs a Shell command. + +The workflow job will wait until the Shell command completes before +continuing to the next action. + +To run the Shell job, you have to configure the `shell` action with the +`job-tracker`, `name-node` and Shell `exec` elements as +well as the necessary arguments and configuration. + +A `shell` action can be configured to create or delete HDFS directories +before starting the Shell job. + +Shell _launcher_ configuration can be specified with a file, using the `job-xml` +element, and inline, using the `configuration` elements. + +Oozie EL expressions can be used in the inline configuration. Property +values specified in the `configuration` element override values specified +in the `job-xml` file. + +Note that YARN `yarn.resourcemanager.address` (`resource-manager`) and HDFS `fs.default.name` (`name-node`) properties +must not be present in the inline configuration. + +As with Hadoop `map-reduce` jobs, it is possible to add files and +archives in order to make them available to the Shell job. Refer to the +[WorkflowFunctionalSpec#FilesArchives][Adding Files and Archives for the Job] +section for more information about this feature. + +The output (STDOUT) of the Shell job can be made available to the workflow job after the Shell job ends. This information +could be used from within decision nodes. If the output of the Shell job is made available to the workflow job the shell +command must follow the following requirements: + + * The format of the output must be a valid Java Properties file. + * The size of the output must not exceed 2KB. + +**Syntax:** + + +``` +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="[NODE-NAME]"> + <shell xmlns="uri:oozie:shell-action:1.0"> + <resource-manager>[RESOURCE-MANAGER]</resource-manager> + <name-node>[NAME-NODE]</name-node> + <prepare> + <delete path="[PATH]"/> + ... + <mkdir path="[PATH]"/> + ... + </prepare> + <job-xml>[SHELL SETTINGS FILE]</job-xml> + <configuration> + <property> + <name>[PROPERTY-NAME]</name> + <value>[PROPERTY-VALUE]</value> + </property> + ... + </configuration> + <exec>[SHELL-COMMAND]</exec> + <argument>[ARG-VALUE]</argument> + ... + <argument>[ARG-VALUE]</argument> + <env-var>[VAR1=VALUE1]</env-var> + ... + <env-var>[VARN=VALUEN]</env-var> + <file>[FILE-PATH]</file> + ... + <archive>[FILE-PATH]</archive> + ... + <capture-output/> + </shell> + <ok to="[NODE-NAME]"/> + <error to="[NODE-NAME]"/> + </action> + ... +</workflow-app> +``` + +The `prepare` element, if present, indicates a list of paths to delete +or create before starting the job. Specified paths must start with `hdfs://HOST:PORT`. + +The `job-xml` element, if present, specifies a file containing configuration +for the Shell job. As of schema 0.2, multiple `job-xml` elements are allowed in order to +specify multiple `job.xml` files. + +The `configuration` element, if present, contains configuration +properties that are passed to the Shell job. + +The `exec` element must contain the path of the Shell command to +execute. The arguments of Shell command can then be specified +using one or more `argument` element. + +The `argument` element, if present, contains argument to be passed to +the Shell command. + +The `env-var` element, if present, contains the environment to be passed +to the Shell command. `env-var` should contain only one pair of environment variable +and value. If the pair contains the variable such as $PATH, it should follow the +Unix convention such as PATH=$PATH:mypath. Don't use ${PATH} which will be +substituted by Oozie's EL evaluator. + +A `shell` action creates a Hadoop configuration. The Hadoop configuration is made available as a local file to the +Shell application in its running directory. The exact file path is exposed to the spawned shell using the environment +variable called `OOZIE_ACTION_CONF_XML`.The Shell application can access the environment variable to read the action +configuration XML file path. + +If the `capture-output` element is present, it indicates Oozie to capture output of the STDOUT of the shell command +execution. The Shell command output must be in Java Properties file format and it must not exceed 2KB. From within the +workflow definition, the output of an Shell action node is accessible via the `String action:output(String node, +String key)` function (Refer to section '4.2.6 Action EL Functions'). + +All the above elements can be parameterized (templatized) using EL +expressions. + +**Example:** + +How to run any shell script or perl script or CPP executable + + +``` +<workflow-app xmlns='uri:oozie:workflow:1.0' name='shell-wf'> + <start to='shell1' /> + <action name='shell1'> + <shell xmlns="uri:oozie:shell-action:1.0"> + <resource-manager>${resourceManager}</resource-manager> + <name-node>${nameNode}</name-node> + <configuration> + <property> + <name>mapred.job.queue.name</name> + <value>${queueName}</value> + </property> + </configuration> + <exec>${EXEC}</exec> + <argument>A</argument> + <argument>B</argument> + <file>${EXEC}#${EXEC}</file> <!--Copy the executable to compute node's current working directory --> + </shell> + <ok to="end" /> + <error to="fail" /> + </action> + <kill name="fail"> + <message>Script failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> + </kill> + <end name='end' /> +</workflow-app> +``` + +The corresponding job properties file used to submit Oozie job could be as follows: + + +``` +oozie.wf.application.path=hdfs://localhost:8020/user/kamrul/workflows/script + +#Execute is expected to be in the Workflow directory. +#Shell Script to run +EXEC=script.sh +#CPP executable. Executable should be binary compatible to the compute node OS. +#EXEC=hello +#Perl script +#EXEC=script.pl + +resourceManager=localhost:8032 +nameNode=hdfs://localhost:8020 +queueName=default + +``` + +How to run any java program bundles in a jar. + + +``` +<workflow-app xmlns='uri:oozie:workflow:1.0' name='shell-wf'> + <start to='shell1' /> + <action name='shell1'> + <shell xmlns="uri:oozie:shell-action:1.0"> + <resource-manager>${resourceManager}</resource-manager> + <name-node>${nameNode}</name-node> + <configuration> + <property> + <name>mapred.job.queue.name</name> + <value>${queueName}</value> + </property> + </configuration> + <exec>java</exec> + <argument>-classpath</argument> + <argument>./${EXEC}:$CLASSPATH</argument> + <argument>Hello</argument> + <file>${EXEC}#${EXEC}</file> <!--Copy the jar to compute node current working directory --> + </shell> + <ok to="end" /> + <error to="fail" /> + </action> + <kill name="fail"> + <message>Script failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> + </kill> + <end name='end' /> +</workflow-app> +``` + +The corresponding job properties file used to submit Oozie job could be as follows: + + +``` +oozie.wf.application.path=hdfs://localhost:8020/user/kamrul/workflows/script + +#Hello.jar file is expected to be in the Workflow directory. +EXEC=Hello.jar + +resourceManager=localhost:8032 +nameNode=hdfs://localhost:8020 +queueName=default +``` + +### Shell Action Configuration + + * `oozie.action.shell.setup.hadoop.conf.dir` - Generates a config directory with various core/hdfs/yarn/mapred-site.xml files and points `HADOOP_CONF_DIR` and `YARN_CONF_DIR` env-vars to it, before the Script is invoked. XML is sourced from the action configuration. Useful when the Shell script passed uses various `hadoop` commands. Default is false. + * `oozie.action.shell.setup.hadoop.conf.dir.write.log4j.properties` - When `oozie.action.shell.setup.hadoop.conf.dir` is enabled, toggle if a log4j.properties file should also be written under the configuration files directory. Default is true. + * `oozie.action.shell.setup.hadoop.conf.dir.log4j.content` - When `oozie.action.shell.setup.hadoop.conf.dir.write.log4j.properties` is enabled, the content to write into the log4j.properties file under the configuration files directory. Default is a simple console based stderr logger, as presented below: + +``` +log4j.rootLogger=${hadoop.root.logger} +hadoop.root.logger=INFO,console +log4j.appender.console=org.apache.log4j.ConsoleAppender +log4j.appender.console.target=System.err +log4j.appender.console.layout=org.apache.log4j.PatternLayout +log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n +``` + +### Shell Action Logging + +Shell action's stdout and stderr output are redirected to the Oozie Launcher map-reduce job task STDOUT that runs the shell command. + +From Oozie web-console, from the Shell action pop up using the 'Console URL' link, it is possible +to navigate to the Oozie Launcher map-reduce job task logs via the Hadoop job-tracker web-console. + +### Shell Action Limitations +Although Shell action can execute any shell command, there are some limitations. + + * No interactive command is supported. + * Command can't be executed as different user using sudo. + * User has to explicitly upload the required 3rd party packages (such as jar, so lib, executable etc). Oozie provides a way using \<file\> and \<archive\> tag through Hadoop's Distributed Cache to upload. + * Since Oozie will execute the shell command into a Hadoop compute node, the default installation of utility in the compute node might not be fixed. However, the most common unix utilities are usually installed on all compute nodes. It is important to note that Oozie could only support the commands that are installed into the compute nodes or that are uploaded through Distributed Cache. + +## Appendix, Shell XML-Schema + +### AE.A Appendix A, Shell XML-Schema + +#### Shell Action Schema Version 1.0 + +``` +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; + xmlns:shell="uri:oozie:shell-action:1.0" + elementFormDefault="qualified" + targetNamespace="uri:oozie:shell-action:1.0"> +. + <xs:include schemaLocation="oozie-common-1.0.xsd"/> +. + <xs:element name="shell" type="shell:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:choice> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:choice> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="shell:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="launcher" type="shell:LAUNCHER" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="shell:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="exec" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="env-var" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="capture-output" type="shell:FLAG" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + </xs:complexType> +. +</xs:schema> +``` + +#### Shell Action Schema Version 0.3 + +``` +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; + xmlns:shell="uri:oozie:shell-action:0.3" elementFormDefault="qualified" + targetNamespace="uri:oozie:shell-action:0.3"> + + <xs:element name="shell" type="shell:ACTION"/> + + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="shell:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="shell:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="exec" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="env-var" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="capture-output" type="shell:FLAG" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="FLAG"/> + + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="shell:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="shell:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> + + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> + +</xs:schema> +``` + +#### Shell Action Schema Version 0.2 + +``` +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; + xmlns:shell="uri:oozie:shell-action:0.2" elementFormDefault="qualified" + targetNamespace="uri:oozie:shell-action:0.2"> + + <xs:element name="shell" type="shell:ACTION"/> + + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="prepare" type="shell:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="shell:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="exec" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="env-var" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="capture-output" type="shell:FLAG" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="FLAG"/> + + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="shell:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="shell:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> + + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> + +</xs:schema> +``` + +#### Shell Action Schema Version 0.1 + +``` +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; + xmlns:shell="uri:oozie:shell-action:0.1" elementFormDefault="qualified" + targetNamespace="uri:oozie:shell-action:0.1"> + + <xs:element name="shell" type="shell:ACTION"/> + + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="prepare" type="shell:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="configuration" type="shell:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="exec" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="env-var" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="capture-output" type="shell:FLAG" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="FLAG"/> + + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="shell:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="shell:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> + + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> + +</xs:schema> +``` + +[::Go back to Oozie Documentation Index::](index.html) + +
