Added: websites/staging/oozie/trunk/content/docs/5.0.0/DG_Hive2ActionExtension.html ============================================================================== --- websites/staging/oozie/trunk/content/docs/5.0.0/DG_Hive2ActionExtension.html (added) +++ websites/staging/oozie/trunk/content/docs/5.0.0/DG_Hive2ActionExtension.html Mon Apr 9 14:26:49 2018 @@ -0,0 +1,470 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at Apr 9, 2018 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie - </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.0.0"> + 5.0.0</a> + <span class="divider">/</span> + </li> + <li class="active ">Oozie - </li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-04-09</li> + <li id="projectVersion" class="pull-right"> + Version: 5.0.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p></p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<hr /> +<a name="Oozie_Hive_2_Action_Extension"></a> +<div class="section"><h2> Oozie Hive 2 Action Extension</h2> +<p><ul><ul><li><a href="#Hive_2_Action">Hive 2 Action</a> +<ul><li><a href="#Security">Security</a> +</li> +</ul> +</li> +<li><a href="#Appendix_Hive_2_XML-Schema">Appendix, Hive 2 XML-Schema</a> +<ul><li><a href="#AE.A_Appendix_A_Hive_2_XML-Schema">AE.A Appendix A, Hive 2 XML-Schema</a> +<ul><li><a href="#Hive_2_Action_Schema_Version_1.0">Hive 2 Action Schema Version 1.0</a> +</li> +<li><a href="#Hive_2_Action_Schema_Version_0.2">Hive 2 Action Schema Version 0.2</a> +</li> +<li><a href="#Hive_2_Action_Schema_Version_0.1">Hive 2 Action Schema Version 0.1</a> +</li> +</ul> +</li> +</ul> +</li> +</ul> +</ul> +</p> +<a name="Hive_2_Action"></a> +<div class="section"><h3>Hive 2 Action</h3> +<p>The <tt>hive2</tt> + action runs Beeline to connect to Hive Server 2.</p> +<p>The workflow job will wait until the Hive Server 2 job completes before +continuing to the next action.</p> +<p>To run the Hive Server 2 job, you have to configure the <tt>hive2</tt> + action with the <tt>resource-manager</tt> +, <tt>name-node</tt> +, <tt>jdbc-url</tt> +, + <tt>password</tt> + elements, and either Hive's <tt>script</tt> + or <tt>query</tt> + element, as well as the necessary parameters and configuration.</p> +<p>A <tt>hive2</tt> + action can be configured to create or delete HDFS directories +before starting the Hive Server 2 job.</p> +<p>Oozie EL expressions can be used in the inline configuration. Property +values specified in the <tt>configuration</tt> + element override values specified +in the <tt>job-xml</tt> + file.</p> +<p>As with Hadoop <tt>map-reduce</tt> + jobs, it is possible to add files and +archives in order to make them available to Beeline. Refer to the +[WorkflowFunctionalSpec#FilesArchives][Adding Files and Archives for the Job] +section for more information about this feature.</p> +<p>Oozie Hive 2 action supports Hive scripts with parameter variables, their +syntax is <tt>${VARIABLES}</tt> +.</p> +<p><b>Syntax:</b> +</p> +<p><pre> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="[NODE-NAME]"> + <hive2 xmlns="uri:oozie:hive2-action:1.0"> + <resource-manager>[RESOURCE-MANAGER]</resource-manager> + <name-node>[NAME-NODE]</name-node> + <prepare> + <delete path="[PATH]"/> + ... + <mkdir path="[PATH]"/> + ... + </prepare> + <job-xml>[HIVE SETTINGS FILE]</job-xml> + <configuration> + <property> + <name>[PROPERTY-NAME]</name> + <value>[PROPERTY-VALUE]</value> + </property> + ... + </configuration> + <jdbc-url>[jdbc:hive2://HOST:10000/default]</jdbc-url> + <password>[PASS]</password> + <script>[HIVE-SCRIPT]</script> + <param>[PARAM-VALUE]</param> + ... + <param>[PARAM-VALUE]</param> + <argument>[ARG-VALUE]</argument> + ... + <argument>[ARG-VALUE]</argument> + <file>[FILE-PATH]</file> + ... + <archive>[FILE-PATH]</archive> + ... + </hive2> + <ok to="[NODE-NAME]"/> + <error to="[NODE-NAME]"/> + </action> + ... +</workflow-app> +</pre></p> +<p>The <tt>prepare</tt> + element, if present, indicates a list of paths to delete +or create before starting the job. Specified paths must start with <tt><a href="./hdfs://HOST:PORT.html">hdfs://HOST:PORT</a> +</tt> +.</p> +<p>The <tt>job-xml</tt> + element, if present, specifies a file containing configuration +for Beeline. Multiple <tt>job-xml</tt> + elements are allowed in order to specify multiple <tt>job.xml</tt> + files.</p> +<p>The <tt>configuration</tt> + element, if present, contains configuration +properties that are passed to the Beeline job.</p> +<p>The <tt>jdbc-url</tt> + element must contain the JDBC URL for the Hive Server 2. Beeline will use this to know where to connect to.</p> +<p>The <tt>password</tt> + element must contain the password of the current user. However, the <tt>password</tt> + is only used if Hive Server 2 is +backed by something requiring a password (e.g. LDAP); non-secured Hive Server 2 or Kerberized Hive Server 2 don't require a password +so in those cases the <tt>password</tt> + is ignored and can be omitted from the action XML. It is up to the user to ensure that a password +is specified when required.</p> +<p>The <tt>script</tt> + element must contain the path of the Hive script to +execute. The Hive script can be templatized with variables of the form +=${VARIABLE}=. The values of these variables can then be specified +using the <tt>params</tt> + element.</p> +<p>The <tt>query</tt> + element available from uri:oozie:hive2-action:0.2, can be used instead of the <tt>script</tt> + element. It allows for embedding +queries within the <tt>worklfow.xml</tt> + directly. Similar to the <tt>script</tt> + element, it also allows for the templatization of variables +in the form <tt>${VARIABLE}</tt> +.</p> +<p>The <tt>params</tt> + element, if present, contains parameters to be passed to +the Hive script.</p> +<p>The <tt>argument</tt> + element, if present, contains arguments to be passed as-is to Beeline.</p> +<p>All the above elements can be parameterized (templatized) using EL +expressions.</p> +<p><b>Example:</b> +</p> +<p><pre> +<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="my-hive2-action"> + <hive2 xmlns="uri:oozie:hive2-action:1.0"> + <resource-manager>foo:8032</resource-manager> + <name-node>bar:8020</name-node> + <prepare> + <delete path="${jobOutput}"/> + </prepare> + <configuration> + <property> + <name>mapred.compress.map.output</name> + <value>true</value> + </property> + </configuration> + <jdbc-url>jdbc:hive2://localhost:10000/default</jdbc-url> + <password>foo</password> + <script>myscript.q</script> + <param>InputDir=/home/rkanter/input-data</param> + <param>OutputDir=${jobOutput}</param> + </hive2> + <ok to="my-other-action"/> + <error to="error-cleanup"/> + </action> + ... +</workflow-app> +</pre></p> +<a name="Security"></a> +<div class="section"><h4>Security</h4> +<p>As mentioned above, <tt>password</tt> + is only used in cases where Hive Server 2 is backed by something requiring a password (e.g. LDAP). +Non-secured Hive Server 2 and Kerberized Hive Server 2 don't require a password so in these cases it can be omitted. +See <a href="./DG_UnifiedCredentialsModule.html">here</a> + for more information on the configuration for using the Hive Server 2 Action +with a Kerberized Hive Server 2.</p> +<a name="Appendix_Hive_2_XML-Schema"></a> +</div> +</div> +<div class="section"><h3>Appendix, Hive 2 XML-Schema</h3> +<a name="AE.A_Appendix_A_Hive_2_XML-Schema"></a> +<div class="section"><h4>AE.A Appendix A, Hive 2 XML-Schema</h4> +<a name="Hive_2_Action_Schema_Version_1.0"></a> +<div class="section"><h5>Hive 2 Action Schema Version 1.0</h5> +<p><pre> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:hive2="uri:oozie:hive2-action:1.0" elementFormDefault="qualified" + targetNamespace="uri:oozie:hive2-action:1.0"> +. + <xs:include schemaLocation="oozie-common-1.0.xsd"/> +. + <xs:element name="hive2" type="hive2:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:choice> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:choice> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="hive2:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="launcher" type="hive2:LAUNCHER" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="hive2:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="jdbc-url" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="password" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:choice minOccurs="1" maxOccurs="1"> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="query" type="xs:string" minOccurs="1" maxOccurs="1"/> + </xs:choice> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. +</xs:schema> +</pre></p> +<a name="Hive_2_Action_Schema_Version_0.2"></a> +</div> +<div class="section"><h5>Hive 2 Action Schema Version 0.2</h5> +<p><pre> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:hive2="uri:oozie:hive2-action:0.2" elementFormDefault="qualified" + targetNamespace="uri:oozie:hive2-action:0.2"> +. + <xs:element name="hive2" type="hive2:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="hive2:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="hive2:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="jdbc-url" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="password" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:choice minOccurs="1" maxOccurs="1"> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="query" type="xs:string" minOccurs="1" maxOccurs="1"/> + </xs:choice> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="hive2:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="hive2:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. +</xs:schema> +</pre></p> +<a name="Hive_2_Action_Schema_Version_0.1"></a> +</div> +<div class="section"><h5>Hive 2 Action Schema Version 0.1</h5> +<p><pre> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:hive2="uri:oozie:hive2-action:0.1" elementFormDefault="qualified" + targetNamespace="uri:oozie:hive2-action:0.1"> +. + <xs:element name="hive2" type="hive2:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="hive2:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="hive2:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="jdbc-url" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="password" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="hive2:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="hive2:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. +</xs:schema> +</pre></p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<p></p> +</div> +</div> +</div> + + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html>
Added: websites/staging/oozie/trunk/content/docs/5.0.0/DG_HiveActionExtension.html ============================================================================== --- websites/staging/oozie/trunk/content/docs/5.0.0/DG_HiveActionExtension.html (added) +++ websites/staging/oozie/trunk/content/docs/5.0.0/DG_HiveActionExtension.html Mon Apr 9 14:26:49 2018 @@ -0,0 +1,637 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at Apr 9, 2018 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie - </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.0.0"> + 5.0.0</a> + <span class="divider">/</span> + </li> + <li class="active ">Oozie - </li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-04-09</li> + <li id="projectVersion" class="pull-right"> + Version: 5.0.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p></p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<hr /> +<a name="Oozie_Hive_Action_Extension"></a> +<div class="section"><h2> Oozie Hive Action Extension</h2> +<p><ul><ul><li><a href="#Hive_Action">Hive Action</a> +<ul><li><a href="#Hive_Default_and_Site_Configuration_Files">Hive Default and Site Configuration Files</a> +</li> +<li><a href="#Hive_Action_Logging">Hive Action Logging</a> +</li> +</ul> +</li> +<li><a href="#Appendix_Hive_XML-Schema">Appendix, Hive XML-Schema</a> +<ul><li><a href="#AE.A_Appendix_A_Hive_XML-Schema">AE.A Appendix A, Hive XML-Schema</a> +<ul><li><a href="#Hive_Action_Schema_Version_1.0">Hive Action Schema Version 1.0</a> +</li> +<li><a href="#Hive_Action_Schema_Version_0.6">Hive Action Schema Version 0.6</a> +</li> +<li><a href="#Hive_Action_Schema_Version_0.5">Hive Action Schema Version 0.5</a> +</li> +<li><a href="#Hive_Action_Schema_Version_0.4">Hive Action Schema Version 0.4</a> +</li> +<li><a href="#Hive_Action_Schema_Version_0.3">Hive Action Schema Version 0.3</a> +</li> +<li><a href="#Hive_Action_Schema_Version_0.2">Hive Action Schema Version 0.2</a> +</li> +</ul> +</li> +</ul> +</li> +</ul> +</ul> +</p> +<p><a name="HiveAction"></a> +</p> +<a name="Hive_Action"></a> +<div class="section"><h3>Hive Action</h3> +<p>The <tt>hive</tt> + action runs a Hive job.</p> +<p>The workflow job will wait until the Hive job completes before +continuing to the next action.</p> +<p>To run the Hive job, you have to configure the <tt>hive</tt> + action with the <tt>resource-manager</tt> +, <tt>name-node</tt> + and Hive <tt>script</tt> + +(or Hive <tt>query</tt> +) elements as well as the necessary parameters and configuration.</p> +<p>A <tt>hive</tt> + action can be configured to create or delete HDFS directories +before starting the Hive job.</p> +<p>Hive configuration can be specified with a file, using the <tt>job-xml</tt> + +element, and inline, using the <tt>configuration</tt> + elements.</p> +<p>Oozie EL expressions can be used in the inline configuration. Property +values specified in the <tt>configuration</tt> + element override values specified +in the <tt>job-xml</tt> + file.</p> +<p>Note that YARN <tt>yarn.resourcemanager.address</tt> + (=resource-manager=) and HDFS <tt>fs.default.name</tt> + (=name-node=) properties +must not be present in the inline configuration.</p> +<p>As with Hadoop <tt>map-reduce</tt> + jobs, it is possible to add files and +archives in order to make them available to the Hive job. Refer to the +[WorkflowFunctionalSpec#FilesArchives][Adding Files and Archives for the Job] +section for more information about this feature.</p> +<p>Oozie Hive action supports Hive scripts with parameter variables, their +syntax is <tt>${VARIABLES}</tt> +.</p> +<p><b>Syntax:</b> +</p> +<p><pre> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="[NODE-NAME]"> + <hive xmlns="uri:oozie:hive-action:1.0"> + <resource-manager>[RESOURCE-MANAGER]</resource-manager> + <name-node>[NAME-NODE]</name-node> + <prepare> + <delete path="[PATH]"/> + ... + <mkdir path="[PATH]"/> + ... + </prepare> + <job-xml>[HIVE SETTINGS FILE]</job-xml> + <configuration> + <property> + <name>[PROPERTY-NAME]</name> + <value>[PROPERTY-VALUE]</value> + </property> + ... + </configuration> + <script>[HIVE-SCRIPT]</script> + <param>[PARAM-VALUE]</param> + ... + <param>[PARAM-VALUE]</param> + <file>[FILE-PATH]</file> + ... + <archive>[FILE-PATH]</archive> + ... + </hive> + <ok to="[NODE-NAME]"/> + <error to="[NODE-NAME]"/> + </action> + ... +</workflow-app> +</pre></p> +<p>The <tt>prepare</tt> + element, if present, indicates a list of paths to delete +or create before starting the job. Specified paths must start with <tt><a href="./hdfs://HOST:PORT.html">hdfs://HOST:PORT</a> +</tt> +.</p> +<p>The <tt>job-xml</tt> + element, if present, specifies a file containing configuration +for the Hive job. As of schema 0.3, multiple <tt>job-xml</tt> + elements are allowed in order to +specify multiple <tt>job.xml</tt> + files.</p> +<p>The <tt>configuration</tt> + element, if present, contains configuration +properties that are passed to the Hive job.</p> +<p>The <tt>script</tt> + element must contain the path of the Hive script to +execute. The Hive script can be templatized with variables of the form +=${VARIABLE}=. The values of these variables can then be specified +using the <tt>params</tt> + element.</p> +<p>The <tt>query</tt> + element available from uri:oozie:hive-action:0.6, can be used instead of the +=script= element. It allows for embedding queries within the <tt>worklfow.xml</tt> + directly. +Similar to the <tt>script</tt> + element, it also allows for the templatization of variables in the +form <tt>${VARIABLE}</tt> +.</p> +<p>The <tt>params</tt> + element, if present, contains parameters to be passed to +the Hive script.</p> +<p>All the above elements can be parameterized (templatized) using EL +expressions.</p> +<p><b>Example:</b> +</p> +<p><pre> +<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="myfirsthivejob"> + <hive xmlns="uri:oozie:hive-action:1.0"> + <resource-manager>foo:8032</resource-manager> + <name-node>bar:8020</name-node> + <prepare> + <delete path="${jobOutput}"/> + </prepare> + <configuration> + <property> + <name>mapred.compress.map.output</name> + <value>true</value> + </property> + </configuration> + <script>myscript.q</script> + <param>InputDir=/home/tucu/input-data</param> + <param>OutputDir=${jobOutput}</param> + </hive> + <ok to="myotherjob"/> + <error to="errorcleanup"/> + </action> + ... +</workflow-app> +</pre></p> +<a name="Hive_Default_and_Site_Configuration_Files"></a> +<div class="section"><h4>Hive Default and Site Configuration Files</h4> +<p>Hive (as of Hive 0.8) ignores a <tt>hive-default.xml</tt> + file. As a result, Oozie (as of Oozie 3.4) ignores the <tt>oozie.hive.defaults</tt> + +property that was previously required by earlier versions of Oozie for the Hive action.</p> +<a name="Hive_Action_Logging"></a> +</div> +<div class="section"><h4>Hive Action Logging</h4> +<p>Hive action logs are redirected to the Oozie Launcher map-reduce job task STDOUT/STDERR that runs Hive.</p> +<p>From Oozie web-console, from the Hive action pop up using the 'Console URL' link, it is possible +to navigate to the Oozie Launcher map-reduce job task logs via the Hadoop job-tracker web-console.</p> +<p>The logging level of the Hive action can set in the Hive action configuration using the +property <tt>oozie.hive.log.level</tt> +. The default value is <tt>INFO</tt> +.</p> +<a name="Appendix_Hive_XML-Schema"></a> +</div> +</div> +<div class="section"><h3>Appendix, Hive XML-Schema</h3> +<a name="AE.A_Appendix_A_Hive_XML-Schema"></a> +<div class="section"><h4>AE.A Appendix A, Hive XML-Schema</h4> +<a name="Hive_Action_Schema_Version_1.0"></a> +<div class="section"><h5>Hive Action Schema Version 1.0</h5> +<p><pre> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:hive="uri:oozie:hive-action:1.0" + elementFormDefault="qualified" + targetNamespace="uri:oozie:hive-action:1.0"> +. + <xs:include schemaLocation="oozie-common-1.0.xsd"/> +. + <xs:element name="hive" type="hive:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:choice> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:choice> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="launcher" type="hive:LAUNCHER" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:choice minOccurs="1" maxOccurs="1"> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="query" type="xs:string" minOccurs="1" maxOccurs="1"/> + </xs:choice> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. +</xs:schema> +</pre></p> +<a name="Hive_Action_Schema_Version_0.6"></a> +</div> +<div class="section"><h5>Hive Action Schema Version 0.6</h5> +<p><pre> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:hive="uri:oozie:hive-action:0.6" elementFormDefault="qualified" + targetNamespace="uri:oozie:hive-action:0.6"> +. + <xs:element name="hive" type="hive:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:choice minOccurs="1" maxOccurs="1"> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="query" type="xs:string" minOccurs="1" maxOccurs="1"/> + </xs:choice> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. +</xs:schema> +</pre></p> +<a name="Hive_Action_Schema_Version_0.5"></a> +</div> +<div class="section"><h5>Hive Action Schema Version 0.5</h5> +<p><pre> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:hive="uri:oozie:hive-action:0.5" elementFormDefault="qualified" + targetNamespace="uri:oozie:hive-action:0.5"> +. + <xs:element name="hive" type="hive:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. +</xs:schema> +</pre></p> +<a name="Hive_Action_Schema_Version_0.4"></a> +</div> +<div class="section"><h5>Hive Action Schema Version 0.4</h5> +<p><pre> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:hive="uri:oozie:hive-action:0.4" elementFormDefault="qualified" + targetNamespace="uri:oozie:hive-action:0.4"> +. + <xs:element name="hive" type="hive:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. +</xs:schema> +</pre></p> +<a name="Hive_Action_Schema_Version_0.3"></a> +</div> +<div class="section"><h5>Hive Action Schema Version 0.3</h5> +<p><pre> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:hive="uri:oozie:hive-action:0.3" elementFormDefault="qualified" + targetNamespace="uri:oozie:hive-action:0.3"> +. + <xs:element name="hive" type="hive:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. +</xs:schema> +</pre></p> +<a name="Hive_Action_Schema_Version_0.2"></a> +</div> +<div class="section"><h5>Hive Action Schema Version 0.2</h5> +<p><pre> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:hive="uri:oozie:hive-action:0.2" elementFormDefault="qualified" + targetNamespace="uri:oozie:hive-action:0.2"> +. + <xs:element name="hive" type="hive:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. +</xs:schema> +</pre></p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<p></p> +</div> +</div> +</div> + + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html> Added: websites/staging/oozie/trunk/content/docs/5.0.0/DG_JMSNotifications.html ============================================================================== --- websites/staging/oozie/trunk/content/docs/5.0.0/DG_JMSNotifications.html (added) +++ websites/staging/oozie/trunk/content/docs/5.0.0/DG_JMSNotifications.html Mon Apr 9 14:26:49 2018 @@ -0,0 +1,345 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at Apr 9, 2018 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie - </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.0.0"> + 5.0.0</a> + <span class="divider">/</span> + </li> + <li class="active ">Oozie - </li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-04-09</li> + <li id="projectVersion" class="pull-right"> + Version: 5.0.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p></p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<a name="JMS_Notifications"></a> +<div class="section"><h2> JMS Notifications</h2> +<p><ul><ul><li><a href="#Overview">Overview</a> +</li> +<li><a href="#Oozie_Server_Configuration">Oozie Server Configuration</a> +</li> +<li><a href="#Consuming_Notifications">Consuming Notifications</a> +<ul><li><a href="#Notification_types">Notification types</a> +</li> +<li><a href="#JMS_Topic">JMS Topic</a> +</li> +<li><a href="#JMS_Message_Format">JMS Message Format</a> +</li> +<li><a href="#JMS_Client">JMS Client</a> +<ul><li><a href="#Example">Example</a> +</li> +<li><a href="#Applying_Selectors">Applying Selectors</a> +</li> +</ul> +</li> +</ul> +</li> +</ul> +</ul> +</p> +<a name="Overview"></a> +<div class="section"><h3>Overview</h3> +<p>Since Oozie 4.0, Oozie supports publishing notifications to a JMS Provider for job status changes and SLA met and miss +events. This provides an alternative to polling Oozie for Job or SLA related information and getting events as they +happen without any delay. Clients can be written to consume these notifications and integrate with different monitoring +and alerting systems.</p> +<a name="Oozie_Server_Configuration"></a> +</div> +<div class="section"><h3>Oozie Server Configuration</h3> +<p>Refer to <a href="./AG_Install.html#Notifications_Configuration">Notifications Configuration</a> + section of <a href="./AG_Install.html">Oozie Install</a> + +documentation for the Oozie server side configuration required to support publishing notifications to a JMS Provider. +The JNDI properties for the JMS provider, the topics to publish to and the notification types to publish (Job and/or SLA) +need to be configured.</p> +<a name="Consuming_Notifications"></a> +</div> +<div class="section"><h3>Consuming Notifications</h3> +<a name="Notification_types"></a> +<div class="section"><h4>Notification types</h4> +<p>Job and SLA notifications are published to the configured JMS Provider on the configured topics.</p> +<p>Job status change notifications include job start, success, failure, suspended, etc. Currently only workflow job and +coordinator action status change notifications are published.</p> +<p>SLA notifications include START_MET, END_MET, DURATION_MET, START_MISS, END_MISS, DURATION_MISS events and are published +for a workflow job, workflow action or coordinator action for which SLA information is configured in the job xml. Refer +to <a href="./DG_SLAMonitoring.html#Configuring_SLA_in_Applications">SLA Configuration</a> + for information on configuring SLA for a workflow or +coordinator.</p> +<a name="JMS_Topic"></a> +</div> +<div class="section"><h4>JMS Topic</h4> +<p>Consumers interested in notification on events will require to know the JNDI properties to connect to the JMS provider. +They will also need to know the JMS topic on which notifications for a particular job are published.</p> +<p>Oozie Client provides the following APIs : +<pre> +public JMSConnectionInfo getJMSConnectionInfo() +public String getJMSTopicName(String jobId) +</pre></p> +<p>The JMSConnectionInfo exposes 3 methods:</p> +<p><pre> +Properties getJNDIProperties(); +String getTopicPattern(AppType appType); +String getTopicPrefix(); +</pre></p> +<p>The topic is obtained by concatenating topic prefix and the substituted value for topic pattern. The topic pattern +can be a constant value like workflow or coordinator which the administrator has configured or a variable (either ${username} +or ${jobId}). If ${username}, it has to be substituted with the name of the user who has submitted the job; and if ${jobId} it has +to be substituted with the job Id. Administrators can chose to publish messages to topics containing user names to avoid having one +topic containing all messages and all users having to apply selectors to filter the message they are interested in.</p> +<p>The getJMSTopicName API can be used if the job id is already known and will give the exact topic name to which the +notifications for that job are published.</p> +<a name="JMS_Message_Format"></a> +</div> +<div class="section"><h4>JMS Message Format</h4> +<p>JMS messages published are <tt>javax.jms.TextMessage</tt> +. The body contains JSON and the header contains multiple properties +that can be used as selectors. The header properties are not repeated in the body of the message to keep the messages +small.</p> +<p><b>Message Header:</b> <br/> +The different header properties are:<ul><li>msgType - Value can be JOB or SLA.</li> +<li>user - The user who submitted the job.</li> +<li>appName - Application name of the job.</li> +<li>appType - Type of the job. One of WORKFLOW_JOB, WORKFLOW_ACTION, COORDINATOR_ACTION</li> +<li>slaStatus - Applicable only to SLA messages. Value is one of NOT_STARTED, IN_PROCESS, MET, MISS</li> +<li>eventStatus - It takes one of the following values- START_MET, START_MISS, DURATION_MET, DURATION_MISS, END_MET, END_MISS for SLA notifications. It can take any of the following values- WAITING, STARTED, SUCCESS, SUSPEND, FAILURE for job notifications. Note that event status is different from the Job status. It is included in the header to provide better filtering. Below is the mapping of event status to the actual status of workflow job or coordinator action.</li> +</ul> +</p> +<p>WAITING = When a Coordinator Action is waiting for input dependency</p> +<p>STARTED = When the Workflow Job or Coordinator Action is in RUNNING state</p> +<p>SUCCESS = When the Workflow Job or Coordinator Action is in SUCCEEDED state</p> +<p>SUSPEND = When the Workflow Job or Coordinator Action is in SUSPENDED state</p> +<p>FAILURE = When the Workflow Job or Coordinator Action is in terminal state other than SUCCEEDED.</p> +<p><b>Message Body for Job Notifications:</b> <br/> + Sample JSON response for different job and sla events as below.</p> +<p><pre> +Workflow Job in RUNNING state: +{"status":"RUNNING","id":"0000042-130618221729631-oozie-oozi-W","startTime":1342915200000} +</pre></p> +<p><pre> +Workflow Job in FAILED state: +{"status":"FAILED","errorCode":"EL_ERROR","errorMessage":"variable [dummyvalue] cannot be resolved", +"id":"0000042-130618221729631-oozie-oozi-W","startTime":1342915200000,"endTime":1366672183543} +</pre></p> +<p><pre> +Workflow Job in SUCCEEDED state: +{"status":"SUCCEEDED","id":"0000039-130618221729631-oozie-oozi-W","startTime":1342915200000, +"parentId":"0000025-130618221729631-oozie-oozi-C@1","endTime":1366676224154} +</pre></p> +<p><pre> +Workflow Job in SUSPENDED state: +{"status":"SUSPENDED","id":"0000039-130618221729631-oozie-oozi-W","startTime":1342915200000, +"parentId":"0000025-130618221729631-oozie-oozi-C@1"} +</pre></p> +<p><pre> +Coordinator Action in WAITING state: +{"status":"WAITING","nominalTime":1310342400000,"missingDependency":"hdfs://gsbl90107.blue.com:8020/user/john/dir1/file1", +"id":"0000025-130618221729631-oozie-oozi-C@1","startTime":1342915200000,"parentId":"0000025-130618221729631-oozie-oozi-C"} +</pre></p> +<p><pre> +Coordinator Action in RUNNING state: +{"status":"RUNNING","nominalTime":1310342400000,"id":"0000025-130618221729631-oozie-oozi-C@1", +"startTime":1342915200000,"parentId":"0000025-130618221729631-oozie-oozi-C"} +</pre></p> +<p><pre> +Coordinator Action in SUCCEEDED state: +{"status":"SUCCEEDED","nominalTime":1310342400000,"id":"0000025-130618221729631-oozie-oozi-C@1", +"startTime":1342915200000,"parentId":"0000025-130618221729631-oozie-oozi-C","endTime":1366677082799} +</pre></p> +<p><pre> +Coordinator Action in FAILED state: +{"status":"FAILED","errorCode":"E0101","errorMessage":"dummyError","nominalTime":1310342400000, +"id":"0000025-130618221729631-oozie-oozi-C@1","startTime":1342915200000, +"parentId":"0000025-130618221729631-oozie-oozi-C","endTime":1366677140818} +</pre></p> +<p><b>Message Body for SLA Notifications:</b> <br/></p> +<p><pre> +Workflow Job in sla END_MISS state: +{"id":"0000000-000000000000001-oozie-wrkf-C@1","parentId":"0000000-000000000000001-oozie-wrkf-C", +"expectedStartTime":1356998400000,"notificationMessage":"notification of start miss","actualStartTime":1357002000000, +"expectedDuration":-1, "actualDuration":3600,"expectedEndTime":1356998400000,"actualEndTime":1357002000000} +</pre></p> +<a name="JMS_Client"></a> +</div> +<div class="section"><h4>JMS Client</h4> +<p>Oozie provides a helper class JMSMessagingUtils for consumers to deserialize the JMS messages back to Java objects. +The below method getEventMessage() expects a sub type of EventMessage. +There are different implementations of EventMessage - WorkflowJobMessage, CoordinatorActionMessage and SLAMessage.</p> +<p><pre> +<T extends EventMessage> T JMSMessagingUtils.getEventMessage(Message jmsMessage) +</pre></p> +<a name="Example"></a> +<div class="section"><h5>Example</h5> +<p>Below is a sample code to consume notifications.</p> +<p>First, create the Oozie client and retrieve the JNDI properties to make a connection to the JMS server. +<pre> + OozieClient oc = new OozieClient("http://localhost:11000/oozie"); + JMSConnectionInfo jmsInfo = oc.getJMSConnectionInfo(); + Properties jndiProperties = jmsInfo.getJNDIProperties(); + Context jndiContext = new InitialContext(jndiProperties); + String connectionFactoryName = (String) jndiContext.getEnvironment().get("connectionFactoryNames"); + ConnectionFactory connectionFactory = (ConnectionFactory) jndiContext.lookup(connectionFactoryName); + Connection connection = connectionFactory.createConnection(); + Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE); + String topicPrefix = jmsInfo.getTopicPrefix(); + String topicPattern = jmsInfo.getTopicPattern(AppType.WORKFLOW_JOB); + // Following code checks if the topic pattern is + //'username', then the topic name is set to the actual user submitting the job + String topicName = null; + if (topicPattern.equals("${username}")) { + topicName = "john"; + // Following code checks if the topic pattern is + //'jobId', then the topic name is set to the job id + } else if (topicPattern.equals("${jobId}")) { + topicName = "0000004-140328125200198-oozie-oozi-W"; + } + Destination topic = session.createTopic(topicPrefix+topicName); + MessageConsumer consumer = session.createConsumer(topic); + consumer.setMessageListener(this); + connection.start(); +</pre></p> +<p>To start receiving messages, the JMS <a class="externalLink" href="http://docs.oracle.com/javaee/6/api/javax/jms/MessageListener.html";>MessageListener</a> + +interface needs to be implemented. Also, its onMessage() method needs to be implemented. +This method will be called whenever a message is available on the JMS bus.</p> +<p><pre> + public void onMessage(Message message) { + if (message.getStringProperty(JMSHeaderConstants.MESSAGE_TYPE).equals(MessageType.SLA.name())){ + SLAMessage slaMessage = JMSMessagingUtils.getEventMessage(message); + // Further processing + } + else if (message.getStringProperty(JMSHeaderConstants.APP_TYPE).equals(AppType.WORKFLOW_JOB.name())){ + WorkflowJobMessage wfJobMessage = JMSMessagingUtils.getEventMessage(message); + // Further processing + } + } +</pre></p> +<a name="Applying_Selectors"></a> +</div> +<div class="section"><h5>Applying Selectors</h5> +<p>Below is a sample ActiveMQ text message header properties section. +<pre> +ActiveMQTextMessage +{properties <tt> {appName = map-reduce-wf, msgType</tt> +JOB, appType=WORKFLOW_JOB, user=john, msgFormat=json, eventStatus=STARTED} ...} +</pre></p> +<p>On the header properties, consumers can apply JMS selectors to filter messages from JMS provider. +They are listed at <a href="../docs/client/apidocs/org/apache/oozie/client/event/jms/JMSHeaderConstants.html">JMSHeaderConstants</a> +</p> +<p>Sample use of selector to filter events related to Job which have failed and has a particular app-name</p> +<p><pre> +String selector=JMSHeaderConstants.EVENT_STATUS + "='FAILURE' AND " + JMSHeaderConstants.APP_NAME + "='app-name'"; +MessageConsumer consumer = session.createConsumer(topic, selector); +</pre></p> +</div> +</div> +</div> + + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html> Added: websites/staging/oozie/trunk/content/docs/5.0.0/DG_Overview.html ============================================================================== --- websites/staging/oozie/trunk/content/docs/5.0.0/DG_Overview.html (added) +++ websites/staging/oozie/trunk/content/docs/5.0.0/DG_Overview.html Mon Apr 9 14:26:49 2018 @@ -0,0 +1,201 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at Apr 9, 2018 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie - </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.0.0"> + 5.0.0</a> + <span class="divider">/</span> + </li> + <li class="active ">Oozie - </li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-04-09</li> + <li id="projectVersion" class="pull-right"> + Version: 5.0.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p></p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<a name="Oozie_Workflow_Overview"></a> +<div class="section"><h2>Oozie Workflow Overview</h2> +<p>Oozie is a server based <i>Workflow Engine</i> + specialized in running workflow jobs with actions that run Hadoop Map/Reduce +and Pig jobs.</p> +<p>Oozie is a Java Web-Application that runs in a Java servlet-container.</p> +<p>For the purposes of Oozie, a workflow is a collection of actions (i.e. Hadoop Map/Reduce jobs, Pig jobs) arranged in +a control dependency DAG (Direct Acyclic Graph). "control dependency" from one action to another means that the second +action can't run until the first action has completed.</p> +<p>Oozie workflows definitions are written in hPDL (a XML Process Definition Language similar to +<a class="externalLink" href="http://www.jboss.org/jbossjbpm/";>JBOSS JBPM</a> + jPDL).</p> +<p>Oozie workflow actions start jobs in remote systems (i.e. Hadoop, Pig). Upon action completion, the remote systems +callback Oozie to notify the action completion, at this point Oozie proceeds to the next action in the workflow.</p> +<p>Oozie workflows contain control flow nodes and action nodes.</p> +<p>Control flow nodes define the beginning and the end of a workflow ( <tt>start</tt> +, <tt>end</tt> + and <tt>fail</tt> + nodes) and provide a +mechanism to control the workflow execution path ( <tt>decision</tt> +, <tt>fork</tt> + and <tt>join</tt> + nodes).</p> +<p>Action nodes are the mechanism by which a workflow triggers the execution of a computation/processing task. Oozie +provides support for different types of actions: Hadoop map-reduce, Hadoop file system, Pig, SSH, HTTP, eMail and +Oozie sub-workflow. Oozie can be extended to support additional type of actions.</p> +<p>Oozie workflows can be parameterized (using variables like <tt>${inputDir}</tt> + within the workflow definition). When +submitting a workflow job values for the parameters must be provided. If properly parameterized (i.e. using different +output directories) several identical workflow jobs can concurrently.</p> +<a name="WordCount_Workflow_Example"></a> +<div class="section"><h3>WordCount Workflow Example</h3> +<p><b>Workflow Diagram:</b> +</p> +<p><img src="./DG_Overview.png"/></p> +<p><b>hPDL Workflow Definition:</b> +</p> +<p><pre> +<workflow-app name='wordcount-wf' xmlns="uri:oozie:workflow:0.1"> + <start to='wordcount'/> + <action name='wordcount'> + <map-reduce> + <job-tracker>${jobTracker}</job-tracker> + <name-node>${nameNode}</name-node> + <configuration> + <property> + <name>mapred.mapper.class</name> + <value>org.myorg.WordCount.Map</value> + </property> + <property> + <name>mapred.reducer.class</name> + <value>org.myorg.WordCount.Reduce</value> + </property> + <property> + <name>mapred.input.dir</name> + <value>${inputDir}</value> + </property> + <property> + <name>mapred.output.dir</name> + <value>${outputDir}</value> + </property> + </configuration> + </map-reduce> + <ok to='end'/> + <error to='end'/> + </action> + <kill name='kill'> + <message>Something went wrong: ${wf:errorCode('wordcount')}</message> + </kill/> + <end name='end'/> +</workflow-app> +</pre></p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<p></p> +</div> +</div> + + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html> Added: websites/staging/oozie/trunk/content/docs/5.0.0/DG_Overview.png ============================================================================== Binary file - no diff available. Propchange: websites/staging/oozie/trunk/content/docs/5.0.0/DG_Overview.png ------------------------------------------------------------------------------ svn:mime-type = image/png
![https://oozie.apache.org/images/oozie_200x.png"](https://oozie.apache.org/images/oozie_200x.png"){kind=link}