Author: dionusos Date: Fri Feb 26 14:14:19 2021 New Revision: 1886952 URL: http://svn.apache.org/viewvc?rev=1886952&view=rev Log: 5.2.1 release
Added: oozie/site/trunk/content/resources/docs/5.2.1/ oozie/site/trunk/content/resources/docs/5.2.1/AG_ActionConfiguration.html oozie/site/trunk/content/resources/docs/5.2.1/AG_HadoopConfiguration.html oozie/site/trunk/content/resources/docs/5.2.1/AG_Install.html oozie/site/trunk/content/resources/docs/5.2.1/AG_Monitoring.html oozie/site/trunk/content/resources/docs/5.2.1/AG_OozieLogging.html oozie/site/trunk/content/resources/docs/5.2.1/AG_OozieUpgrade.html oozie/site/trunk/content/resources/docs/5.2.1/BundleFunctionalSpec.html oozie/site/trunk/content/resources/docs/5.2.1/CoordinatorFunctionalSpec.html oozie/site/trunk/content/resources/docs/5.2.1/DG_ActionAuthentication.html oozie/site/trunk/content/resources/docs/5.2.1/DG_CommandLineTool.html oozie/site/trunk/content/resources/docs/5.2.1/DG_CoordinatorRerun.html oozie/site/trunk/content/resources/docs/5.2.1/DG_CustomActionExecutor.html oozie/site/trunk/content/resources/docs/5.2.1/DG_DistCpActionExtension.html oozie/site/trunk/content/resources/docs/5.2.1/DG_EmailActionExtension.html oozie/site/trunk/content/resources/docs/5.2.1/DG_Examples.html oozie/site/trunk/content/resources/docs/5.2.1/DG_FluentJobAPI.html oozie/site/trunk/content/resources/docs/5.2.1/DG_GitActionExtension.html oozie/site/trunk/content/resources/docs/5.2.1/DG_HCatalogIntegration.html oozie/site/trunk/content/resources/docs/5.2.1/DG_Hive2ActionExtension.html oozie/site/trunk/content/resources/docs/5.2.1/DG_HiveActionExtension.html oozie/site/trunk/content/resources/docs/5.2.1/DG_JMSNotifications.html oozie/site/trunk/content/resources/docs/5.2.1/DG_Overview.html oozie/site/trunk/content/resources/docs/5.2.1/DG_Overview.png (with props) oozie/site/trunk/content/resources/docs/5.2.1/DG_QuickStart.html oozie/site/trunk/content/resources/docs/5.2.1/DG_SLAMonitoring.html oozie/site/trunk/content/resources/docs/5.2.1/DG_ShellActionExtension.html oozie/site/trunk/content/resources/docs/5.2.1/DG_SparkActionExtension.html oozie/site/trunk/content/resources/docs/5.2.1/DG_SqoopActionExtension.html oozie/site/trunk/content/resources/docs/5.2.1/DG_SshActionExtension.html oozie/site/trunk/content/resources/docs/5.2.1/DG_WorkflowReRun.html oozie/site/trunk/content/resources/docs/5.2.1/ENG_Building.html oozie/site/trunk/content/resources/docs/5.2.1/ENG_Custom_Authentication.html oozie/site/trunk/content/resources/docs/5.2.1/ENG_MiniOozie.html oozie/site/trunk/content/resources/docs/5.2.1/WebServicesAPI.html oozie/site/trunk/content/resources/docs/5.2.1/WorkflowFunctionalSpec.html oozie/site/trunk/content/resources/docs/5.2.1/css/ oozie/site/trunk/content/resources/docs/5.2.1/css/apache-maven-fluido-1.4.min.css oozie/site/trunk/content/resources/docs/5.2.1/css/print.css oozie/site/trunk/content/resources/docs/5.2.1/css/site.css oozie/site/trunk/content/resources/docs/5.2.1/fonts/ oozie/site/trunk/content/resources/docs/5.2.1/fonts/glyphicons-halflings-regular.eot (with props) oozie/site/trunk/content/resources/docs/5.2.1/fonts/glyphicons-halflings-regular.svg oozie/site/trunk/content/resources/docs/5.2.1/fonts/glyphicons-halflings-regular.ttf (with props) oozie/site/trunk/content/resources/docs/5.2.1/fonts/glyphicons-halflings-regular.woff (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/ oozie/site/trunk/content/resources/docs/5.2.1/images/accessories-text-editor.png (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/add.gif (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/apache-maven-project-2.png (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/application-certificate.png (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/contact-new.png (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/document-properties.png (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/drive-harddisk.png (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/fix.gif (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/icon_error_sml.gif (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/icon_help_sml.gif (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/icon_info_sml.gif (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/icon_success_sml.gif (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/icon_warning_sml.gif (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/image-x-generic.png (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/internet-web-browser.png (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/logos/ oozie/site/trunk/content/resources/docs/5.2.1/images/logos/build-by-maven-black.png (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/logos/build-by-maven-white.png (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/logos/maven-feather.png (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/network-server.png (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/package-x-generic.png (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/profiles/ oozie/site/trunk/content/resources/docs/5.2.1/images/profiles/pre-release.png (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/profiles/retired.png (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/profiles/sandbox.png (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/remove.gif (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/rss.png (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/update.gif (with props) oozie/site/trunk/content/resources/docs/5.2.1/images/window-new.png (with props) oozie/site/trunk/content/resources/docs/5.2.1/img/ oozie/site/trunk/content/resources/docs/5.2.1/img/glyphicons-halflings-white.png (with props) oozie/site/trunk/content/resources/docs/5.2.1/img/glyphicons-halflings.png (with props) oozie/site/trunk/content/resources/docs/5.2.1/index.html oozie/site/trunk/content/resources/docs/5.2.1/js/ oozie/site/trunk/content/resources/docs/5.2.1/js/apache-maven-fluido-1.4.min.js oozie/site/trunk/content/resources/docs/5.2.1/oozie-default.xml oozie/site/trunk/content/resources/docs/5.2.1/release-log.txt Modified: oozie/site/trunk/content/site.xml oozie/site/trunk/doap_Oozie.rdf Added: oozie/site/trunk/content/resources/docs/5.2.1/AG_ActionConfiguration.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.2.1/AG_ActionConfiguration.html?rev=1886952&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.2.1/AG_ActionConfiguration.html (added) +++ oozie/site/trunk/content/resources/docs/5.2.1/AG_ActionConfiguration.html Fri Feb 26 14:14:19 2021 @@ -0,0 +1,182 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2021-02-26 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20210226" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie – </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/" id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png" alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org/" class="externalLink" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.2.1"> + 5.2.1</a> + <span class="divider">/</span> + </li> + <li class="active "></li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2021-02-26</li> + <li id="projectVersion" class="pull-right"> + Version: 5.2.1 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p><a href="index.html">::Go back to Oozie Documentation Index::</a></p> +<h1>Action Configuration</h1> +<p>Oozie supports providing default configuration for actions of a particular action type and default configuration for all actions</p> +<div class="section"> +<h2><a name="Hadoop_Default_Configuration_Values"></a>Hadoop Default Configuration Values</h2> +<p>Oozie supports action configuration equivalent to the component’s <tt>*-site.xml</tt> and <tt>*.properties</tt> files.</p> +<p>The configuration property in the <tt>oozie-site.xml</tt> is <tt>oozie.service.HadoopAccessorService.action.configurations</tt> and its value must follow the pattern <i>[AUTHORITY=ACTION-CONF-DIR</i>,]*. Where <i>AUTHORITY</i> is the <i>HOST:PORT</i> of the Hadoop service (JobTracker/ResourceManager or HDFS). The <i>ACTION-CONF-DIR</i> is the action configuration directory. If the specified directory is a relative path, it will be looked under the Oozie configuration directory. An absolute path can also be specified. Oozie will load and process the action configuration files in the following order.</p> +<ol style="list-style-type: decimal"> + +<li>All files in <i>default</i>/*.xml (sorted by lexical name, files with names lexically lower have lesser precedence than the following ones), if present.</li> +<li><i>default</i>.xml, if present.</li> +<li>All supported files in <i>actionname</i>/*, e.g. <i>actionname</i>/*.xml and <i>actionname</i>/*.properties (based on filename extension, sorted by lexical name, files with names lexically lower have lesser precedence than the following ones), if present.</li> +<li><i>actionname</i>.xml, if present.</li> +</ol> +<p>For example, for <i>Hive</i> action (which has the <i>actionname</i> defined as <i>hive</i> ), the list of files (under relevant <i>ACTION-CONF-DIR</i> ) processed would be,</p> +<ol style="list-style-type: decimal"> + +<li>All files in <i>default</i>/*.xml, if present</li> +<li><i>default</i>.xml, if present.</li> +<li>All files in <i>hive</i>/*.xml and <i>hive</i>/*.properties, if present</li> +<li><i>hive</i>.xml, if present.</li> +</ol> +<p>Files processed earlier for an action have the lowest precedence and can have the configuration parameters redefined. All files and directories are relative to the <i>ACTION-CONF-DIR</i> directory.</p> +<p>In addition to explicit authorities, a ‘*’ wildcard is supported. The configuration file associated with the wildcard will be used as default if there is no action configuration for the requested Hadoop service.</p> +<p>For example, the configuration in the <tt>oozie-site.xml</tt> would look like:</p> + +<div> +<div> +<pre class="source">... + <property> + <name>oozie.service.HadoopAccessorService.action.configurations</name> + <value>*=hadoop-conf,jt-bar:8021=bar-cluster,nn-bar:8020=bar-cluster</value> + </property> +... +</pre></div></div> + +<p>The action configuration files use the Hadoop configuration syntax.</p> +<p>By default Oozie does not define any default action configurations.</p></div> +<div class="section"> +<h2><a name="Dependency_deduplication"></a>Dependency deduplication</h2> +<p>Using Oozie with Hadoop 3 may require to have dependency file names distinguishable, which means having two files on sharelib and in your app’s dependencies with identical names, leads to job submission failure. To avoid this you can enable the deduplicator by setting oozie.action.dependency.deduplicate=true in oozie-site.xml (false, by default). Dependencies which are closer to your application has higher priority: action jar > user workflow libs > action libs > system lib, where dependency with greater prio is used.</p> +<p>Real world example: You have an application workflow which is uploaded to HDFS in /apps/app directory. You have your app.jar and dependency jars. You also define a spark action in your workflow and set use system libs; the HDFS tree is similar to this:</p> + +<div> +<div> +<pre class="source"> + /apps/app/ + - app.jar + - workflow.xml + + libs + - app.jar + - jackson-annotations-1.0.jar + + share/lib/ + + spark + - app.jar + - jackson-annotations-1.0.jar + + oozie + - jackson-annotations-1.0.jar +</pre></div></div> + +<p>The deduplicator code will create the following list of files: <tt>/apps/app/app.jar,/apps/app/libs/jackson-annotations-1.0.jar</tt> And no other files will be passed at job submission.</p> +<p><a href="index.html">::Go back to Oozie Documentation Index::</a></p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2021 + <a href="https://www.apache.org">Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html> Added: oozie/site/trunk/content/resources/docs/5.2.1/AG_HadoopConfiguration.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.2.1/AG_HadoopConfiguration.html?rev=1886952&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.2.1/AG_HadoopConfiguration.html (added) +++ oozie/site/trunk/content/resources/docs/5.2.1/AG_HadoopConfiguration.html Fri Feb 26 14:14:19 2021 @@ -0,0 +1,208 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2021-02-26 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20210226" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie – </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/" id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png" alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org/" class="externalLink" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.2.1"> + 5.2.1</a> + <span class="divider">/</span> + </li> + <li class="active "></li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2021-02-26</li> + <li id="projectVersion" class="pull-right"> + Version: 5.2.1 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p><a href="index.html">::Go back to Oozie Documentation Index::</a></p> +<h1>Hadoop Configuration</h1> +<div class="section"> +<h2><a name="Hadoop_Services_Whitelisting"></a>Hadoop Services Whitelisting</h2> +<p>Oozie supports whitelisting Hadoop services (JobTracker, HDFS), via 2 configuration properties:</p> + +<div> +<div> +<pre class="source">... + <property> + <name>oozie.service.HadoopAccessorService.jobTracker.whitelist</name> + <value> </value> + <description> + Whitelisted job tracker for Oozie service. + </description> + </property> + <property> + <name>oozie.service.HadoopAccessorService.nameNode.whitelist</name> + <value> </value> + <description> + Whitelisted job tracker for Oozie service. + </description> + </property> +... +</pre></div></div> + +<p>The value must follow the pattern <tt>[AUTHORITY,...]</tt>. Where <tt>AUTHORITY</tt> is the <tt>HOST:PORT</tt> of the Hadoop service (JobTracker, HDFS).</p> +<p>If the value is empty any HOST:PORT is accepted. Empty is the default value.</p></div> +<div class="section"> +<h2><a name="Hadoop_Default_Configuration_Values"></a>Hadoop Default Configuration Values</h2> +<p>Oozie supports Hadoop configuration equivalent to the Hadoop <tt>*-site.xml</tt> files.</p> +<p>The configuration property in the <tt>oozie-site.xml</tt> is <tt>oozie.service.HadoopAccessorService.hadoop.configurations</tt> and its value must follow the pattern <tt>[<AUTHORITY>=<HADOOP_CONF_DIR>,]*</tt>. Where <tt><AUTHORITY></tt> is the <tt>HOST:PORT</tt> of the Hadoop service (JobTracker, HDFS). The <tt><HADOOP_CONF_DIR></tt> is a Hadoop configuration directory. If the specified directory is a relative path, it will be looked under the Oozie configuration directory. And absolute path can also be specified. Oozie will load the Hadoop <tt>*-site.xml</tt> files in the following order: core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml, hadoop-site.xml, ssl-client.xml.</p> +<p>In addition to explicit authorities, a ‘*’ wildcard is supported. The configuration file associated with the wildcard will be used as default if there is no configuration for the requested Hadoop service.</p> +<p>For example, the configuration in the <tt>oozie-site.xml</tt> would look like:</p> + +<div> +<div> +<pre class="source">... + <property> + <name>oozie.service.HadoopAccessorService.hadoop.configurations</name> + <value>*=hadoop-conf,jt-bar:8021=bar-cluster,nn-bar:8020=bar-cluster</value> + </property> +... +</pre></div></div> + +<p>The Hadoop configuration files use the Hadoop configuration syntax.</p> +<p>By default Oozie defines <tt>*=hadoop-conf</tt> and the default values of the <tt>hadoop-site.xml</tt> file are:</p> + +<div> +<div> +<pre class="source"><configuration> + <property> + <name>mapreduce.jobtracker.kerberos.principal</name> + <value>mapred/_HOST@LOCALREALM</value> + </property> + <property> + <name>yarn.resourcemanager.principal</name> + <value>yarn/_HOST@LOCALREALM</value> + </property> + <property> + <name>dfs.namenode.kerberos.principal</name> + <value>hdfs/_HOST@LOCALREALM</value> + </property> + <property> + <name>mapreduce.framework.name</name> + <value>yarn</value> + </property> +</configuration> +</pre></div></div> +</div> +<div class="section"> +<h2><a name="File_system_custom_properties"></a>File system custom properties</h2> +<p>Some users notified us about issues when they started using Amazon S3A file system - see <a class="externalLink" href="https://issues.apache.org/jira/browse/OOZIE-3529">OOZIE-3529</a>. Oozie from version 5.2.0 supports custom file system properties which can be defined in the following way. The example shows how to resolve issues mentioned in OOZIE-3529 by setting the following in oozie-site.xml:</p> + +<div> +<div> +<pre class="source"> <property> + <name>oozie.service.HadoopAccessorService.fs.s3a</name> + <value>fs.s3a.fast.upload.buffer=bytebuffer,fs.s3a.impl.disable.cache=true</value> + </property> +</pre></div></div> + +<p>Use <tt>oozie.service.HadoopAccessorService.fs.%s</tt> where <tt>%s</tt> is the schema of the file system. The value shall be a list of key=value pairs separated using a comma (,). You can use properties as describe below: * <tt>property_name=property_value_1=property_value2</tt> will be read as: * name: property_name * value: property_value_1=property_value2 * <tt>property1_name=value1,property2_name</tt> * name: property1_name * value: value1 * property2_name will be ignored * Limitation: the custom file system properties cannot contain comma neither in key nor in value. See <a class="externalLink" href="https://issues.apache.org/jira/browse/OOZIE-3547">OOZIE-3547</a>.</p></div> +<div class="section"> +<h2><a name="Limitations"></a>Limitations</h2> +<p>All actions in a workflow application must interact with the same Hadoop JobTracker and NameNode.</p> +<p><a href="index.html">::Go back to Oozie Documentation Index::</a></p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2021 + <a href="https://www.apache.org">Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html>