Added: oozie/site/trunk/content/resources/docs/5.0.0/AG_Monitoring.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.0.0/AG_Monitoring.html?rev=1828722&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.0.0/AG_Monitoring.html (added) +++ oozie/site/trunk/content/resources/docs/5.0.0/AG_Monitoring.html Mon Apr 9 14:12:36 2018 @@ -0,0 +1,380 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at Apr 9, 2018 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie - </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.0.0"> + 5.0.0</a> + <span class="divider">/</span> + </li> + <li class="active ">Oozie - </li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-04-09</li> + <li id="projectVersion" class="pull-right"> + Version: 5.0.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p></p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<a name="Oozie_Monitoring"></a> +<div class="section"><h2> Oozie Monitoring</h2> +<p><ul><ul><li><a href="#Oozie_Instrumentation">Oozie Instrumentation</a> +<ul><li><a href="#Variables">Variables</a> +</li> +<li><a href="#Samplers_-_Poll_data_at_a_fixed_interval_default_1_sec_and_report_an_average_utilization_over_a_longer_period_of_time_default_60_seconds.">Samplers - Poll data at a fixed interval (default 1 sec) and report an average utilization over a longer period of time (default 60 seconds).</a> +</li> +<li><a href="#Counters_-_Maintain_statistics_about_the_number_of_times_an_event_has_occurred_for_the_running_Oozie_instance._The_values_are_reset_if_the_Oozie_instance_is_restarted.">Counters - Maintain statistics about the number of times an event has occurred, for the running Oozie instance. The values are reset if the Oozie instance is restarted.</a> +</li> +<li><a href="#Timers_-_Maintain_information_about_the_time_spent_in_various_operations.">Timers - Maintain information about the time spent in various operations.</a> +</li> +</ul> +</li> +<li><a href="#Oozie_JVM_Thread_Dump">Oozie JVM Thread Dump</a> +</li> +<li><a href="#Monitoring_Database_Schema_Integrity">Monitoring Database Schema Integrity</a> +</li> +</ul> +</ul> +</p> +<a name="Oozie_Instrumentation"></a> +<div class="section"><h3>Oozie Instrumentation</h3> +<p>Oozie code is instrumented in several places to collect runtime metrics. The instrumentation data can be used to +determine the health of the system, performance of the system, and to tune the system.</p> +<p>This comes in two flavors:<ul><li>metrics (by default enabled since 5.0.0)</li> +<li>instrumentation (deprecated and by default disabled since 5.0.0)</li> +</ul> +</p> +<p>The instrumentation is accessible via the Admin web-services API (see the <a href="./WebServicesAPI.html#Oozie_Metrics">metrics</a> + and +<a href="./WebServicesAPI.html#Oozie_Instrumentation">instrumentation</a> + Web Services API documentations for more details) and is also written on +regular intervals to an instrumentation log.</p> +<p>Instrumentation data includes variables, samplers, timers and counters.</p> +<a name="Variables"></a> +<div class="section"><h4>Variables</h4> +<p><ul><li>oozie<ul><li>version: Oozie build version.</li> +</ul> +</li> +</ul> +</p> +<p><ul><li>configuration<ul><li>config.dir: directory from where the configuration files are loaded. If null, all configuration files are loaded from the classpath. <a href="./AG_Install.html#Oozie_Configuration">Configuration files are described here</a> +.</li> +<li>config.file: the Oozie custom configuration for the instance.</li> +</ul> +</li> +</ul> +</p> +<p><ul><li>jvm<ul><li>free.memory</li> +<li>max.memory</li> +<li>total.memory</li> +</ul> +</li> +</ul> +</p> +<p><ul><li>locks<ul><li>locks: Locks are used by Oozie to synchronize access to workflow and action entries when the database being used does not support 'select for update' queries. (MySQL supports 'select for update').</li> +</ul> +</li> +</ul> +</p> +<p><ul><li>logging<ul><li>config.file: Log4j '.properties' configuration file.</li> +<li>from.classpath: whether the config file has been read from the classpath or from the config directory.</li> +<li>reload.interval: interval at which the config file will be reloaded. 0 if the config file will never be reloaded, when loaded from the classpath is never reloaded.</li> +</ul> +</li> +</ul> +</p> +<a name="Samplers_-_Poll_data_at_a_fixed_interval_default_1_sec_and_report_an_average_utilization_over_a_longer_period_of_time_default_60_seconds."></a> +</div> +<div class="section"><h4>Samplers - Poll data at a fixed interval (default 1 sec) and report an average utilization over a longer period of time (default 60 seconds).</h4> +<p>Poll for data over fixed interval and generate an average over the time interval. Unless specified, all samplers in +Oozie work on a 1 minute interval.</p> +<p><ul><li>callablequeue<ul><li>delayed.queue.size: The size of the delayed command queue.</li> +<li>queue.size: The size of the command queue.</li> +<li>threads.active: The number of threads processing callables.</li> +</ul> +</li> +</ul> +</p> +<p><ul><li>jdbc:<ul><li>connections.active: Active Connections over the past minute.</li> +</ul> +</li> +</ul> +</p> +<p><ul><li>webservices: Requests to the Oozie HTTP endpoints over the last minute.<ul><li>admin</li> +<li>callback</li> +<li>job</li> +<li>jobs</li> +<li>requests</li> +<li>version</li> +</ul> +</li> +</ul> +</p> +<a name="Counters_-_Maintain_statistics_about_the_number_of_times_an_event_has_occurred_for_the_running_Oozie_instance._The_values_are_reset_if_the_Oozie_instance_is_restarted."></a> +</div> +<div class="section"><h4>Counters - Maintain statistics about the number of times an event has occurred, for the running Oozie instance. The values are reset if the Oozie instance is restarted.</h4> +<p><ul><li>action.executors - Counters related to actions.<ul><li>[action_type]#action.[operation_performed] (start, end, check, kill)</li> +<li>[action_type]#ex.[exception_type] (transient, non-transient, error, failed)</li> +<li>e.g. <pre></li> +</ul> +</li> +</ul> + ssh#action.end: 306 + ssh#action.start: 316 </pre> +</p> +<p><ul><li>callablequeue - count of events in various execution queues.<ul><li>delayed.queued: Number of commands queued with a delay.</li> +<li>executed: Number of executions from the queue.</li> +<li>failed: Number of queue attempts which failed.</li> +<li>queued: Number of queued commands.</li> +</ul> +</li> +</ul> +</p> +<p><ul><li>commands: Execution Counts for various commands. This data is generated for all commands.<ul><li>action.end</li> +<li>action.notification</li> +<li>action.start</li> +<li>callback</li> +<li>job.info</li> +<li>job.notification</li> +<li>purge</li> +<li>signal</li> +<li>start</li> +<li>submit</li> +</ul> +</li> +</ul> +</p> +<p><ul><li>jobs: Job Statistics<ul><li>start: Number of started jobs.</li> +<li>submit: Number of submitted jobs.</li> +<li>succeeded: Number of jobs which succeeded.</li> +<li>kill: Number of killed jobs.</li> +</ul> +</li> +</ul> +</p> +<p><ul><li>authorization<ul><li>failed: Number of failed authorization attempts.</li> +</ul> +</li> +</ul> +</p> +<p><ul><li>webservices: Number of request to various web services along with the request type.<ul><li>failed: total number of failed requests.</li> +<li>requests: total number of requests.</li> +<li>admin</li> +<li>admin-GET</li> +<li>callback</li> +<li>callback-GET</li> +<li>jobs</li> +<li>jobs-GET</li> +<li>jobs-POST</li> +<li>version</li> +<li>version-GET</li> +</ul> +</li> +</ul> +</p> +<a name="Timers_-_Maintain_information_about_the_time_spent_in_various_operations."></a> +</div> +<div class="section"><h4>Timers - Maintain information about the time spent in various operations.</h4> +<p><ul><li>action.executors - Counters related to actions.<ul><li>[action_type]#action.[operation_performed] (start, end, check, kill)</li> +</ul> +</li> +</ul> +</p> +<p><ul><li>callablequeue<ul><li>time.in.queue: Time a callable spent in the queue before being processed.</li> +</ul> +</li> +</ul> +</p> +<p><ul><li>commands: Generated for all Commands.<ul><li>action.end</li> +<li>action.notification</li> +<li>action.start</li> +<li>callback</li> +<li>job.info</li> +<li>job.notification</li> +<li>purge</li> +<li>signal</li> +<li>start</li> +<li>submit</li> +</ul> +</li> +</ul> +</p> +<p><ul><li>db - Timers related to various database operations.<ul><li>create-workflow</li> +<li>load-action</li> +<li>load-pending-actions</li> +<li>load-running-actions</li> +<li>load-workflow</li> +<li>load-workflows</li> +<li>purge-old-workflows</li> +<li>save-action</li> +<li>update-action</li> +<li>update-workflow</li> +</ul> +</li> +</ul> +</p> +<p><ul><li>webservices<ul><li>admin</li> +<li>admin-GET</li> +<li>callback</li> +<li>callback-GET</li> +<li>jobs</li> +<li>jobs-GET</li> +<li>jobs-POST</li> +<li>version</li> +<li>version-GET</li> +</ul> +</li> +</ul> +</p> +<a name="Oozie_JVM_Thread_Dump"></a> +</div> +</div> +<div class="section"><h3>Oozie JVM Thread Dump</h3> +<p>The <tt>admin/jvminfo.jsp</tt> + servlet can be used to get some basic jvm stats and thread dump. +For eg: <a class="externalLink" href="http://localhost:11000/oozie/admin/jvminfo.jsp?cpuwatch=1000&threadsort=cpu.";>http://localhost:11000/oozie/admin/jvminfo.jsp?cpuwatch=1000&threadsort=cpu.</a> + It takes the following optional +query parameters:<ul><li>threadsort - The order in which the threads are sorted for display. Valid values are name, cpu, state. Default is state.</li> +<li>cpuwatch - Time interval in milliseconds to monitor cpu usage of threads. Default value is 0.</li> +</ul> +</p> +<a name="Monitoring_Database_Schema_Integrity"></a> +</div> +<div class="section"><h3>Monitoring Database Schema Integrity</h3> +<p>Oozie stores all of its state in a database. Hence, ensuring that the database schema is correct is very important to ensuring that +Oozie is healthy and behaves correctly. To help with this, Oozie includes a <tt>SchemaCheckerService</tt> + which periodically runs and +performs a series of checks on the database schema. More specifically, it checks the following:<ul><li>Existence of the required tables</li> +<li>Existence of the required columns in each table</li> +<li>Each column has the correct type and default value</li> +<li>Existence of the required primary keys and indexes</li> +</ul> +</p> +<p>After each run, the <tt>SchemaCheckerService</tt> + writes the result of the checks to the Oozie log and to the "schema-checker.status" +instrumentation variable. If there's a problem, it will be logged at the ERROR level, while correct checks are logged at the DEBUG +level.</p> +<p>By default, the <tt>SchemaCheckerService</tt> + runs every 7 days. This can be configured +by <tt>oozie.service.SchemaCheckerService.check.interval</tt> +</p> +<p>By default, the <tt>SchemaCheckerService</tt> + will consider "extra" tables, columns, and indexes to be incorrect. Advanced users who have +added additional tables, columns, and indexes can tell Oozie to ignore these by +setting <tt>oozie.service.SchemaCheckerService.ignore.extras</tt> + to <tt>false</tt> +.</p> +<p>The <tt>SchemaCheckerService</tt> + currently only supports MySQL, PostgreSQL, and Oracle databases. SQL Server and Derby are currently not +supported.</p> +<p>When Oozie HA is enabled, only one of the Oozie servers will perform the checks.</p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<p></p> +</div> + + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html>
Added: oozie/site/trunk/content/resources/docs/5.0.0/AG_OozieLogging.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.0.0/AG_OozieLogging.html?rev=1828722&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.0.0/AG_OozieLogging.html (added) +++ oozie/site/trunk/content/resources/docs/5.0.0/AG_OozieLogging.html Mon Apr 9 14:12:36 2018 @@ -0,0 +1,239 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at Apr 9, 2018 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie - </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.0.0"> + 5.0.0</a> + <span class="divider">/</span> + </li> + <li class="active ">Oozie - </li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-04-09</li> + <li id="projectVersion" class="pull-right"> + Version: 5.0.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p></p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<a name="Oozie_Logging"></a> +<div class="section"><h2> Oozie Logging</h2> +<p><ul><ul><li><a href="#Default_Oozie_Logging">Default Oozie Logging</a> +<ul><li><a href="#Default_Configuration">Default Configuration</a> +</li> +<li><a href="#Restrictions">Restrictions</a> +</li> +</ul> +</li> +<li><a href="#Previous_Default_Oozie_Logging">Previous Default Oozie Logging</a> +<ul><li><a href="#Previous_Default_Configuration">Previous Default Configuration</a> +</li> +<li><a href="#Restrictions_">Restrictions</a> +</li> +</ul> +</li> +<li><a href="#Other_Oozie_Logging">Other Oozie Logging</a> +</li> +</ul> +</ul> +</p> +<a name="Default_Oozie_Logging"></a> +<div class="section"><h3>Default Oozie Logging</h3> +<p>Oozie's logging properties can be configured in its log4j properties file (default is <tt>oozie-log4j.properties</tt> +). Most log messages +are configured by default to be written to the <tt>oozie</tt> + appender.</p> +<p>The default configuration for the <tt>oozie</tt> + appender is shown below.</p> +<a name="Default_Configuration"></a> +<div class="section"><h4>Default Configuration</h4> +<p><pre> +log4j.appender.oozie=org.apache.log4j.rolling.RollingFileAppender +log4j.appender.oozie.RollingPolicy=org.apache.oozie.util.OozieRollingPolicy +log4j.appender.oozie.File=${oozie.log.dir}/oozie.log +log4j.appender.oozie.Append=true +log4j.appender.oozie.layout=org.apache.log4j.PatternLayout +log4j.appender.oozie.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n +log4j.appender.oozie.RollingPolicy.FileNamePattern=${log4j.appender.oozie.File}-%d{yyyy-MM-dd-HH} +log4j.appender.oozie.RollingPolicy.MaxHistory=720 +</pre></p> +<p>In this configuration, the active log file will be named <tt>oozie.log</tt> + and all old log files will be named <tt>oozie.log-yyyy-MM-dd-HH</tt> + +(where <tt>yyyy-MM-dd-HH</tt> + is the time that that log file was created; e.g. 2012-07-21-05). All log files are in the same directory +(whatever <tt>oozie.log.dir</tt> + is assigned to). A maximum of 720 older log files will be retained. The active log file is rolled every +hour, so 720 old logs means that they are kept for 30 days before being deleted.</p> +<p>To keep all old logs instead of deleting them, <tt>log4j.appender.oozie.RollingPolicy.MaxHistory</tt> + can be set to <tt>-1</tt> +. +Additionally, <tt>log4j.appender.oozie.RollingPolicy</tt> + can be set to <tt>org.apache.log4j.rolling.TimeBasedRollingPolicy</tt> +, which has the +same exact behavior as <tt>org.apache.oozie.util.OozieRollingPolicy</tt> + except that it does not delete old logs.</p> +<a name="Restrictions"></a> +</div> +<div class="section"><h4>Restrictions</h4> +<p>In order for Oozie logging to work 100% correctly, the following restrictions must be observed (described below and in +the <tt>oozie-log4j.properties</tt> + file):</p> +<p>* The appender that Oozie uses must be named "oozie" (i.e. <tt>log4j.appender.oozie</tt> +)</p> +<p>* <tt>log4j.appender.oozie.RollingPolicy.FileNamePattern</tt> + must end with "-%d{yyyy-MM-dd-HH}.gz" or "-%d{yyyy-MM-dd-HH}". +If it ends with ".gz" the old logs will be compressed when rolled</p> +<p>* <tt>log4j.appender.oozie.RollingPolicy.FileNamePattern</tt> + must start with the value of <tt>log4j.appender.oozie.File</tt> +</p> +<a name="Previous_Default_Oozie_Logging"></a> +</div> +</div> +<div class="section"><h3>Previous Default Oozie Logging</h3> +<p>Oozie previously used the logging configuration shown below as the default for the <tt>oozie</tt> + appender. The other appender that Oozie +writes to still use a configuration similar to this.</p> +<a name="Previous_Default_Configuration"></a> +<div class="section"><h4>Previous Default Configuration</h4> +<p><pre> +log4j.appender.oozie=org.apache.log4j.DailyRollingFileAppender +log4j.appender.oozie.File=${oozie.log.dir}/oozie.log +log4j.appender.oozie.Append=true +log4j.appender.oozie.layout=org.apache.log4j.PatternLayout +log4j.appender.oozie.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n +log4j.appender.oozie.DatePattern='.'yyyy-MM-dd-HH +</pre></p> +<p>In this configuration, the active log file will be named <tt>oozie.log</tt> + and all old log files will be named <tt>oozie.log.yyyy-MM-dd-HH</tt> + +(where <tt>yyyy-MM-dd-HH</tt> + is the time that the log file was created; e.g. 2012-07-21-05). All log files are in the same directory +(whatever <tt>oozie.log.dir</tt> + is assigned to). All older log files are retained. The active log file is rolled every hour.</p> +<a name="Restrictions_"></a> +</div> +<div class="section"><h4>Restrictions</h4> +<p>In order for Oozie logging to work 100% correctly, the following restrictions must be observed (described below and in the +=oozie-log4j.properties= file):</p> +<p>* The appender that Oozie uses must be named "oozie" (i.e. <tt>log4j.appender.oozie</tt> +)</p> +<p>* <tt>log4j.appender.oozie.DatePattern</tt> + must end with either "dd" or "HH". If it ends with "HH", the log will be rolled every hour; +if it ends with "dd", the log will be rolled every day.</p> +<a name="Other_Oozie_Logging"></a> +</div> +</div> +<div class="section"><h3>Other Oozie Logging</h3> +<p>While Oozie can technically use any valid log4j Appender or configurations that violate the above restrictions, certain +features related to logs may be disabled and/or not work correctly, and is thus not advised.</p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<p></p> +</div> + + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html> Added: oozie/site/trunk/content/resources/docs/5.0.0/AG_OozieUpgrade.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.0.0/AG_OozieUpgrade.html?rev=1828722&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.0.0/AG_OozieUpgrade.html (added) +++ oozie/site/trunk/content/resources/docs/5.0.0/AG_OozieUpgrade.html Mon Apr 9 14:12:36 2018 @@ -0,0 +1,203 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at Apr 9, 2018 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie - </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.0.0"> + 5.0.0</a> + <span class="divider">/</span> + </li> + <li class="active ">Oozie - </li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-04-09</li> + <li id="projectVersion" class="pull-right"> + Version: 5.0.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p></p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<a name="Oozie_Upgrade"></a> +<div class="section"><h2> Oozie Upgrade</h2> +<p><ul><li><a href="#Preparation">Preparation</a> +</li> +<li><a href="#Oozie_Server_Upgrade">Oozie Server Upgrade</a> +</li> +<li><a href="#Oozie_Client_Upgrade">Oozie Client Upgrade</a> +</li> +</ul> +</p> +<a name="Preparation"></a> +<div class="section"><h2>Preparation</h2> +<p>Make sure there are not Workflows in RUNNING or SUSPENDED status, otherwise the database upgrade will fail.</p> +<p>Shutdown Oozie and backup the Oozie database.</p> +<p>Copy the oozie-site.xml from your current setup.</p> +<a name="Oozie_Server_Upgrade"></a> +</div> +<div class="section"><h2>Oozie Server Upgrade</h2> +<p>Expand the new Oozie tarball in a new location.</p> +<p>Edit the new <tt>oozie-site.xml</tt> + setting all custom properties values from the old <tt>oozie-site.xml</tt> +</p> +<p>IMPORTANT: From Oozie 2.x to Oozie 3.x the names of the database configuration properties have +changed. Their prefix has changed from =oozie.service.StoreService.*= to =oozie.service.JPAService.*=. +Make sure you are using the new prefix.</p> +<p>After upgrading the Oozie server, the <tt>oozie-setup.sh</tt> + MUST be rerun before starting the +upgraded Oozie server.</p> +<p>Oozie database migration is required when there Oozie database schema changes, like +upgrading from Oozie 2.x to Oozie 3.x.</p> +<p>Configure the oozie-site.xml with the correct database configuration properties as +explained in the 'Database Configuration' section in <a href="./AG_Install.html">Oozie Install</a> +.</p> +<p>Once <tt>oozie-site.xml</tt> + has been configured with the database configuration execute the <tt>ooziedb.sh</tt> + +command line tool to upgrade the database:</p> +<p><pre> +$ bin/ooziedb.sh upgrade -runValidate DB Connection. +DONE +Check DB schema exists +DONE +Check OOZIE_SYS table does not exist +DONE +Verify there are not active Workflow Jobs +DONE +Create SQL schema +DONE +DONE +Create OOZIE_SYS table +DONE +Upgrade COORD_JOBS new columns default values. +DONE +Upgrade COORD_JOBS & COORD_ACTIONS status values. +DONE +Table 'WF_ACTIONS' column 'execution_path', length changed to 1024 +DONE +Oozie DB has been upgraded to Oozie version '3.2.0' +The SQL commands have been written to: /tmp/ooziedb-5737263881793872034.sql +$ +</pre> +</p> +<p>The new version of the Oozie server is ready to be started.</p> +<p>NOTE: If using MySQL or Oracle, copy the corresponding JDBC driver JAR file to the <tt>libext/</tt> + directory before running +the <tt>ooziedb.sh</tt> + command line tool.</p> +<p>NOTE: If instead using the '-run' option, the '-sqlfile <FILE>' option is used, then all the +database changes will be written to the specified file and the database won't be modified.</p> +<a name="Oozie_Client_Upgrade"></a> +</div> +<div class="section"><h2>Oozie Client Upgrade</h2> +<p>While older Oozie clients work with newer Oozie server, to have access to all the +functionality of the Oozie server the same version of Oozie client should be installed +and used by users.</p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<p></p> +</div> + + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html> Added: oozie/site/trunk/content/resources/docs/5.0.0/BundleFunctionalSpec.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.0.0/BundleFunctionalSpec.html?rev=1828722&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.0.0/BundleFunctionalSpec.html (added) +++ oozie/site/trunk/content/resources/docs/5.0.0/BundleFunctionalSpec.html Mon Apr 9 14:12:36 2018 @@ -0,0 +1,628 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at Apr 9, 2018 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie - </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.0.0"> + 5.0.0</a> + <span class="divider">/</span> + </li> + <li class="active ">Oozie - </li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-04-09</li> + <li id="projectVersion" class="pull-right"> + Version: 5.0.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p></p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<hr /> +<a name="Oozie_Bundle_Specification"></a> +<div class="section"><h2> Oozie Bundle Specification</h2> +<p>The goal of this document is to define a new oozie abstraction called bundle system specialized in submitting and maintaining a set of coordinator applications.</p> +<p><ul><ul><li><a href="#Changelog">Changelog</a> +</li> +<li><a href="#a1._Bundle_Overview">1. Bundle Overview</a> +</li> +<li><a href="#a2._Definitions">2. Definitions</a> +</li> +<li><a href="#a3._Expression_Language_for_Parameterization">3. Expression Language for Parameterization</a> +</li> +<li><a href="#a4._Bundle_Job">4. Bundle Job</a> +<ul><li><a href="#a4.1._Bundle_Job_Status">4.1. Bundle Job Status</a> +</li> +<li><a href="#a4.2._Transitions_of_Bundle_Job_Status">4.2. Transitions of Bundle Job Status </a> +</li> +<li><a href="#a4.3._Details_of_Status_Transitions">4.3. Details of Status Transitions</a> +</li> +<li><a href="#a4.3.__Bundle_Application_Definition">4.3. Bundle Application Definition</a> +</li> +<li><a href="#a4.4.__Bundle_Formal_Parameters">4.4. Bundle Formal Parameters</a> +</li> +</ul> +</li> +<li><a href="#a5._User_Propagation">5. User Propagation</a> +</li> +<li><a href="#a6._Bundle_Application_Deployment">6. Bundle Application Deployment</a> +<ul><li><a href="#a6.1._Organizing_Bundle_Applications">6.1. Organizing Bundle Applications</a> +</li> +</ul> +</li> +<li><a href="#a7._Bundle_Job_Submission">7. Bundle Job Submission</a> +</li> +<li><a href="#a8._Bundle_Rerun">8. Bundle Rerun</a> +<ul><li><a href="#Rerunning_a_Bundle_Job">Rerunning a Bundle Job</a> +</li> +<li><a href="#Rerun_Arguments">Rerun Arguments</a> +</li> +</ul> +</li> +<li><a href="#Appendixes">Appendixes</a> +<ul><li><a href="#Appendix_A_Oozie_Bundle_XML-Schema">Appendix A, Oozie Bundle XML-Schema</a> +<ul><li><a href="#Oozie_Bundle_Schema_0.1">Oozie Bundle Schema 0.1</a> +</li> +<li><a href="#Oozie_Bundle_Schema_0.2">Oozie Bundle Schema 0.2</a> +</li> +</ul> +</li> +</ul> +</li> +</ul> +</ul> +</p> +<a name="Changelog"></a> +<div class="section"><h3>Changelog</h3> +<a name="a1._Bundle_Overview"></a> +</div> +<div class="section"><h3>1. Bundle Overview</h3> +<p>Bundle is a higher-level oozie abstraction that will batch a set of coordinator applications. The user will be able to start/stop/suspend/resume/rerun in the bundle level resulting a better and easy operational control.</p> +<p>More specifically, the oozie <b>Bundle</b> + system allows the user to define and execute a bunch of coordinator applications often called a data pipeline. There is no explicit dependency among the coordinator applications in a bundle. However, a user could use the data dependency of coordinator applications to create an implicit data application pipeline.</p> +<a name="a2._Definitions"></a> +</div> +<div class="section"><h3>2. Definitions</h3> +<p><b>Kick-off-time:</b> + The time when a bundle should start and submit coordinator applications.</p> +<p><b>Bundle Application:</b> + A bundle application defines a set of coordinator applications and when to start those. Normally, bundle applications are parameterized. A bundle application is written in XML.</p> +<p><b>Bundle Job:</b> + A bundle job is an executable instance of a bundle application. A job submission is done by submitting a job configuration that resolves all parameters in the application definition.</p> +<p><b>Bundle Definition Language:</b> + The language used to describe bundle applications.</p> +<a name="a3._Expression_Language_for_Parameterization"></a> +</div> +<div class="section"><h3>3. Expression Language for Parameterization</h3> +<p>Bundle application definitions can be parameterized with variables.</p> +<p>At job submission time all the parameters are resolved into concrete values.</p> +<p>The parameterization of bundle definitions is done using JSP Expression Language syntax from the <a class="externalLink" href="http://jcp.org/aboutJava/communityprocess/final/jsr152/";>JSP 2.0 Specification (JSP.2.3)</a> +, allowing not only to support variables as parameters but also complex expressions.</p> +<p>EL expressions can be used in XML attribute values and XML text element values. They cannot be used in XML element and XML attribute names.</p> +<a name="a4._Bundle_Job"></a> +</div> +<div class="section"><h3>4. Bundle Job</h3> +<a name="a4.1._Bundle_Job_Status"></a> +<div class="section"><h4>4.1. Bundle Job Status</h4> +<p>At any time, a bundle job is in one of the following status: <b>PREP, RUNNING, RUNNINGWITHERROR, SUSPENDED, PREPSUSPENDED, SUSPENDEDWITHERROR, PAUSED, PAUSEDWITHERROR, PREPPAUSED, SUCCEEDED, DONEWITHERROR, KILLED, FAILED</b> +.</p> +<a name="a4.2._Transitions_of_Bundle_Job_Status"></a> +</div> +<div class="section"><h4>4.2. Transitions of Bundle Job Status</h4> +<p>Valid bundle job status transitions are:</p> +<p><ul><li><b>PREP --> PREPSUSPENDED | PREPPAUSED | RUNNING | KILLED</b> +</li> +<li><b>RUNNING --> RUNNINGWITHERROR | SUSPENDED | PAUSED | SUCCEEDED | KILLED</b> +</li> +<li><b>RUNNINGWITHERROR --> RUNNING | SUSPENDEDWITHERROR | PAUSEDWITHERROR | DONEWITHERROR | FAILED | KILLED</b> +</li> +<li><b>PREPSUSPENDED --> PREP | KILLED</b> +</li> +<li><b>SUSPENDED --> RUNNING | KILLED</b> +</li> +<li><b>SUSPENDEDWITHERROR --> RUNNINGWITHERROR | KILLED</b> +</li> +<li><b>PREPPAUSED --> PREP | KILLED</b> +</li> +<li><b>PAUSED --> SUSPENDED | RUNNING | KILLED</b> +</li> +<li><b>PAUSEDWITHERROR --> SUSPENDEDWITHERROR | RUNNINGWITHERROR | KILLED</b> +</li> +</ul> +</p> +<a name="a4.3._Details_of_Status_Transitions"></a> +</div> +<div class="section"><h4>4.3. Details of Status Transitions</h4> +<p>When a bundle job is submitted, oozie parses the bundle job XML. Oozie then creates a record for the bundle with status <b>PREP</b> + and returns a unique ID.</p> +<p>When a user requests to suspend a bundle job that is in <b>PREP</b> + state, oozie puts the job in status <b>PREPSUSPENDED</b> +. Similarly, when pause time reaches for a bundle job with <b>PREP</b> + status, oozie puts the job in status <b>PREPPAUSED</b> +.</p> +<p>Conversely, when a user requests to resume a <b>PREPSUSPENDED</b> + bundle job, oozie puts the job in status <b>PREP</b> +. And when pause time is reset for a bundle job that is in <b>PREPPAUSED</b> + state, oozie puts the job in status <b>PREP</b> +.</p> +<p>There are two ways a bundle job could be started.</p> +<p>* If <tt>kick-off-time</tt> + (defined in the bundle xml) reaches. The default value is null which means starts coordinators NOW.</p> +<p>* If user sends a start request to START the bundle.</p> +<p>When a bundle job starts, oozie puts the job in status <b>RUNNING</b> + and it submits all the coordinator jobs. If any coordinator job goes to <b>FAILED/KILLED/DONEWITHERROR</b> + state, the bundle job is put in <b>RUNNINGWITHERROR</b> +</p> +<p>When a user requests to kill a bundle job, oozie puts the job in status <b>KILLED</b> + and it sends kill to all submitted coordinator jobs.</p> +<p>When a user requests to suspend a bundle job that is in <b>RUNNING</b> + status, oozie puts the job in status <b>SUSPENDED</b> + and it suspends all submitted coordinator jobs. Similarly, when a user requests to suspend a bundle job that is in <b>RUNNINGWITHERROR</b> + status, oozie puts the job in status <b>SUSPENDEDWITHERROR</b> + and it suspends all submitted coordinator jobs.</p> +<p>When pause time reaches for a bundle job that is in <b>RUNNING</b> + status, oozie puts the job in status <b>PAUSED</b> +. When pause time reaches for a bundle job that is in <b>RUNNINGWITHERROR</b> + status, oozie puts the job in status <b>PAUSEDWITHERROR</b> +.</p> +<p>Conversely, when a user requests to resume a <b>SUSPENDED</b> + bundle job, oozie puts the job in status <b>RUNNING</b> +. Similarly, when a user requests to resume a <b>SUSPENDEDWITHERROR</b> + bundle job, oozie puts the job in status <b>RUNNINGWITHERROR</b> +. And when pause time is reset for a bundle job and job status is <b>PAUSED</b> +, oozie puts the job in status <b>RUNNING</b> +. Similarly, when the pause time is reset for a bundle job and job status is <b>PAUSEDWITHERROR</b> +, oozie puts the job in status <b>RUNNINGWITHERROR</b> +</p> +<p>When all the coordinator jobs finish, oozie updates the bundle status accordingly. If all coordinators reaches to the <i>same</i> + terminal state, bundle job status also move to the same status. For example, if all coordinators are <b>SUCCEEDED</b> +, oozie puts the bundle job into <b>SUCCEEDED</b> + status. However, if all coordinator jobs don't finish with the same status, oozie puts the bundle job into <b>DONEWITHERROR</b> +.</p> +<a name="a4.3.__Bundle_Application_Definition"></a> +</div> +<div class="section"><h4>4.3. Bundle Application Definition</h4> +<p>A bundle definition is defined in XML by a name, controls and one or more coordinator application specifications:</p> +<p><ul><li><b><font color="#0000ff"> name: </font></b> + The name for the bundle job.</li> +</ul> +<b> *<font color="#0000ff"> controls: </font></b> + The control specification for the bundle.<ul><li><ul><li><b><font color="#0000ff"> kick-off-time: </font></b> + It defines when the bundle job should start and submit the coordinator applications. This field is optional and the default is <b>NOW</b> + that means the job should start right-a-way.</li> +</ul> +</li> +<li><b><font color="#0000ff"> coordinator: </font></b> + Coordinator application specification. There should be at least one coordinator application in any bundle.<ul><li><b><font color="#0000ff"> name: </font></b> + Name of the coordinator application. It can be used for referring this application through bundle to control such as kill, suspend, rerun.</li> +<li><b><font color="#0000ff"> enabled: </font></b> + Enabled can be used to enable or disable a coordinator. It is optional. The default value for enabled is true.</li> +<li><b><font color="#0000ff"> app-path: </font></b> + Path of the coordinator application definition in hdfs. This is a mandatory element.</li> +<li><b><font color="#0000ff"> configuration: </font></b> + A hadoop like configuration to parameterize corresponding coordinator application. This is optional.</li> +</ul> +</li> +</ul> +<b> *<font color="#0000ff"> Parameterization: </font></b> + Configuration properties that are a valid Java identifier, [A-Za-z_][0-9A-Za-z_]*, are available as <tt>${NAME}</tt> + variables within the bundle application definition. Configuration properties that are not a valid Java identifier, for example <tt>job.tracker</tt> +, are available via the <tt>${bundle:conf(String name)}</tt> + function. Valid Java identifier properties are available via this function as well.</p> +<p><b><font color="#800080"> Syntax: </font></b> +</p> +<p><pre> + <bundle-app name=[NAME] xmlns='uri:oozie:bundle:0.1'> + <controls> + <kick-off-time>[DATETIME]</kick-off-time> + </controls> + <coordinator name=[NAME] enabled=[TRUE | FALSE] > + <app-path>[COORD-APPLICATION-PATH]</app-path> + <configuration> + <property> + <name>[PROPERTY-NAME]</name> + <value>[PROPERTY-VALUE]</value> + </property> + ... + </configuration> + </coordinator> + ... +</bundle-app> +</pre></p> +<p><b><font color="#008000"> Examples: </font></b> +</p> +<p><b>A Bundle Job that maintains two coordinator applications:</b> +</p> +<p><pre> +<bundle-app name='APPNAME' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns='uri:oozie:bundle:0.1'> + <controls> + <kick-off-time>${kickOffTime}</kick-off-time> + </controls> + <coordinator name="${bundle:conf('coordName1')}" > + <app-path>${appPath}</app-path> + <configuration> + <property> + <name>startTime1</name> + <value>${bundle:conf('coord1.startTime1')}</value> + </property> + <property> + <name>endTime1</name> + <value>${END_TIME}</value> + </property> + </configuration> + </coordinator> + <coordinator name='coordJobFromBundle2' > + <app-path>${appPath2}</app-path> + <configuration> + <property> + <name>startTime2</name> + <value>${START_TIME2}</value> + </property> + <property> + <name>endTime2</name> + <value>${END_TIME2}</value> + </property> + </configuration> + </coordinator> +</bundle-app> +</pre></p> +<a name="a4.4.__Bundle_Formal_Parameters"></a> +</div> +<div class="section"><h4>4.4. Bundle Formal Parameters</h4> +<p>As of schema 0.2, a list of formal parameters can be provided which will allow Oozie to verify, at submission time, that said +properties are actually specified (i.e. before the job is executed and fails). Default values can also be provided.</p> +<p><b>Example:</b> +</p> +<p>The previous Bundle Job application definition with formal parameters:</p> +<p><pre> +<bundle-app name='APPNAME' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns='uri:oozie:bundle:0.2'> + <parameters> + <property> + <name>appPath</name> + </property> + <property> + <name>appPath2</name> + <value>hdfs://foo:8020/user/joe/job/job.properties</value> + </property> + </parameters> + <controls> + <kick-off-time>${kickOffTime}</kick-off-time> + </controls> + <coordinator name='coordJobFromBundle1' > + <app-path>${appPath}</app-path> + <configuration> + <property> + <name>startTime1</name> + <value>${START_TIME}</value> + </property> + <property> + <name>endTime1</name> + <value>${END_TIME}</value> + </property> + </configuration> + </coordinator> + <coordinator name='coordJobFromBundle2' > + <app-path>${appPath2}</app-path> + <configuration> + <property> + <name>startTime2</name> + <value>${START_TIME2}</value> + </property> + <property> + <name>endTime2</name> + <value>${END_TIME2}</value> + </property> + </configuration> + </coordinator> +</bundle-app> +</pre></p> +<p>In the above example, if <tt>appPath</tt> + is not specified, Oozie will print an error message instead of submitting the job. If +=appPath2= is not specified, Oozie will use the default value, <tt><a href="./hdfs://foo:8020/user/joe/job/job.properties.html">hdfs://foo:8020/user/joe/job/job.properties</a> +</tt> +.</p> +<a name="a5._User_Propagation"></a> +</div> +</div> +<div class="section"><h3>5. User Propagation</h3> +<p>When submitting a bundle job, the configuration must contain a <tt>user.name</tt> + property. If security is enabled, Oozie must ensure that the value of the <tt>user.name</tt> + property in the configuration match the user credentials present in the protocol (web services) request.</p> +<p>When submitting a bundle job, the configuration may contain the <tt>oozie.job.acl</tt> + property (the <tt>group.name</tt> + property +has been deprecated). If authorization is enabled, this property is treated as as the ACL for the job, it can contain +user and group IDs separated by commas.</p> +<p>The specified user and ACL are assigned to the created bundle job.</p> +<p>Oozie must propagate the specified user and ACL to the system executing its children jobs (coordinator jobs).</p> +<a name="a6._Bundle_Application_Deployment"></a> +</div> +<div class="section"><h3>6. Bundle Application Deployment</h3> +<p>A bundle application consist exclusively of bundle application definition and associated coordinator application specifications. They must be installed in an HDFS directory. To submit a job for a bundle application, the full HDFS path to bundle application definition must be specified.</p> +<a name="a6.1._Organizing_Bundle_Applications"></a> +<div class="section"><h4>6.1. Organizing Bundle Applications</h4> +<p>TBD.</p> +<a name="a7._Bundle_Job_Submission"></a> +</div> +</div> +<div class="section"><h3>7. Bundle Job Submission</h3> +<p>When a bundle job is submitted to Oozie, the submitter must specified all the required job properties plus the HDFS path to the bundle application definition for the job.</p> +<p>The bundle application definition HDFS path must be specified in the 'oozie.bundle.application.path' job property.</p> +<p>All the bundle job properties, the HDFS path for the bundle application, the 'user.name' and 'oozie.job.acl' must be +submitted to the Oozie using an XML configuration file (Hadoop XML configuration file).</p> +<p><b><font color="#008000"> Example: </font></b> +:</p> +<p><pre> +<?xml version="1.0" encoding="UTF-8"?> +<configuration> + <property> + <name>user.name</name> + <value>joe</value> + </property> + <property> + <name>oozie.bundle.application.path</name> + <value>hdfs://foo:8020/user/joe/mybundles/hello-bundle1.xml</value> + </property> + ... +</configuration> +</pre></p> +<a name="a8._Bundle_Rerun"></a> +</div> +<div class="section"><h3>8. Bundle Rerun</h3> +<a name="Rerunning_a_Bundle_Job"></a> +<div class="section"><h4>Rerunning a Bundle Job</h4> +<p>Oozie provides a way of rerunning a bundle job. The user could request to rerun a subset of coordinators within a bundle by defining a list of coordinator's names. In addition, a user could define a list of dates or ranges of dates (in UTC format) to rerun for those time windows. +There is a way of asking whether to cleanup all output directories before rerun. By default, oozie will remove all output directories. Moreover, there is an option by which a user could ask to re-calculate the dynamic input directories defined by latest function in coordinators.</p> +<a name="Rerun_Arguments"></a> +</div> +<div class="section"><h4>Rerun Arguments</h4> +<p><pre> +$oozie job -rerun <bundle_Job_id> [-coordinator <list of coordinator name separate by comma> +[-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z] + [-nocleanup] [-refresh] +</pre></p> +<p><ul><li>The <tt>rerun</tt> + option reruns a bundle job that is <b>not</b> + in (=KILLED=, <tt>FAILED</tt> +, <tt>PREP</tt> +, <tt>PREPPAUSED</tt> +, <tt>PREPSUSPENDED</tt> +).</li> +<li>Rerun a bundle job that is in <tt>PAUSED</tt> + state will reset the paused time.</li> +<li>The option -coordinator determines the name of coordinator that will be rerun. By default all coordinators are rerun.</li> +<li>Multiple ranges can be used in -date. See the above examples.</li> +<li>The dates specified in -date must be UTC.</li> +<li>If -nocleanup is given, corresponding coordinator directories will not be removed; otherwise the 'output-event' will be deleted.</li> +<li>If -refresh is set, new dataset is re-evaluated for latest() and future() for the corresponding coordinators.</li> +<li>If -refresh is set, all dependencies will be re-checked; otherwise only missed dependencies will be checked for the corresponding coordinators.</li> +</ul> +</p> +<p>After the command is executed the rerun bundle job will be in <tt>RUNNING</tt> + status.</p> +<p>Refer to the <a href="./DG_CoordinatorRerun.html">Rerunning Coordinator Actions</a> + for details on rerun of coordinator job.</p> +<a name="Appendixes"></a> +</div> +</div> +<div class="section"><h3>Appendixes</h3> +<a name="Appendix_A_Oozie_Bundle_XML-Schema"></a> +<div class="section"><h4>Appendix A, Oozie Bundle XML-Schema</h4> +<a name="Oozie_Bundle_Schema_0.1"></a> +<div class="section"><h5>Oozie Bundle Schema 0.1</h5> +<p><pre> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:bundle="uri:oozie:bundle:0.1" + elementFormDefault="qualified" targetNamespace="uri:oozie:bundle:0.1"> <xs:element name="bundle-app" type="bundle:BUNDLE-APP"/> + <xs:simpleType name="IDENTIFIER"> + <xs:restriction base="xs:string"> + <xs:pattern value="([a-zA-Z]([\-_a-zA-Z0-9])*){1,39})"/> + </xs:restriction> + </xs:simpleType> + <xs:complexType name="BUNDLE-APP"> + <xs:sequence> + <xs:element name="controls" type="bundle:CONTROLS" minOccurs="0" maxOccurs="1"/> + <xs:element name="coordinator" type="bundle:COORDINATOR" minOccurs="1" maxOccurs="unbounded"/> + </xs:sequence> + <xs:attribute name="name" type="bundle:IDENTIFIER" use="required"/> + </xs:complexType> + <xs:complexType name="CONTROLS"> + <xs:sequence minOccurs="0" maxOccurs="1"> + <xs:element name="kick-off-time" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + </xs:complexType> + <xs:complexType name="COORDINATOR"> + <xs:sequence minOccurs="1" maxOccurs="1"> + <xs:element name="app-path" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="configuration" type="bundle:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + <xs:attribute name="name" type="bundle:IDENTIFIER" use="required"/> + <xs:attribute name="critical" type="xs:string" use="optional"/> + </xs:complexType> + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +</xs:schema> +</pre> +</p> +<a name="Oozie_Bundle_Schema_0.2"></a> +</div> +<div class="section"><h5>Oozie Bundle Schema 0.2</h5> +<p><pre> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:bundle="uri:oozie:bundle:0.2" + elementFormDefault="qualified" targetNamespace="uri:oozie:bundle:0.2"> <xs:element name="bundle-app" type="bundle:BUNDLE-APP"/> + <xs:simpleType name="IDENTIFIER"> + <xs:restriction base="xs:string"> + <xs:pattern value="([a-zA-Z]([\-_a-zA-Z0-9])*){1,39}"/> + </xs:restriction> + </xs:simpleType> + <xs:complexType name="BUNDLE-APP"> + <xs:sequence> + <xs:element name="parameters" type="bundle:PARAMETERS" minOccurs="0" maxOccurs="1"/> + <xs:element name="controls" type="bundle:CONTROLS" minOccurs="0" maxOccurs="1"/> + <xs:element name="coordinator" type="bundle:COORDINATOR" minOccurs="1" maxOccurs="unbounded"/> + </xs:sequence> + <xs:attribute name="name" type="xs:string" use="required"/> + </xs:complexType> + <xs:complexType name="PARAMETERS"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="0" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + <xs:complexType name="CONTROLS"> + <xs:sequence minOccurs="0" maxOccurs="1"> + <xs:element name="kick-off-time" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + </xs:complexType> + <xs:complexType name="COORDINATOR"> + <xs:sequence minOccurs="1" maxOccurs="1"> + <xs:element name="app-path" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="configuration" type="bundle:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + <xs:attribute name="name" type="bundle:IDENTIFIER" use="required"/> + <xs:attribute name="critical" type="xs:string" use="optional"/> + <xs:attribute name="enabled" type="xs:string" use="optional"/> + </xs:complexType> + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +</xs:schema> +</pre> +</p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<p></p> +</div> +</div> +</div> + + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html>
![https://oozie.apache.org/images/oozie_200x.png"](https://oozie.apache.org/images/oozie_200x.png"){kind=link}