Added: oozie/site/trunk/content/resources/docs/5.2.1/DG_CommandLineTool.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.2.1/DG_CommandLineTool.html?rev=1886952&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.2.1/DG_CommandLineTool.html (added) +++ oozie/site/trunk/content/resources/docs/5.2.1/DG_CommandLineTool.html Fri Feb 26 14:14:19 2021 @@ -0,0 +1,2290 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2021-02-26 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20210226" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie – </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org/"; class="externalLink" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.2.1"> + 5.2.1</a> + <span class="divider">/</span> + </li> + <li class="active "></li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2021-02-26</li> + <li id="projectVersion" class="pull-right"> + Version: 5.2.1 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p><a href="index.html">::Go back to Oozie Documentation Index::</a></p> +<h1>Command Line Interface Utilities</h1> +<ul> +<li><a href="#Introduction">Introduction</a></li> +<li><a href="#Oozie_Command_Line_Usage">Oozie Command Line Usage</a> +<ul> +<li><a href="#Oozie_basic_commands">Oozie basic commands</a></li> +<li><a href="#Oozie_job_operation_commands">Oozie job operation commands</a></li> +<li><a href="#Oozie_jobs_operation_commands">Oozie jobs operation commands</a></li> +<li><a href="#Oozie_admin_operation_commands">Oozie admin operation commands</a></li> +<li><a href="#Oozie_validate_command">Oozie validate command</a></li> +<li><a href="#Oozie_SLA_operation_commands">Oozie SLA operation commands</a></li> +<li><a href="#Oozie_Pig_submit_command">Oozie Pig submit command</a></li> +<li><a href="#Oozie_Hive_submit_command">Oozie Hive submit command</a></li> +<li><a href="#Oozie_Sqoop_submit_command">Oozie Sqoop submit command</a></li> +<li><a href="#Oozie_info_command">Oozie info command</a></li> +<li><a href="#Oozie_MapReduce_job_command">Oozie MapReduce job command</a></li></ul></li> +<li><a href="#Common_CLI_Options">Common CLI Options</a> +<ul> +<li><a href="#Authentication">Authentication</a></li> +<li><a href="#Impersonation_doAs">Impersonation, doAs</a></li> +<li><a href="#Oozie_URL">Oozie URL</a></li> +<li><a href="#Time_zone">Time zone</a></li> +<li><a href="#Debug_Mode">Debug Mode</a></li> +<li><a href="#CLI_retry">CLI retry</a></li></ul></li> +<li><a href="#Job_Operations">Job Operations</a> +<ul> +<li><a href="#Submitting_a_Workflow_Coordinator_or_Bundle_Job">Submitting a Workflow, Coordinator or Bundle Job</a></li> +<li><a href="#Starting_a_Workflow_Coordinator_or_Bundle_Job">Starting a Workflow, Coordinator or Bundle Job</a></li> +<li><a href="#Running_a_Workflow_Coordinator_or_Bundle_Job">Running a Workflow, Coordinator or Bundle Job</a></li> +<li><a href="#Suspending_a_Workflow_Coordinator_or_Bundle_Job">Suspending a Workflow, Coordinator or Bundle Job</a></li> +<li><a href="#Resuming_a_Workflow_Coordinator_or_Bundle_Job">Resuming a Workflow, Coordinator or Bundle Job</a></li> +<li><a href="#Killing_a_Workflow_Coordinator_or_Bundle_Job">Killing a Workflow, Coordinator or Bundle Job</a></li> +<li><a href="#Killing_a_Coordinator_Action_or_Multiple_Actions">Killing a Coordinator Action or Multiple Actions</a></li> +<li><a href="#Changing_endtimeconcurrencypausetimestatus_of_a_Coordinator_Job">Changing endtime/concurrency/pausetime/status of a Coordinator Job</a></li> +<li><a href="#Changing_endtimepausetime_of_a_Bundle_Job">Changing endtime/pausetime of a Bundle Job</a></li> +<li><a href="#Rerunning_a_Workflow_Job">Rerunning a Workflow Job</a></li> +<li><a href="#Rerunning_a_Coordinator_Action_or_Multiple_Actions">Rerunning a Coordinator Action or Multiple Actions</a></li> +<li><a href="#Rerunning_a_Bundle_Job">Rerunning a Bundle Job</a></li> +<li><a href="#Checking_the_Information_and_Status_of_a_Workflow_Coordinator_or_Bundle_Job_or_a_Coordinator_Action">Checking the Information and Status of a Workflow, Coordinator or Bundle Job or a Coordinator Action</a></li> +<li><a href="#Listing_all_the_Workflows_for_a_Coordinator_Action">Listing all the Workflows for a Coordinator Action</a></li> +<li><a href="#Listing_all_retry_attempts_of_a_workflow_action">Listing all retry attempts of a workflow action</a></li> +<li><a href="#Checking_the_xml_definition_of_a_Workflow_Coordinator_or_Bundle_Job">Checking the xml definition of a Workflow, Coordinator or Bundle Job</a></li> +<li><a href="#Checking_the_server_logs_of_a_Workflow_Coordinator_or_Bundle_Job">Checking the server logs of a Workflow, Coordinator or Bundle Job</a></li> +<li><a href="#Checking_the_server_error_logs_of_a_Workflow_Coordinator_or_Bundle_Job">Checking the server error logs of a Workflow, Coordinator or Bundle Job</a></li> +<li><a href="#Checking_the_audit_logs_of_a_Workflow_Coordinator_or_Bundle_Job">Checking the audit logs of a Workflow, Coordinator or Bundle Job</a></li> +<li><a href="#Checking_the_server_logs_for_particular_actions_of_a_Coordinator_Job">Checking the server logs for particular actions of a Coordinator Job</a></li> +<li><a href="#Filtering_the_server_logs_with_logfilter_options">Filtering the server logs with logfilter options</a></li> +<li><a href="#Dryrun_of_Coordinator_Job">Dryrun of Coordinator Job</a></li> +<li><a href="#Dryrun_of_Workflow_Job">Dryrun of Workflow Job</a></li> +<li><a href="#Dryrun_of_Bundle_Job">Dryrun of Bundle Job</a></li> +<li><a href="#Updating_coordinator_definition_and_properties">Updating coordinator definition and properties</a></li> +<li><a href="#Ignore_a_Coordinator_Job">Ignore a Coordinator Job</a></li> +<li><a href="#Ignore_a_Coordinator_Action_or_Multiple_Coordinator_Actions">Ignore a Coordinator Action or Multiple Coordinator Actions</a></li> +<li><a href="#Polling_an_Oozie_job">Polling an Oozie job</a></li> +<li><a href="#Changing_job_SLA_definition_and_alerting">Changing job SLA definition and alerting</a></li> +<li><a href="#Getting_missing_dependencies_of_coordinator_actions">Getting missing dependencies of coordinator action(s)</a></li> +<li><a href="#Checking_a_workflow_definition_generated_by_a_Fluent_Job_API_jar_file">Checking a workflow definition generated by a Fluent Job API jar file</a></li> +<li><a href="#Submitting_a_workflow_definition_generated_by_a_Fluent_Job_API_jar_file">Submitting a workflow definition generated by a Fluent Job API jar file</a></li> +<li><a href="#Running_a_workflow_definition_generated_by_a_Fluent_Job_API_jar_file">Running a workflow definition generated by a Fluent Job API jar file</a></li></ul></li> +<li><a href="#Jobs_Operations">Jobs Operations</a> +<ul> +<li><a href="#Checking_the_Status_of_multiple_Workflow_Jobs">Checking the Status of multiple Workflow Jobs</a></li> +<li><a href="#Checking_the_Status_of_multiple_Coordinator_Jobs">Checking the Status of multiple Coordinator Jobs</a></li> +<li><a href="#Checking_the_Status_of_multiple_Bundle_Jobs">Checking the Status of multiple Bundle Jobs</a></li> +<li><a href="#Bulk_kill_suspend_or_resume_multiple_jobs">Bulk kill, suspend or resume multiple jobs</a></li> +<li><a href="#Bulk_monitoring_for_jobs_launched_via_Bundles">Bulk monitoring for jobs launched via Bundles</a></li></ul></li> +<li><a href="#Admin_Operations">Admin Operations</a> +<ul> +<li><a href="#Checking_the_Status_of_the_Oozie_System">Checking the Status of the Oozie System</a></li> +<li><a href="#Changing_the_Status_of_the_Oozie_System">Changing the Status of the Oozie System</a></li> +<li><a href="#Displaying_the_Build_Version_of_the_Oozie_System">Displaying the Build Version of the Oozie System</a></li> +<li><a href="#Displaying_the_queue_dump_of_the_Oozie_System">Displaying the queue dump of the Oozie System</a></li> +<li><a href="#Displaying_the_list_of_available_Oozie_Servers">Displaying the list of available Oozie Servers</a></li> +<li><a href="#Displaying_the_Oozie_server_configuration">Displaying the Oozie server configuration</a></li> +<li><a href="#Displaying_the_Oozie_server_OS_environment">Displaying the Oozie server OS environment</a></li> +<li><a href="#Displaying_the_Oozie_server_Java_system_properties">Displaying the Oozie server Java system properties</a></li> +<li><a href="#Displaying_the_Oozie_server_Instrumentation">Displaying the Oozie server Instrumentation</a></li> +<li><a href="#Displaying_the_Oozie_server_Metrics">Displaying the Oozie server Metrics</a></li> +<li><a href="#Running_purge_command">Running purge command</a></li></ul></li> +<li><a href="#Validate_Operations">Validate Operations</a> +<ul> +<li><a href="#Validating_a_Workflow_XML">Validating a Workflow XML</a></li> +<li><a href="#Getting_list_of_available_sharelib">Getting list of available sharelib</a></li> +<li><a href="#Update_system_sharelib">Update system sharelib</a></li></ul></li> +<li><a href="#SLA_Operations">SLA Operations</a> +<ul> +<li><a href="#Getting_a_list_of_SLA_events">Getting a list of SLA events</a></li> +<li><a href="#Getting_the_SLA_event_with_particular_sequenceID">Getting the SLA event with particular sequenceID</a></li> +<li><a href="#Getting_information_about_SLA_events_using_filter">Getting information about SLA events using filter</a></li></ul></li> +<li><a href="#Pig_Operations">Pig Operations</a> +<ul> +<li><a href="#Submitting_a_pig_job_through_HTTP">Submitting a pig job through HTTP</a></li></ul></li> +<li><a href="#Hive_Operations">Hive Operations</a> +<ul> +<li><a href="#Submitting_a_hive_job_through_HTTP">Submitting a hive job through HTTP</a></li></ul></li> +<li><a href="#Sqoop_Operations">Sqoop Operations</a> +<ul> +<li><a href="#Submitting_a_sqoop_job_through_HTTP">Submitting a sqoop job through HTTP</a></li></ul></li> +<li><a href="#Info_Operations">Info Operations</a> +<ul> +<li><a href="#Getting_a_list_of_time_zones">Getting a list of time zones</a></li></ul></li> +<li><a href="#Map-reduce_Operations">Map-reduce Operations</a> +<ul> +<li><a href="#Submitting_a_map-reduce_job">Submitting a map-reduce job</a></li></ul></li> +<li><a href="#Getting_Oozie_diagnostics_bundles">Getting Oozie diagnostics bundles</a></li></ul> + +<div class="section"> +<h2><a name="Introduction"></a>Introduction</h2> +<p>Oozie provides a command line utility, <tt>oozie</tt>, to perform job and admin tasks. All operations are done via sub-commands of the <tt>oozie</tt> CLI.</p> +<p>The <tt>oozie</tt> CLI interacts with Oozie via its WS API.</p></div> +<div class="section"> +<h2><a name="Oozie_Command_Line_Usage"></a>Oozie Command Line Usage</h2> + +<div> +<div> +<pre class="source">usage: + the env variable 'OOZIE_URL' is used as default value for the '-oozie' option + the env variable 'OOZIE_TIMEZONE' is used as default value for the '-timezone' option + the env variable 'OOZIE_AUTH' is used as default value for the '-auth' option + custom headers for Oozie web services can be specified using '-Dheader:NAME=VALUE' +</pre></div></div> + +<div class="section"> +<h3><a name="Oozie_basic_commands"></a>Oozie basic commands</h3> + +<div> +<div> +<pre class="source">oozie help : display usage + +oozie version : show client version + +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Oozie_job_operation_commands"></a>Oozie job operation commands</h3> + +<div> +<div> +<pre class="source">oozie job <OPTIONS> : job operations + -action <arg> coordinator rerun/kill on action ids (requires -rerun/-kill); + coordinator log retrieval on action ids (requires -log) + -allruns Get workflow jobs corresponding to a coordinator action + including all the reruns + -auditlog <arg> job audit log + -auth <arg> select authentication type [SIMPLE|BASIC|KERBEROS] + -change <arg> change a coordinator or bundle job + -config <arg> job configuration file '.xml' or '.properties' + -configcontent <arg> job configuration + -coordinator <arg> bundle rerun on coordinator names (requires -rerun) + -D <property=value> set/override value for given property + -date <arg> coordinator/bundle rerun on action dates (requires -rerun); + coordinator log retrieval on action dates (requires -log) + -debug Use debug mode to see debugging statements on stdout + -definition <arg> job definition + -diff <arg> Show diff of the new coord definition and properties with the + existing one (default true) + -doas <arg> doAs user, impersonates as the specified user + -dryrun Dryrun a workflow (since 3.3.2), a coordinator (since 2.0) + or a bundle (since 5.1) job without actually executing it + -errorlog <arg> job error log + -failed runs the failed workflow actions of the coordinator actions + (requires -rerun) + -filter <arg> <key><comparator><value>[;<key><comparator><value>]* + (All Coordinator actions satisfying the filters will be + retrieved). + key: status or nominaltime + comparator: =, !=, <, <=, >, >=. = is used as OR and others + as AND + status: values are valid status like SUCCEEDED, KILLED etc. + Only = and != apply for status + nominaltime: time of format yyyy-MM-dd'T'HH:mm'Z' + -ignore <arg> change status of a coordinator job or action to IGNORED + (-action required to ignore coord actions) + -info <arg> info of a job + -interval <arg> polling interval in minutes (default is 5, requires -poll) + -kill <arg> kill a job (coordinator can mention -action or -date) + -len <arg> number of actions (default TOTAL ACTIONS, requires -info) + -localtime use local time (same as passing your time zone to -timezone). + Overrides -timezone option + -log <arg> job log + -logfilter <arg> job log search parameter. Can be specified as -logfilter + opt1=val1;opt2=val1;opt3=val1. Supported options are recent, + start, end, loglevel, text, limit and debug + -missingdeps <arg> List missing dependencies of a coord action. To specify + multiple actions, use with -action or -date option. + -nocleanup do not clean up output-events of the coordinator rerun + actions (requires -rerun) + -offset <arg> job info offset of actions (default '1', requires -info) + -oozie <arg> Oozie URL + -order <arg> order to show coord actions (default ascending order, 'desc' + for descending order, requires -info) + -password <arg> password for BASIC authentication + -poll <arg> poll Oozie until a job reaches a terminal state or a timeout + occurs + -refresh re-materialize the coordinator rerun actions (requires + -rerun) + -rerun <arg> rerun a job (coordinator requires -action or -date, bundle + requires -coordinator or -date) + -resume <arg> resume a job + -retries <arg> Get information of the retry attempts for a given workflow + action + -run run a job + -runjar <arg> generate and run job definition + -slachange <arg> Update sla param for jobs, supported param are should-start, + should-end, nominal-time and max-duration + -sladisable <arg> disables sla alerts for the job and its children + -slaenable <arg> enables sla alerts for the job and its children + -start <arg> start a job + -submit submit a job + -submitjar <arg> generate and submit job definition + -suspend <arg> suspend a job + -timeout <arg> timeout in minutes (default is 30, negative values indicate + no timeout, requires -poll) + -timezone <arg> use time zone with the specified ID (default GMT). + See 'oozie info -timezones' for a list + -update <arg> Update coord definition and properties + -username <arg> username for BASIC authentication + -validatejar <arg> generate and check job definition + -value <arg> new endtime/concurrency/pausetime value for changing a + coordinator job + -verbose verbose mode +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Oozie_jobs_operation_commands"></a>Oozie jobs operation commands</h3> + +<div> +<div> +<pre class="source">oozie jobs <OPTIONS> : jobs status + -auth <arg> select authentication type [SIMPLE|BASIC|KERBEROS] + -bulk <arg> key-value pairs to filter bulk jobs response. e.g. + bundle=<B>\;coordinators=<C>\;actionstatus=<S>\;startcreatedtime= + <SC>\;endcreatedtime=<EC>\;startscheduledtime=<SS>\;endscheduledt + ime=<ES>\; bundle, coordinators and actionstatus can be multiple + comma separated values. Bundle and coordinators can be id(s) or + appName(s) of those jobs. Specifying bundle is mandatory, other + params are optional + -doas <arg> doAs user, impersonates as the specified user + -filter <arg> + text=<*>\;user=<U>\;name=<N>\;group=<G>\;status=<S>\;frequency=<F + >\;unit=<M>\;startcreatedtime=<SC>\;endcreatedtime=<EC> + \;sortBy=<SB> + (text filter: matches partially with name and user or complete + match with job ID. Valid unit values are 'months', 'days', + 'hours' or 'minutes'. startcreatedtime, endcreatedtime: time of + format yyyy-MM-dd'T'HH:mm'Z'. Valid values for sortBy are + 'createdTime' or 'lastModifiedTime'.) + -jobtype <arg> job type ('Supported in Oozie-2.0 or later versions ONLY - + 'coordinator' or 'bundle' or 'wf'(default)) + -kill bulk kill operation + -len <arg> number of jobs (default '100') + -localtime use local time (same as passing your time zone to -timezone). + Overrides -timezone option + -offset <arg> jobs offset (default '1') + -oozie <arg> Oozie URL + -password <arg> password for BASIC authentication + -resume bulk resume operation + -suspend bulk suspend operation + -timezone <arg> use time zone with the specified ID (default GMT). + See 'oozie info -timezones' for a list + -username <arg> username for BASIC authentication + -verbose verbose mode +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Oozie_admin_operation_commands"></a>Oozie admin operation commands</h3> + +<div> +<div> +<pre class="source">oozie admin <OPTIONS> : admin operations + -auth <arg> select authentication type [SIMPLE|BASIC|KERBEROS] + -configuration show Oozie system configuration + -doas <arg> doAs user, impersonates as the specified user + -instrumentation show Oozie system instrumentation + -javasysprops show Oozie Java system properties + -metrics show Oozie system metrics + -oozie <arg> Oozie URL + -osenv show Oozie system OS environment + -password <arg> password for BASIC authentication + -purge <arg> purge old oozie workflow, coordinator and bundle records from + DB (parameter unit: day) + -queuedump show Oozie server queue elements + -servers list available Oozie servers (more than one only if HA is + enabled) + -shareliblist List available sharelib that can be specified in a workflow + action + -sharelibupdate Update server to use a newer version of sharelib + -status show the current system status + -systemmode <arg> Supported in Oozie-2.0 or later versions ONLY. Change oozie + system mode [NORMAL|NOWEBSERVICE|SAFEMODE] + -username <arg> username for BASIC authentication + -version show Oozie server build version +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Oozie_validate_command"></a>Oozie validate command</h3> + +<div> +<div> +<pre class="source">oozie validate <OPTIONS> <ARGS> : validate a workflow, coordinator, bundle XML file + -auth <arg> select authentication type [SIMPLE|BASIC|KERBEROS] + -oozie <arg> Oozie URL + -password <arg> password for BASIC authentication + -username <arg> username for BASIC authentication +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Oozie_SLA_operation_commands"></a>Oozie SLA operation commands</h3> + +<div> +<div> +<pre class="source">oozie sla <OPTIONS> : sla operations (Deprecated with Oozie 4.0) + -auth <arg> select authentication type [SIMPLE|BASIC|KERBEROS] + -filter <arg> filter of SLA events. e.g., jobid=<J>\;appname=<A> + -len <arg> number of results (default '100', max '1000') + -offset <arg> start offset (default '0') + -oozie <arg> Oozie URL + -password <arg> password for BASIC authentication + -username <arg> username for BASIC authentication +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Oozie_Pig_submit_command"></a>Oozie Pig submit command</h3> + +<div> +<div> +<pre class="source">oozie pig <OPTIONS> -X <ARGS> : submit a pig job, everything after '-X' are pass-through parameters to pig, any '-D' arguments + after '-X' are put in <configuration> + -auth <arg> select authentication type [SIMPLE|BASIC|KERBEROS] + -config <arg> job configuration file '.properties' + -D <property=value> set/override value for given property + -doas <arg> doAs user, impersonates as the specified user + -file <arg> pig script + -oozie <arg> Oozie URL + -P <property=value> set parameters for script + -password <arg> password for BASIC authentication + -username <arg> username for BASIC authentication +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Oozie_Hive_submit_command"></a>Oozie Hive submit command</h3> + +<div> +<div> +<pre class="source">oozie hive <OPTIONS> -X<ARGS> : submit a hive job, everything after '-X' are pass-through parameters to hive, + any '-D' arguments after '-X' are put in <configuration> + -auth <arg> select authentication type [SIMPLE|BASIC|KERBEROS] + -config <arg> job configuration file '.properties' + -D <property=value> set/override value for given property + -doas <arg> doAs user, impersonates as the specified user + -file <arg> hive script + -oozie <arg> Oozie URL + -P <property=value> set parameters for script + -password <arg> password for BASIC authentication + -username <arg> username for BASIC authentication +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Oozie_Sqoop_submit_command"></a>Oozie Sqoop submit command</h3> + +<div> +<div> +<pre class="source">oozie sqoop <OPTIONS> -X <ARGS> : submit a sqoop job, everything after '-X' are pass-through parameters to sqoop, any '-D' + arguments after '-X' are put in <configuration> + -auth <arg> select authentication type [SIMPLE|BASIC|KERBEROS] + -command <command> sqoop command + -config <arg> job configuration file '.properties' + -D <property=value> set/override value for given property + -doas <arg> doAs user, impersonates as the specified user + -oozie <arg> Oozie URL + -password <arg> password for BASIC authentication + -username <arg> username for BASIC authentication +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Oozie_info_command"></a>Oozie info command</h3> + +<div> +<div> +<pre class="source">oozie info <OPTIONS> : get more detailed info about specific topics + -timezones display a list of available time zones +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Oozie_MapReduce_job_command"></a>Oozie MapReduce job command</h3> + +<div> +<div> +<pre class="source">oozie mapreduce <OPTIONS> : submit a mapreduce job + -auth <arg> select authentication type [SIMPLE|BASIC|KERBEROS] + -config <arg> job configuration file '.properties' + -D <property=value> set/override value for given property + -doas <arg> doAs user, impersonates as the specified user + -oozie <arg> Oozie URL + -password <arg> password for BASIC authentication + -username <arg> username for BASIC authentication +</pre></div></div> +</div></div> +<div class="section"> +<h2><a name="Common_CLI_Options"></a>Common CLI Options</h2> +<div class="section"> +<h3><a name="Authentication"></a>Authentication</h3> +<p>The <tt>oozie</tt> CLI automatically perform authentication if the Oozie server requests it. By default it supports both pseudo/simple authentication and Kerberos HTTP SPNEGO authentication.</p> +<p>To perform a specific authentication, the <tt>auth</tt> option with authentication type requests Oozie client to run the specified authentication mechanism only. Oozie client provides three types <tt>simple</tt>, <tt>basic</tt> and <tt>kerberos</tt> which supports <tt>pseudo/simple</tt> and <tt>Kerberos</tt>. Basic authentication can be used, when the server sits behind a proxy which accepts basic authentication, and use Kerberos to authorize to Oozie itself.</p> +<p>For pseudo/simple authentication the <tt>oozie</tt> CLI uses the user name of the current OS user.</p> +<p>For Kerberos HTTP SPNEGO authentication the <tt>oozie</tt> CLI uses the default principal for the OS Kerberos cache (normally the principal that did <tt>kinit</tt>).</p> +<p>Oozie uses Apache Hadoop-Auth (Java HTTP SPNEGO) library for authentication. This library can be extended to support other authentication mechanisms.</p> +<p>Once authentication is performed successfully the received authentication token is cached in the user home directory in the <tt>.oozie-auth-token</tt> file with owner-only permissions. Subsequent requests reuse the cached token while valid.</p> +<p>The use of the cache file can be disabled by invoking the <tt>oozie</tt> CLI with the <tt>-Doozie.auth.token.cache</tt>=false option.</p> +<p>To use an custom authentication mechanism, a Hadoop-Auth <tt>Authenticator</tt> implementation must be specified with the <tt>-Dauthenticator.class=CLASS</tt> option.</p></div> +<div class="section"> +<h3><a name="Impersonation_doAs"></a>Impersonation, doAs</h3> +<p>The <tt>-doas</tt> option allows the current user to impersonate other users when interacting with the Oozie system. The current user must be configured as a proxyuser in the Oozie system. The proxyuser configuration may restrict from which hosts a user may impersonate users, as well as users of which groups can be impersonated.</p></div> +<div class="section"> +<h3><a name="Oozie_URL"></a>Oozie URL</h3> +<p>All <tt>oozie</tt> CLI sub-commands expect the <tt>-oozie OOZIE_URL</tt> option indicating the URL of the Oozie system to run the command against. If the OOZIE_URL environment variable has not been set, <tt>oozie</tt> will use the default URL specified in oozie-client-env.sh (equivalent to <tt>!http://$(hostname -f):11000/oozie</tt>).</p> +<p>If the <tt>-oozie</tt> option is not specified, the <tt>oozie</tt> CLI will look for the <tt>OOZIE_URL</tt> environment variable and uses it if set.</p> +<p>If the option is not provided and the environment variable is not set, the <tt>oozie</tt> CLI will fail.</p></div> +<div class="section"> +<h3><a name="Time_zone"></a>Time zone</h3> +<p>The <tt>-timezone TIME_ZONE_ID</tt> option in the <tt>job</tt> and <tt>jobs</tt> sub-commands allows you to specify the time zone to use in the output of those sub-commands. The <tt>TIME_ZONE_ID</tt> should be one of the standard Java Time Zone IDs. You can get a list of the available time zones with the command <tt>oozie info -timezones</tt>.</p> +<p>If the <tt>-localtime</tt> option is used, it will cause Oozie to use whatever the time zone is of the machine. If both <tt>-localtime</tt> and <tt>-timezone TIME_ZONE_ID</tt> are used, the <tt>-localtime</tt> option will override the <tt>-timezone TIME_ZONE_ID</tt> option. If neither option is given, Oozie will look for the <tt>OOZIE_TIMEZONE</tt> environment variable and uses it if set. If neither option is given and the environment variable is not set, or if Oozie is given an invalid time zone, it will use GMT.</p></div> +<div class="section"> +<h3><a name="Debug_Mode"></a>Debug Mode</h3> +<p>If you export <tt>OOZIE_DEBUG=1</tt> then the Oozie CLI will output the Web Services API details used by any commands you execute. This is useful for debugging purposes to or see how the Oozie CLI works with the WS API.</p></div> +<div class="section"> +<h3><a name="CLI_retry"></a>CLI retry</h3> +<p>Oozie CLI retries connection to Oozie servers for transparent high availability failover when one of the Oozie servers go down. <tt>Oozie</tt> CLI command will retry for all commands in case of ConnectException. In case of SocketException, all commands except <tt>PUT</tt> and <tt>POST</tt> will have retry logic. All job submit are POST call, examples of PUT and POST commands can be find out from <a href="WebServicesAPI.html">WebServicesAPI</a>. Retry count can be configured with system property <tt>oozie.connection.retry.count</tt>. Default count is 4.</p></div></div> +<div class="section"> +<h2><a name="Job_Operations"></a>Job Operations</h2> +<div class="section"> +<h3><a name="Submitting_a_Workflow_Coordinator_or_Bundle_Job"></a>Submitting a Workflow, Coordinator or Bundle Job</h3> +<ul> + +<li>Submitting bundle feature is only supported in Oozie 3.0 or later. Similarly, all bundle operation features below are only supported in Oozie 3.0 or later.</li> +</ul> +<p>Example:</p> + +<div> +<div> +<pre class="source">$ oozie job -oozie http://localhost:11000/oozie -config job.properties -submit +. +job: 14-20090525161321-oozie-joe +</pre></div></div> + +<p>The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML Configuration file (.xml). This file must be specified with the <tt>-config</tt> option.</p> +<p>The workflow application path must be specified in the file with the <tt>oozie.wf.application.path</tt> property. The coordinator application path must be specified in the file with the <tt>oozie.coord.application.path</tt> property.The bundle application path must be specified in the file with the <tt>oozie.bundle.application.path</tt> property. Specified path must be an HDFS path.</p> +<p>The job will be created, but it will not be started, it will be in <tt>PREP</tt> status.</p></div> +<div class="section"> +<h3><a name="Starting_a_Workflow_Coordinator_or_Bundle_Job"></a>Starting a Workflow, Coordinator or Bundle Job</h3> +<p>Example:</p> + +<div> +<div> +<pre class="source">$ oozie job -oozie http://localhost:11000/oozie -start 14-20090525161321-oozie-joe +</pre></div></div> + +<p>The <tt>start</tt> option starts a previously submitted workflow job or bundle job that is in <tt>PREP</tt> status.</p> +<p>After the command is executed the workflow job will be in <tt>RUNNING</tt> status and bundle job will be in <tt>RUNNING</tt>status.</p> +<p>A coordinator job does not support the <tt>start</tt> action. It will show the following error message when trying to start it via the CLI:</p> + +<div> +<div> +<pre class="source">Error: E0303 : E0303: Invalid parameter value, [action] = [start] +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Running_a_Workflow_Coordinator_or_Bundle_Job"></a>Running a Workflow, Coordinator or Bundle Job</h3> +<p>Example:</p> + +<div> +<div> +<pre class="source">$ oozie job -oozie http://localhost:11000/oozie -config job.properties -run +. +job: 15-20090525161321-oozie-joe +</pre></div></div> + +<p>The <tt>run</tt> option creates and starts a workflow job, coordinator job or bundle job.</p> +<p>The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML Configuration file (.xml). This file must be specified with the <tt>-config</tt> option.</p> +<p>The workflow application path must be specified in the file with the <tt>oozie.wf.application.path</tt> property. The coordinator application path must be specified in the file with the <tt>oozie.coord.application.path</tt> property. The bundle application path must be specified in the file with the <tt>oozie.bundle.application.path</tt> property.The specified path must be an HDFS path.</p> +<p>The job will be created and it will started, the job will be in <tt>RUNNING</tt> status.</p></div> +<div class="section"> +<h3><a name="Suspending_a_Workflow_Coordinator_or_Bundle_Job"></a>Suspending a Workflow, Coordinator or Bundle Job</h3> +<p>Example:</p> + +<div> +<div> +<pre class="source">$ oozie job -oozie http://localhost:11000/oozie -suspend 14-20090525161321-oozie-joe +</pre></div></div> + +<p>The <tt>suspend</tt> option suspends a workflow job in <tt>RUNNING</tt> status. After the command is executed the workflow job will be in <tt>SUSPENDED</tt> status.</p> +<p>The <tt>suspend</tt> option suspends a coordinator/bundle job in <tt>RUNNING</tt>, <tt>RUNNINGWITHERROR</tt> or <tt>PREP</tt> status. When the coordinator job is suspended, running coordinator actions will stay in running and the workflows will be suspended. If the coordinator job is in <tt>RUNNING</tt>status, it will transit to <tt>SUSPENDED</tt>status; if it is in <tt>RUNNINGWITHERROR</tt>status, it will transit to <tt>SUSPENDEDWITHERROR</tt>; if it is in <tt>PREP</tt>status, it will transit to <tt>PREPSUSPENDED</tt>status.</p> +<p>When the bundle job is suspended, running coordinators will be suspended. If the bundle job is in <tt>RUNNING</tt>status, it will transit to <tt>SUSPENDED</tt>status; if it is in <tt>RUNNINGWITHERROR</tt>status, it will transit to <tt>SUSPENDEDWITHERROR</tt>; if it is in <tt>PREP</tt>status, it will transit to <tt>PREPSUSPENDED</tt>status.</p></div> +<div class="section"> +<h3><a name="Resuming_a_Workflow_Coordinator_or_Bundle_Job"></a>Resuming a Workflow, Coordinator or Bundle Job</h3> +<p>Example:</p> + +<div> +<div> +<pre class="source">$ oozie job -oozie http://localhost:11000/oozie -resume 14-20090525161321-oozie-joe +</pre></div></div> + +<p>The <tt>resume</tt> option resumes a workflow job in <tt>SUSPENDED</tt> status.</p> +<p>After the command is executed the workflow job will be in <tt>RUNNING</tt> status.</p> +<p>The <tt>suspend</tt> option suspends a coordinator/bundle job in <tt>SUSPENDED</tt>, <tt>SUSPENDEDWITHERROR</tt> or <tt>PREPSUSPENDED</tt> status. If the coordinator job is in <tt>SUSPENDED</tt>status, it will transit to <tt>RUNNING</tt>status; if it is in <tt>SUSPENDEDWITHERROR</tt>status, it will transit to <tt>RUNNINGWITHERROR</tt>; if it is in <tt>PREPSUSPENDED</tt>status, it will transit to <tt>PREP</tt>status.</p> +<p>When the coordinator job is resumed it will create all the coordinator actions that should have been created during the time it was suspended, actions will not be lost, they will delayed.</p> +<p>When the bundle job is resumed, suspended coordinators will resume running. If the bundle job is in <tt>SUSPENDED</tt>status, it will transit to <tt>RUNNING</tt>status; if it is in <tt>SUSPENDEDWITHERROR</tt>status, it will transit to <tt>RUNNINGWITHERROR</tt>; if it is in <tt>PREPSUSPENDED</tt>status, it will transit to <tt>PREP</tt>status.</p></div> +<div class="section"> +<h3><a name="Killing_a_Workflow_Coordinator_or_Bundle_Job"></a>Killing a Workflow, Coordinator or Bundle Job</h3> +<p>Example:</p> + +<div> +<div> +<pre class="source">$ oozie job -oozie http://localhost:11000/oozie -kill 14-20090525161321-oozie-joe +</pre></div></div> + +<p>The <tt>kill</tt> option kills a workflow job in <tt>PREP</tt>, <tt>SUSPENDED</tt> or <tt>RUNNING</tt> status and a coordinator/bundle job in <tt>PREP</tt>, <tt>RUNNING</tt>, <tt>PREPSUSPENDED</tt>, <tt>SUSPENDED</tt>, <tt>PREPPAUSED</tt>, or <tt>PAUSED</tt> status.</p> +<p>After the command is executed the job will be in <tt>KILLED</tt> status.</p></div> +<div class="section"> +<h3><a name="Killing_a_Coordinator_Action_or_Multiple_Actions"></a>Killing a Coordinator Action or Multiple Actions</h3> +<p>Example:</p> + +<div> +<div> +<pre class="source">$oozie job -kill <coord_Job_id> [-action 1, 3-4, 7-40] [-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z] +</pre></div></div> + +<ul> + +<li>The <tt>kill</tt> option here for a range of coordinator actions kills a non-terminal (<tt>RUNNING</tt>, <tt>WAITING</tt>, <tt>READY</tt>, <tt>SUSPENDED</tt>) coordinator action when coordinator job is not in <tt>FAILED</tt> or <tt>KILLED</tt> state.</li> +<li>Either -action or -date should be given.</li> +<li>If neither -action nor -date is given, the exception will be thrown. Also if BOTH -action and -date are given, an error will be thrown.</li> +<li>Multiple ranges can be used in -action or -date. See the above example.</li> +<li>If one of the actions in the given list of -action is already in terminal state, the output of this command will only include the other actions.</li> +<li>The dates specified in -date must be UTC.</li> +<li>Single date specified in -date must be able to find an action with matched nominal time to be effective.</li> +<li>After the command is executed the killed coordinator action will have <tt>KILLED</tt> status.</li> +</ul></div> +<div class="section"> +<h3><a name="Changing_endtimeconcurrencypausetimestatus_of_a_Coordinator_Job"></a>Changing endtime/concurrency/pausetime/status of a Coordinator Job</h3> +<p>Example:</p> + +<div> +<div> +<pre class="source">$ oozie job -oozie http://localhost:11000/oozie -change 14-20090525161321-oozie-joe -value endtime=2011-12-01T05:00Z\;concurrency=100\;2011-10-01T05:00Z +$ oozie job -oozie http://localhost:11000/oozie -change 0000001-140321155112907-oozie-puru-C -value status=RUNNING +</pre></div></div> + +<p>The <tt>endtime/concurrency/pausetime</tt> option changes a coordinator job that is not in <tt>KILLED</tt> status.</p> +<p>Valid value names are:</p> +<ul> + +<li>endtime: the end time of the coordinator job.</li> +<li>concurrency: the concurrency of the coordinator job.</li> +<li>pausetime: the pause time of the coordinator job.</li> +<li>status: new status for coordinator job.</li> +</ul> +<p>Conditions and usage:</p> +<ul> + +<li>Repeated value names are not allowed.</li> +<li>New end time should not be before job’s start time and last action time.</li> +<li>If end time is before job start time and if the job has not materialized any actions, then job status is changed to SUCCEEDED.</li> +<li>Currently status only takes RUNNING and can be used to change the status of FAILED, KILLED, IGNORED coordinator job to RUNNING and resuming materialization. This status change command does not affect the status of already materialized actions in the coordinator. If there are FAILED, KILLED or IGNORED coordinator actions they have to be rerun separately.</li> +<li>New concurrency value has to be a valid integer.</li> +<li>All lookahead actions which are in WAITING/READY state will be revoked according to the new pause/end time. If any action after new pause/end time is not in WAITING/READY state, an exception will be thrown.</li> +<li>Also empty string "" can be used to reset pause time to none.</li> +<li>Endtime/concurrency/pausetime of IGNORED Job cannot be changed.</li> +</ul> +<p>After the command is executed the job’s end time, concurrency or pause time should be changed. If an already-succeeded job changes its end time, its status will become running.</p></div> +<div class="section"> +<h3><a name="Changing_endtimepausetime_of_a_Bundle_Job"></a>Changing endtime/pausetime of a Bundle Job</h3> +<p>Example:</p> + +<div> +<div> +<pre class="source">$ oozie job -oozie http://localhost:11000/oozie -change 14-20090525161321-oozie-joe -value pausetime=2011-12-01T05:00Z +</pre></div></div> + +<p>The <tt>change</tt> option changes a bundle job that is not in <tt>KILLED</tt> status.</p> +<p>Valid value names are:</p> +<ul> + +<li>pausetime: the pause time of the bundle job.</li> +<li>endtime: the end time of the bundle job.</li> +</ul> +<p>Repeated value names are not allowed. An empty string "" can be used to reset pause time to none. New end time should not be before job’s kickoff time.</p> +<p>Bundle will execute pause/end date change command on each coordinator job. Refer conditions and usage section of coordinator change command for more details <a href="DG_CommandLineTool.html#Changing_endtimeconcurrencypausetimestatus_of_a_Coordinator_Job">Coordinator job change command</a>.</p></div> +<div class="section"> +<h3><a name="Rerunning_a_Workflow_Job"></a>Rerunning a Workflow Job</h3> +<p>Example:</p> + +<div> +<div> +<pre class="source">$ oozie job -oozie http://localhost:11000/oozie -config job.properties -rerun 14-20090525161321-oozie-joe +</pre></div></div> + +<p>The <tt>rerun</tt> option reruns a completed ( <tt>SUCCEEDED</tt>, <tt>FAILED</tt> or <tt>KILLED</tt> ) job skipping the specified nodes.</p> +<p>The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML Configuration file (.xml). This file must be specified with the <tt>-config</tt> option.</p> +<p>The workflow application path must be specified in the file with the <tt>oozie.wf.application.path</tt> property. The specified path must be an HDFS path.</p> +<p>The list of nodes to skipped must be provided in the <tt>oozie.wf.rerun.skip.nodes</tt> property separated by commas.</p> +<p>After the command is executed the job will be in <tt>RUNNING</tt> status.</p> +<p>Refer to the <a href="DG_WorkflowReRun.html">Rerunning Workflow Jobs</a> for details on rerun.</p></div> +<div class="section"> +<h3><a name="Rerunning_a_Coordinator_Action_or_Multiple_Actions"></a>Rerunning a Coordinator Action or Multiple Actions</h3> +<p>Example:</p> + +<div> +<div> +<pre class="source">$oozie job -rerun <coord_Job_id> [-nocleanup] [-refresh] [-failed] [-config <arg>] +[-action 1, 3-4, 7-40] (-action or -date is required to rerun.) +[-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z] +(if neither -action nor -date is given, the exception will be thrown.) +</pre></div></div> + +<p>The <tt>rerun</tt> option reruns a terminated (<tt>TIMEDOUT</tt>, <tt>SUCCEEDED</tt>, <tt>KILLED</tt>, <tt>FAILED</tt>, <tt>IGNORED</tt>) coordinator action when coordinator job is not in <tt>FAILED</tt> or <tt>KILLED</tt> state.</p> +<p>After the command is executed the rerun coordinator action will be in <tt>WAITING</tt> status.</p> +<p>Refer to the <a href="DG_CoordinatorRerun.html">Rerunning Coordinator Actions</a> for details on rerun.</p></div> +<div class="section"> +<h3><a name="Rerunning_a_Bundle_Job"></a>Rerunning a Bundle Job</h3> +<p>Example:</p> + +<div> +<div> +<pre class="source">$oozie job -rerun <bundle_Job_id> [-nocleanup] [-refresh] +[-coordinator c1, c3, c4] (-coordinator or -date is required to rerun.) +[-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z] +(if neither -coordinator nor -date is given, the exception will be thrown.) +</pre></div></div> + +<p>The <tt>rerun</tt> option reruns coordinator actions belonging to specified coordinators within the specified date range.</p> +<p>After the command is executed the rerun coordinator action will be in <tt>WAITING</tt> status.</p></div> +<div class="section"> +<h3><a name="Checking_the_Information_and_Status_of_a_Workflow_Coordinator_or_Bundle_Job_or_a_Coordinator_Action"></a>Checking the Information and Status of a Workflow, Coordinator or Bundle Job or a Coordinator Action</h3> +<p>Example:</p> + +<div> +<div> +<pre class="source">$ oozie job -oozie http://localhost:11000/oozie -info 14-20090525161321-oozie-joe +. +.---------------------------------------------------------------------------------------------------------------------------------------------------------------- +Workflow Name : map-reduce-wf +App Path : hdfs://localhost:8020/user/joe/workflows/map-reduce +Status : SUCCEEDED +Run : 0 +User : joe +Group : users +Created : 2009-05-26 05:01 +0000 +Started : 2009-05-26 05:01 +0000 +Ended : 2009-05-26 05:01 +0000 +Actions +.---------------------------------------------------------------------------------------------------------------------------------------------------------------- +Action Name Type Status Transition External Id External Status Error Code Start End +.---------------------------------------------------------------------------------------------------------------------------------------------------------------- +hadoop1 map-reduce OK end job_200904281535_0254 SUCCEEDED - 2009-05-26 05:01 +0000 2009-05-26 05:01 +0000 +.---------------------------------------------------------------------------------------------------------------------------------------------------------------- +</pre></div></div> + +<p>The <tt>info</tt> option can display information about a workflow job or coordinator job or coordinator action. The <tt>info</tt> option for a Coordinator job will retrieve the Coordinator actions ordered by nominal time. However, the <tt>info</tt> command may timeout if the number of Coordinator actions are very high. In that case, <tt>info</tt> should be used with <tt>offset</tt> and <tt>len</tt> option.</p> +<p>The <tt>offset</tt> and <tt>len</tt> option should be used for pagination. offset determines the start offset of the action returned among all the actions that matched the filter criteria. len determines number of actions to be returned.</p> +<p>The <tt>localtime</tt> option displays times in local time, if not specified times are displayed in GMT.</p> +<p>The <tt>filter</tt> option can be used to filter coordinator actions based on some criteria. The filter option syntax is: <tt><key><comparator><value>[;<key><comparator><value>]*</tt>. (Note escape <tt>\</tt> needed before semicolon to specify multiple names for filter in shell) key: status or nominalTime comparator: <tt>, !</tt>, <, <<tt>, >, ></tt> value: valid status like SUCCEEDED, KILLED, RUNNING etc. Only <tt>and !</tt> apply for status value for nominalTime is valid date of the format yyyy-MM-dd’T’HH:mm’Z’ (like 2014-06-01T00:00Z)</p> +<p>Multiple values must be specified as different name value pairs. The query is formed by doing AND of all conditions, with the exception of = which uses OR if there are multiple values for the same key. For example, filter ’status<tt>RUNNING;status</tt>WAITING;nominalTime><tt>2014-06-01T00:00Z' maps to query (status</tt> RUNNING OR status = WAITING) AND nominalTime ><tt>2014-06-01T00:00Z which returns all waiting or running actions with nominalTime ></tt> 2014-06-01T00:00Z.</p> +<p>Currently, the filter option can be used only with an <tt>info</tt> option on Coordinator job.</p> +<p>The <tt>verbose</tt> option gives more detailed information for all the actions, if checking for workflow job or coordinator job. An example below shows how the <tt>verbose</tt> option can be used to gather action statistics information for a job:</p> + +<div> +<div> +<pre class="source">$ oozie job -oozie http://localhost:11000/oozie -info 0000001-111219170928042-oozie-para-W@mr-node -verbose +ID : 0000001-111219170928042-oozie-para-W@mr-node +------------------------------------------------------------------------------------------------------------------------------------ +Console URL : http://localhost:50030/jobdetails.jsp?jobid=job_201112191708_0006 +Error Code : - +Error Message : - +External ID : job_201112191708_0006 +External Status : SUCCEEDED +Name : mr-node +Retries : 0 +Tracker URI : localhost:8021 +Type : map-reduce +Started : 2011-12-20 01:12 +Status : OK +Ended : 2011-12-20 01:12 +External Stats : {"org.apache.hadoop.mapred.JobInProgress$Counter":{"TOTAL_LAUNCHED_REDUCES":1,"TOTAL_LAUNCHED_MAPS":1,"DATA_LOCAL_MAPS":1},"ACTION_TYPE":"MAP_REDUCE","FileSystemCounters":{"FILE_BYTES_READ":1746,"HDFS_BYTES_READ":1409,"FILE_BYTES_WRITTEN":3524,"HDFS_BYTES_WRITTEN":1547},"org.apache.hadoop.mapred.Task$Counter":{"REDUCE_INPUT_GROUPS":33,"COMBINE_OUTPUT_RECORDS":0,"MAP_INPUT_RECORDS":33,"REDUCE_SHUFFLE_BYTES":0,"REDUCE_OUTPUT_RECORDS":33,"SPILLED_RECORDS":66,"MAP_OUTPUT_BYTES":1674,"MAP_INPUT_BYTES":1409,"MAP_OUTPUT_RECORDS":33,"COMBINE_INPUT_RECORDS":0,"REDUCE_INPUT_RECORDS":33}} +External ChildIDs : null +------------------------------------------------------------------------------------------------------------------------------------ +</pre></div></div> + +<p>The two fields External Stats and External ChildIDs display the action statistics information (that includes counter information in case of MR action and PigStats information in case of a pig action) and child ids of the given job.</p> +<p>Note that the user can turn on/off External Stats by specifying the property <i>oozie.action.external.stats.write</i> as <i>true</i> or <i>false</i> in workflow.xml. By default, it is set to false (not to collect External Stats). External ChildIDs will always be stored.</p></div> +<div class="section"> +<h3><a name="Listing_all_the_Workflows_for_a_Coordinator_Action"></a>Listing all the Workflows for a Coordinator Action</h3> +<p>A coordinator action kicks off different workflows for its original run and all subsequent reruns. Getting a list of those workflow ids is a useful tool to keep track of your actions’ runs and to go debug the workflow job logs if required. Along with ids, it also lists their statuses, and start and end times for quick reference.</p> +<p>This is achieved by using the Coordinator Action info command and specifying a flag <b><tt>allruns</tt></b> along with the <tt>info</tt> command.</p> + +<div> +<div> +<pre class="source">$ oozie job -info 0000001-111219170928042-oozie-joe-C@1 -allruns -oozie http://localhost:11000/oozie +. +Job ID Status Started Ended +.---------------------------------------------------------------------------------------------------- +0000001-140324163709596-oozie-joe-W SUCCEEDED 2014-03-24 23:40 GMT 2014-03-24 23:40 GMT +.---------------------------------------------------------------------------------------------------- +0000000-140324164318985-oozie-joe-W SUCCEEDED 2014-03-24 23:44 GMT 2014-03-24 23:44 GMT +.---------------------------------------------------------------------------------------------------- +0000001-140324164318985-oozie-joe-W SUCCEEDED 2014-03-24 23:44 GMT 2014-03-24 23:44 GMT +.---------------------------------------------------------------------------------------------------- +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Listing_all_retry_attempts_of_a_workflow_action"></a>Listing all retry attempts of a workflow action</h3> +<p>When retry-max is specified for an action in the workflow definition, and there is a failure, it will be retried till it succeeds or retry-max attempts are exhausted. To get information on all the retry attempts, <tt>-retries</tt> command can be used.</p> + +<div> +<div> +<pre class="source">$ oozie job -retries 0000000-161212175234862-oozie-puru-W@pig-node -oozie http://localhost:11000/oozie + +ID : 0000000-161212175234862-oozie-puru-W@pig-node +------------------------------------------------------------------------------------------------------------------------------------ +Attempt : 1 +Start Time : Tue, 13 Dec 2016 01:56:24 GMT +End Time : Tue, 13 Dec 2016 01:56:27 GMT +Console URL : http://localhost:50030/jobdetails.jsp?jobid=job_201612051339_2650 +------------------------------------------------------------------------------------------------------------------------------------ +Attempt : 2 +Start Time : Tue, 13 Dec 2016 01:56:24 GMT +End Time : Tue, 13 Dec 2016 01:56:27 GMT +Console URL : http://localhost:50030/jobdetails.jsp?jobid=job_201612051339_2650 +------------------------------------------------------------------------------------------------------------------------------------ +Attempt : 3 +Start Time : Tue, 13 Dec 2016 01:56:24 GMT +End Time : Tue, 13 Dec 2016 01:56:27 GMT +Console URL : http://localhost:50030/jobdetails.jsp?jobid=job_201612051339_2650 +------------------------------------------------------------------------------------------------------------------------------------ +$ +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Checking_the_xml_definition_of_a_Workflow_Coordinator_or_Bundle_Job"></a>Checking the xml definition of a Workflow, Coordinator or Bundle Job</h3> +<p>Example:</p> + +<div> +<div> +<pre class="source">$ oozie job -oozie http://localhost:11000/oozie -definition 14-20090525161321-oozie-joe +. +<workflow-app xmlns="uri:oozie:workflow:0.2" name="sm3-segment-2413"> + <start to="p0"/> + <action name="p0"> + </action> + <end name="end"/> +</workflow-app> + +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Checking_the_server_logs_of_a_Workflow_Coordinator_or_Bundle_Job"></a>Checking the server logs of a Workflow, Coordinator or Bundle Job</h3> +<p>Example:</p> + +<div> +<div> +<pre class="source">$ oozie job -oozie http://localhost:11000/oozie -log 14-20090525161321-oozie-joe + +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Checking_the_server_error_logs_of_a_Workflow_Coordinator_or_Bundle_Job"></a>Checking the server error logs of a Workflow, Coordinator or Bundle Job</h3> +<p>Example:</p> + +<div> +<div> +<pre class="source">$ oozie job -oozie http://localhost:11000/oozie -errorlog 0000000-150121110331712-oozie-puru-B +2015-01-21 11:33:29,090 WARN CoordSubmitXCommand:523 - SERVER[-] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-150121110331712-oozie-puru-B] ACTION[] SAXException : +org.xml.sax.SAXParseException; lineNumber: 20; columnNumber: 22; cvc-complex-type.2.4.a: Invalid content was found starting with element 'concurrency'. One of '{"uri:oozie:coordinator:0.2":controls, "uri:oozie:coordinator:0.2":datasets, "uri:oozie:coordinator:0.2":input-events, "uri:oozie:coordinator:0.2":output-events, "uri:oozie:coordinator:0.2":action}' is expected. + at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source) + at org.apache.xerces.util.ErrorHandlerWrapper.error(Unknown Source) + at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) + at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) + at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) + at org.apache.xerces.impl.xs.XMLSchemaValidator$XSIErrorReporter.reportError(Unknown Source) + at org.apache.xerces.impl.xs.XMLSchemaValidator.reportSchemaError(Unknown Source) + at org.apache.xerces.impl.xs.XMLSchemaValidator.handleStartElement(Unknown Source) + at org.apache.xerces.impl.xs.XMLSchemaValidator.startElement(Unknown Source) + at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source) + at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) + at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) + at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) + at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) + at org.apache.xerces.jaxp.validation.StreamValidatorHelper.validate(Unknown Source) + at org.apache.xerces.jaxp.validation.ValidatorImpl.validate(Unknown Source) +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Checking_the_audit_logs_of_a_Workflow_Coordinator_or_Bundle_Job"></a>Checking the audit logs of a Workflow, Coordinator or Bundle Job</h3> +<p>Example:</p> + +<div> +<div> +<pre class="source">$ oozie job -oozie http://localhost:11000/oozie -auditlog 0000000-150322000230582-oozie-puru-C +2015-03-22 00:04:35,494 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [start], PARAMETER [null], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null] +2015-03-22 00:05:13,823 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null] +2015-03-22 00:06:59,561 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null] +2015-03-22 23:22:20,012 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null] +2015-03-22 23:28:48,218 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [resume], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null] +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Checking_the_server_logs_for_particular_actions_of_a_Coordinator_Job"></a>Checking the server logs for particular actions of a Coordinator Job</h3> +<p>Example:</p> + +<div> +<div> +<pre class="source">$ oozie job -log <coord_job_id> [-action 1, 3-4, 7-40] (-action is optional.) + +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Filtering_the_server_logs_with_logfilter_options"></a>Filtering the server logs with logfilter options</h3> +<p>User can provide multiple option to filter logs using -logfilter opt1=val1;opt2=val1;opt3=val1. This can be used to fetch only just logs of interest faster as fetching Oozie server logs is slow due to the overhead of pattern matching. Available options are:</p> +<ul> + +<li>recent: Specify recent hours/min of logs to scan. The recent offset specified is taken relative to the <tt>end</tt> time specified, job end time or the current system time if the job is still running in that order of precedence. For eg: recent=3h or recent=30m will fetch logs starting 3 hours/30 minutes before the end time and up to the end time. H/h is used to denote hour and M/m is used to denote minutes. If no suffix is specified default is hours.</li> +<li>start: Start time for scanning logs. Default is start time of the job. User can provide a valid date or offset similar to <tt>recent</tt> option. Valid date formats are “yyyy-MM-dd’T’HH:mm’Z’” and “yyyy-MM-dd HH:mm:ss,SSS”. When an offset is specified, it is calculated relative to the start time of the job. For eg: start=2h will fetch logs starting 2 hours after the job was started.</li> +<li>end: End time for scanning logs. Default is end time of the job or current system time if the job is still running. User can provide a valid date or offset similar to <tt>start</tt> option. When an offset is specified, it is calculated relative to start time i.e job start time . For eg: end=2h will fetch logs from start time and start time plus 2 hours.</li> +<li>loglevel : Multiple log levels separated by “|” can be specified. Supported log levels are ALL, DEBUG, ERROR, INFO, TRACE, WARN, FATAL.</li> +<li>text: String to search in logs.</li> +<li>limit : Limit number of line to be searched. Log search will end when when n lines(excluding stack-trace) have been matched.</li> +<li>debug : Prints debug information on the log files, time ranges and patterns being searched for. Can be used to debug if expected logs are not shown with the filter options provided.</li> +</ul> +<p>Examples. Searching log with log level ERROR or WARN will only give log with Error and Warning (with stack-trace) only. This will be useful if job has failed and user want to find error logs with exception.</p> + +<div> +<div> +<pre class="source">$ ./oozie job -log 0000006-140319184715726-oozie-puru-W -logfilter loglevel=WARN\;limit=3 -oozie http://localhost:11000/oozie/ +2014-03-20 10:01:52,977 WARN ActionStartXCommand:542 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000006-140319184715726-oozie-puru-W] ACTION[0000006-140319184715726-oozie-puru-W@:start:] [***0000006-140319184715726-oozie-puru-W@:start:***]Action status=DONE +2014-03-20 10:01:52,977 WARN ActionStartXCommand:542 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000006-140319184715726-oozie-puru-W] ACTION[0000006-140319184715726-oozie-puru-W@:start:] [***0000006-140319184715726-oozie-puru-W@:start:***]Action updated in DB! +2014-03-20 10:01:53,189 WARN ActionStartXCommand:542 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000006-140319184715726-oozie-puru-W] ACTION[0000006-140319184715726-oozie-puru-W@mr-node-1] Error starting action [mr-node-1]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: java.io.IOException: java.io.IOException: Queue "aaadefault" does not exist + at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3615) + at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3561) + at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) + at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) + at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) + at java.lang.reflect.Method.invoke(Method.java:597) + at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587) + at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432) + at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428) + at java.security.AccessController.doPrivileged(Native Method) + at javax.security.auth.Subject.doAs(Subject.java:394) + at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) + at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426) +Caused by: java.io.IOException: Queue "aaadefault" does not exist + at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:433) + at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3613) + ... 12 more +$ +</pre></div></div> + +<p>Search with specific text and recent option.</p> + +<div> +<div> +<pre class="source">$ ./oozie job -log 0000003-140319184715726-oozie-puru-C -logfilter text=Missing\;limit=4\;recent=1h -oozie http://localhost:11000/oozie/ +2014-03-20 09:59:50,329 INFO CoordActionInputCheckXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-140319184715726-oozie-puru-C] ACTION[0000003-140319184715726-oozie-puru-C@1] [0000003-140319184715726-oozie-puru-C@1]::CoordActionInputCheck:: Missing deps:hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/ +2014-03-20 09:59:50,330 INFO CoordActionInputCheckXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-140319184715726-oozie-puru-C] ACTION[0000003-140319184715726-oozie-puru-C@1] [0000003-140319184715726-oozie-puru-C@1]::ActionInputCheck:: In checkListOfPaths: hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/ is Missing. +2014-03-20 10:02:19,087 INFO CoordActionInputCheckXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-140319184715726-oozie-puru-C] ACTION[0000003-140319184715726-oozie-puru-C@2] [0000003-140319184715726-oozie-puru-C@2]::CoordActionInputCheck:: Missing deps:hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/ +2014-03-20 10:02:19,088 INFO CoordActionInputCheckXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-140319184715726-oozie-puru-C] ACTION[0000003-140319184715726-oozie-puru-C@2] [0000003-140319184715726-oozie-puru-C@2]::ActionInputCheck:: In checkListOfPaths: hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/ is Missing. +$ +</pre></div></div> + +<p>Search example with specific date range.</p> + +<div> +<div> +<pre class="source">$ ./oozie job -log 0000003-140319184715726-oozie-puru-C -logfilter "start=2014-03-20 10:00:57,063;end=2014-03-20 10:10:57,063" -oozie http://localhost:11000/oozie/ +2014-03-20 10:00:57,063 INFO CoordActionUpdateXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-140319184715726-oozie-puru-C] ACTION[0000003-140319184715726-oozie-puru-C@1] Updating Coordinator action id :0000003-140319184715726-oozie-puru-C@1 status to KILLED, pending = 0 +2014-03-20 10:02:18,967 INFO CoordMaterializeTransitionXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[aggregator-coord] JOB[0000003-140319184715726-oozie-puru-C] ACTION[-] materialize actions for tz=Coordinated Universal Time, + start=Thu Dec 31 18:00:00 PST 2009, end=Thu Dec 31 19:00:00 PST 2009, + timeUnit 12, + frequency :60:MINUTE, + lastActionNumber 1 +2014-03-20 10:02:18,971 WARN CoordELFunctions:542 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[aggregator-coord] JOB[0000003-140319184715726-oozie-puru-C] ACTION[-] If the initial instance of the dataset is later than the current-instance specified, such as coord:current(-200) in this case, an empty string is returned. This means that no data is available at the current-instance specified by the user and the user could try modifying his initial-instance to an earlier time. +2014-03-20 10:02:18,975 INFO CoordMaterializeTransitionXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[aggregator-coord] JOB[0000003-140319184715726-oozie-puru-C] ACTION[-] [0000003-140319184715726-oozie-puru-C]: all actions have been materialized, set pending to true +2014-03-20 10:02:18,975 INFO CoordMaterializeTransitionXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[aggregator-coord] JOB[0000003-140319184715726-oozie-puru-C] ACTION[-] Coord Job status updated to = RUNNING +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Dryrun_of_Coordinator_Job"></a>Dryrun of Coordinator Job</h3> +<ul> + +<li>This feature is only supported in Oozie 2.0 or later.</li> +</ul> +<p>Example:</p> + +<div> +<div> +<pre class="source">$ oozie job -oozie http://localhost:11000/oozie -dryrun -config job.properties +***coordJob after parsing: *** +<coordinator-app xmlns="uri:oozie:coordinator:0.1" name="sla_coord" frequency="20" +start="2009-03-06T010:00Z" end="2009-03-20T11:00Z" timezone="America/Los_Angeles"> + <output-events> + <data-out name="Output" dataset="DayLogs"> + <dataset name="DayLogs" frequency="1440" initial-instance="2009-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE" end_of_duration="NONE"> + <uri-template>hdfs://localhost:8020/user/angeloh/coord_examples/${YEAR}/${MONTH}/${DAY}</uri-template> + </dataset> + <instance>${coord:current(0)}</instance> + </data-out> + </output-events> + <action> + </action> +</coordinator-app> + +***actions for instance*** +***total coord actions is 1 *** +------------------------------------------------------------------------------------------------------------------------------------ +coordAction instance: 1: +<coordinator-app xmlns="uri:oozie:coordinator:0.1" name="sla_coord" frequency="20" +start="2009-03-06T010:00Z" end="2009-03-20T11:00Z" timezone="America/Los_Angeles"> + <output-events> + <data-out name="Output" dataset="DayLogs"> + <uris>hdfs://localhost:8020/user/angeloh/coord_examples/2009/03/06</uris> + <dataset name="DayLogs" frequency="1440" initial-instance="2009-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE" end_of_duration="NONE"> + <uri-template>hdfs://localhost:8020/user/angeloh/coord_examples/${YEAR}/${MONTH}/${DAY}</uri-template> + </dataset> + </data-out> + </output-events> + <action> + </action> +</coordinator-app> +------------------------------------------------------------------------------------------------------------------------------------ + +</pre></div></div> + +<p>The <tt>dryrun</tt> option tests running a coordinator job with given job properties and does not create the job.</p> +<p>The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML Configuration file (.xml). This file must be specified with the <tt>-config</tt> option.</p> +<p>The coordinator application path must be specified in the file with the <tt>oozie.coord.application.path</tt> property. The specified path must be an HDFS path.</p></div> +<div class="section"> +<h3><a name="Dryrun_of_Workflow_Job"></a>Dryrun of Workflow Job</h3> +<ul> + +<li>This feature is only supported in Oozie 3.3.2 or later.</li> +</ul> +<p>Example:</p> + +<div> +<div> +<pre class="source">$ oozie job -oozie http://localhost:11000/oozie -dryrun -config job.properties +OK + +</pre></div></div> + +<p>The <tt>dryrun</tt> option tests running a workflow job with given job properties and does not create the job.</p> +<p>The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML Configuration file (.xml). This file must be specified with the <tt>-config</tt> option.</p> +<p>The workflow application path must be specified in the file with the <tt>oozie.wf.application.path</tt> property. The specified path must be an HDFS path.</p> +<p>If the workflow is accepted (i.e. Oozie is able to successfully read and parse it), it will return <tt>"OK"</tt>; otherwise, it will return an error message describing why it was rejected.</p></div> +<div class="section"> +<h3><a name="Dryrun_of_Bundle_Job"></a>Dryrun of Bundle Job</h3> +<ul> + +<li>This feature is only supported in Oozie 5.1 or later.</li> +</ul> +<p>Example:</p> + +<div> +<div> +<pre class="source">$ oozie job -oozie http://localhost:11000/oozie -dryrun -config job.properties +***Bundle job after parsing: *** +<bundle-app xmlns="uri:oozie:bundle:0.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="My Bundle"> + <parameters> + <property> + <name>oozie.use.system.libpath</name> + <value>true</value> + </property> + </parameters> + <controls> + <kick-off-time>2017-02-14T00:13Z</kick-off-time> + </controls> + <coordinator name="My Coordinator-0"> + <app-path>hdfs://localhost:8020/user/hue/oozie/deployments/_admin_-oozie-24-1487060026.39</app-path> + <configuration> + <property> + <name>wf_application_path</name> + <value>hdfs://localhost:8020/user/hue/oozie/workspaces/hue-oozie-1486786607.01</value> + </property> + <property> + <name>start_date</name> + <value>2017-02-14T08:12Z</value> + </property> + <property> + <name>end_date</name> + <value>2017-02-21T08:12Z</value> + </property> + </configuration> + </coordinator> +</bundle-app> + +</pre></div></div> + +<p>The <tt>dryrun</tt> option tests running a bundle job with given job properties and does not create the job.</p> +<p>The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML Configuration file (.xml). This file must be specified with the <tt>-config</tt> option.</p> +<p>If the bundle job is accepted (i.e. Oozie is able to successfully read and parse it), it will return the parsed bundle job in xml; otherwise, it will return an error message describing why it was rejected.</p></div> +<div class="section"> +<h3><a name="Updating_coordinator_definition_and_properties"></a>Updating coordinator definition and properties</h3> +<p>Existing coordinator definition will be replaced by new definition. The refreshed coordinator would keep the same coordinator ID, state, and coordinator actions. All created coord action(including in WAITING) will use old configuration. One can rerun actions with -refresh option, -refresh option will use new configuration to rerun coord action</p> +<p>Update command also verifies coordinator definition like submit command, if there is any issue with definition, update will fail. Update command with -dryrun will show coordinator definition and properties differences. Config option is optional, if not specified existing coordinator property is used to find coordinator path.</p> +<p>Update command doesn’t allow update of coordinator name, frequency, start time, end time and timezone and will fail on an attempt to change any of them. To change end time of coordinator use the <tt>-change</tt> command.</p> +<p>To change the entire XML for a running coordinator, hdfs path for the new XML can be specified as <tt>oozie.coord.application.path</tt> in job.properties. Then, use <tt>-config job.properties</tt> in the update command.</p> + +<div> +<div> +<pre class="source">$ oozie job -oozie http://localhost:11000/oozie -config job.properties -update 0000005-140402104721140-oozie-puru-C -dryrun +. +**********Job definition changes********** +@@ -3,8 +3,8 @@ + <concurrency>1</concurrency> + </controls> + <input-events> +- <data-in name="input" dataset="raw-logs"> +- <dataset name="raw-logs" frequency="20" initial-instance="2010-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE" end_of_duration="NONE"> ++ <data-in name="input" dataset="raw-logs-rename"> ++ <dataset name="raw-logs-rename" frequency="20" initial-instance="2010-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE" end_of_duration="NONE"> + <uri-template>hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/</uri-template> + <done-flag /> + </dataset> +********************************** +**********Job conf changes********** +@@ -8,10 +8,6 @@ + <value>hdfs://localhost:9000/user/purushah/examples/apps/aggregator/coordinator.xml</value> + </property> + <property> +- <name>old</name> +- <value>test</value> +- </property> +- <property> + <name>user.name</name> + <value>purushah</value> + </property> +@@ -28,6 +24,10 @@ + <value>hdfs://localhost:9000</value> + </property> + <property> ++ <name>adding</name> ++ <value>new</value> ++ </property> ++ <property> + <name>jobTracker</name> + <value>localhost:9001</value> + </property> +********************************** +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Ignore_a_Coordinator_Job"></a>Ignore a Coordinator Job</h3> +<p>Example:</p> + +<div> +<div> +<pre class="source">$oozie job -ignore <coord_Job_id> +</pre></div></div> + +<p>The <tt>ignore</tt> option changes a coordinator job in <tt>KILLED</tt>, <tt>FAILED</tt> to <tt>IGNORED</tt> state. When a coordinator job in a bundle is in <tt>IGNORED</tt> state, the coordinator job doesn’t impact the state of the bundle job. For example, when a coordinator job in a bundle failed and afterwards ignored, the bundle job becomes <tt>SUCCEEDED</tt> instead of <tt>DONEWITHERROR</tt> as long as other coordinator jobs in the bundle succeeded. A ignored coordinator job can be changed to <tt>RUNNING</tt> using -change command. Refer to the <a href="DG_CommandLineTool.html#Changing_endtimeconcurrencypausetimestatus_of_a_Coordinator_Job">Coordinator job change command</a> for details.</p></div> +<div class="section"> +<h3><a name="Ignore_a_Coordinator_Action_or_Multiple_Coordinator_Actions"></a>Ignore a Coordinator Action or Multiple Coordinator Actions</h3> +<p>Example:</p> + +<div> +<div> +<pre class="source">$oozie job -ignore <coord_Job_id> -action 1,3-4,7-40 +</pre></div></div> + +<p>The <tt>ignore</tt> option changes a coordinator action(s) in terminal state (<tt>KILLED</tt>, <tt>FAILED</tt>, <tt>TIMEDOUT</tt>) to <tt>IGNORED</tt> state, while not changing the state of the coordinator job. When a coordinator action is in <tt>IGNORED</tt> state, the action doesn’t impact the state of a coordinator job. For example, when a coordinator action failed and afterwards ignored, a coordinator job becomes <tt>SUCCEEDED</tt> instead of <tt>DONEWITHERROR</tt> as long as other coordinator actions succeeded.</p> +<p>A ignored coordinator action can be rerun using -rerun command. Refer to the <a href="DG_CoordinatorRerun.html">Rerunning Coordinator Actions</a> for details. When a workflow job of a ignored coordinator action is rerun, the coordinator action becomes <tt>RUNNING</tt> state.</p></div> +<div class="section"> +<h3><a name="Polling_an_Oozie_job"></a>Polling an Oozie job</h3> +<p>This command allows polling the Oozie server for an Oozie job until it reaches a completed status (e.g. <tt>SUCCEEDED</tt>, <tt>KILLED</tt>, etc).</p> +<p>Example:</p> + +<div> +<div> +<pre class="source">$ oozie job -poll <job_id> -interval 10 -timeout 60 -verbose +. +RUNNING +RUNNING +RUNNING +SUCCEEDED +</pre></div></div> + +<p>The <tt>-poll</tt> argument takes a valid Workflow Job ID, Coordinator Job ID, Coordinator Action ID, or Bundle Job ID.</p> +<p>All other arguments are optional:</p> +<ul> + +<li><tt>verbose</tt> will cause the job status to be printed at each poll; otherwise, there will be no output</li> +<li><tt>interval</tt> allows specifying the polling interval in minutes (default is 5)</li> +<li><tt>timeout</tt> allows specifying the timeout in minutes (default is 30 minutes); negative values indicate no timeout</li> +</ul></div> +<div class="section"> +<h3><a name="Changing_job_SLA_definition_and_alerting"></a>Changing job SLA definition and alerting</h3> +<ul> + +<li>slaenable command can be used to enable job sla alerts.</li> +<li>sladisable command can be used to disable job sla alerts.</li> +<li>slachange command can be used to change sla job definition.</li> +<li>Supported parameters for sla change command are should-start, should-end and max-duration. Please specify the value in single quotes instead of double quotes in command line to avoid bash interpreting braces in EL functions and causing error.</li> +<li>All sla commands takes -action or -date parameter. For bundle jobs additional -coordinator (coord_name/id) parameter can be passed. Sla change command need extra parameter -value to specify new sla definition.</li> +<li>Sla commands without -action or -date parameter is applied to all non terminated actions and all future actions.</li> +<li>Sla commands with -action or -date parameter will be applied to only non terminated actions.</li> +</ul> +<p>Eg.</p> + +<div> +<div> +<pre class="source"> $oozie job -slaenable <coord_Job_id> [-action 1,3-4,7-40] [-date 2009-01-01T01:00Z::2009-05-31T23:59Z,2009-11-10T01:00Z::2009-12-31T22:00Z] + $oozie job -sladisable <coord_Job_id> [-action 1,3-4,7-40] [-date 2009-01-01T01:00Z::2009-05-31T23:59Z,2009-11-10T01:00Z::2009-12-31T22:00Z] + $oozie job -slachange <coord_Job_id> [-action 1,3-4,7-40] [-date 2009-01-01T01:00Z::2009-05-31T23:59Z,2009-11-10T01:00Z::2009-12-31T22:00Z] -value 'sla-max-duration=${10 ** MINUTES};sla-should-end=${30 ** MINUTES};sla-max-duration=${30 * MINUTES}' + $oozie job -slaenable <bundle_job_id> [-action 1,3-4,7-40] [-date 2009-01-01T01:00Z::2009-05-31T23:59Z,2009-11-10T01:00Z::2009-12-31T22:00Z] [-coordinator <List_of_coord_names/ids] +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Getting_missing_dependencies_of_coordinator_actions"></a>Getting missing dependencies of coordinator action(s)</h3> +<ul> + +<li>Coordination action id can be specified directly for getting missing dependencies of a single action.</li> +<li>To get information on multiple actions, either -action or -date option can be specified with the coordinator job id.</li> +<li>missingdeps command doesn’t recompute dependencies. It list missing dependencies which were last computed.</li> +<li>Oozie checks missing dependencies sequentially, and it will stop on first missing dependency. <tt>Blocked On</tt> is the first missing dependency for action. So, there could be a chance that Oozie will report some missing dependencies, but it might be present. To resolve the waiting issue, one should fix the blockedOn missing dependency.</li> +<li>For input logic, missingdeps command doesn’t compute input-logic expression. It will report everything which is missing or not computed.</li> +</ul> + +<div> +<div> +<pre class="source">$oozie job -oozie http://localhost:11000/oozie -missingdeps 0000000-170104141851590-oozie-puru-C -action 1 +$oozie job -oozie http://localhost:11000/oozie -missingdeps 0000000-170104141851590-oozie-puru-C@1 +. +CoordAction : 1 +Blocked on : hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/00/00/_SUCCESS +. +Dataset : input-1 +Pending Dependencies : + hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/01/00/_SUCCESS +. +Dataset : input-2 +Pending Dependencies : + hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/01/00/_SUCCESS + hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/00/40/_SUCCESS + hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/00/20/_SUCCESS + hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/00/00/_SUCCESS +$ +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Checking_a_workflow_definition_generated_by_a_Fluent_Job_API_jar_file"></a>Checking a workflow definition generated by a Fluent Job API jar file</h3> +<p>Since Oozie 5.1.0.</p>
[... 1040 lines stripped ...]
![https://oozie.apache.org/images/oozie_200x.png"](https://oozie.apache.org/images/oozie_200x.png"){kind=link}