OOZIE-2674 Improve oozie commads documentation (abhishekbafna via rkanter)
Project: http://git-wip-us.apache.org/repos/asf/oozie/repo Commit: http://git-wip-us.apache.org/repos/asf/oozie/commit/1bf91be7 Tree: http://git-wip-us.apache.org/repos/asf/oozie/tree/1bf91be7 Diff: http://git-wip-us.apache.org/repos/asf/oozie/diff/1bf91be7 Branch: refs/heads/oya Commit: 1bf91be7b3f824ca4a9fb3e705764574f5dae4da Parents: 64ea00f Author: Robert Kanter <rkan...@cloudera.com> Authored: Fri Oct 21 13:08:54 2016 -0700 Committer: Robert Kanter <rkan...@cloudera.com> Committed: Fri Oct 21 13:08:54 2016 -0700 ---------------------------------------------------------------------- docs/src/site/twiki/DG_CommandLineTool.twiki | 353 ++++++++++--------- .../src/site/twiki/WorkflowFunctionalSpec.twiki | 2 +- release-log.txt | 1 + 3 files changed, 198 insertions(+), 158 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/oozie/blob/1bf91be7/docs/src/site/twiki/DG_CommandLineTool.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/DG_CommandLineTool.twiki b/docs/src/site/twiki/DG_CommandLineTool.twiki index 567e260..e7b165c 100644 --- a/docs/src/site/twiki/DG_CommandLineTool.twiki +++ b/docs/src/site/twiki/DG_CommandLineTool.twiki @@ -21,168 +21,207 @@ usage: the env variable 'OOZIE_TIMEZONE' is used as default value for the '-timezone' option the env variable 'OOZIE_AUTH' is used as default value for the '-auth' option custom headers for Oozie web services can be specified using '-Dheader:NAME=VALUE' +</verbatim> - oozie help : display usage -. - oozie version : show client version -. - oozie job <OPTIONS> : job operations - -action <arg> coordinator rerun/kill on action ids (requires -rerun/-kill); - coordinator log retrieval on action ids (requires -log) - -allruns Get workflow jobs corresponding to a coordinator action - including all the reruns - -auth <arg> select authentication type [SIMPLE|KERBEROS] - -change <arg> change a coordinator or bundle job - -config <arg> job configuration file '.xml' or '.properties' - -configcontent <arg> job configuration - -coordinator <arg> bundle rerun on coordinator names (requires -rerun) - -D <property=value> set/override value for given property - -date <arg> coordinator/bundle rerun on action dates (requires -rerun); - coordinator log retrieval on action dates (requires -log) - -debug Use debug mode to see debugging statements on stdout - -definition <arg> job definition - -diff <arg> Show diff of the new coord definition and properties with the - existing one (default true) - -doas <arg> doAs user, impersonates as the specified user - -dryrun Dryrun a workflow (since 3.3.2) or coordinator (since 2.0) - job without actually executing it - -failed re-runs the failed workflow actions of the coordinator actions (requires -rerun) - -filter <arg> <key><comparator><value>[;<key><comparator><value>]* - (All Coordinator actions satisfying the filters will be retrieved). - key: status or nominaltime - comparator: =, !=, <, <=, >, >=. = is used as OR and others as AND - status: values are valid status like SUCCEEDED, KILLED etc. Only = and != apply - for status. - nominaltime: time of format yyyy-MM-dd'T'HH:mm'Z' - -ignore <arg> change status of a coordinator job or action to IGNORED - (-action required to ignore coord actions) - -info <arg> info of a job - -interval <arg> polling interval in minutes (default is 5, requires -poll) - -kill <arg> kill a job (coordinator can mention -action or -date) - -len <arg> number of actions (default TOTAL ACTIONS, requires -info) - -localtime use local time (same as passing your time zone to -timezone). - Overrides -timezone option - -log <arg> job log - -errorlog <arg> job error log - -auditlog <arg> job audit log - -logfilter <arg> job log search parameter. Can be specified as -logfilter - opt1=val1;opt2=val1;opt3=val1. Supported options are recent, - start, end, loglevel, text, limit and debug - -nocleanup do not clean up output-events of the coordinator rerun - actions (requires -rerun) - -offset <arg> job info offset of actions (default '1', requires -info) - -oozie <arg> Oozie URL - -order <arg> order to show coord actions (default ascending order, 'desc' - for descending order, requires -info) - -poll <arg> poll Oozie until a job reaches a terminal state or a timeout - occurs - -refresh re-materialize the coordinator rerun actions (requires - -rerun) - -rerun <arg> rerun a job (coordinator requires -action or -date, bundle - requires -coordinator or -date) - -resume <arg> resume a job - -run run a job - -start <arg> start a job - -submit submit a job - -suspend <arg> suspend a job - -timeout <arg> timeout in minutes (default is 30, negative values indicate - no timeout, requires -poll) - -timezone <arg> use time zone with the specified ID (default GMT). - See 'oozie info -timezones' for a list - -update <arg> Update coord definition and properties - -value <arg> new endtime/concurrency/pausetime value for changing a - coordinator job - -verbose verbose mode - -sladisable disables sla alerts for the job and its children - -slaenable enables sla alerts for the job and its children - -slachange Update sla param for jobs, supported param are should-start, should-end and max-duration - - oozie jobs <OPTIONS> : jobs status - -auth <arg> select authentication type [SIMPLE|KERBEROS] - -doas <arg> doAs user, impersonates as the specified user. - -filter <arg> user=<U>\;name=<N>\;group=<G>\;status=<S>\;frequency=<F>\;unit=<M>\;startcreatedtime=<SC>\; - endcreatedtime=<EC>\;sortby=<SB> - -jobtype <arg> job type ('Supported in Oozie-2.0 or later versions ONLY - coordinator' or 'wf' (default)) - -len <arg> number of jobs (default '100') - -localtime use local time (same as passing your time zone to -timezone). Overrides -timezone option - -offset <arg> jobs offset (default '1') - -oozie <arg> Oozie URL - -timezone <arg> use time zone with the specified ID (default GMT). See 'oozie info -timezones' for a list - -kill kill all jobs that satisfy the filter, len, offset, or/and jobtype options. If it's used without - other options, it will kill all the first 50 workflow jobs. Command will fail if one or more - of the jobs is in wrong state. - -suspend suspend all jobs that satisfy the filter, len, offset, or/and jobtype options. If it's used without - other options, it will suspend all the first 50 workflow jobs. Command will fail if one or more - of the jobs is in wrong state. - -resume resume all jobs that satisfy the filter, len, offset, or/and jobtype options. If it's used without - other options, it will resume all the first 50 workflow jobs. Command will fail if one or more - of the jobs is in wrong state. - -verbose verbose mode -. - oozie admin <OPTIONS> : admin operations - -auth <arg> select authentication type [SIMPLE|KERBEROS] - -configuration show Oozie system configuration - -doas <arg> doAs user, impersonates as the specified user - -instrumentation show Oozie system instrumentation - -javasysprops show Oozie Java system properties - -metrics show Oozie system metrics - -oozie <arg> Oozie URL - -osenv show Oozie system OS environment - -queuedump show Oozie server queue elements - -servers list available Oozie servers (more than one only if HA is enabled) - -shareliblist List available sharelib that can be specified in a workflow action - -sharelibupdate Update server to use a newer version of sharelib - -status show the current system status - -systemmode <arg> Supported in Oozie-2.0 or later versions ONLY. Change oozie - system mode [NORMAL|NOWEBSERVICE|SAFEMODE] - -version show Oozie server build version -. - oozie validate <OPTIONS> <ARGS> : validate a workflow, coordinator, bundle XML file - -auth <arg> select authentication type [SIMPLE|KERBEROS] - -oozie <arg> Oozie URL -. - oozie sla <OPTIONS> : sla operations (Deprecated as of Oozie 4.0) - -auth <arg> select authentication type [SIMPLE|KERBEROS] - -len <arg> number of results (default '100', max limited by oozie server setting which defaults to '1000') - -offset <arg> start offset (default '0') - -oozie <arg> Oozie URL - -filter <arg> jobid=<JobID/ActionID>\;appname=<Application Name> -. - oozie pig <OPTIONS> -X <ARGS> : submit a pig job, everything after '-X' are pass-through parameters to pig, any '-D' arguments after '-X' are put in <configuration> +---+++ Oozie basic commands +<verbatim> + +oozie help : display usage + +oozie version : show client version + +</verbatim> + +---+++ Oozie job operation commands +<verbatim> + +oozie job <OPTIONS> : job operations + -action <arg> coordinator rerun/kill on action ids (requires -rerun/-kill); + coordinator log retrieval on action ids (requires -log) + -allruns Get workflow jobs corresponding to a coordinator action + including all the reruns + -auth <arg> select authentication type [SIMPLE|KERBEROS] + -change <arg> change a coordinator or bundle job + -config <arg> job configuration file '.xml' or '.properties' + -configcontent <arg> job configuration + -coordinator <arg> bundle rerun on coordinator names (requires -rerun) + -D <property=value> set/override value for given property + -date <arg> coordinator/bundle rerun on action dates (requires -rerun); + coordinator log retrieval on action dates (requires -log) + -debug Use debug mode to see debugging statements on stdout + -definition <arg> job definition + -diff <arg> Show diff of the new coord definition and properties with the + existing one (default true) + -doas <arg> doAs user, impersonates as the specified user + -dryrun Dryrun a workflow (since 3.3.2) or coordinator (since 2.0) + job without actually executing it + -failed re-runs the failed workflow actions of the coordinator actions (requires -rerun) + -filter <arg> <key><comparator><value>[;<key><comparator><value>]* + (All Coordinator actions satisfying the filters will be retrieved). + key: status or nominaltime + comparator: =, !=, <, <=, >, >=. = is used as OR and others as AND + status: values are valid status like SUCCEEDED, KILLED etc. Only = and != apply + for status. + nominaltime: time of format yyyy-MM-dd'T'HH:mm'Z' + -ignore <arg> change status of a coordinator job or action to IGNORED + (-action required to ignore coord actions) + -info <arg> info of a job + -interval <arg> polling interval in minutes (default is 5, requires -poll) + -kill <arg> kill a job (coordinator can mention -action or -date) + -len <arg> number of actions (default TOTAL ACTIONS, requires -info) + -localtime use local time (same as passing your time zone to -timezone). + Overrides -timezone option + -log <arg> job log + -errorlog <arg> job error log + -auditlog <arg> job audit log + -logfilter <arg> job log search parameter. Can be specified as -logfilter + opt1=val1;opt2=val1;opt3=val1. Supported options are recent, + start, end, loglevel, text, limit and debug + -nocleanup do not clean up output-events of the coordinator rerun + actions (requires -rerun) + -offset <arg> job info offset of actions (default '1', requires -info) + -oozie <arg> Oozie URL + -order <arg> order to show coord actions (default ascending order, 'desc' + for descending order, requires -info) + -poll <arg> poll Oozie until a job reaches a terminal state or a timeout + occurs + -refresh re-materialize the coordinator rerun actions (requires + -rerun) + -rerun <arg> rerun a job (coordinator requires -action or -date, bundle + requires -coordinator or -date) + -resume <arg> resume a job + -run run a job + -start <arg> start a job + -submit submit a job + -suspend <arg> suspend a job + -timeout <arg> timeout in minutes (default is 30, negative values indicate + no timeout, requires -poll) + -timezone <arg> use time zone with the specified ID (default GMT). + See 'oozie info -timezones' for a list + -update <arg> Update coord definition and properties + -value <arg> new endtime/concurrency/pausetime value for changing a + coordinator job + -verbose verbose mode + -sladisable disables sla alerts for the job and its children + -slaenable enables sla alerts for the job and its children + -slachange Update sla param for jobs, supported param are should-start, should-end and max-duration + +</verbatim> + +---+++ Oozie jobs operation commands +<verbatim> +oozie jobs <OPTIONS> : jobs status + -auth <arg> select authentication type [SIMPLE|KERBEROS] + -doas <arg> doAs user, impersonates as the specified user. + -filter <arg> user=<U>\;name=<N>\;group=<G>\;status=<S>\;frequency=<F>\;unit=<M>\;startcreatedtime=<SC>\; + endcreatedtime=<EC>\;sortby=<SB> + -jobtype <arg> job type ('Supported in Oozie-2.0 or later versions ONLY - coordinator' or 'wf' (default)) + -len <arg> number of jobs (default '100') + -localtime use local time (same as passing your time zone to -timezone). Overrides -timezone option + -offset <arg> jobs offset (default '1') + -oozie <arg> Oozie URL + -timezone <arg> use time zone with the specified ID (default GMT). See 'oozie info -timezones' for a list + -kill kill all jobs that satisfy the filter, len, offset, or/and jobtype options. If it's used without + other options, it will kill all the first 50 workflow jobs. Command will fail if one or more + of the jobs is in wrong state. + -suspend suspend all jobs that satisfy the filter, len, offset, or/and jobtype options. If it's used without + other options, it will suspend all the first 50 workflow jobs. Command will fail if one or more + of the jobs is in wrong state. + -resume resume all jobs that satisfy the filter, len, offset, or/and jobtype options. If it's used without + other options, it will resume all the first 50 workflow jobs. Command will fail if one or more + of the jobs is in wrong state. + -verbose verbose mode +</verbatim> + +---+++ Oozie admin operation commands +<verbatim> +oozie admin <OPTIONS> : admin operations + -auth <arg> select authentication type [SIMPLE|KERBEROS] + -configuration show Oozie system configuration + -doas <arg> doAs user, impersonates as the specified user + -instrumentation show Oozie system instrumentation + -javasysprops show Oozie Java system properties + -metrics show Oozie system metrics + -oozie <arg> Oozie URL + -osenv show Oozie system OS environment + -queuedump show Oozie server queue elements + -servers list available Oozie servers (more than one only if HA is enabled) + -shareliblist List available sharelib that can be specified in a workflow action + -sharelibupdate Update server to use a newer version of sharelib + -status show the current system status + -systemmode <arg> Supported in Oozie-2.0 or later versions ONLY. Change oozie + system mode [NORMAL|NOWEBSERVICE|SAFEMODE] + -version show Oozie server build version +</verbatim> + +---+++ Oozie validate command +<verbatim> +oozie validate <OPTIONS> <ARGS> : validate a workflow, coordinator, bundle XML file + -auth <arg> select authentication type [SIMPLE|KERBEROS] + -oozie <arg> Oozie URL +</verbatim> + +---+++ Oozie SLA operation commands +<verbatim> +oozie sla <OPTIONS> : sla operations (Deprecated as of Oozie 4.0) + -auth <arg> select authentication type [SIMPLE|KERBEROS] + -len <arg> number of results (default '100', max limited by oozie server setting which defaults to '1000') + -offset <arg> start offset (default '0') + -oozie <arg> Oozie URL + -filter <arg> jobid=<JobID/ActionID>\;appname=<Application Name> +</verbatim> + +---+++ Oozie Pig submit command +<verbatim> +oozie pig <OPTIONS> -X <ARGS> : submit a pig job, everything after '-X' are pass-through parameters to pig, any '-D' arguments + after '-X' are put in <configuration> + -auth <arg> select authentication type [SIMPLE|KERBEROS] + -doas <arg> doAs user, impersonates as the specified user. + -config <arg> job configuration file '.properties' + -D <property=value> set/override value for given property + -file <arg> Pig script + -oozie <arg> Oozie URL + -P <property=value> set parameters for script +</verbatim> + +---+++ Oozie Hive submit command +<verbatim> +oozie hive <OPTIONS> -X<ARGS> : submit a hive job, everything after '-X' are pass-through parameters to hive, any '-D' arguments + after '-X' are put in <configuration> + -auth <arg> select authentication type [SIMPLE|KERBEROS] + -config <arg> job configuration file '.properties' + -D <property=value> set/override value for given property + -doas <arg> doAs user, impersonates as the specified user + -file <arg> hive script + -oozie <arg> Oozie URL + -P <property=value> set parameters for script +</verbatim> + +---+++ Oozie Sqoop submit command +<verbatim> +oozie sqoop <OPTIONS> -X<ARGS> : submit a sqoop job, any '-D' arguments after '-X' are put in <configuration> + -auth <arg> select authentication type [SIMPLE|KERBEROS] + -config <arg> job configuration file '.properties' + -D <property=value> set/override value for given property + -doas <arg> doAs user, impersonates as the specified user + -command <arg> sqoop command + -oozie <arg> Oozie URL +</verbatim> + +---+++ Oozie info command +<verbatim> +oozie info <OPTIONS> : get more detailed info about specific topics + -timezones display a list of available time zones +</verbatim> + +---+++ Oozie MapReduce job command +<verbatim> +oozie mapreduce <OPTIONS> : submit a mapreduce job -auth <arg> select authentication type [SIMPLE|KERBEROS] - -doas <arg> doAs user, impersonates as the specified user. -config <arg> job configuration file '.properties' -D <property=value> set/override value for given property - -file <arg> Pig script + -doas <arg> doAs user, impersonates as the specified user -oozie <arg> Oozie URL - -P <property=value> set parameters for script -. - oozie hive <OPTIONS> -X<ARGS> : submit a hive job, everything after '-X' are pass-through parameters to hive, any '-D' arguments after '-X' are put in <configuration> - -auth <arg> select authentication type [SIMPLE|KERBEROS] - -config <arg> job configuration file '.properties' - -D <property=value> set/override value for given property - -doas <arg> doAs user, impersonates as the specified user - -file <arg> hive script - -oozie <arg> Oozie URL - -P <property=value> set parameters for script -. - oozie sqoop <OPTIONS> -X<ARGS> : submit a sqoop job, any '-D' arguments after '-X' are put in <configuration> - -auth <arg> select authentication type [SIMPLE|KERBEROS] - -config <arg> job configuration file '.properties' - -D <property=value> set/override value for given property - -doas <arg> doAs user, impersonates as the specified user - -command <arg> sqoop command - -oozie <arg> Oozie URL -. - oozie info <OPTIONS> : get more detailed info about specific topics - -timezones display a list of available time zones -. - oozie mapreduce <OPTIONS> : submit a mapreduce job - -auth <arg> select authentication type [SIMPLE|KERBEROS] - -config <arg> job configuration file '.properties' - -D <property=value> set/override value for given property - -doas <arg> doAs user, impersonates as the specified user - -oozie <arg> Oozie URL </verbatim> ---++ Common CLI Options http://git-wip-us.apache.org/repos/asf/oozie/blob/1bf91be7/docs/src/site/twiki/WorkflowFunctionalSpec.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/WorkflowFunctionalSpec.twiki b/docs/src/site/twiki/WorkflowFunctionalSpec.twiki index 7474ea5..b7eaf5e 100644 --- a/docs/src/site/twiki/WorkflowFunctionalSpec.twiki +++ b/docs/src/site/twiki/WorkflowFunctionalSpec.twiki @@ -213,7 +213,7 @@ The workflow definition language is XML based and it is called hPDL (Hadoop Proc Refer to the Appendix A for the[[WorkflowFunctionalSpec#OozieWFSchema][Oozie Workflow Definition XML Schema]]. Appendix B has [[WorkflowFunctionalSpec#OozieWFExamples][Workflow Definition Examples]]. ----++ 2.1 Cycles in Workflow Definitions +---+++ 2.1 Cycles in Workflow Definitions Oozie does not support cycles in workflow definitions, workflow definitions must be a strict DAG. http://git-wip-us.apache.org/repos/asf/oozie/blob/1bf91be7/release-log.txt ---------------------------------------------------------------------- diff --git a/release-log.txt b/release-log.txt index d4dbb74..77cc5ad 100644 --- a/release-log.txt +++ b/release-log.txt @@ -6,6 +6,7 @@ OOZIE-2634 Queue dump command message is confusing when the queue is empty (andr -- Oozie 4.3.0 release +OOZIE-2674 Improve oozie commads documentation (abhishekbafna via rkanter) OOZIE-2710 Oozie HCatalog example workflow fails (abhishekbafna via shwethags) OOZIE-2705 Oozie Spark action ignores spark.executor.extraJavaOptions and spark.driver.extraJavaOptions (gezapeti via rkanter) OOZIE-2621 Use hive-exec-<version>-core instead of hive-exec in oozie-core (gezapeti via rkanter)