OOZIE-2734 [docs] Switch from TWiki to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Project: http://git-wip-us.apache.org/repos/asf/oozie/repo Commit: http://git-wip-us.apache.org/repos/asf/oozie/commit/4e5b3cb5 Tree: http://git-wip-us.apache.org/repos/asf/oozie/tree/4e5b3cb5 Diff: http://git-wip-us.apache.org/repos/asf/oozie/diff/4e5b3cb5 Branch: refs/heads/master Commit: 4e5b3cb596ef5ffb68f401b050b30484ba21b268 Parents: d5c4f3b Author: Andras Piros <andras.pi...@cloudera.com> Authored: Fri Sep 14 16:05:17 2018 +0200 Committer: Andras Piros <andras.pi...@cloudera.com> Committed: Fri Sep 14 16:05:17 2018 +0200 ---------------------------------------------------------------------- README.txt | 4 +- docs/pom.xml | 22 +- docs/src/site/markdown/AG_OozieLogging.md | 89 + docs/src/site/markdown/DG_QuickStart.md | 240 ++ docs/src/site/markdown/ENG_Building.md | 281 ++ .../site/markdown/ENG_Custom_Authentication.md | 80 + docs/src/site/markdown/index.md | 109 + .../src/site/twiki/AG_ActionConfiguration.twiki | 38 +- .../src/site/twiki/AG_HadoopConfiguration.twiki | 47 +- docs/src/site/twiki/AG_Install.twiki | 1012 +++--- docs/src/site/twiki/AG_Monitoring.twiki | 61 +- docs/src/site/twiki/AG_OozieLogging.twiki | 87 - docs/src/site/twiki/AG_OozieUpgrade.twiki | 39 +- docs/src/site/twiki/BundleFunctionalSpec.twiki | 203 +- .../site/twiki/CoordinatorFunctionalSpec.twiki | 2938 +++++++++--------- .../site/twiki/DG_ActionAuthentication.twiki | 76 +- docs/src/site/twiki/DG_CommandLineTool.twiki | 1027 +++--- docs/src/site/twiki/DG_CoordinatorRerun.twiki | 23 +- .../site/twiki/DG_CustomActionExecutor.twiki | 59 +- .../site/twiki/DG_DistCpActionExtension.twiki | 76 +- .../site/twiki/DG_EmailActionExtension.twiki | 81 +- docs/src/site/twiki/DG_Examples.twiki | 102 +- docs/src/site/twiki/DG_FluentJobAPI.twiki | 164 +- .../src/site/twiki/DG_HCatalogIntegration.twiki | 77 +- .../site/twiki/DG_Hive2ActionExtension.twiki | 109 +- .../src/site/twiki/DG_HiveActionExtension.twiki | 134 +- docs/src/site/twiki/DG_JMSNotifications.twiki | 117 +- docs/src/site/twiki/DG_Overview.twiki | 31 +- docs/src/site/twiki/DG_QuickStart.twiki | 230 -- docs/src/site/twiki/DG_SLAMonitoring.twiki | 230 +- .../site/twiki/DG_ShellActionExtension.twiki | 151 +- .../site/twiki/DG_SparkActionExtension.twiki | 173 +- .../site/twiki/DG_SqoopActionExtension.twiki | 118 +- docs/src/site/twiki/DG_SshActionExtension.twiki | 78 +- docs/src/site/twiki/DG_WorkflowReRun.twiki | 27 +- docs/src/site/twiki/ENG_Building.twiki | 270 -- .../site/twiki/ENG_Custom_Authentication.twiki | 79 - docs/src/site/twiki/ENG_MiniOozie.twiki | 52 +- docs/src/site/twiki/WebServicesAPI.twiki | 1205 +++---- .../src/site/twiki/WorkflowFunctionalSpec.twiki | 1694 +++++----- docs/src/site/twiki/index.twiki | 111 - pom.xml | 33 +- release-log.txt | 1 + 43 files changed, 6252 insertions(+), 5526 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/README.txt ---------------------------------------------------------------------- diff --git a/README.txt b/README.txt index 8cc0f59..f9c6bdf 100644 --- a/README.txt +++ b/README.txt @@ -42,7 +42,7 @@ More inforamtion could be found at: http://oozie.apache.org/ Oozie Quick Start: -http://oozie.apache.org/docs/4.3.0/DG_QuickStart.html +http://oozie.apache.org/docs/5.0.0/DG_QuickStart.html Supported Hadoop Versions: @@ -50,7 +50,7 @@ Supported Hadoop Versions: This version of Oozie was primarily tested against Hadoop 2.4.x and 2.6.x. --------------------------------------- + If you have any questions/issues, please send an email to: http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/pom.xml ---------------------------------------------------------------------- diff --git a/docs/pom.xml b/docs/pom.xml index ae2e94c..35b868f 100644 --- a/docs/pom.xml +++ b/docs/pom.xml @@ -78,7 +78,7 @@ <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-project-info-reports-plugin</artifactId> <!-- Using version because plugin management does not work for reporting plugins --> - <version>2.2</version> + <version>2.9</version> <reportSets> <reportSet> <reports> @@ -89,7 +89,12 @@ </plugins> </reporting> <build> - <plugins> + <plugins> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-project-info-reports-plugin</artifactId> + <version>2.9</version> + </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-site-plugin</artifactId> @@ -99,16 +104,19 @@ </configuration> <dependencies> <dependency> - <!-- Customized Doxia Maven Plugin for twiki documentation --> + <groupId>org.apache.velocity</groupId> + <artifactId>velocity</artifactId> + <version>1.5</version> + </dependency> + <dependency> <groupId>org.apache.maven.doxia</groupId> - <artifactId>doxia-module-twiki</artifactId> - <version>1.0-alpha-9.2y</version> + <artifactId>doxia-module-markdown</artifactId> + <version>1.8</version> </dependency> <dependency> - <!-- Customized Doxia Maven Plugin for twiki documentation --> <groupId>org.apache.maven.doxia</groupId> <artifactId>doxia-core</artifactId> - <version>1.0-alpha-9.2y</version> + <version>1.7</version> </dependency> </dependencies> <executions> http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/markdown/AG_OozieLogging.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/AG_OozieLogging.md b/docs/src/site/markdown/AG_OozieLogging.md new file mode 100644 index 0000000..873e9e3 --- /dev/null +++ b/docs/src/site/markdown/AG_OozieLogging.md @@ -0,0 +1,89 @@ + + +[::Go back to Oozie Documentation Index::](index.html) + +# Oozie Logging + +<!-- MACRO{toc|fromDepth=1|toDepth=4} --> + +## Default Oozie Logging + +Oozie's logging properties can be configured in its log4j properties file (default is `oozie-log4j.properties`). Most log messages +are configured by default to be written to the `oozie` appender. + +The default configuration for the `oozie` appender is shown below. + +### Default Configuration + + +``` +log4j.appender.oozie=org.apache.log4j.rolling.RollingFileAppender +log4j.appender.oozie.RollingPolicy=org.apache.oozie.util.OozieRollingPolicy +log4j.appender.oozie.File=${oozie.log.dir}/oozie.log +log4j.appender.oozie.Append=true +log4j.appender.oozie.layout=org.apache.log4j.PatternLayout +log4j.appender.oozie.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n +log4j.appender.oozie.RollingPolicy.FileNamePattern=${log4j.appender.oozie.File}-%d{yyyy-MM-dd-HH} +log4j.appender.oozie.RollingPolicy.MaxHistory=720 +``` + +In this configuration, the active log file will be named `oozie.log` and all old log files will be named `oozie.log-yyyy-MM-dd-HH` +(where `yyyy-MM-dd-HH` is the time that that log file was created; e.g. 2012-07-21-05). All log files are in the same directory +(whatever `oozie.log.dir` is assigned to). A maximum of 720 older log files will be retained. The active log file is rolled every +hour, so 720 old logs means that they are kept for 30 days before being deleted. + +To keep all old logs instead of deleting them, `log4j.appender.oozie.RollingPolicy.MaxHistory` can be set to `-1`. +Additionally, `log4j.appender.oozie.RollingPolicy` can be set to `org.apache.log4j.rolling.TimeBasedRollingPolicy`, which has the +same exact behavior as `org.apache.oozie.util.OozieRollingPolicy` except that it does not delete old logs. + +### Restrictions + +In order for Oozie logging to work 100% correctly, the following restrictions must be observed (described below and in +the `oozie-log4j.properties` file): + +* The appender that Oozie uses must be named "oozie" (i.e. `log4j.appender.oozie`) + +* `log4j.appender.oozie.RollingPolicy.FileNamePattern` must end with "-%d{yyyy-MM-dd-HH}.gz" or "-%d{yyyy-MM-dd-HH}". +If it ends with ".gz" the old logs will be compressed when rolled + +* `log4j.appender.oozie.RollingPolicy.FileNamePattern` must start with the value of `log4j.appender.oozie.File` + +## Previous Default Oozie Logging + +Oozie previously used the logging configuration shown below as the default for the `oozie` appender. The other appender that Oozie +writes to still use a configuration similar to this. + +### Previous Default Configuration + + +``` +log4j.appender.oozie=org.apache.log4j.DailyRollingFileAppender +log4j.appender.oozie.File=${oozie.log.dir}/oozie.log +log4j.appender.oozie.Append=true +log4j.appender.oozie.layout=org.apache.log4j.PatternLayout +log4j.appender.oozie.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n +log4j.appender.oozie.DatePattern='.'yyyy-MM-dd-HH +``` + +In this configuration, the active log file will be named `oozie.log` and all old log files will be named `oozie.log.yyyy-MM-dd-HH` +(where `yyyy-MM-dd-HH` is the time that the log file was created; e.g. 2012-07-21-05). All log files are in the same directory +(whatever `oozie.log.dir` is assigned to). All older log files are retained. The active log file is rolled every hour. + +### Restrictions + +In order for Oozie logging to work 100% correctly, the following restrictions must be observed (described below and in the +`oozie-log4j.properties` file): + +* The appender that Oozie uses must be named "oozie" (i.e. `log4j.appender.oozie`) + +* `log4j.appender.oozie.DatePattern` must end with either "dd" or "HH". If it ends with "HH", the log will be rolled every hour; +if it ends with "dd", the log will be rolled every day. + +## Other Oozie Logging + +While Oozie can technically use any valid log4j Appender or configurations that violate the above restrictions, certain +features related to logs may be disabled and/or not work correctly, and is thus not advised. + +[::Go back to Oozie Documentation Index::](index.html) + + http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/markdown/DG_QuickStart.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/DG_QuickStart.md b/docs/src/site/markdown/DG_QuickStart.md new file mode 100644 index 0000000..64f67ce --- /dev/null +++ b/docs/src/site/markdown/DG_QuickStart.md @@ -0,0 +1,240 @@ + + +[::Go back to Oozie Documentation Index::](index.html) + +# Oozie Quick Start + +These instructions install and run Oozie using an embedded Jetty server and an embedded Derby database. + +For detailed install and configuration instructions refer to [Oozie Install](AG_Install.html). + +<!-- MACRO{toc|fromDepth=1|toDepth=4} --> + +## Building Oozie + +### System Requirements: + * Unix box (tested on Mac OS X and Linux) + * Java JDK 1.8+ + * Maven 3.0.1+ + * Hadoop 2.6.0+ + * Pig 0.10.1+ + +JDK commands (java, javac) must be in the command path. + +The Maven command (mvn) must be in the command path. + +### Building Oozie + +Download a source distribution of Oozie from the "Releases" drop down menu on the [Oozie site](http://oozie.apache.org). + +Expand the source distribution `tar.gz` and change directories into it. + +The simplest way to build Oozie is to run the `mkdistro.sh` script: + +``` +$ bin/mkdistro.sh [-DskipTests] +``` + + +Running `mkdistro.sh` will create the binary distribution of Oozie. By default, oozie war will not contain hadoop and +hcatalog libraries, however they are required for oozie to work. There are 2 options to add these libraries: + +1. At install time, copy the hadoop and hcatalog libraries to libext and run oozie-setup.sh to setup Oozie. This is + suitable when same oozie package needs to be used in multiple set-ups with different hadoop/hcatalog versions. + +2. Build with -Puber which will bundle the required libraries in the oozie war. Further, the following options are + available to customise the versions of the dependencies: + ``` + -Dhadoop.version=<version> - default 2.6.0 + -Ptez - Bundle tez jars in hive and pig sharelibs. Useful if you want to use tez + +as the execution engine for those applications. + -Dpig.version=<version> - default 0.16.0 + -Dpig.classifier=<classifier> - default h2 + -Dsqoop.version=<version> - default 1.4.3 + -Dsqoop.classifier=<classifier> - default hadoop100 + -Djetty.version=<version> - default 9.3.20.v20170531 + -Dopenjpa.version=<version> - default 2.2.2 + -Dxerces.version=<version> - default 2.10.0 + -Dcurator.version=<version> - default 2.5.0 + -Dhive.version=<version - default 1.2.0 + -Dhbase.version=<version> - default 1.2.3 + -Dtez.version=<version> - default 0.8.4 + + *IMPORTANT:* Profile hadoop-3 must be activated if building against Hadoop 3 + ``` + + More details on building Oozie can be found on the [Building Oozie](ENG_Building.html) page. + +## Server Installation + +### System Requirements + + * Unix (tested in Linux and Mac OS X) + * Java 1.8+ + * Hadoop + * [Apache Hadoop](http://hadoop.apache.org) (tested with 1.2.1 & 2.6.0+) + * ExtJS library (optional, to enable Oozie webconsole) + * [ExtJS 2.2](http://archive.cloudera.com/gplextras/misc/ext-2.2.zip) + +The Java 1.8+ `bin` directory should be in the command path. + +### Server Installation + +**IMPORTANT:** Oozie ignores any set value for `OOZIE_HOME`, Oozie computes its home automatically. + + * Build an Oozie binary distribution + * Download a Hadoop binary distribution + * Download ExtJS library (it must be version 2.2) + +**NOTE:** The ExtJS library is not bundled with Oozie because it uses a different license. + +**NOTE:** Oozie UI browser compatibility Chrome (all), Firefox (3.5), Internet Explorer (8.0), Opera (10.5). + +**NOTE:** It is recommended to use a Oozie Unix user for the Oozie server. + +Expand the Oozie distribution `tar.gz`. + +Expand the Hadoop distribution `tar.gz` (as the Oozie Unix user). + +<a name="HadoopProxyUser"></a> + +**NOTE:** Configure the Hadoop cluster with proxyuser for the Oozie process. + +The following two properties are required in Hadoop core-site.xml: + + +``` + <!-- OOZIE --> + <property> + <name>hadoop.proxyuser.[OOZIE_SERVER_USER].hosts</name> + <value>[OOZIE_SERVER_HOSTNAME]</value> + </property> + <property> + <name>hadoop.proxyuser.[OOZIE_SERVER_USER].groups</name> + <value>[USER_GROUPS_THAT_ALLOW_IMPERSONATION]</value> + </property> +``` + +Replace the capital letter sections with specific values and then restart Hadoop. + +The ExtJS library is optional (only required for the Oozie web-console to work) + +**IMPORTANT:** all Oozie server scripts (`oozie-setup.sh`, `oozied.sh`, `oozie-start.sh`, `oozie-run.sh` +and `oozie-stop.sh`) run only under the Unix user that owns the Oozie installation directory, +if necessary use `sudo -u OOZIE_USER` when invoking the scripts. + +As of Oozie 3.3.2, use of `oozie-start.sh`, `oozie-run.sh`, and `oozie-stop.sh` has +been deprecated and will print a warning. The `oozied.sh` script should be used +instead; passing it `start`, `run`, or `stop` as an argument will perform the +behaviors of `oozie-start.sh`, `oozie-run.sh`, and `oozie-stop.sh` respectively. + +Create a **libext/** directory in the directory where Oozie was expanded. + +If using the ExtJS library copy the ZIP file to the **libext/** directory. If hadoop and hcatalog libraries are not +already included in the war, add the corresponding libraries to **libext/** directory. + +A "sharelib create -fs fs_default_name [-locallib sharelib]" command is available when running oozie-setup.sh +for uploading new sharelib into hdfs where the first argument is the default fs name +and the second argument is the Oozie sharelib to install, it can be a tarball or the expanded version of it. +If the second argument is omitted, the Oozie sharelib tarball from the Oozie installation directory will be used. +Upgrade command is deprecated, one should use create command to create new version of sharelib. +Sharelib files are copied to new `lib_<timestamped>` directory. At start, server picks the sharelib from latest time-stamp directory. +While starting server also purge sharelib directory which is older than sharelib retention days +(defined as oozie.service.ShareLibService.temp.sharelib.retention.days and 7 days is default). + +db create|upgrade|postupgrade -run [-sqlfile \<FILE\>] command is for create, upgrade or postupgrade oozie db with an +optional sql file + +Run the `oozie-setup.sh` script to configure Oozie with all the components added to the **libext/** directory. + + +``` +$ bin/oozie-setup.sh sharelib create -fs <FS_URI> [-locallib <PATH>] + sharelib upgrade -fs <FS_URI> [-locallib <PATH>] + db create|upgrade|postupgrade -run [-sqlfile <FILE>] +``` + +**IMPORTANT**: If the Oozie server needs to establish secure connection with an external server with a self-signed certificate, +make sure you specify the location of a truststore that contains required certificates. It can be done by configuring +`oozie.https.truststore.file` in `oozie-site.xml`, or by setting the `javax.net.ssl.trustStore` system property. +If it is set in both places, the value passed as system property will be used. + +Create the Oozie DB using the 'ooziedb.sh' command line tool: + + +``` +$ bin/ooziedb.sh create -sqlfile oozie.sql -run + +Validate DB Connection. +DONE +Check DB schema does not exist +DONE +Check OOZIE_SYS table does not exist +DONE +Create SQL schema +DONE +DONE +Create OOZIE_SYS table +DONE + +Oozie DB has been created for Oozie version '3.2.0' + +$ +``` + +Start Oozie as a daemon process run: + + +``` +$ bin/oozied.sh start +``` + +To start Oozie as a foreground process run: + + +``` +$ bin/oozied.sh run +``` + +Check the Oozie log file `logs/oozie.log` to ensure Oozie started properly. + +Using the Oozie command line tool check the status of Oozie: + + +``` +$ bin/oozie admin -oozie http://localhost:11000/oozie -status +``` + +Using a browser go to the [Oozie web console](http://localhost:11000/oozie.html), Oozie status should be **NORMAL**. + +Refer to the [Running the Examples](DG_Examples.html) document for details on running the examples. + +## Client Installation + +### System Requirements + + * Unix (tested in Linux and Mac OS X) + * Java 1.8+ + +The Java 1.8+ `bin` directory should be in the command path. + +### Client Installation + +Copy and expand the `oozie-client` TAR.GZ file bundled with the distribution. Add the `bin/` directory to the `PATH`. + +Refer to the [Command Line Interface Utilities](DG_CommandLineTool.html) document for a full reference of the `oozie` +command line tool. + +NOTE: The Oozie server installation includes the Oozie client. The Oozie client should be installed in remote machines +only. + +<a name="OozieShareLib"></a> +## Oozie Share Lib Installation + +Oozie share lib has been installed by oozie-setup.sh create command explained in the earlier section. + +See the [Workflow Functional Specification](WorkflowFunctionalSpec.html#ShareLib) and [Installation](AG_Install.html#Oozie_Share_Lib) for more information about the Oozie ShareLib. + +[::Go back to Oozie Documentation Index::](index.html) + + http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/markdown/ENG_Building.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/ENG_Building.md b/docs/src/site/markdown/ENG_Building.md new file mode 100644 index 0000000..92f988a --- /dev/null +++ b/docs/src/site/markdown/ENG_Building.md @@ -0,0 +1,281 @@ + + +[::Go back to Oozie Documentation Index::](index.html) + +# Building Oozie + +<!-- MACRO{toc|fromDepth=1|toDepth=4} --> + +## System Requirements + + * Unix box (tested on Mac OS X and Linux) + * Java JDK 1.8+ + * [Maven 3.0.1+](http://maven.apache.org/) + * [Hadoop 2.6.0+](http://hadoop.apache.org/core/releases.html) + * [Pig 0.10.1+](http://hadoop.apache.org/pig/releases.html) + +JDK commands (java, javac) must be in the command path. + +The Maven command (mvn) must be in the command path. + +## Oozie Documentation Generation + +To generate the documentation, Oozie uses a Doxia plugin for Maven with markdown support. + + + + + + +``` +$ mvn install +``` + +<a name="SshSetup"></a> +## Passphrase-less SSH Setup + +**NOTE: SSH actions are deprecated in Oozie 2.** + +To run SSH Testcases and for easier Hadoop start/stop configure SSH to localhost to be passphrase-less. + +Create your SSH keys without a passphrase and add the public key to the authorized file: + + +``` +$ ssh-keygen -t dsa +$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys2 +``` + +Test that you can ssh without password: + + +``` +$ ssh localhost +``` + +## Building with different Java Versions + +Oozie requires a minimum Java version of 1.8. Any newer version can be used but by default bytecode will be generated +which is compatible with 1.8. This can be changed by specifying the build property **targetJavaVersion**. + +## Building and Testing Oozie + +The JARs for the specified Hadoop and Pig versions must be available in one of the Maven repositories defined in Oozie +main 'pom.xml' file. Or they must be installed in the local Maven cache. + +### Examples Running Oozie Testcases with Different Configurations + +**Using embedded Hadoop minicluster with 'simple' authentication:** + + +``` +$ mvn clean test +``` + +**Using a Hadoop cluster with 'simple' authentication:** + + +``` +$ mvn clean test -Doozie.test.hadoop.minicluster=false +``` + +**Using embedded Hadoop minicluster with 'simple' authentication and Derby database:** + + +``` +$ mvn clean test -Doozie.test.hadoop.minicluster=false -Doozie.test.db=derby +``` + +**Using a Hadoop cluster with 'kerberos' authentication:** + + +``` +$ mvn clean test -Doozie.test.hadoop.minicluster=false -Doozie.test.hadoop.security=kerberos +``` + +NOTE: The embedded minicluster cannot be used when testing with 'kerberos' authentication. + +**Using a custom Oozie configuration for testcases:** + + +``` +$ mvn clean test -Doozie.test.config.file=/home/tucu/custom-oozie-sitel.xml +``` + +**Running the testcases with different databases:** + + +``` +$ mvn clean test -Doozie.test.db=[hsqldb*|derby|mysql|postgres|oracle] +``` + +Using `mysql` and `oracle` enables profiles that will include their JARs files in the build. If using + `oracle`, the Oracle JDBC JAR file must be manually installed in the local Maven cache (the JAR is +not available in public Maven repos). + +### Build Options Reference + +All these options can be set using **-D**. + +Except for the options marked with `(*)`, the options can be specified in the `test.properties` in the root +of the Oozie project. The options marked with `(*)` are used in Maven POMs, thus they don't take effect if +specified in the `test.properties` file (which is loaded by the `XTestCase` class at class initialization time). + +**hadoop.version** `(*)`: indicates the Hadoop version you wish to build Oozie against specifically. It will +substitute this value in the Oozie POM properties and pull the corresponding Hadoop artifacts from Maven. +The default version is 2.6.0 and that is the minimum supported Hadoop version. + +**generateSite** (*): generates Oozie documentation, default is undefined (no documentation is generated) + +**skipTests** (*): skips the execution of all testcases, no value required, default is undefined + +**test**= (*): runs a single test case, to run a test give the test class name without package and extension, no default + +**oozie.test.db**= (*): indicates the database to use for running the testcases, supported values are 'hsqldb', 'derby', + 'mysql', 'postgres' and 'oracle'; default value is 'hsqldb'. For each database there is + `core/src/test/resources/DATABASE-oozie-site.xml` file preconfigured. + +**oozie.test.properties** (*): indicates the file to load the test properties from, by default is `test.properties`. +Having this option allows having different test properties sets, for example: minicluster, simple & kerberos. + +**oozie.test.waitfor.ratio**= : multiplication factor for testcases using waitfor, the ratio is used to adjust the +effective time out. For slow machines the ratio should be increased. The default value is `1`. + +**oozie.test.config.file**= : indicates a custom Oozie configuration file for running the testcases. The specified file +must be an absolute path. For example, it can be useful to specify different database than HSQL for running the +testcases. + +**oozie.test.hadoop.minicluster**= : indicates if Hadoop minicluster should be started for testcases, default value 'true' + +**oozie.test.job.tracker**= : indicates the URI of the JobTracker when using a Hadoop cluster for testing, default value +'localhost:8021' + +**oozie.test.name.node**= : indicates the URI of the NameNode when using a Hadoop cluster for testing, default value +'`hdfs://localhost:8020`' + +**oozie.test.hadoop.security**= : indicates the type of Hadoop authentication for testing, valid values are 'simple' or +'kerberos, default value 'simple' + +**oozie.test.kerberos.keytab.file**= : indicates the location of the keytab file, default value +'${user.home}/oozie.keytab' + +**oozie.test.kerberos.realm**= : indicates the Kerberos real, default value 'LOCALHOST' + +**oozie.test.kerberos.oozie.principal**= : indicates the Kerberos principal for oozie, default value +'${user.name}/localhost' + +**oozie.test.kerberos.jobtracker.principal**= : indicates the Kerberos principal for the JobTracker, default value +'mapred/localhost' + +**oozie.test.kerberos.namenode.principal**= : indicates the Kerberos principal for the NameNode, default value +'hdfs/localhost' + +**oozie.test.user.oozie**= : specifies the user ID used to start Oozie server in testcases, default value +is `${user.name}`. + +**oozie.test.user.test**= : specifies primary user ID used as the user submitting jobs to Oozie Server in testcases, +default value is `test`. + +**oozie.test.user.test2**= : specifies secondary user ID used as the user submitting jobs to Oozie Server in testcases, +default value is `test2`. + +**oozie.test.user.test3**= : specifies secondary user ID used as the user submitting jobs to Oozie Server in testcases, +default value is `test3`. + +**oozie.test.group**= : specifies group ID used as group when submitting jobs to Oozie Server in testcases, +default value is `testg`. + +NOTE: The users/group specified in **oozie.test.user.test2**, **oozie.test.user.test3** and **oozie.test.user.group** +are used for the authorization testcases only. + +**oozie.test.dir**` : specifies the directory where the `oozietests` directory will be created, default value is `/tmp=. +The `oozietests` directory is used by testcases when they need a local filesystem directory. + +**hadoop.log.dir**= : specifies the directory where Hadoop minicluster will write its logs during testcases, default +value is `/tmp`. + +**test.exclude**` : specifies a testcase class (just the class name) to exclude for the tests run, for example =TestSubmitCommand`. + +**test.exclude.pattern**` : specifies one or more patterns for testcases to exclude, for example =**/Test*Command.java`. + +### Testing Map Reduce Pipes Action + +Pipes testcases require Hadoop's **wordcount-simple** pipes binary example to run. The **wordcount-simple** pipes binary +should be compiled for the build platform and copied into Oozie's **core/src/test/resources/** directory. The binary file +must be named **wordcount-simple**. + +If the **wordcount-simple** pipes binary file is not available the testcase will do a NOP and it will print to its output +file the following message 'SKIPPING TEST: TestPipesMain, binary 'wordcount-simple' not available in the classpath'. + +There are 2 testcases that use the **wordcount-simple** pipes binary, **TestPipesMain** and **TestMapReduceActionExecutor**, +the 'SKIPPING TEST..." message would appear in the testcase log file of both testcases. + +### Testing using dist_test and grind + +Testing using [dist_test](https://github.com/cloudera/dist_test) framework with +[grind](https://github.com/cloudera/dist_test/blob/master/docs/grind.md) front end might not work using the default 3.0.2 +version of the maven dependency plugin. It is necessary to downgrade to version 2.10 using +`-Dmaven-dependency-plugin.version=2.10` . + +Maven flags for grind can be specified using `GRIND_MAVEN_FLAGS` environment variable: + + +``` +export GRIND_MAVEN_FLAGS=-Dmaven.dependency.plugin.version=2.10 +grind test --java-version 8 +``` + +## Building an Oozie Distribution + +An Oozie distribution bundles an embedded Jetty server. + +The simplest way to build Oozie is to run the `mkdistro.sh` script: + +``` +$ bin/mkdistro.sh [-DskipTests] +Running =mkdistro.sh= will create the binary distribution of Oozie. The following options are available to customise +the versions of the dependencies: +-Puber - Bundle required hadoop and hcatalog libraries in oozie war +-Dhadoop.version=<version> - default 2.6.0 +-Ptez - Bundle tez jars in hive and pig sharelibs. Useful if you want to use tez +as the execution engine for those applications. +-Dpig.version=<version> - default 0.16.0 +-Dpig.classifier=<classifier> - default h2 +-Dsqoop.version=<version> - default 1.4.3 +-Dsqoop.classifier=<classifier> - default hadoop100 +-jetty.version=<version> - default 9.3.20.v20170531 +-Dopenjpa.version=<version> - default 2.2.2 +-Dxerces.version=<version> - default 2.10.0 +-Dcurator.version=<version> - default 2.5.0 +-Dhive.version=<version> - default 1.2.0 +-Dhbase.version=<version> - default 1.2.3 +-Dtez.version=<version> - default 0.8.4 +``` + +**IMPORTANT:** Profile hadoop-3 must be activated if building against Hadoop 3 + +The following properties should be specified when building a release: + + * -DgenerateDocs : forces the generation of Oozie documentation + * -Dbuild.time= : timestamps the distribution + * -Dvc.revision= : specifies the source control revision number of the distribution + * -Dvc.url= : specifies the source control URL of the distribution + +The provided `bin/mkdistro.sh` script runs the above Maven invocation setting all these properties to the +right values (the 'vc.*' properties are obtained from the local git repository). + +## IDE Setup + +Eclipse and IntelliJ can use directly Oozie Maven project files. + +The only special consideration is that the following source directories from the `client` module must be added to +the `core` module source path: + + * `client/src/main/java` : as source directory + * `client/src/main/resources` : as source directory + * `client/src/test/java` : as test-source directory + * `client/src/test/resources` : as test-source directory + +[::Go back to Oozie Documentation Index::](index.html) + + http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/markdown/ENG_Custom_Authentication.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/ENG_Custom_Authentication.md b/docs/src/site/markdown/ENG_Custom_Authentication.md new file mode 100644 index 0000000..c70355f --- /dev/null +++ b/docs/src/site/markdown/ENG_Custom_Authentication.md @@ -0,0 +1,80 @@ + + +[::Go back to Oozie Documentation Index::](index.html) + +# Creating Custom Authentication + +<!-- MACRO{toc|fromDepth=1|toDepth=4} --> + +## Hadoop-Auth Authentication Interfaces and classes + +1. `org.apache.hadoop.security.authentication.client.Authenticator:` Interface for client authentication mechanisms. + + The following authenticators are provided in hadoop-auth: + + * KerberosAuthenticator : the authenticator implements the Kerberos SPNEGO authentication sequence. + * PseudoAuthenticator : the authenticator implementation provides an authentication equivalent to Hadoop's Simple + authentication, it trusts the value of the 'user.name' Java System property. + +2. `org.apache.hadoop.security.authentication.server.AuthenticationHandler:` Interface for server authentication mechanisms. + + * KerberosAuthenticationHandler : the authenticator handler implements the Kerberos SPNEGO authentication mechanism for HTTP. + * PseudoAuthenticationHandler : the authenticator handler provides a pseudo authentication mechanism that accepts the user + name specified as a query string parameter. + +3. `org.apache.hadoop.security.authentication.server.AuthenticationFilter:` A servlet filter enables protecting web application + resources with different authentication mechanisms provided by AuthenticationHandler. To enable the filter, web application + resources file (ex. web.xml) needs to include a filter class derived from `AuthenticationFilter`. + + For more information have a look at the appropriate + [Hadoop documentation](https://hadoop.apache.org/docs/r2.7.2/hadoop-auth/index.html). + +## Provide Custom Authentication to Oozie Client + +Apache Oozie contains a default class `org.apache.oozie.client.AuthOozieClient` to support Kerberos HTTP SPNEGO authentication, +pseudo/simple authentication and anonymous access for client connections. + +To provide other authentication mechanisms, an Oozie client should extend from `AuthOozieClient` and provide the following +methods should be overridden by derived classes to provide custom authentication: + + * getAuthenticator() : return corresponding Authenticator based on value specified by user at `auth` command option. + * createConnection() : create a singleton class at Authenticator to allow client set and get key-value configuration for + authentication. + +## Provide Custom Authentication to Oozie Server + +To accept custom authentication in Oozie server, a filter extends from AuthenticationFilter must be provided. This filter +delegates to the configured authentication handler for authentication and once it obtains an `AuthenticationToken` from it, sets +a signed HTTP cookie with the token. If HTTP cookie is provided with different key name, its cookie value can be retrieved by +overriding `getToken()` method. Please note, only when `getToken()` return NULL, a custom authentication can be invoked and +processed in `AuthenticationFilter.doFilter()`. + +The following method explains how to read it and return NULL token. + +``` +protected AuthenticationToken getToken(HttpServletRequest request) throws IOException, AuthenticationException { + String tokenStr = null; + Cookie[] cookies = request.getCookies(); + + if (cookies != null) { + for (Cookie cookie : cookies) { + if (cookie.getName().equals(AuthenticatedURL.AUTH_COOKIE)) { + tokenStr = cookie.getValue(); + LOG.info("Got 'hadoop.auth' cookie from request = " + tokenStr); + if (tokenStr != null && !tokenStr.trim().isEmpty()) { + AuthenticationToken retToken = super.getToken(request); + return retToken; + } + } else if (cookie.getName().equals("NEWAUTH")) { + tokenStr = cookie.getValue(); + // DO NOT return the token string so request can authenticated. + } + } + } + return null; + } +``` + +[::Go back to Oozie Documentation Index::](index.html) + + http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/markdown/index.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/index.md b/docs/src/site/markdown/index.md new file mode 100644 index 0000000..0216222 --- /dev/null +++ b/docs/src/site/markdown/index.md @@ -0,0 +1,109 @@ + + +# Oozie, Workflow Engine for Apache Hadoop + +Oozie v3 is a server based _Bundle Engine_ that provides a higher-level oozie abstraction that will batch a set of coordinator applications. The user will be able to start/stop/suspend/resume/rerun a set coordinator jobs in the bundle level resulting a better and easy operational control. + +Oozie v2 is a server based _Coordinator Engine_ specialized in running workflows based on time and data triggers. +(e.g. wait for my input data to exist before running my workflow). + +Oozie v1 is a server based _Workflow Engine_ specialized in running workflow jobs with actions that +execute Hadoop Map/Reduce and Pig jobs. + +<!-- MACRO{toc|fromDepth=1|toDepth=4} --> + +## Distribution Contents + +Oozie distribution consists of a single 'tar.gz' file containing: + + * Readme, license, notice & [Release log](./release-log.txt) files. + * Oozie server: `oozie-server` directory. + * Scripts: `bin/` directory, client and server scripts. + * Binaries: `lib/` directory, client JAR files. + * Configuration: `conf/` server configuration directory. + * Archives: + * `oozie-client-*.tar.gz`: Client tools. + * `oozie.war`: Oozie WAR file. + * `docs.zip`: Documentation. + * `oozie-examples-*.tar.gz`: Examples. + * `oozie-sharelib-*.tar.gz`: Share libraries (with Streaming, Pig JARs). + +## Quick Start + +Enough reading already? Follow the steps in [Oozie Quick Start](DG_QuickStart.html) to get Oozie up and running. + +## Developer Documentation + + * [Overview](DG_Overview.html) + * [Oozie Quick Start](DG_QuickStart.html) + * [Running the Examples](DG_Examples.html) + * [Workflow Functional Specification](WorkflowFunctionalSpec.html) + * [Coordinator Functional Specification](CoordinatorFunctionalSpec.html) + * [Bundle Functional Specification](BundleFunctionalSpec.html) + * [EL Expression Language Quick Reference](https://docs.oracle.com/javaee/7/tutorial/jsf-el.htm) + * [Command Line Tool](DG_CommandLineTool.html) + * [Workflow Re-runs Explained](DG_WorkflowReRun.html) + * [HCatalog Integration Explained](DG_HCatalogIntegration.html) + * [Oozie Client Javadocs](./client/apidocs/index.html) + * [Oozie Core Javadocs](./core/apidocs/index.html) + * [Oozie Web Services API](WebServicesAPI.html) + * [Action Authentication](DG_ActionAuthentication.html) + +### Action Extensions + + * [Email Action](DG_EmailActionExtension.html) + * [Shell Action](DG_ShellActionExtension.html) + * [Hive Action](DG_HiveActionExtension.html) + * [Hive 2 Action](DG_Hive2ActionExtension.html) + * [Sqoop Action](DG_SqoopActionExtension.html) + * [Ssh Action](DG_SshActionExtension.html) + * [DistCp Action](DG_DistCpActionExtension.html) + * [Spark Action](DG_SparkActionExtension.html) + * [Writing a Custom Action Executor](DG_CustomActionExecutor.html) + +### Job Status and SLA Monitoring + + * [JMS Notifications for Job and SLA](DG_JMSNotifications.html) + * [Configuring and Monitoring SLA](DG_SLAMonitoring.html) + +## Administrator Documentation + + * [Oozie Install](AG_Install.html) + * [Oozie Logging](AG_OozieLogging.html) + * [Hadoop Configuration](AG_HadoopConfiguration.html) + * [Action Configuration](AG_ActionConfiguration.html) + * [Oozie Monitoring](AG_Monitoring.html) + * [Command Line Tool](DG_CommandLineTool.html) + * [Oozie Upgrade](AG_OozieUpgrade.html) + +<a name="LicenseInfo"></a> +## Licensing Information + +Oozie is distributed under [Apache License 2.0](http://www.apache.org/licenses/LICENSE-2.0). + +For details on the license of the dependent components, refer to the +[Dependencies Report, Licenses section](./dependencies.html#Licenses). + +Oozie bundles an embedded Jetty 9.x. + +Some of the components in the dependencies report don't mention their license in the published POM. They are: + + * JDOM: [JDOM License](http://www.jdom.org/docs/faq.html#a0030) (Apache style). + * Oro: [Apache License 2.0](http://www.apache.org/licenses/LICENSE-2.0). + + + +## Engineering Documentation + + * [Building Oozie](ENG_Building.html) + * [Dependencies Report](./dependencies.html) + +## MiniOozie Documentation + + * [Testing User Oozie Applications Using MiniOozie](ENG_MiniOozie.html) + +## Oozie User Authentication Documentation + + * [Create Custom Oozie Authentication](ENG_Custom_Authentication.html) + + http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/AG_ActionConfiguration.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/AG_ActionConfiguration.twiki b/docs/src/site/twiki/AG_ActionConfiguration.twiki index 8c032a7..e020d1a 100644 --- a/docs/src/site/twiki/AG_ActionConfiguration.twiki +++ b/docs/src/site/twiki/AG_ActionConfiguration.twiki @@ -1,24 +1,24 @@ -<noautolink> -[[index][::Go back to Oozie Documentation Index::]] ----+!! Action Configuration +[::Go back to Oozie Documentation Index::](index.html) + +# Action Configuration Oozie supports providing default configuration for actions of a particular action type and default configuration for all actions ----++ Hadoop Default Configuration Values +## Hadoop Default Configuration Values -Oozie supports action configuration equivalent to the component's =*-site.xml= and =*.properties= files. +Oozie supports action configuration equivalent to the component's `*-site.xml` and `*.properties` files. -The configuration property in the =oozie-site.xml= is =oozie.service.HadoopAccessorService.action.configurations= -and its value must follow the pattern _[AUTHORITY=ACTION-CONF-DIR_,]*. Where _AUTHORITY_ is the _HOST:PORT_ of +The configuration property in the `oozie-site.xml` is `oozie.service.HadoopAccessorService.action.configurations` +and its value must follow the pattern _\[AUTHORITY=ACTION-CONF-DIR_,\]*. Where _AUTHORITY_ is the _HOST:PORT_ of the Hadoop service (JobTracker/ResourceManager or HDFS). The _ACTION-CONF-DIR_ is the action configuration directory. If the specified directory is a relative path, it will be looked under the Oozie configuration directory. An absolute path can also be specified. Oozie will load and process the action configuration files in the following order. 1. All files in _default_/*.xml (sorted by lexical name, files with names lexically lower have lesser precedence than the following ones), if present. 1. _default_.xml, if present. - 1. All supported files in _actionname_/*, e.g. _actionname_/*.xml and _actionname_/*.properties (based on filename extension, sorted by lexical name, files with names lexically lower have lesser precedence than the following ones), if present. + 1. All supported files in _actionname_/\*, e.g. _actionname_/\*.xml and _actionname_/*.properties (based on filename extension, sorted by lexical name, files with names lexically lower have lesser precedence than the following ones), if present. 1. _actionname_.xml, if present. @@ -26,7 +26,7 @@ For example, for _Hive_ action (which has the _actionname_ defined as _hive_ ), 1. All files in _default_/*.xml, if present 1. _default_.xml, if present. - 1. All files in _hive_/*.xml and _hive_/*.properties, if present + 1. All files in _hive_/\*.xml and _hive_/\*.properties, if present 1. _hive_.xml, if present. @@ -35,22 +35,23 @@ Files processed earlier for an action have the lowest precedence and can have th In addition to explicit authorities, a '*' wildcard is supported. The configuration file associated with the wildcard will be used as default if there is no action configuration for the requested Hadoop service. -For example, the configuration in the =oozie-site.xml= would look like: +For example, the configuration in the `oozie-site.xml` would look like: + -<verbatim> +``` ... <property> <name>oozie.service.HadoopAccessorService.action.configurations</name> <value>*=hadoop-conf,jt-bar:8021=bar-cluster,nn-bar:8020=bar-cluster</value> </property> ... -</verbatim> +``` The action configuration files use the Hadoop configuration syntax. By default Oozie does not define any default action configurations. ----++ Dependency deduplication +## Dependency deduplication Using Oozie with Hadoop 3 may require to have dependency file names distinguishable, which means having two files on sharelib and in your app's dependencies with identical names, leads to job submission failure. @@ -62,7 +63,8 @@ where dependency with greater prio is used. Real world example: You have an application workflow which is uploaded to HDFS in /apps/app directory. You have your app.jar and dependency jars. You also define a spark action in your workflow and set use system libs; the HDFS tree is similar to this: -<verbatim> + +``` + /apps/app/ - app.jar - workflow.xml @@ -75,11 +77,11 @@ You also define a spark action in your workflow and set use system libs; the HDF - jackson-annotations-1.0.jar + oozie - jackson-annotations-1.0.jar -</verbatim> +``` The deduplicator code will create the following list of files: -=/apps/app/app.jar,/apps/app/libs/jackson-annotations-1.0.jar= +`/apps/app/app.jar,/apps/app/libs/jackson-annotations-1.0.jar` And no other files will be passed at job submission. -[[index][::Go back to Oozie Documentation Index::]] +[::Go back to Oozie Documentation Index::](index.html) + -</noautolink> http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/AG_HadoopConfiguration.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/AG_HadoopConfiguration.twiki b/docs/src/site/twiki/AG_HadoopConfiguration.twiki index 528bf4a..ab71d7c 100644 --- a/docs/src/site/twiki/AG_HadoopConfiguration.twiki +++ b/docs/src/site/twiki/AG_HadoopConfiguration.twiki @@ -1,14 +1,15 @@ -<noautolink> -[[index][::Go back to Oozie Documentation Index::]] ----+!! Hadoop Configuration +[::Go back to Oozie Documentation Index::](index.html) ----++ Hadoop Services Whitelisting +# Hadoop Configuration + +## Hadoop Services Whitelisting Oozie supports whitelisting Hadoop services (JobTracker, HDFS), via 2 configuration properties: -<verbatim> + +``` ... <property> <name>oozie.service.HadoopAccessorService.jobTracker.whitelist</name> @@ -25,43 +26,45 @@ Oozie supports whitelisting Hadoop services (JobTracker, HDFS), via 2 configurat </description> </property> ... -</verbatim> +``` -The value must follow the pattern =[AUTHORITY,...]=. Where =AUTHORITY= is the =HOST:PORT= of +The value must follow the pattern `[AUTHORITY,...]`. Where `AUTHORITY` is the `HOST:PORT` of the Hadoop service (JobTracker, HDFS). If the value is empty any HOST:PORT is accepted. Empty is the default value. ----++ Hadoop Default Configuration Values +## Hadoop Default Configuration Values -Oozie supports Hadoop configuration equivalent to the Hadoop =*-site.xml= files. +Oozie supports Hadoop configuration equivalent to the Hadoop `*-site.xml` files. -The configuration property in the =oozie-site.xml= is =oozie.service.HadoopAccessorService.hadoop.configurations= -and its value must follow the pattern =[<AUTHORITY>=<HADOOP_CONF_DIR>,]*=. Where =<AUTHORITY>= is the =HOST:PORT= of -the Hadoop service (JobTracker, HDFS). The =<HADOOP_CONF_DIR>= is a Hadoop configuration directory. If the specified +The configuration property in the `oozie-site.xml` is `oozie.service.HadoopAccessorService.hadoop.configurations` +and its value must follow the pattern `[<AUTHORITY>=<HADOOP_CONF_DIR>,]*`. Where `<AUTHORITY>` is the `HOST:PORT` of +the Hadoop service (JobTracker, HDFS). The `<HADOOP_CONF_DIR>` is a Hadoop configuration directory. If the specified directory is a relative path, it will be looked under the Oozie configuration directory. And absolute path can - also be specified. Oozie will load the Hadoop =*-site.xml= files in the following order: core-site.xml, hdfs-site.xml, + also be specified. Oozie will load the Hadoop `*-site.xml` files in the following order: core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml, hadoop-site.xml, ssl-client.xml. In addition to explicit authorities, a '*' wildcard is supported. The configuration file associated with the wildcard will be used as default if there is no configuration for the requested Hadoop service. -For example, the configuration in the =oozie-site.xml= would look like: +For example, the configuration in the `oozie-site.xml` would look like: + -<verbatim> +``` ... <property> <name>oozie.service.HadoopAccessorService.hadoop.configurations</name> <value>*=hadoop-conf,jt-bar:8021=bar-cluster,nn-bar:8020=bar-cluster</value> </property> ... -</verbatim> +``` The Hadoop configuration files use the Hadoop configuration syntax. -By default Oozie defines =*=hadoop-conf= and the default values of the =hadoop-site.xml= file are: +By default Oozie defines `*=hadoop-conf` and the default values of the `hadoop-site.xml` file are: -<verbatim> + +``` <configuration> <property> <name>mapreduce.jobtracker.kerberos.principal</name> @@ -80,12 +83,12 @@ By default Oozie defines =*=hadoop-conf= and the default values of the =hadoop-s <value>yarn</value> </property> </configuration> -</verbatim> +``` ----++ Limitations +## Limitations All actions in a workflow application must interact with the same Hadoop JobTracker and NameNode. -[[index][::Go back to Oozie Documentation Index::]] +[::Go back to Oozie Documentation Index::](index.html) + -</noautolink>
