OOZIE-2710 Oozie HCatalog example workflow fails (abhishekbafna via shwethags)
Project: http://git-wip-us.apache.org/repos/asf/oozie/repo Commit: http://git-wip-us.apache.org/repos/asf/oozie/commit/64ea00fc Tree: http://git-wip-us.apache.org/repos/asf/oozie/tree/64ea00fc Diff: http://git-wip-us.apache.org/repos/asf/oozie/diff/64ea00fc Branch: refs/heads/oya Commit: 64ea00fc3e38fc54dd1fcdceb96a482575bd2746 Parents: 1901435 Author: Shwetha GS <sshivalingamur...@hortonworks.com> Authored: Fri Oct 21 10:54:14 2016 +0530 Committer: Shwetha GS <sshivalingamur...@hortonworks.com> Committed: Fri Oct 21 10:54:14 2016 +0530 ---------------------------------------------------------------------- examples/src/main/apps/hcatalog/README | 29 +++++++-------------- examples/src/main/apps/hcatalog/job.properties | 2 +- release-log.txt | 1 + sharelib/pig/pom.xml | 24 +++++++++++++++++ 4 files changed, 36 insertions(+), 20 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/oozie/blob/64ea00fc/examples/src/main/apps/hcatalog/README ---------------------------------------------------------------------- diff --git a/examples/src/main/apps/hcatalog/README b/examples/src/main/apps/hcatalog/README index 5f65840..499a90e 100644 --- a/examples/src/main/apps/hcatalog/README +++ b/examples/src/main/apps/hcatalog/README @@ -2,25 +2,16 @@ Running the coordinator example with HCatalog (End-to-end test) 1. Setup services - Hive with JMS (e.g. ActiveMQ Server), HCatalog, database e.g. MySQL, and of course Hadoop 2. This example points to hive.metastore.uris=thrift://localhost:11002. Change it in job.properties if required -3. Create 2 tables 'invites' (input) and 'oozie' (output) with this structure: "create table invites (foo INT, bar INT, car VARCHAR(10)) partitioned by (ds STRING, region STRING)" -4. Oozie distro should be built this way $> bin/mkdistro.sh -Dhcatalog.version=0.4.1 -DskipTests -5. The 'libext' dir used by oozie-setup should contain the following list JARS - hcatalog-core.jar - webhcat-java-client.jar - jackson-mapper-asl-1.8.8.jar - jackson-core-asl-1.8.8.jar - hive-common.jar - hive-metastore.jar - hive-exec.jar - hive-serde.jar - hive-shims.jar - libfb303.jar - (Note) hcatalog JARs will be automatically injected -6. Upload this application directory to HDFS -7. Run Oozie job using the job.properties. Coordinator actions will be in WAITING -8. Make input dependencies available throught HCat client by "alter table invites add partition (ds='2010-01-01', region='usa')". This event will start the workflows with pig action -9. First workflow will SUCCEED as expected, however second one will fail due to 'partition already exists' error. Disregard this. The example demonstrates working as expected. -10. When running in a secure cluster +3. Create 2 tables 'invites' (input) and 'oozie' (output) with this structure: + "create table invites (foo INT, bar INT, car VARCHAR(10)) partitioned by (ds STRING, region STRING)" + "create table oozie (foo INT, bar INT, car VARCHAR(10)) partitioned by (ds STRING, region STRING)" +4. Upload this application directory to HDFS +5. Run Oozie job using the job.properties. Coordinator actions will be in WAITING +6. Make input dependencies available throught HCat client by "alter table invites add partition (ds='2010-01-01', region='usa')". + This event will start the workflows with pig action +7. First workflow will SUCCEED as expected, however second one will fail due to 'partition already exists' error. Disregard this. + The example demonstrates working as expected. +8. When running in a secure cluster a. The files job.properties.security and workflow.xml.security can be used and modified as needed. Please refer to those two files and follow the instructions. http://git-wip-us.apache.org/repos/asf/oozie/blob/64ea00fc/examples/src/main/apps/hcatalog/job.properties ---------------------------------------------------------------------- diff --git a/examples/src/main/apps/hcatalog/job.properties b/examples/src/main/apps/hcatalog/job.properties index e8d1f1b..e0aa18c 100644 --- a/examples/src/main/apps/hcatalog/job.properties +++ b/examples/src/main/apps/hcatalog/job.properties @@ -21,7 +21,7 @@ jobTracker=localhost:8021 queueName=default examplesRoot=examples oozie.use.system.libpath=true - +oozie.action.sharelib.for.pig=pig,hive oozie.coord.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/hcatalog hcatNode=hcat://localhost:11002 http://git-wip-us.apache.org/repos/asf/oozie/blob/64ea00fc/release-log.txt ---------------------------------------------------------------------- diff --git a/release-log.txt b/release-log.txt index bcf2f3a..d4dbb74 100644 --- a/release-log.txt +++ b/release-log.txt @@ -6,6 +6,7 @@ OOZIE-2634 Queue dump command message is confusing when the queue is empty (andr -- Oozie 4.3.0 release +OOZIE-2710 Oozie HCatalog example workflow fails (abhishekbafna via shwethags) OOZIE-2705 Oozie Spark action ignores spark.executor.extraJavaOptions and spark.driver.extraJavaOptions (gezapeti via rkanter) OOZIE-2621 Use hive-exec-<version>-core instead of hive-exec in oozie-core (gezapeti via rkanter) OOZIE-2613 Upgrade hive version from 0.13.1 to 1.2.0 (abhishekbafna via rkanter) http://git-wip-us.apache.org/repos/asf/oozie/blob/64ea00fc/sharelib/pig/pom.xml ---------------------------------------------------------------------- diff --git a/sharelib/pig/pom.xml b/sharelib/pig/pom.xml index e437621..fb656fe 100644 --- a/sharelib/pig/pom.xml +++ b/sharelib/pig/pom.xml @@ -117,6 +117,30 @@ <artifactId>joda-time</artifactId> <scope>compile</scope> </dependency> + <dependency> + <groupId>org.apache.hive.hcatalog</groupId> + <artifactId>hive-hcatalog-pig-adapter</artifactId> + <version>${hive.version}</version> + <exclusions> + <exclusion> + <groupId>org.apache.hive</groupId> + <artifactId>hive-exec</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.hive</groupId> + <artifactId>hive-shims</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.curator</groupId> + <artifactId>curator-framework</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.curator</groupId> + <artifactId>curator-recipes</artifactId> + </exclusion> + </exclusions> + </dependency> + </dependencies> <build>
