[ https://issues.apache.org/jira/browse/HIVE-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909967#comment-13909967 ]
Swarnim Kulkarni commented on HIVE-6147: ---------------------------------------- I am not sure what's going on here. I checked out the latest trunk, applied the patch and re-ran the failing tests locally and they all passed! {noformat} mac-swarnim:hive swarnim$ git pull --rebase Current branch trunk is up to date. mac-swarnim:hive swarnim$ wget https://issues.apache.org/jira/secure/attachment/12629556/HIVE-6147.3.patch.txt --2014-02-23 18:40:27-- https://issues.apache.org/jira/secure/attachment/12629556/HIVE-6147.3.patch.txt Resolving issues.apache.org... 140.211.11.121 Connecting to issues.apache.org|140.211.11.121|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 212743 (208K) [text/plain] Saving to: `HIVE-6147.3.patch.txt' 100%[=====================================================================================================================>] 212,743 29.5K/s in 7.0s 2014-02-23 18:40:37 (29.5 KB/s) - `HIVE-6147.3.patch.txt' saved [212743/212743] mac-swarnim:hive swarnim$ git status # On branch trunk # Untracked files: # (use "git add <file>..." to include in what will be committed) # # HIVE-6147.3.patch.txt nothing added to commit but untracked files present (use "git add" to track) mac-swarnim:hive swarnim$ patch -p0 < HIVE-6147.3.patch.txt patching file hbase-handler/pom.xml patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Address.java patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/ContactInfo.java patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Employee.java patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/EmployeeAvro.java patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Gender.java patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/HomePhone.java patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Magic.java patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/OfficePhone.java patching file hbase-handler/src/if/avro/avro_test.avpr patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseCompositeKey.java patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeHelper.java patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java patching file hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestAvroSchemaRetriever.java patching file hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestCompositeKey.java patching file hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroGenericRecordWritable.java patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroLazyObjectInspector.java patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorException.java patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorGenerator.java patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaRetriever.java patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUnion.java patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyObjectInspectorFactory.java patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazySimpleStructObjectInspector.java patching file serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java mac-swarnim:hive swarnim$ git status # On branch trunk # Changes not staged for commit: # (use "git add <file>..." to update what will be committed) # (use "git checkout -- <file>..." to discard changes in working directory) # # modified: hbase-handler/pom.xml # modified: hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseCompositeKey.java # modified: hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java # modified: hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java # modified: hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestCompositeKey.java # modified: hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroGenericRecordWritable.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorGenerator.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUnion.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyObjectInspectorFactory.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazySimpleStructObjectInspector.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java # # Untracked files: # (use "git add <file>..." to include in what will be committed) # # HIVE-6147.3.patch.txt # hbase-handler/src/gen/ # hbase-handler/src/if/ # hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeHelper.java # hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestAvroSchemaRetriever.java # serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroLazyObjectInspector.java # serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorException.java # serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaRetriever.java mac-swarnim:hive swarnim$ cd hcatalog/ mac-swarnim:hcatalog swarnim$ mvn clean install -pl hcatalog-pig-adapter -P hadoop-1 [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Hive HCatalog Pig Adapter 0.13.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-hcatalog-pig-adapter --- [INFO] Deleting /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target [INFO] Deleting /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter (includes = [datanucleus.log, derby.log], excludes = []) [INFO] Deleting /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/build (includes = [], excludes = []) [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-hcatalog-pig-adapter --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/src/main/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-hcatalog-pig-adapter --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-hcatalog-pig-adapter --- [INFO] Compiling 10 source files to /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/classes [WARNING] Note: Some input files use or override a deprecated API. [WARNING] Note: Recompile with -Xlint:deprecation for details. [WARNING] Note: Some input files use unchecked or unsafe operations. [WARNING] Note: Recompile with -Xlint:unchecked for details. [INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-hcatalog-pig-adapter --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/src/test/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-hcatalog-pig-adapter --- [INFO] Executing tasks main: [mkdir] Created dir: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/tmp [mkdir] Created dir: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/warehouse [mkdir] Created dir: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/tmp/conf [copy] Copying 5 files to /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-hcatalog-pig-adapter --- [INFO] Compiling 26 source files to /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/test-classes [WARNING] Note: Some input files use or override a deprecated API. [WARNING] Note: Recompile with -Xlint:deprecation for details. [INFO] [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-hcatalog-pig-adapter --- [INFO] Surefire report directory: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/surefire-reports ------------------------------------------------------- T E S T S ------------------------------------------------------- ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hcatalog.pig.TestHCatLoader 2014-02-23 16:42:49.909 java[65062:1903] Unable to load realm info from SCDynamicStore Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.476 sec - in org.apache.hcatalog.pig.TestHCatLoader Running org.apache.hcatalog.pig.TestHCatLoaderComplexSchema 2014-02-23 16:43:45.590 java[65112:1903] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.887 sec - in org.apache.hcatalog.pig.TestHCatLoaderComplexSchema Running org.apache.hcatalog.pig.TestHCatLoaderStorer 2014-02-23 16:44:14.765 java[65319:1903] Unable to load realm info from SCDynamicStore Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.618 sec - in org.apache.hcatalog.pig.TestHCatLoaderStorer Running org.apache.hcatalog.pig.TestHCatStorer 2014-02-23 16:44:26.599 java[65328:1903] Unable to load realm info from SCDynamicStore Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.412 sec - in org.apache.hcatalog.pig.TestHCatStorer Running org.apache.hcatalog.pig.TestHCatStorerMulti 2014-02-23 16:44:47.482 java[65363:1903] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.681 sec - in org.apache.hcatalog.pig.TestHCatStorerMulti Running org.apache.hcatalog.pig.TestHCatStorerWrapper 2014-02-23 16:45:01.495 java[65377:1903] Unable to load realm info from SCDynamicStore Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.07 sec - in org.apache.hcatalog.pig.TestHCatStorerWrapper Running org.apache.hcatalog.pig.TestOrcHCatLoader 2014-02-23 16:45:06.985 java[65384:1903] Unable to load realm info from SCDynamicStore Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.718 sec - in org.apache.hcatalog.pig.TestOrcHCatLoader Running org.apache.hcatalog.pig.TestOrcHCatStorer 2014-02-23 16:46:04.004 java[65438:1903] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.293 sec - in org.apache.hcatalog.pig.TestOrcHCatStorer Running org.apache.hcatalog.pig.TestPigHCatUtil Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.512 sec - in org.apache.hcatalog.pig.TestPigHCatUtil Running org.apache.hive.hcatalog.pig.TestE2EScenarios 2014-02-23 16:46:16.502 java[65454:1903] Unable to load realm info from SCDynamicStore Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.143 sec - in org.apache.hive.hcatalog.pig.TestE2EScenarios Running org.apache.hive.hcatalog.pig.TestHCatLoader 2014-02-23 16:46:25.991 java[65461:1903] Unable to load realm info from SCDynamicStore Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.261 sec - in org.apache.hive.hcatalog.pig.TestHCatLoader Running org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema 2014-02-23 16:47:30.547 java[65536:1903] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.867 sec - in org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema Running org.apache.hive.hcatalog.pig.TestHCatLoaderStorer 2014-02-23 16:47:59.693 java[65744:1903] Unable to load realm info from SCDynamicStore Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.167 sec - in org.apache.hive.hcatalog.pig.TestHCatLoaderStorer Running org.apache.hive.hcatalog.pig.TestHCatStorer 2014-02-23 16:48:11.201 java[65754:1903] Unable to load realm info from SCDynamicStore Tests run: 27, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 101.175 sec - in org.apache.hive.hcatalog.pig.TestHCatStorer Running org.apache.hive.hcatalog.pig.TestHCatStorerMulti 2014-02-23 16:49:50.434 java[65845:1903] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.798 sec - in org.apache.hive.hcatalog.pig.TestHCatStorerMulti Running org.apache.hive.hcatalog.pig.TestHCatStorerWrapper 2014-02-23 16:50:05.471 java[65860:1903] Unable to load realm info from SCDynamicStore Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.097 sec - in org.apache.hive.hcatalog.pig.TestHCatStorerWrapper Running org.apache.hive.hcatalog.pig.TestOrcHCatLoader 2014-02-23 16:50:11.008 java[65866:1903] Unable to load realm info from SCDynamicStore Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.125 sec - in org.apache.hive.hcatalog.pig.TestOrcHCatLoader Running org.apache.hive.hcatalog.pig.TestOrcHCatLoaderComplexSchema 2014-02-23 16:51:15.432 java[65940:1903] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.779 sec - in org.apache.hive.hcatalog.pig.TestOrcHCatLoaderComplexSchema Running org.apache.hive.hcatalog.pig.TestOrcHCatStorer 2014-02-23 16:51:42.577 java[66137:1903] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.179 sec - in org.apache.hive.hcatalog.pig.TestOrcHCatStorer Running org.apache.hive.hcatalog.pig.TestPigHCatUtil Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.513 sec - in org.apache.hive.hcatalog.pig.TestPigHCatUtil Results : Tests run: 101, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-hcatalog-pig-adapter --- [INFO] Building jar: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.jar [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-hcatalog-pig-adapter --- [INFO] Installing /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.jar to /Users/swarnim/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-pig-adapter/0.13.0-SNAPSHOT/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.jar [INFO] Installing /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/pom.xml to /Users/swarnim/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-pig-adapter/0.13.0-SNAPSHOT/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.pom [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 9:11.629s [INFO] Finished at: Sun Feb 23 18:51:54 CST 2014 [INFO] Final Memory: 24M/81M [INFO] ------------------------------------------------------------------------ {noformat} > Support avro data stored in HBase columns > ----------------------------------------- > > Key: HIVE-6147 > URL: https://issues.apache.org/jira/browse/HIVE-6147 > Project: Hive > Issue Type: Bug > Components: HBase Handler > Affects Versions: 0.12.0 > Reporter: Swarnim Kulkarni > Assignee: Swarnim Kulkarni > Attachments: HIVE-6147.1.patch.txt, HIVE-6147.2.patch.txt, > HIVE-6147.3.patch.txt > > > Presently, the HBase Hive integration supports querying only primitive data > types in columns. It would be nice to be able to store and query Avro objects > in HBase columns by making them visible as structs to Hive. This will allow > Hive to perform ad hoc analysis of HBase data which can be deeply structured. -- This message was sent by Atlassian JIRA (v6.1.5#6160)