[jira] [Commented] (HIVE-2589) Newly created partition should inherit properties from table
[ https://issues.apache.org/jira/browse/HIVE-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13190365#comment-13190365 ] Hudson commented on HIVE-2589: -- Integrated in Hive-trunk-h0.21 #1212 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1212/]) HIVE-2589 : Newly created partition should inherit properties from table (Ashutosh Chauhan) hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1234235 Files : * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/conf/hive-default.xml.template * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java * /hive/trunk/ql/src/test/queries/clientpositive/part_inherit_tbl_props.q * /hive/trunk/ql/src/test/queries/clientpositive/part_inherit_tbl_props_empty.q * /hive/trunk/ql/src/test/queries/clientpositive/part_inherit_tbl_props_with_star.q * /hive/trunk/ql/src/test/results/clientpositive/part_inherit_tbl_props.q.out * /hive/trunk/ql/src/test/results/clientpositive/part_inherit_tbl_props_empty.q.out * /hive/trunk/ql/src/test/results/clientpositive/part_inherit_tbl_props_with_star.q.out Newly created partition should inherit properties from table Key: HIVE-2589 URL: https://issues.apache.org/jira/browse/HIVE-2589 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.8.1, 0.9.0 Attachments: HIVE-2589.D1335.1.patch, HIVE-2589.D1335.2.patch, hive-2589.patch, hive-2589.patch, hive-2589_1.patch, hive-2589_2.patch, hive-2589_3.patch, hive-2589_4.patch This will make all the info contained in table properties available to partitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 1212 - Still Failing
Changes for Build #1178 Changes for Build #1179 Changes for Build #1180 Changes for Build #1181 Changes for Build #1182 [heyongqiang] HIVE-2621:Allow multiple group bys with the same input data and spray keys to be run on the same reducer. (Kevin via He Yongqiang) Changes for Build #1184 [namit] HIVE-2690 a bug in 'alter table concatenate' that causes filenames getting double url encoded (He Yongqiang via namit) Changes for Build #1185 Changes for Build #1186 Changes for Build #1187 Changes for Build #1188 Changes for Build #1189 Changes for Build #1190 [amareshwari] HIVE-2629. Make a single Hive binary work with both 0.20.x and 0.23.0. (Thomas Weise via amareshwari) Changes for Build #1191 [amareshwari] HIVE-2629. Reverting previous commit Changes for Build #1192 [heyongqiang] HIVE-2706 [jira] StackOverflowError when using custom UDF after adding archive after adding jars (Kevin Wilfong via Yongqiang He) Summary: https://issues.apache.org/jira/browse/HIVE-2706 The issue was that the current thread's classloader and the classloader in the conf differed due to the prehook updating only the current thread's classloader with new jars. Now, it updates both classloaders, fixing the issue. When a custom UDF is used in a query after add an archive, such as a zip file, after adding jars, the XMLEncoder enters an infinite loop when serializing the map reduce task, as part of sending it to be executed. This results in a stack overflow error. Test Plan: Verified it fixed the stack overflow error. Reviewers: JIRA, heyongqiang, njain Reviewed By: heyongqiang CC: heyongqiang Differential Revision: https://reviews.facebook.net/D1167 Changes for Build #1193 [hashutosh] HIVE-2705: SemanticAnalyzer twice swallows an exception it shouldn't (jghoman via hashutosh) Changes for Build #1194 Changes for Build #1195 [hashutosh] HIVE-2589: Newly created partition should inherit properties from table (Ashutosh Chauhan) [hashutosh] HIVE-2682: Clean-up logs (Rajat Goel via Ashutosh Chauhan) Changes for Build #1196 [amareshwari] HIVE-2629. Make a single Hive binary work with both 0.20.x and 0.23.0. (Thomas Weise via amareshwari) Changes for Build #1197 Changes for Build #1198 [namit] HIVE-2504 Warehouse table subdirectories should inherit the group permissions of the warehouse parent directory (Chinna Rao Lalam via namit) [namit] HIVE-2695 Add PRINTF() Udf (Zhenxiao Luo via namit) Changes for Build #1199 Changes for Build #1200 Changes for Build #1201 Changes for Build #1202 Changes for Build #1203 Changes for Build #1204 [cws] HIVE-2719. Revert HIVE-2589 (He Yongqiang via cws) Changes for Build #1205 Changes for Build #1207 [namit] HIVE-2718 NPE in union followed by join (He Yongqiang via namit) Changes for Build #1208 Changes for Build #1209 Changes for Build #1210 [namit] HIVE-2674 get_partitions_ps throws TApplicationException if table doesn't exist (Kevin Wilfong via namit) Changes for Build #1211 [cws] HIVE-2203. Extend concat_ws() UDF to support arrays of strings (Zhenxiao Luo via cws) [cws] HIVE-2279. Implement sort(array) UDF (Zhenxiao Luo via cws) Changes for Build #1212 [hashutosh] HIVE-2589 : Newly created partition should inherit properties from table (Ashutosh Chauhan) 2 tests failed. FAILED: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppr_pushdown Error Message: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. Stack Trace: junit.framework.AssertionFailedError: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. at junit.framework.Assert.fail(Assert.java:50) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppr_pushdown(TestCliDriver.java:26051) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) FAILED:
[jira] [Commented] (HIVE-1634) Allow access to Primitive types stored in binary format in HBase
[ https://issues.apache.org/jira/browse/HIVE-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13190370#comment-13190370 ] Developer EA commented on HIVE-1634: It's unclear from the last comment whether the issue is resolved or not.I tried with hive-hbase-handler-0.7.1.jar, but the issue still exists. Can you suggest work around for this. Allow access to Primitive types stored in binary format in HBase Key: HIVE-1634 URL: https://issues.apache.org/jira/browse/HIVE-1634 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.7.0 Reporter: Basab Maulik Assignee: Basab Maulik Attachments: HIVE-1634.0.patch, HIVE-1634.1.patch, TestHiveHBaseExternalTable.java This addresses HIVE-1245 in part, for atomic or primitive types. The serde property hbase.columns.storage.types = -,b,b,b,b,b,b,b,b is a specification of the storage option for the corresponding column in the serde property hbase.columns.mapping. Allowed values are '-' for table default, 's' for standard string storage, and 'b' for binary storage as would be obtained from o.a.h.hbase.utils.Bytes. Map types for HBase column families use a colon separated pair such as 's:b' for the key and value part specifiers respectively. See the test cases and queries for HBase handler for additional examples. There is also a table property hbase.table.default.storage.type = string to specify a table level default storage type. The other valid specification is binary. The table level default is overridden by a column level specification. This control is available for the boolean, tinyint, smallint, int, bigint, float, and double primitive types. The attached patch also relaxes the mapping of map types to HBase column families to allow any primitive type to be the map key. Attached is a program for creating a table and populating it in HBase. The external table in Hive can access the data as shown in the example below. hive create external table TestHiveHBaseExternalTable (key string, c_bool boolean, c_byte tinyint, c_short smallint, c_int int, c_long bigint, c_string string, c_float float, c_double double) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties (hbase.columns.mapping = :key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double) tblproperties (hbase.table.name = TestHiveHBaseExternalTable); OK Time taken: 0.691 seconds hive select * from TestHiveHBaseExternalTable; OK key-1 NULLNULLNULLNULLNULLTest-String NULLNULL Time taken: 0.346 seconds hive drop table TestHiveHBaseExternalTable; OK Time taken: 0.139 seconds hive create external table TestHiveHBaseExternalTable (key string, c_bool boolean, c_byte tinyint, c_short smallint, c_int int, c_long bigint, c_string string, c_float float, c_double double) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties ( hbase.columns.mapping = :key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double, hbase.columns.storage.types = -,b,b,b,b,b,b,b,b ) tblproperties ( hbase.table.name = TestHiveHBaseExternalTable, hbase.table.default.storage.type = string); OK Time taken: 0.139 seconds hive select * from TestHiveHBaseExternalTable; OK key-1 true-128-32768 -2147483648 -9223372036854775808 Test-String -2.1793132E-11 2.01345E291 Time taken: 0.151 seconds hive drop table TestHiveHBaseExternalTable; OK Time taken: 0.154 seconds hive create external table TestHiveHBaseExternalTable (key string, c_bool boolean, c_byte tinyint, c_short smallint, c_int int, c_long bigint, c_string string, c_float float, c_double double) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties ( hbase.columns.mapping = :key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double, hbase.columns.storage.types = -,b,b,b,b,b,-,b,b ) tblproperties (hbase.table.name = TestHiveHBaseExternalTable); OK Time taken: 0.347 seconds hive select * from TestHiveHBaseExternalTable; OK key-1 true-128-32768 -2147483648 -9223372036854775808 Test-String -2.1793132E-11 2.01345E291 Time taken: 0.245 seconds hive -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hbase vs Hive ve Hbase/Hive vs Pig
H all,I know I have asked this question before.. But I want to state more specific...Whats the difference in performance between Hive, Hbase, Hive/Hbase Integration, Pig in case of transaction Processing versus Query Processing??Which suits transaction processing most and which for Query Processing?And Can I use any of these with each others i.e use API of Pig and API of Hive and that of Hbase for three different methods..Thanks and waiting for your help asap
Hive-trunk-h0.21 - Build # 1213 - Still Failing
Changes for Build #1178 Changes for Build #1179 Changes for Build #1180 Changes for Build #1181 Changes for Build #1182 [heyongqiang] HIVE-2621:Allow multiple group bys with the same input data and spray keys to be run on the same reducer. (Kevin via He Yongqiang) Changes for Build #1184 [namit] HIVE-2690 a bug in 'alter table concatenate' that causes filenames getting double url encoded (He Yongqiang via namit) Changes for Build #1185 Changes for Build #1186 Changes for Build #1187 Changes for Build #1188 Changes for Build #1189 Changes for Build #1190 [amareshwari] HIVE-2629. Make a single Hive binary work with both 0.20.x and 0.23.0. (Thomas Weise via amareshwari) Changes for Build #1191 [amareshwari] HIVE-2629. Reverting previous commit Changes for Build #1192 [heyongqiang] HIVE-2706 [jira] StackOverflowError when using custom UDF after adding archive after adding jars (Kevin Wilfong via Yongqiang He) Summary: https://issues.apache.org/jira/browse/HIVE-2706 The issue was that the current thread's classloader and the classloader in the conf differed due to the prehook updating only the current thread's classloader with new jars. Now, it updates both classloaders, fixing the issue. When a custom UDF is used in a query after add an archive, such as a zip file, after adding jars, the XMLEncoder enters an infinite loop when serializing the map reduce task, as part of sending it to be executed. This results in a stack overflow error. Test Plan: Verified it fixed the stack overflow error. Reviewers: JIRA, heyongqiang, njain Reviewed By: heyongqiang CC: heyongqiang Differential Revision: https://reviews.facebook.net/D1167 Changes for Build #1193 [hashutosh] HIVE-2705: SemanticAnalyzer twice swallows an exception it shouldn't (jghoman via hashutosh) Changes for Build #1194 Changes for Build #1195 [hashutosh] HIVE-2589: Newly created partition should inherit properties from table (Ashutosh Chauhan) [hashutosh] HIVE-2682: Clean-up logs (Rajat Goel via Ashutosh Chauhan) Changes for Build #1196 [amareshwari] HIVE-2629. Make a single Hive binary work with both 0.20.x and 0.23.0. (Thomas Weise via amareshwari) Changes for Build #1197 Changes for Build #1198 [namit] HIVE-2504 Warehouse table subdirectories should inherit the group permissions of the warehouse parent directory (Chinna Rao Lalam via namit) [namit] HIVE-2695 Add PRINTF() Udf (Zhenxiao Luo via namit) Changes for Build #1199 Changes for Build #1200 Changes for Build #1201 Changes for Build #1202 Changes for Build #1203 Changes for Build #1204 [cws] HIVE-2719. Revert HIVE-2589 (He Yongqiang via cws) Changes for Build #1205 Changes for Build #1207 [namit] HIVE-2718 NPE in union followed by join (He Yongqiang via namit) Changes for Build #1208 Changes for Build #1209 Changes for Build #1210 [namit] HIVE-2674 get_partitions_ps throws TApplicationException if table doesn't exist (Kevin Wilfong via namit) Changes for Build #1211 [cws] HIVE-2203. Extend concat_ws() UDF to support arrays of strings (Zhenxiao Luo via cws) [cws] HIVE-2279. Implement sort(array) UDF (Zhenxiao Luo via cws) Changes for Build #1212 [hashutosh] HIVE-2589 : Newly created partition should inherit properties from table (Ashutosh Chauhan) Changes for Build #1213 2 tests failed. REGRESSION: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert2_overwrite_partitions Error Message: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. Stack Trace: junit.framework.AssertionFailedError: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. at junit.framework.Assert.fail(Assert.java:50) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert2_overwrite_partitions(TestCliDriver.java:17171) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at
HBQL Questions
Hiii all, Do anyone know any info about HBQL ?? Good/ Bad ?? Performance??
