Review Request: HIVE-1696
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/189/ --- Review request for hive. Summary --- HIVE-1696 This addresses bug HIVE-1696. https://issues.apache.org/jira/browse/HIVE-1696 Diffs - http://svn.apache.org/repos/asf/hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/metastore/if/hive_metastore.thrift 1050266 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/shims/src/0.20S/java/org/apache/hadoop/hive/thrift/DelegationTokenIdentifier.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/shims/src/0.20S/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/shims/src/0.20S/java/org/apache/hadoop/hive/thrift/DelegationTokenSelector.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/shims/src/0.20S/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge20S.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/shims/src/common/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge.java 1050266 Diff: https://reviews.apache.org/r/189/diff Testing --- Thanks, John
Review Request: HIVE-78 patch 10
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/187/ --- Review request for hive. Summary --- review by JVS This addresses bug HIVE-78. https://issues.apache.org/jira/browse/HIVE-78 Diffs - http://svn.apache.org/repos/asf/hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/conf/hive-default.xml 1050266 http://svn.apache.org/repos/asf/hive/trunk/metastore/if/hive_metastore.thrift 1050266 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MColumnPrivilege.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MDBPrivilege.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MGlobalPrivilege.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MRole.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MRoleMap.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MTablePartitionPrivilege.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/package.jdo 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/AuthorizationException.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveUtils.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/GrantDesc.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/GrantRevokeRoleDDL.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PrincipalDesc.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PrivilegeDesc.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PrivilegeObjectDesc.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/RevokeDesc.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/RoleDDLDesc.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ShowGrantDesc.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/security/HadoopDefaultAuthenticator.java
Re: Review Request: HIVE-78 patch 10
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/187/#review76 --- http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java https://reviews.apache.org/r/187/#comment114 Here and elsewhere in this file and ObjectStore, why are we throwing RuntimeExceptions instead of MetaExceptions? http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java https://reviews.apache.org/r/187/#comment115 We should just ignore attempts to create a duplicate grant (of either role or privilege). But if the old one does not have grant option and the new one does, then we should upgrade it in place. (Spoke with Yongqiang; we'll defer this to a followup since it requires adding an update API to the metastore for the WITH GRANT OPTION upgrade.) http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/package.jdo https://reviews.apache.org/r/187/#comment101 Make this 128. http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/package.jdo https://reviews.apache.org/r/187/#comment113 When we define column-level privs (independent of partition), will PART_ID be null? That would be bad since a unique index ignores duplicates in the presence of any nulls. Do we have something like a NO_PARTITION_ID reserved value? Another option is to use a single key column TABLE_OR_PART_ID and add an addition boolean IS_PARTITION (true for partition-level, false for table-level). http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/package.jdo https://reviews.apache.org/r/187/#comment102 Make this 128. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java https://reviews.apache.org/r/187/#comment103 Don't use printStackTrace http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveUtils.java https://reviews.apache.org/r/187/#comment104 should be Class.forName(className, true, JavaUtils.getClassLoader()) http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveUtils.java https://reviews.apache.org/r/187/#comment105 should be Class.forName(className, true, JavaUtils.getClassLoader()) http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g https://reviews.apache.org/r/187/#comment106 should be revoke role http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g https://reviews.apache.org/r/187/#comment107 should be show role grants http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/security/HiveAuthenticationProvider.java https://reviews.apache.org/r/187/#comment108 What is the return value of this method supposed to indicate? Make it void and expect an exception instead. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProvider.java https://reviews.apache.org/r/187/#comment110 Should be Hive's pluggable authorization provider interface http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProvider.java https://reviews.apache.org/r/187/#comment109 This should take no parameters, and should rely on setConf already having been called. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/Privilege.java https://reviews.apache.org/r/187/#comment111 Javadoc? http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeRegistry.java https://reviews.apache.org/r/187/#comment112 Javadoc? - John On 2010-12-20 14:30:53, John Sichi wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/187/ --- (Updated 2010-12-20 14:30:53) Review request for hive. Summary --- review by JVS This addresses bug HIVE-78. https://issues.apache.org/jira/browse/HIVE-78 Diffs - http://svn.apache.org/repos/asf/hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/conf/hive-default.xml 1050266 http://svn.apache.org/repos/asf/hive/trunk/metastore/if/hive_metastore.thrift 1050266 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive
[jira] Commented: (HIVE-78) Authorization infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-78?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12973388#action_12973388 ] John Sichi commented on HIVE-78: A few more comments on patch 10 in https://reviews.apache.org/r/187/ Authorization infrastructure for Hive - Key: HIVE-78 URL: https://issues.apache.org/jira/browse/HIVE-78 Project: Hive Issue Type: New Feature Components: Metastore, Query Processor, Server Infrastructure Reporter: Ashish Thusoo Assignee: He Yongqiang Attachments: createuser-v1.patch, hive-78-metadata-v1.patch, hive-78-syntax-v1.patch, HIVE-78.1.nothrift.patch, HIVE-78.1.thrift.patch, HIVE-78.10.no_thrift.patch, HIVE-78.2.nothrift.patch, HIVE-78.2.thrift.patch, HIVE-78.4.complete.patch, HIVE-78.4.no_thrift.patch, HIVE-78.5.complete.patch, HIVE-78.5.no_thrift.patch, HIVE-78.6.complete.patch, HIVE-78.6.no_thrift.patch, HIVE-78.7.no_thrift.patch, HIVE-78.7.patch, HIVE-78.9.no_thrift.patch, HIVE-78.9.patch, hive-78.diff Allow hive to integrate with existing user repositories for authentication and authorization infromation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1857) mixed case tablename on lefthand side of LATERAL VIEW results in query failing with confusing error message
mixed case tablename on lefthand side of LATERAL VIEW results in query failing with confusing error message --- Key: HIVE-1857 URL: https://issues.apache.org/jira/browse/HIVE-1857 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi Assignee: John Sichi Fix For: 0.7.0 Attachments: HIVE-1857.1.patch For the modified query below in lateral_view.q, the exception org.apache.hadoop.hive.ql.parse.SemanticException: line 3:7 Invalid Table Alias or Column Reference myCol is thrown. The query should succeed. SELECT myCol from tmp_PYANG_lv LATERAL VIEW explode(array(1,2,3)) myTab as myCol limit 3; -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1857) mixed case tablename on lefthand side of LATERAL VIEW results in query failing with confusing error message
[ https://issues.apache.org/jira/browse/HIVE-1857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1857: - Attachment: HIVE-1857.1.patch mixed case tablename on lefthand side of LATERAL VIEW results in query failing with confusing error message --- Key: HIVE-1857 URL: https://issues.apache.org/jira/browse/HIVE-1857 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi Assignee: John Sichi Fix For: 0.7.0 Attachments: HIVE-1857.1.patch For the modified query below in lateral_view.q, the exception org.apache.hadoop.hive.ql.parse.SemanticException: line 3:7 Invalid Table Alias or Column Reference myCol is thrown. The query should succeed. SELECT myCol from tmp_PYANG_lv LATERAL VIEW explode(array(1,2,3)) myTab as myCol limit 3; -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1857) mixed case tablename on lefthand side of LATERAL VIEW results in query failing with confusing error message
[ https://issues.apache.org/jira/browse/HIVE-1857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1857: - Status: Patch Available (was: Open) Fix is in HIVE-1857.1.patch. mixed case tablename on lefthand side of LATERAL VIEW results in query failing with confusing error message --- Key: HIVE-1857 URL: https://issues.apache.org/jira/browse/HIVE-1857 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi Assignee: John Sichi Fix For: 0.7.0 Attachments: HIVE-1857.1.patch For the modified query below in lateral_view.q, the exception org.apache.hadoop.hive.ql.parse.SemanticException: line 3:7 Invalid Table Alias or Column Reference myCol is thrown. The query should succeed. SELECT myCol from tmp_PYANG_lv LATERAL VIEW explode(array(1,2,3)) myTab as myCol limit 3; -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Review Request: HIVE-1856
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/188/ --- Review request for hive. Summary --- review by JVS This addresses bug HIVE-1856. https://issues.apache.org/jira/browse/HIVE-1856 Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2508f2b conf/hive-default.xml fdc9a82 ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java cbd9d62 ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 6b47702 ql/src/test/queries/clientnegative/drop_table_failure1.q PRE-CREATION ql/src/test/queries/clientnegative/drop_table_failure2.q PRE-CREATION ql/src/test/queries/clientnegative/drop_view_failure2.q e8e78a3 ql/src/test/queries/clientpositive/drop_table.q PRE-CREATION ql/src/test/queries/clientpositive/drop_view.q PRE-CREATION ql/src/test/results/clientnegative/drop_table_failure1.q.out PRE-CREATION ql/src/test/results/clientnegative/drop_table_failure2.q.out PRE-CREATION ql/src/test/results/clientnegative/drop_view_failure2.q.out 80903f9 ql/src/test/results/clientpositive/drop_table.q.out PRE-CREATION ql/src/test/results/clientpositive/drop_view.q.out PRE-CREATION Diff: https://reviews.apache.org/r/188/diff Testing --- Thanks, John
[jira] Updated: (HIVE-1856) Implement DROP TABLE/VIEW ... IF EXISTS
[ https://issues.apache.org/jira/browse/HIVE-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1856: - Status: Open (was: Patch Available) Review comments in https://reviews.apache.org/r/188/ Note that there are also DROP commands for databases, indexes, partitions, and temporary functions. This config property should cover those too, so we should either address those as well in this patch, or open a followup JIRA. Implement DROP TABLE/VIEW ... IF EXISTS Key: HIVE-1856 URL: https://issues.apache.org/jira/browse/HIVE-1856 Project: Hive Issue Type: New Feature Affects Versions: 0.7.0 Reporter: Marcel Kornacker Assignee: Marcel Kornacker Attachments: hive-1856.patch This issue combines issues HIVE-1550/1165/1542/1551: - augment DROP TABLE/VIEW with IF EXISTS - signal an error if the table/view doesn't exist and IF EXISTS wasn't specified - introduce a flag in the configuration that allows you to turn off the new behavior -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Review Request: HIVE-78 patch 9
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/183/ --- Review request for hive. Summary --- Review by JVS. Note that the patch had some conflicts, so I wasn't able to test this version; I'll do more testing and commenting after the next patch. Diffs - http://svn.apache.org/repos/asf/hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/conf/hive-default.xml 1050266 http://svn.apache.org/repos/asf/hive/trunk/metastore/if/hive_metastore.thrift 1050266 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MColumnPrivilege.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MDBPrivilege.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MGlobalPrivilege.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MRole.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MRoleMap.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MTablePartitionPrivilege.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/package.jdo 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/AuthorizationException.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java 1050266 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/GrantDesc.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/GrantRevokeRoleDDL.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PrincipalDesc.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PrivilegeDesc.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PrivilegeObjectDesc.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/RevokeDesc.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/RoleDDLDesc.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ShowGrantDesc.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/security/Authenticator.java PRE-CREATION
[jira] Commented: (HIVE-1853) downgrade JDO version
[ https://issues.apache.org/jira/browse/HIVE-1853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12972669#action_12972669 ] John Sichi commented on HIVE-1853: -- Yeah, Hudson agrees with Ashutosh: https://hudson.apache.org/hudson/job/Hive-trunk-h0.20/445/testReport/junit/org.apache.hadoop.hive.metastore/TestEmbeddedHiveMetaStore/testPartitionFilter/ downgrade JDO version - Key: HIVE-1853 URL: https://issues.apache.org/jira/browse/HIVE-1853 Project: Hive Issue Type: Bug Affects Versions: 0.7.0 Reporter: Namit Jain Assignee: Paul Yang Fix For: 0.7.0 Attachments: HIVE-1853.1.patch, HIVE-1853.2.patch After HIVE-1609, we are seeing some table not found errors intermittently. We have a test case where 5 processes are concurrently issueing the same query - explain extended insert .. select from T and once in a while, we get a error T not found - When we revert back the JDO version, the error is gone. We can investigate later to find the JDO bug, but for now this is a show-stopper for facebook, and needs to be reverted back immediately. This also means, that the filters will not be pushed to mysql. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1542) DROP TABLE tablename should raise an error when tablename does not exist
[ https://issues.apache.org/jira/browse/HIVE-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12972269#action_12972269 ] John Sichi commented on HIVE-1542: -- Hi Marcel :) DROP TABLE tablename should raise an error when tablename does not exist Key: HIVE-1542 URL: https://issues.apache.org/jira/browse/HIVE-1542 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Carl Steinbach Assignee: Marcel Kornacker See http://dev.mysql.com/doc/refman/5.1/en/drop-table.html -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-78) Authorization infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-78?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12971161#action_12971161 ] John Sichi commented on HIVE-78: Some more from me: * There's a bug when attempting to grant multiple privileges at once; only one of them is getting granted (what I showed you in CLI) * Multiple grants from the same grantor to the same grantee should not result in duplicates (verify against Oracle), and we should collapse everything into one row no matter whether the grants were made at the same or different times (sort privilege names for determinism) * revokeAllPrivileges should revoke role grants as well * Role cycle is not being prevented * try/finally around transactions in ObjectStore should be used consistently (I know there are some cases which were already missing them, but we shouldn't make it worse) * Don't use printStackTrace * show [role] grant role unknown should fail (even though we have to tolerate unknown for user/group since we don't have a table for those) Some additional points noted at code review session: * Need many many negative tests * Provide a way to make partitions inherit from table (and make it the default) * Define a UNIQUE key for the priv tables in JDO * GRANT should mark WriteEntity for replication etc More Typos: * candicate * anaylze I have some more code-level comments but not all of them may be relevant after the issues above have been resolved, so I'll do another pass after the next patch. Authorization infrastructure for Hive - Key: HIVE-78 URL: https://issues.apache.org/jira/browse/HIVE-78 Project: Hive Issue Type: New Feature Components: Metastore, Query Processor, Server Infrastructure Reporter: Ashish Thusoo Assignee: He Yongqiang Attachments: createuser-v1.patch, hive-78-metadata-v1.patch, hive-78-syntax-v1.patch, HIVE-78.1.nothrift.patch, HIVE-78.1.thrift.patch, HIVE-78.2.nothrift.patch, HIVE-78.2.thrift.patch, HIVE-78.4.complete.patch, HIVE-78.4.no_thrift.patch, HIVE-78.5.complete.patch, HIVE-78.5.no_thrift.patch, HIVE-78.6.complete.patch, HIVE-78.6.no_thrift.patch, hive-78.diff Allow hive to integrate with existing user repositories for authentication and authorization infromation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-78) Authorization infrastructure for Hive
[
https://issues.apache.org/jira/browse/HIVE-78?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12970662#action_12970662
]
John Sichi commented on HIVE-78:
First batch of review comments.
JDO:
* Do we want roles to be contained by databases? Let's discuss this
at next design review.
* Instead of two separate flags (IS_ROLE/IS_GROUP) should we instead use
an enum for principal type { USER, GROUP, ROLE }?
* Naming suggestions (if accepted, propagate to Thrift API also):
** SECURITYROLE - ROLES
** SECURITYROLEMAP - ROLE_MAP
** SECURITYUSER - GLOBAL_PRIVS
** SECURITYDB - DB_PRIVS
** SECURITYTBLPART - TBLPART_PRIVS
** SECURITYCOLUMN - COL_PRIVS
* VARCHAR precision for privileges fields should be 4000
* Since we're going to need to record GRANT OPTION eventually, maybe
we should add it now so that we don't have to ALTER TABLE later?
Thrift API:
* Avoid embedding objects inside of other objects except where
necessary. For example, in the definition of struct Role, use
dbName instead of a Database object (assuming we keep roles as
contained by databases). Likewise, in PrivilegeBag, the map keys
should be identifiers, not objects. This applies to quite a few of
the new structs.
* Can we reduce the number of new structs and API calls by
consolidating different object types? For example, for the
get_XXX_privilege_set calls, just have one, and take object
type+identifier.
* Add comments for all new methods.
Config:
* Why is hive.exec.security used for some config params instead of
hive.security? Also, those parameter names should make it clear
that they are default grants. Also, do we really need owner grants
(don't owners automatically have full privileges implicitly)?
* Looks like hive.variable.substitute crept in from some other patch.
* Comments for plugin-loading parameters should make it explicit
exactly which interface they are supposed to implement.
* Comment for role grants says to some groups instead.
Pluggable Interfaces:
* I don't think we need the factory classes; just add new methods to
HiveUtils (and follow the classloading pattern used there)
* Rename AuthorizationProvider to HiveAuthorizationProvider
and make it extend Configurable
* Rename AuthorizationProviderManager to AbstractAuthorizationProvider
* All outside references should be to the interface (HiveAuthorizationProvider)
not the abstract class.
* Rename Authenticator to HiveAuthenticationProvider and make it
extend Configurable
* Javadoc?
Typos:
* principla
* Authrization
* GrantInfor
* privielges
* Table is partitioned, but partition spec found
* DummpyAuthenticator
* detroy
* wheenve
Implementation:
* why does doAuthorization return a boolean when it just throws
anyway?
* more coming...
Authorization infrastructure for Hive
-
Key: HIVE-78
URL: https://issues.apache.org/jira/browse/HIVE-78
Project: Hive
Issue Type: New Feature
Components: Metastore, Query Processor, Server Infrastructure
Reporter: Ashish Thusoo
Assignee: He Yongqiang
Attachments: createuser-v1.patch, hive-78-metadata-v1.patch,
hive-78-syntax-v1.patch, HIVE-78.1.nothrift.patch, HIVE-78.1.thrift.patch,
HIVE-78.2.nothrift.patch, HIVE-78.2.thrift.patch, HIVE-78.4.complete.patch,
HIVE-78.4.no_thrift.patch, HIVE-78.5.complete.patch,
HIVE-78.5.no_thrift.patch, HIVE-78.6.complete.patch,
HIVE-78.6.no_thrift.patch, hive-78.diff
Allow hive to integrate with existing user repositories for authentication
and authorization infromation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-842) Authentication Infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12969866#action_12969866 ] John Sichi commented on HIVE-842: - Click Submit Patch to get it into the review queue. Authentication Infrastructure for Hive -- Key: HIVE-842 URL: https://issues.apache.org/jira/browse/HIVE-842 Project: Hive Issue Type: New Feature Components: Security, Server Infrastructure Reporter: Edward Capriolo Assignee: Todd Lipcon Attachments: hive-842.txt, hive-842_2.patch, HiveSecurityThoughts.pdf This issue deals with the authentication (user name,password) infrastructure. Not the authorization components that specify what a user should be able to do. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1731) Improve miscellaneous error messages
[ https://issues.apache.org/jira/browse/HIVE-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1731: - Comment: was deleted (was: hive select actiontype, sourcetype, destinationtype, get_json_object(payloaddata, '$.product'), regexp_extract(fullurl, '^.*?php') from nectar_contentaction_merged_partitioned where ds = '2010-11-30' and payloadtype = 'comment' and get_json_object(payloaddata, '$.product') = 'groups' limit 50; Total MapReduce jobs = 1 Launching Job 1 out of 1 ... Ended Job = job_20101201_220674 with errors FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask The actual error was something to do with my regexp_extract syntax: index out of bounds error, no group 1 I didn't specify a group in the regexp, and when I made the adjustment everything worked as expected. It seems like this should be caught earlier, or at least the error trace for the task with the most failures (or a random task with max_failures if there's more than one) should be dumped to the CLI. This one might belong in a different task, since it's a much harder problem to deal with the error traces for badly specified functions (or, worse, custom scripts). However, in general return code 2 is not useful. It only tells me I need to do a bunch of work to find out what's broken, and the only reason I know that is because I've been using Hive for a long time.) Improve miscellaneous error messages Key: HIVE-1731 URL: https://issues.apache.org/jira/browse/HIVE-1731 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: John Sichi Fix For: 0.7.0 This is a place for accumulating error message improvements so that we can update a bunch in batch. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-78) Authorization infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-78?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12970002#action_12970002 ] John Sichi commented on HIVE-78: HIVE-78.6.no_thrift.patch has the thrift-generated code. Authorization infrastructure for Hive - Key: HIVE-78 URL: https://issues.apache.org/jira/browse/HIVE-78 Project: Hive Issue Type: New Feature Components: Metastore, Query Processor, Server Infrastructure Reporter: Ashish Thusoo Assignee: He Yongqiang Attachments: createuser-v1.patch, hive-78-metadata-v1.patch, hive-78-syntax-v1.patch, HIVE-78.1.nothrift.patch, HIVE-78.1.thrift.patch, HIVE-78.2.nothrift.patch, HIVE-78.2.thrift.patch, HIVE-78.4.complete.patch, HIVE-78.4.no_thrift.patch, HIVE-78.5.complete.patch, HIVE-78.5.no_thrift.patch, HIVE-78.6.complete.patch, HIVE-78.6.no_thrift.patch, hive-78.diff Allow hive to integrate with existing user repositories for authentication and authorization infromation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-78) Authorization infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-78?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-78: --- Comment: was deleted (was: HIVE-78.6.no_thrift.patch has the thrift-generated code. ) Authorization infrastructure for Hive - Key: HIVE-78 URL: https://issues.apache.org/jira/browse/HIVE-78 Project: Hive Issue Type: New Feature Components: Metastore, Query Processor, Server Infrastructure Reporter: Ashish Thusoo Assignee: He Yongqiang Attachments: createuser-v1.patch, hive-78-metadata-v1.patch, hive-78-syntax-v1.patch, HIVE-78.1.nothrift.patch, HIVE-78.1.thrift.patch, HIVE-78.2.nothrift.patch, HIVE-78.2.thrift.patch, HIVE-78.4.complete.patch, HIVE-78.4.no_thrift.patch, HIVE-78.5.complete.patch, HIVE-78.5.no_thrift.patch, HIVE-78.6.complete.patch, HIVE-78.6.no_thrift.patch, hive-78.diff Allow hive to integrate with existing user repositories for authentication and authorization infromation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1415) add CLI command for executing a SQL script
[ https://issues.apache.org/jira/browse/HIVE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1415: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed. Thanks Ed! Since docs still aren't converted, can you update the wiki too (noting it as only available starting with 0.7)? add CLI command for executing a SQL script -- Key: HIVE-1415 URL: https://issues.apache.org/jira/browse/HIVE-1415 Project: Hive Issue Type: Improvement Components: Clients Affects Versions: 0.5.0 Reporter: John Sichi Assignee: Edward Capriolo Fix For: 0.7.0 Attachments: hive-1415-1-patch.txt, hive-1415-2-patch.txt Suggestion in HIVE-1405 was source, e.g. source somescript.sql; -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1763) drop table (or view) should issue warning if table doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12968855#action_12968855 ] John Sichi commented on HIVE-1763: -- See HIVE-1542 for my suggested approach. drop table (or view) should issue warning if table doesn't exist Key: HIVE-1763 URL: https://issues.apache.org/jira/browse/HIVE-1763 Project: Hive Issue Type: Improvement Components: Metastore Reporter: dan f Assignee: Paul Butler Priority: Minor Attachments: HIVE-1763.patch drop table reports OK even if the table doesn't exist. Better to report something like mysql's Unknown table 'foo' so that, e.g., unwanted tables (especially ones with names prone to typos) don't persist. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1079) CREATE VIEW followup: derive dependencies on underlying base table partitions from view definition
[ https://issues.apache.org/jira/browse/HIVE-1079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi reassigned HIVE-1079: Assignee: John Sichi (was: Carl Steinbach) CREATE VIEW followup: derive dependencies on underlying base table partitions from view definition --- Key: HIVE-1079 URL: https://issues.apache.org/jira/browse/HIVE-1079 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi Assignee: John Sichi When querying a view, it would be useful to know which underlying base table partitions it depends on in order to know how fresh the result is (or to be able to wait until all of those partitions have been loaded consistently). The task is to come up with a way to perform this analysis automatically (possibly overconservatively), or alternately to let the view creator annotate the view definition with this dependency information, or some combination of the two. Note that this would be useful for any complex query which directly accesses base tables (not just view definitions). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1539) Concurrent metastore threading problem
[ https://issues.apache.org/jira/browse/HIVE-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12965515#action_12965515 ] John Sichi commented on HIVE-1539: -- According to the datanucleus website, 2.2.0 is scheduled for 10-Dec, so let's wait for it. Concurrent metastore threading problem --- Key: HIVE-1539 URL: https://issues.apache.org/jira/browse/HIVE-1539 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Attachments: ClassLoaderResolver.patch, HIVE-1539.patch, thread_dump_hanging.txt When running hive as a service and running a high number of queries concurrently I end up with multiple threads running at 100% cpu without any progress. Looking at these threads I notice this thread(484e): at org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:598) But on a different thread(63a2): at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoReplaceField(MStorageDescriptor.java) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1812) spurious rmr failure messages when inserting with dynamic partitioning
spurious rmr failure messages when inserting with dynamic partitioning -- Key: HIVE-1812 URL: https://issues.apache.org/jira/browse/HIVE-1812 Project: Hive Issue Type: Bug Components: Logging Affects Versions: 0.6.0 Reporter: John Sichi Assignee: Ning Zhang Fix For: 0.7.0 Running a test such as load_dyn_part1.q, there is a lot of noise like this: test: [junit] Copying data from file:/Users/jsichi/open/hive-trunk/data/files/kv1.txt [junit] Loading data to table srcpart partition (ds=2008-04-08, hr=11) [junit] rmr: cannot remove pfile:/Users/jsichi/open/hive-trunk/build/ql/test/data/warehouse/srcpart/ds=2008-04-08/hr=11: No such file or directory. This shows up in production logs as well, which confuses users, making them thinking that the job is encountering problems when it isn't. Is there a way we can squelch these? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-1785) change Pre/Post Query Hooks to take in 1 parameter: HookContext
[ https://issues.apache.org/jira/browse/HIVE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi resolved HIVE-1785. -- Resolution: Fixed Release Note: PreExecute and PostExecute have been deprecated in favor of ExecuteWithHookContext. Committed. Thanks Liyin! Could you explain this change on the user mailing list? Also, we need a followup patch for changing the description of hive.exec.pre/post.hooks in conf/hive-default.xml (I just remembered that). change Pre/Post Query Hooks to take in 1 parameter: HookContext --- Key: HIVE-1785 URL: https://issues.apache.org/jira/browse/HIVE-1785 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.7.0 Reporter: Namit Jain Assignee: Liyin Tang Fix For: 0.7.0 Attachments: hive-1785_3.patch, hive-1785_4.patch, hive-1785_6.patch, hive_1785_1.patch, hive_1785_2.patch This way, it would be possible to add new parameters to the hooks without changing the existing hooks. This will be a incompatible change, and all the hooks need to change to the new API -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1785) change Pre/Post Query Hooks to take in 1 parameter: HookContext
[ https://issues.apache.org/jira/browse/HIVE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1785: - Fix Version/s: 0.7.0 Affects Version/s: 0.7.0 Status: Open (was: Patch Available) * It looks like you mixed in some changes from other patches? (And the patch name is HIVE-1758 instead of HIVE-1785.) Please resubmit with only the changes relevant to HIVE-1785. Also, I don't see where you addressed two of my previous comments: * rewrite existing hook implementations in Hive codebase to implement new interface * add Javadoc for Hook, HookContext, and ExecuteWithHookContext One new one while we're at it: * The old hook interface provided SessionState, SetReadEntity, SetWriteEntity, LineageInfo and UserGroupInformation. HookContext should provide getters for these as well. I realize that some of these are obtainable via QueryPlan and HiveConf, but we should not force users to become dependent on internals like QueryPlan unless they want to be; also, providing the equivalent getters makes upgrade to the new interface easier. You can make the HookContext getters call through to QueryPlan getters etc. change Pre/Post Query Hooks to take in 1 parameter: HookContext --- Key: HIVE-1785 URL: https://issues.apache.org/jira/browse/HIVE-1785 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.7.0 Reporter: Namit Jain Assignee: Liyin Tang Fix For: 0.7.0 Attachments: hive-1785_3.patch, hive-1785_4.patch, hive_1758_5.patch, hive_1785_1.patch, hive_1785_2.patch This way, it would be possible to add new parameters to the hooks without changing the existing hooks. This will be a incompatible change, and all the hooks need to change to the new API -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1785) change Pre/Post Query Hooks to take in 1 parameter: HookContext
[ https://issues.apache.org/jira/browse/HIVE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1785: - Attachment: (was: hive_1758_5.patch) change Pre/Post Query Hooks to take in 1 parameter: HookContext --- Key: HIVE-1785 URL: https://issues.apache.org/jira/browse/HIVE-1785 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.7.0 Reporter: Namit Jain Assignee: Liyin Tang Fix For: 0.7.0 Attachments: hive-1785_3.patch, hive-1785_4.patch, hive-1785_6.patch, hive_1785_1.patch, hive_1785_2.patch This way, it would be possible to add new parameters to the hooks without changing the existing hooks. This will be a incompatible change, and all the hooks need to change to the new API -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1746) Support for using ALTER to set IDXPROPERTIES
[ https://issues.apache.org/jira/browse/HIVE-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1746: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed. Thanks Marquis! Support for using ALTER to set IDXPROPERTIES Key: HIVE-1746 URL: https://issues.apache.org/jira/browse/HIVE-1746 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.7.0 Reporter: Marquis Wang Assignee: Marquis Wang Fix For: 0.7.0 Attachments: 1746.prelim.patch, HIVE-1746.2.patch, HIVE-1746.3.patch, HIVE-1746.4.patch Hive-1498 has support for IDXPROPERTIES on index creation, so now we want to support ALTERing those properties. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1785) change Pre/Post Query Hooks to take in 1 parameter: HookContext
[ https://issues.apache.org/jira/browse/HIVE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1785: - Status: Open (was: Patch Available) Discussed with Liyin, and there are a few more things we should do here: * add Javadoc for Hook, HookContext, and ExecuteWithHookContext * add deprecation annotation for old PreExecute/PostExecute * rewrite existing hook implementations in Hive codebase to implement new interface * run Facebook-specific tests to validate backwards-compatibility during the transition change Pre/Post Query Hooks to take in 1 parameter: HookContext --- Key: HIVE-1785 URL: https://issues.apache.org/jira/browse/HIVE-1785 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Liyin Tang Attachments: hive-1785_3.patch, hive-1785_4.patch, hive_1785_1.patch, hive_1785_2.patch This way, it would be possible to add new parameters to the hooks without changing the existing hooks. This will be a incompatible change, and all the hooks need to change to the new API -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1785) change Pre/Post Query Hooks to take in 1 parameter: HookContext
[ https://issues.apache.org/jira/browse/HIVE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12933081#action_12933081 ] John Sichi commented on HIVE-1785: -- The way the new interface is defined, people writing the hooks will need to keep implementing the old Pre/PostExec interfaces, even though we will no longer be calling those methods. This will make it harder to remove cruft. change Pre/Post Query Hooks to take in 1 parameter: HookContext --- Key: HIVE-1785 URL: https://issues.apache.org/jira/browse/HIVE-1785 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Liyin Tang Attachments: hive-1785_3.patch, hive_1785_1.patch, hive_1785_2.patch This way, it would be possible to add new parameters to the hooks without changing the existing hooks. This will be a incompatible change, and all the hooks need to change to the new API -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1785) change Pre/Post Query Hooks to take in 1 parameter: HookContext
[ https://issues.apache.org/jira/browse/HIVE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12933230#action_12933230 ] John Sichi commented on HIVE-1785: -- +1. change Pre/Post Query Hooks to take in 1 parameter: HookContext --- Key: HIVE-1785 URL: https://issues.apache.org/jira/browse/HIVE-1785 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Liyin Tang Attachments: hive-1785_3.patch, hive-1785_4.patch, hive_1785_1.patch, hive_1785_2.patch This way, it would be possible to add new parameters to the hooks without changing the existing hooks. This will be a incompatible change, and all the hooks need to change to the new API -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1497) support COMMENT clause on CREATE INDEX, and add new command for SHOW INDEXES
[ https://issues.apache.org/jira/browse/HIVE-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931512#action_12931512 ] John Sichi commented on HIVE-1497: -- +1. Will commit when tests pass. You could have used MetaStoreUtils.getColumnNamesFromFieldSchema, but I'll put a cleanup note for that on HIVE-1764 so we can get this one committed as is (assuming tests pass). support COMMENT clause on CREATE INDEX, and add new command for SHOW INDEXES Key: HIVE-1497 URL: https://issues.apache.org/jira/browse/HIVE-1497 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Russell Melick Fix For: 0.7.0 Attachments: HIVE-1497.4.patch, HIVE-1497.5.patch, HIVE-1497.6.patch, HIVE-1497.7.patch, HIVE-1497.8.patch, hive-1497.p1.patch, hive-1497.p2.patch, hive-1497.p3.patch We need to work out the syntax for SHOW/DESCRIBE, taking partitioning into account. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1764) Add command for DESCRIBE indexes
[ https://issues.apache.org/jira/browse/HIVE-1764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931628#action_12931628 ] John Sichi commented on HIVE-1764: -- Also, one other followup from HIVE-1497: * Use MetaStoreUtils.getColumnNamesFromFieldSchema in getAllColumnsInformation(Index) Add command for DESCRIBE indexes Key: HIVE-1764 URL: https://issues.apache.org/jira/browse/HIVE-1764 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.7.0 Reporter: Russell Melick Assignee: Jeffrey Lym Fix For: 0.7.0 Followup for HIVE-1497 * Work out the syntax for DESCRIBE, taking partitioning into account. * Allow regular expressions for selecting tables (like SHOW TABLES ) * Enable showing indexes which are only on specific partitions -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1497) support COMMENT clause on CREATE INDEX, and add new command for SHOW INDEXES
[ https://issues.apache.org/jira/browse/HIVE-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1497: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed. Thanks Russell! support COMMENT clause on CREATE INDEX, and add new command for SHOW INDEXES Key: HIVE-1497 URL: https://issues.apache.org/jira/browse/HIVE-1497 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Russell Melick Fix For: 0.7.0 Attachments: HIVE-1497.4.patch, HIVE-1497.5.patch, HIVE-1497.6.patch, HIVE-1497.7.patch, HIVE-1497.8.patch, hive-1497.p1.patch, hive-1497.p2.patch, hive-1497.p3.patch We need to work out the syntax for SHOW/DESCRIBE, taking partitioning into account. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1497) support COMMENT clause on CREATE INDEX, and add new command for SHOW INDEXES
[ https://issues.apache.org/jira/browse/HIVE-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931149#action_12931149 ] John Sichi commented on HIVE-1497: -- @Russell: I meant pluralize it unconditionally (i.e. just edit the schema in your patch). support COMMENT clause on CREATE INDEX, and add new command for SHOW INDEXES Key: HIVE-1497 URL: https://issues.apache.org/jira/browse/HIVE-1497 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Russell Melick Fix For: 0.7.0 Attachments: HIVE-1497.4.patch, HIVE-1497.5.patch, HIVE-1497.6.patch, hive-1497.p1.patch, hive-1497.p2.patch, hive-1497.p3.patch We need to work out the syntax for SHOW/DESCRIBE, taking partitioning into account. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1526) Hive should depend on a release version of Thrift
[ https://issues.apache.org/jira/browse/HIVE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1526: - Status: Open (was: Patch Available) Hive should depend on a release version of Thrift - Key: HIVE-1526 URL: https://issues.apache.org/jira/browse/HIVE-1526 Project: Hive Issue Type: Task Components: Build Infrastructure, Clients Reporter: Carl Steinbach Assignee: Todd Lipcon Fix For: 0.7.0 Attachments: HIVE-1526.2.patch.txt, hive-1526.txt, libfb303.jar, libthrift.jar Hive should depend on a release version of Thrift, and ideally it should use Ivy to resolve this dependency. The Thrift folks are working on adding Thrift artifacts to a maven repository here: https://issues.apache.org/jira/browse/THRIFT-363 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Delayed email updates from JIRA
Just a note that (at least for facebook.com email addresses), email notifications from JIRA have become very slow over the past few days (I'm pretty sure there must be a lot queued up for me based on what I've seen while browsing JIRA). I don't know what the cause is, but just letting everyone know in case you've posted a comment and not gotten any response. JVS
[jira] Commented: (HIVE-1747) Make case-insensitive substring search as fast as case-sensitive variant
[
https://issues.apache.org/jira/browse/HIVE-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931227#action_12931227
]
John Sichi commented on HIVE-1747:
--
Also: while at ApacheCon, I heard that the Pig team was using this as a faster
replacement for Java regexp:
http://www.brics.dk/automaton/
We might want to do the same. See PIG-965 for more details.
Make case-insensitive substring search as fast as case-sensitive variant
Key: HIVE-1747
URL: https://issues.apache.org/jira/browse/HIVE-1747
Project: Hive
Issue Type: Improvement
Components: Query Processor
Affects Versions: 0.7.0
Reporter: John Sichi
{noformat}
WHERE str_col like '%XYZ%'
{noformat}
is very fast because the pattern matching code is special-cased. However,
{noformat}
WHERE str_col rlike '(?i).*XYZ.*'
{noformat}
is very slow because it goes through the generic Java pattern matching.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1337) Support HAVING clause
[ https://issues.apache.org/jira/browse/HIVE-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931281#action_12931281 ] John Sichi commented on HIVE-1337: -- Testing JIRA email notifications. Support HAVING clause - Key: HIVE-1337 URL: https://issues.apache.org/jira/browse/HIVE-1337 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Carl Steinbach Assignee: Carl Steinbach Support the HAVING clause in HiveQL. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1586) prune operators for empty inputs
[ https://issues.apache.org/jira/browse/HIVE-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931283#action_12931283 ] John Sichi commented on HIVE-1586: -- Testing email notifications. prune operators for empty inputs Key: HIVE-1586 URL: https://issues.apache.org/jira/browse/HIVE-1586 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi Assignee: John Sichi Fix For: 0.7.0 Given this: create view v as select * from ( select *,'T1' as m from pokes union all select *,'T2' as m from pokes2) u; select * from v where m='T2'; When we optimize the query, we can prune away all inputs to the union other than the one which references pokes2. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1785) change Pre/Post Query Hooks to take in 1 parameter: HookContext
[ https://issues.apache.org/jira/browse/HIVE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931287#action_12931287 ] John Sichi commented on HIVE-1785: -- Can we use reflection to make this change backwards-compatible? change Pre/Post Query Hooks to take in 1 parameter: HookContext --- Key: HIVE-1785 URL: https://issues.apache.org/jira/browse/HIVE-1785 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Liyin Tang This way, it would be possible to add new parameters to the hooks without changing the existing hooks. This will be a incompatible change, and all the hooks need to change to the new API -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1337) Support HAVING clause
[ https://issues.apache.org/jira/browse/HIVE-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931290#action_12931290 ] John Sichi commented on HIVE-1337: -- Testing again. Support HAVING clause - Key: HIVE-1337 URL: https://issues.apache.org/jira/browse/HIVE-1337 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Carl Steinbach Assignee: Carl Steinbach Support the HAVING clause in HiveQL. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Review Request: HIVE-78: Authorization infrastructure for Hive
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55/ --- (Updated 2010-11-11 17:18:49.328105) Review request for hive. Changes --- HIVE-78.2.nothrift.patch (this time actually without generated code) Summary --- Review for patch without generated thrift code (note that the index is still split over three pages due to size). This addresses bug HIVE-78. https://issues.apache.org/jira/browse/HIVE-78 Diffs (updated) - http://svn.apache.org/repos/asf/hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/if/hive_metastore.thrift 1033775 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1033775 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MSecurityColumn.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MSecurityDB.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MSecurityRoleEntity.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MSecurityTablePartition.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MSecurityUser.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MSecurityUserRoleMap.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/package.jdo 1030336 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1033775 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1033775 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1033775 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRProcContext.java 1033775 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1033775 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1031914 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1031676 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/GrantDesc.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/GrantRevokeRoleDDL.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveQueryReadWrite.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PrincipalDesc.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PrivilegeDesc.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PrivilegeObjectDesc.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/RevokeDesc.java PRE-CREATION
[jira] Updated: (HIVE-1497) support COMMENT clause on CREATE INDEX, and add new command for SHOW INDEXES
[ https://issues.apache.org/jira/browse/HIVE-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1497: - Status: Open (was: Patch Available) For idx_compound, the col_names should be key,value1 not just key. support COMMENT clause on CREATE INDEX, and add new command for SHOW INDEXES Key: HIVE-1497 URL: https://issues.apache.org/jira/browse/HIVE-1497 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Russell Melick Fix For: 0.7.0 Attachments: HIVE-1497.4.patch, HIVE-1497.5.patch, HIVE-1497.6.patch, HIVE-1497.7.patch, hive-1497.p1.patch, hive-1497.p2.patch, hive-1497.p3.patch We need to work out the syntax for SHOW/DESCRIBE, taking partitioning into account. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1501) when generating reentrant INSERT for index rebuild, quote identifiers using backticks
[ https://issues.apache.org/jira/browse/HIVE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930677#action_12930677 ] John Sichi commented on HIVE-1501: -- +1 on the latest. Will commit when tests pass. when generating reentrant INSERT for index rebuild, quote identifiers using backticks - Key: HIVE-1501 URL: https://issues.apache.org/jira/browse/HIVE-1501 Project: Hive Issue Type: Bug Components: Indexing Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Skye Berghel Fix For: 0.7.0 Attachments: 1501.patch, 1501_new_tests.patch, 1501_with_tests.patch, HIVE-1501.4.patch, HIVE-1501.5.patch, HIVE-1501.6.patch Yongqiang, you mentioned that you weren't able to do this due to SORT BY not accepting them. The SORT BY is gone now as of HIVE-1494 (and SORT BY needs to be fixed anyway). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1501) when generating reentrant INSERT for index rebuild, quote identifiers using backticks
[ https://issues.apache.org/jira/browse/HIVE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1501: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed. Thanks Skye! when generating reentrant INSERT for index rebuild, quote identifiers using backticks - Key: HIVE-1501 URL: https://issues.apache.org/jira/browse/HIVE-1501 Project: Hive Issue Type: Bug Components: Indexing Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Skye Berghel Fix For: 0.7.0 Attachments: 1501.patch, 1501_new_tests.patch, 1501_with_tests.patch, HIVE-1501.4.patch, HIVE-1501.5.patch, HIVE-1501.6.patch Yongqiang, you mentioned that you weren't able to do this due to SORT BY not accepting them. The SORT BY is gone now as of HIVE-1494 (and SORT BY needs to be fixed anyway). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1497) support COMMENT clause on CREATE INDEX, and add new commands for SHOW/DESCRIBE indexes
[ https://issues.apache.org/jira/browse/HIVE-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12929674#action_12929674 ] John Sichi commented on HIVE-1497: -- Right, we decided to do more details for a single index in a separate DESCRIBE command. Russell, please open a followup JIRA for that one now, link it to this one as related, and then edit the title for this one. support COMMENT clause on CREATE INDEX, and add new commands for SHOW/DESCRIBE indexes -- Key: HIVE-1497 URL: https://issues.apache.org/jira/browse/HIVE-1497 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Russell Melick Fix For: 0.7.0 Attachments: HIVE-1497.4.patch, HIVE-1497.5.patch, hive-1497.p1.patch, hive-1497.p2.patch, hive-1497.p3.patch We need to work out the syntax for SHOW/DESCRIBE, taking partitioning into account. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1497) support COMMENT clause on CREATE INDEX, and add new commands for SHOW/DESCRIBE indexes
[ https://issues.apache.org/jira/browse/HIVE-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12929763#action_12929763 ] John Sichi commented on HIVE-1497: -- Oops, the followup already exists (HIVE-1764). I'll edit the title of this one. support COMMENT clause on CREATE INDEX, and add new commands for SHOW/DESCRIBE indexes -- Key: HIVE-1497 URL: https://issues.apache.org/jira/browse/HIVE-1497 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Russell Melick Fix For: 0.7.0 Attachments: HIVE-1497.4.patch, HIVE-1497.5.patch, hive-1497.p1.patch, hive-1497.p2.patch, hive-1497.p3.patch We need to work out the syntax for SHOW/DESCRIBE, taking partitioning into account. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1497) support COMMENT clause on CREATE INDEX, and add new command for SHOW INDEXES
[ https://issues.apache.org/jira/browse/HIVE-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1497: - Summary: support COMMENT clause on CREATE INDEX, and add new command for SHOW INDEXES (was: support COMMENT clause on CREATE INDEX, and add new commands for SHOW/DESCRIBE indexes) support COMMENT clause on CREATE INDEX, and add new command for SHOW INDEXES Key: HIVE-1497 URL: https://issues.apache.org/jira/browse/HIVE-1497 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Russell Melick Fix For: 0.7.0 Attachments: HIVE-1497.4.patch, HIVE-1497.5.patch, hive-1497.p1.patch, hive-1497.p2.patch, hive-1497.p3.patch We need to work out the syntax for SHOW/DESCRIBE, taking partitioning into account. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1746) Support for using ALTER to set IDXPROPERTIES
[ https://issues.apache.org/jira/browse/HIVE-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1746: - Component/s: Indexing Affects Version/s: 0.7.0 Fix Version/s: 0.7.0 Support for using ALTER to set IDXPROPERTIES Key: HIVE-1746 URL: https://issues.apache.org/jira/browse/HIVE-1746 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.7.0 Reporter: Marquis Wang Assignee: Marquis Wang Fix For: 0.7.0 Attachments: 1746.prelim.patch, HIVE-1746.2.patch Hive-1498 has support for IDXPROPERTIES on index creation, so now we want to support ALTERing those properties. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1501) when generating reentrant INSERT for index rebuild, quote identifiers using backticks
[ https://issues.apache.org/jira/browse/HIVE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1501: - Status: Open (was: Patch Available) Whitespace nitpicks: * MetaStoreUtils.java no longer has any changes, so it should not be in the patch at all * when generating reentrant INSERT for index rebuild, quote identifiers using backticks - Key: HIVE-1501 URL: https://issues.apache.org/jira/browse/HIVE-1501 Project: Hive Issue Type: Bug Components: Indexing Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Skye Berghel Fix For: 0.7.0 Attachments: 1501.patch, 1501_new_tests.patch, 1501_with_tests.patch, HIVE-1501.4.patch, HIVE-1501.5.patch Yongqiang, you mentioned that you weren't able to do this due to SORT BY not accepting them. The SORT BY is gone now as of HIVE-1494 (and SORT BY needs to be fixed anyway). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1501) when generating reentrant INSERT for index rebuild, quote identifiers using backticks
[ https://issues.apache.org/jira/browse/HIVE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12929766#action_12929766 ] John Sichi commented on HIVE-1501: -- * HiveUtils.getUnparsedColumnNamesFromFieldSchema has an extra line of whitespace in between the Javadoc and the method declaration. (If you're using Eclipse for editing, it's easy to end up with extraneous whitespace.) when generating reentrant INSERT for index rebuild, quote identifiers using backticks - Key: HIVE-1501 URL: https://issues.apache.org/jira/browse/HIVE-1501 Project: Hive Issue Type: Bug Components: Indexing Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Skye Berghel Fix For: 0.7.0 Attachments: 1501.patch, 1501_new_tests.patch, 1501_with_tests.patch, HIVE-1501.4.patch, HIVE-1501.5.patch Yongqiang, you mentioned that you weren't able to do this due to SORT BY not accepting them. The SORT BY is gone now as of HIVE-1494 (and SORT BY needs to be fixed anyway). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1501) when generating reentrant INSERT for index rebuild, quote identifiers using backticks
[ https://issues.apache.org/jira/browse/HIVE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12929769#action_12929769 ] John Sichi commented on HIVE-1501: -- (Other than those, this one should be ready to go.) when generating reentrant INSERT for index rebuild, quote identifiers using backticks - Key: HIVE-1501 URL: https://issues.apache.org/jira/browse/HIVE-1501 Project: Hive Issue Type: Bug Components: Indexing Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Skye Berghel Fix For: 0.7.0 Attachments: 1501.patch, 1501_new_tests.patch, 1501_with_tests.patch, HIVE-1501.4.patch, HIVE-1501.5.patch Yongqiang, you mentioned that you weren't able to do this due to SORT BY not accepting them. The SORT BY is gone now as of HIVE-1494 (and SORT BY needs to be fixed anyway). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1746) Support for using ALTER to set IDXPROPERTIES
[ https://issues.apache.org/jira/browse/HIVE-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1746: - Status: Open (was: Patch Available) In DDLTask: * Eliminate System.out.println calls * Instead of duplicating last_modified_by/time code from ALTER TABLE, factor it out to a private helper method instead * You have a bug in the case where the table is in a non-default database. You need to remember the base table's database name in AlterIndexDesc and use it later. Support for using ALTER to set IDXPROPERTIES Key: HIVE-1746 URL: https://issues.apache.org/jira/browse/HIVE-1746 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.7.0 Reporter: Marquis Wang Assignee: Marquis Wang Fix For: 0.7.0 Attachments: 1746.prelim.patch, HIVE-1746.2.patch Hive-1498 has support for IDXPROPERTIES on index creation, so now we want to support ALTERing those properties. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1771) ROUND(infinity) chokes
[ https://issues.apache.org/jira/browse/HIVE-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1771: - Attachment: HIVE-1771.1.patch This patch fixes the two-arg version, but leaves the one-arg version as is. ROUND(infinity) chokes -- Key: HIVE-1771 URL: https://issues.apache.org/jira/browse/HIVE-1771 Project: Hive Issue Type: Bug Components: UDF Affects Versions: 0.6.0 Reporter: John Sichi Assignee: John Sichi Fix For: 0.7.0 Attachments: HIVE-1771.1.patch Since 1-arg ROUND returns an integer, it's hard to fix this without either losing data (return NULL) or making a backwards-incompatible change (return DOUBLE instead of BIGINT). In any case, we should definitely fix 2-arg ROUND to preserve infinity/NaN/etc, since it is already returning double. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1771) ROUND(infinity) chokes
[ https://issues.apache.org/jira/browse/HIVE-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1771: - Status: Patch Available (was: Open) ROUND(infinity) chokes -- Key: HIVE-1771 URL: https://issues.apache.org/jira/browse/HIVE-1771 Project: Hive Issue Type: Bug Components: UDF Affects Versions: 0.6.0 Reporter: John Sichi Assignee: John Sichi Fix For: 0.7.0 Attachments: HIVE-1771.1.patch Since 1-arg ROUND returns an integer, it's hard to fix this without either losing data (return NULL) or making a backwards-incompatible change (return DOUBLE instead of BIGINT). In any case, we should definitely fix 2-arg ROUND to preserve infinity/NaN/etc, since it is already returning double. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1771) ROUND(infinity) chokes
[ https://issues.apache.org/jira/browse/HIVE-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12929887#action_12929887 ] John Sichi commented on HIVE-1771: -- https://reviews.apache.org/r/53/diff/#index_header ROUND(infinity) chokes -- Key: HIVE-1771 URL: https://issues.apache.org/jira/browse/HIVE-1771 Project: Hive Issue Type: Bug Components: UDF Affects Versions: 0.6.0 Reporter: John Sichi Assignee: John Sichi Fix For: 0.7.0 Attachments: HIVE-1771.1.patch Since 1-arg ROUND returns an integer, it's hard to fix this without either losing data (return NULL) or making a backwards-incompatible change (return DOUBLE instead of BIGINT). In any case, we should definitely fix 2-arg ROUND to preserve infinity/NaN/etc, since it is already returning double. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-78) Authorization infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-78?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12929888#action_12929888 ] John Sichi commented on HIVE-78: https://reviews.apache.org/r/55/diff/#index_header Authorization infrastructure for Hive - Key: HIVE-78 URL: https://issues.apache.org/jira/browse/HIVE-78 Project: Hive Issue Type: New Feature Components: Metastore, Query Processor, Server Infrastructure Reporter: Ashish Thusoo Assignee: He Yongqiang Attachments: createuser-v1.patch, hive-78-metadata-v1.patch, hive-78-syntax-v1.patch, HIVE-78.1.nothrift.patch, HIVE-78.1.thrift.patch, hive-78.diff Allow hive to integrate with existing user repositories for authentication and authorization infromation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Review Request: HIVE-78: Authorization infrastructure for Hive
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55/ --- Review request for hive. Summary --- Review for patch without generated thrift code (note that the index is still split over three pages due to size). This addresses bug HIVE-78. https://issues.apache.org/jira/browse/HIVE-78 Diffs - http://svn.apache.org/repos/asf/hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/if/hive_metastore.thrift 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnPrivilegeBag.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen-javabean/org/apache/hadoop/hive/metastore/api/PrivilegeBag.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen-javabean/org/apache/hadoop/hive/metastore/api/PrivilegeSet.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen-javabean/org/apache/hadoop/hive/metastore/api/Role.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen-javabean/org/apache/hadoop/hive/metastore/api/SecurityColumn.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen-javabean/org/apache/hadoop/hive/metastore/api/SecurityDB.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen-javabean/org/apache/hadoop/hive/metastore/api/SecurityTablePartition.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen-javabean/org/apache/hadoop/hive/metastore/api/SecurityUser.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MSecurityColumn.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MSecurityDB.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MSecurityRoleEntity.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MSecurityTablePartition.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MSecurityUser.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MSecurityUserRoleMap.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/package.jdo 1030336 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1030336
Re: Review Request: HIVE-1771: ROUND(infinity) chokes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53/ --- (Updated 2010-11-08 18:36:25.984539) Review request for hive. Summary --- Review request from jvs. This addresses bug HIVE-1771. https://issues.apache.org/jira/browse/HIVE-1771 Diffs - http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFRound.java 1032795 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/udf_round.q 1032795 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_round.q.out 1032795 Diff: https://reviews.apache.org/r/53/diff Testing --- Thanks, John
[jira] Commented: (HIVE-1775) Assertation on inputObjInspectors.length in Groupy operator
[ https://issues.apache.org/jira/browse/HIVE-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12929438#action_12929438 ] John Sichi commented on HIVE-1775: -- Probably a dup of HIVE-1215. Assertation on inputObjInspectors.length in Groupy operator --- Key: HIVE-1775 URL: https://issues.apache.org/jira/browse/HIVE-1775 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.6.0, 0.7.0 Reporter: Liyin Tang Assignee: Liyin Tang Fix For: 0.7.0 In the Groupby Operator: Line 188: assert (inputObjInspectors.length == 1); But this assertion may not necessary true -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1771) ROUND(infinity) chokes
ROUND(infinity) chokes -- Key: HIVE-1771 URL: https://issues.apache.org/jira/browse/HIVE-1771 Project: Hive Issue Type: Bug Components: UDF Affects Versions: 0.6.0 Reporter: John Sichi Assignee: John Sichi Fix For: 0.7.0 Since 1-arg ROUND returns an integer, it's hard to fix this without either losing data (return NULL) or making a backwards-incompatible change (return DOUBLE instead of BIGINT). In any case, we should definitely fix 2-arg ROUND to preserve infinity/NaN/etc, since it is already returning double. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1501) when generating reentrant INSERT for index rebuild, quote identifiers using backticks
[ https://issues.apache.org/jira/browse/HIVE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12928295#action_12928295 ] John Sichi commented on HIVE-1501: -- +1. Will commit when tests pass. when generating reentrant INSERT for index rebuild, quote identifiers using backticks - Key: HIVE-1501 URL: https://issues.apache.org/jira/browse/HIVE-1501 Project: Hive Issue Type: Bug Components: Indexing Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Skye Berghel Fix For: 0.7.0 Attachments: 1501.patch, 1501_new_tests.patch, 1501_with_tests.patch, HIVE-1501.4.patch Yongqiang, you mentioned that you weren't able to do this due to SORT BY not accepting them. The SORT BY is gone now as of HIVE-1494 (and SORT BY needs to be fixed anyway). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1634) Allow access to Primitive types stored in binary format in HBase
[ https://issues.apache.org/jira/browse/HIVE-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12928334#action_12928334 ] John Sichi commented on HIVE-1634: -- OK, I finally got some time to look into the Lazy* classes. I see what you mean about the class hierarchy, and I agree that we can leave any refactoring of the existing classes for a followup patch. Also, I was wrong to think that we could reuse the existing binary classes, since they do things such as VInt zero-compression, and that's incompatible with the HBase Bytes format. However, for this patch, I want to at least get the new classes into their final destination with respect to package name and class name (so that we don't have to move them later, even if we adjust their inheritance). To this end, I suggest a new package serde2.lazydio, and name the classes on the pattern LazyDioInteger. The Dio is to indicate DataInput/DataOutput format. (I was thinking of lazybytes and LazyByteInteger, to indicate HBase Bytes format, but then I saw that Byte is also one of the datatypes, and LazyBytesByte would be puzzling.) Having both LazyIntegerBinary and LazyBinaryInteger, as in the current patch, would just be too confusing. Also, regarding the implementation of the new classes, most of the init method code is duplicated from class to class. The only thing specific to each class is the actual read+set. Should we factor out a LazyDioObject (similar to the existing pattern for LazyObject and LazyBinaryObject)? Likewise for LazyDioPrimitive and LazyDioNonPrimitive. I will ask some others to chime in on this as well. Allow access to Primitive types stored in binary format in HBase Key: HIVE-1634 URL: https://issues.apache.org/jira/browse/HIVE-1634 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.7.0 Reporter: Basab Maulik Assignee: Basab Maulik Attachments: HIVE-1634.0.patch, TestHiveHBaseExternalTable.java This addresses HIVE-1245 in part, for atomic or primitive types. The serde property hbase.columns.storage.types = -,b,b,b,b,b,b,b,b is a specification of the storage option for the corresponding column in the serde property hbase.columns.mapping. Allowed values are '-' for table default, 's' for standard string storage, and 'b' for binary storage as would be obtained from o.a.h.hbase.utils.Bytes. Map types for HBase column families use a colon separated pair such as 's:b' for the key and value part specifiers respectively. See the test cases and queries for HBase handler for additional examples. There is also a table property hbase.table.default.storage.type = string to specify a table level default storage type. The other valid specification is binary. The table level default is overridden by a column level specification. This control is available for the boolean, tinyint, smallint, int, bigint, float, and double primitive types. The attached patch also relaxes the mapping of map types to HBase column families to allow any primitive type to be the map key. Attached is a program for creating a table and populating it in HBase. The external table in Hive can access the data as shown in the example below. hive create external table TestHiveHBaseExternalTable (key string, c_bool boolean, c_byte tinyint, c_short smallint, c_int int, c_long bigint, c_string string, c_float float, c_double double) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties (hbase.columns.mapping = :key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double) tblproperties (hbase.table.name = TestHiveHBaseExternalTable); OK Time taken: 0.691 seconds hive select * from TestHiveHBaseExternalTable; OK key-1 NULLNULLNULLNULLNULLTest-String NULLNULL Time taken: 0.346 seconds hive drop table TestHiveHBaseExternalTable; OK Time taken: 0.139 seconds hive create external table TestHiveHBaseExternalTable (key string, c_bool boolean, c_byte tinyint, c_short smallint, c_int int, c_long bigint, c_string string, c_float float, c_double double) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties ( hbase.columns.mapping = :key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double, hbase.columns.storage.types = -,b,b,b,b,b,b,b,b ) tblproperties ( hbase.table.name = TestHiveHBaseExternalTable, hbase.table.default.storage.type = string); OK Time taken: 0.139 seconds hive select * from TestHiveHBaseExternalTable; OK key-1 true-128-32768 -2147483648 -9223372036854775808 Test-String -2.1793132E-11
[jira] Commented: (HIVE-1694) Accelerate query execution using indexes
[ https://issues.apache.org/jira/browse/HIVE-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12927070#action_12927070 ] John Sichi commented on HIVE-1694: -- Hey guys, I haven't gone through all the code yet, but reading through the slides just now, there's one problem I should point out with using the existing compact indexes for aggregate rewrite. Namely, we store only the distinct block offsets, not the distinct row offsets. So, if the same key appears more than once within the same block, you'll get the wrong answer for COUNT. One way to address this would be to compute the COUNT per index entry at the time we are building the index, and then SUM that later for aggregation. But currently the compact index does not store that, so we would need to add it as a new index type. One smaller item is that for the DISTINCT rewrite (slide 10), you still need to keep a DISTINCT on the rewritten query since the same l_shipdate may be repeated in the index table if it appears in multiple buckets. Accelerate query execution using indexes Key: HIVE-1694 URL: https://issues.apache.org/jira/browse/HIVE-1694 Project: Hive Issue Type: New Feature Components: Indexing, Query Processor Affects Versions: 0.7.0 Reporter: Nikhil Deshpande Attachments: demo_q1.hql, demo_q2.hql, HIVE-1694_2010-10-28.diff The index building patch (Hive-417) is checked into trunk, this JIRA issue tracks supporting indexes in Hive compiler execution engine for SELECT queries. This is in ref. to John's comment at https://issues.apache.org/jira/browse/HIVE-417?focusedCommentId=12884869page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12884869 on creating separate JIRA issue for tracking index usage in optimizer query execution. The aim of this effort is to use indexes to accelerate query execution (for certain class of queries). E.g. - Filters and range scans (already being worked on by He Yongqiang as part of HIVE-417?) - Joins (index based joins) - Group By, Order By and other misc cases The proposal is multi-step: 1. Building index based operators, compiler and execution engine changes 2. Optimizer enhancements (e.g. cost-based optimizer to compare and choose between index scans, full table scans etc.) This JIRA initially focuses on the first step. This JIRA is expected to hold the information about index based plans operator implementations for above mentioned cases. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1694) Accelerate query execution using indexes
[ https://issues.apache.org/jira/browse/HIVE-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12927077#action_12927077 ] John Sichi commented on HIVE-1694: -- +1 to what Namit said. Doing the rewrites at the relational algebra Operator level (similar to the way optimizer transformations such as predicate pushdown already work) will have two big advantages: * more general (syntax-independent) * much easier to maintain (as you noted in your presentation, the SemanticAnalyzer data structures can be very difficult to analyze and manipulate, whereas the Operator tree are a lot cleaner) BTW, thanks for the very clear explanation of the work you've done so far. Accelerate query execution using indexes Key: HIVE-1694 URL: https://issues.apache.org/jira/browse/HIVE-1694 Project: Hive Issue Type: New Feature Components: Indexing, Query Processor Affects Versions: 0.7.0 Reporter: Nikhil Deshpande Attachments: demo_q1.hql, demo_q2.hql, HIVE-1694_2010-10-28.diff The index building patch (Hive-417) is checked into trunk, this JIRA issue tracks supporting indexes in Hive compiler execution engine for SELECT queries. This is in ref. to John's comment at https://issues.apache.org/jira/browse/HIVE-417?focusedCommentId=12884869page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12884869 on creating separate JIRA issue for tracking index usage in optimizer query execution. The aim of this effort is to use indexes to accelerate query execution (for certain class of queries). E.g. - Filters and range scans (already being worked on by He Yongqiang as part of HIVE-417?) - Joins (index based joins) - Group By, Order By and other misc cases The proposal is multi-step: 1. Building index based operators, compiler and execution engine changes 2. Optimizer enhancements (e.g. cost-based optimizer to compare and choose between index scans, full table scans etc.) This JIRA initially focuses on the first step. This JIRA is expected to hold the information about index based plans operator implementations for above mentioned cases. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1497) support COMMENT clause on CREATE INDEX, and add new commands for SHOW/DESCRIBE indexes
[ https://issues.apache.org/jira/browse/HIVE-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12927086#action_12927086 ] John Sichi commented on HIVE-1497: -- As discussed in last week's conf call, we'll need a followup for a separate DESCRIBE command (which will cover properties and partitions). support COMMENT clause on CREATE INDEX, and add new commands for SHOW/DESCRIBE indexes -- Key: HIVE-1497 URL: https://issues.apache.org/jira/browse/HIVE-1497 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Russell Melick Fix For: 0.7.0 Attachments: hive-1497.p1.patch, hive-1497.p2.patch, hive-1497.p3.patch We need to work out the syntax for SHOW/DESCRIBE, taking partitioning into account. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1694) Accelerate query execution using indexes
[ https://issues.apache.org/jira/browse/HIVE-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi reassigned HIVE-1694: Assignee: Nikhil Deshpande Accelerate query execution using indexes Key: HIVE-1694 URL: https://issues.apache.org/jira/browse/HIVE-1694 Project: Hive Issue Type: New Feature Components: Indexing, Query Processor Affects Versions: 0.7.0 Reporter: Nikhil Deshpande Assignee: Nikhil Deshpande Attachments: demo_q1.hql, demo_q2.hql, HIVE-1694_2010-10-28.diff The index building patch (Hive-417) is checked into trunk, this JIRA issue tracks supporting indexes in Hive compiler execution engine for SELECT queries. This is in ref. to John's comment at https://issues.apache.org/jira/browse/HIVE-417?focusedCommentId=12884869page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12884869 on creating separate JIRA issue for tracking index usage in optimizer query execution. The aim of this effort is to use indexes to accelerate query execution (for certain class of queries). E.g. - Filters and range scans (already being worked on by He Yongqiang as part of HIVE-417?) - Joins (index based joins) - Group By, Order By and other misc cases The proposal is multi-step: 1. Building index based operators, compiler and execution engine changes 2. Optimizer enhancements (e.g. cost-based optimizer to compare and choose between index scans, full table scans etc.) This JIRA initially focuses on the first step. This JIRA is expected to hold the information about index based plans operator implementations for above mentioned cases. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1332) Archiving partitions
[ https://issues.apache.org/jira/browse/HIVE-1332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12927228#action_12927228 ] John Sichi commented on HIVE-1332: -- Paul, is there wiki documentation available for this feature? Archiving partitions Key: HIVE-1332 URL: https://issues.apache.org/jira/browse/HIVE-1332 Project: Hive Issue Type: New Feature Components: Metastore Reporter: Paul Yang Assignee: Paul Yang Fix For: 0.6.0 Attachments: HIVE-1332.1.patch, HIVE-1332.2.patch, HIVE-1332.3.patch, HIVE-1332.4.patch, HIVE-1332.5.patch, HIVE-1332.6.patch Partitions and tables in Hive typically consist of many files on HDFS. An issue is that as the number of files increase, there will be higher memory/load requirements on the namenode. Partitions in bucketed tables are a particular problem because they consist of many files, one for each of the buckets. One way to drastically reduce the number of files is to use hadoop archives: http://hadoop.apache.org/common/docs/current/hadoop_archives.html This feature would introduce an ALTER TABLE table_name ARCHIVE PARTITION spec that would automatically put the files for the partition into a HAR file. We would also have an UNARCHIVE option to convert the files in the partition back to the original files. Archived partitions would be slower to access, but they would have the same functionality and decrease the number of files drastically. Typically, only seldom accessed partitions would be archived. Hadoop archives are still somewhat new, so we'll only put in support for the latest released major version (0.20). Here are some bug fixes: https://issues.apache.org/jira/browse/HADOOP-6591 (Important - could potentially cause data loss without this fix) https://issues.apache.org/jira/browse/HADOOP-6645 https://issues.apache.org/jira/browse/MAPREDUCE-1585 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1759) Many important broken links on Hive web page
[ https://issues.apache.org/jira/browse/HIVE-1759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12926399#action_12926399 ] John Sichi commented on HIVE-1759: -- I committed a fix for the change log links (per Carl's suggestion, just use JIRA release notes alone and omit CHANGES.txt). I also changed the nav bar trail to be Apache Hive instead of Apache Hadoop Hive, but kept the Hadoop logo. I'll leave the resolution of the documentation links to Ed since that's still under discussion. Once we fix that, we can do another push of the site. Many important broken links on Hive web page Key: HIVE-1759 URL: https://issues.apache.org/jira/browse/HIVE-1759 Project: Hive Issue Type: Bug Components: Documentation Reporter: Jeff Hammerbacher Assignee: Edward Capriolo The change log links are broken, perhaps because of the move to a TLP, and the Jira issue log links all point to the 0.5 issue log. Also, all of the documentation links are broken. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: release 0.6.0 wrapup
Ed mentioned in IRC that he would not be able to get to this for a few days, so I'll see if I can update it today so that you can announce the release. JVS On Oct 27, 2010, at 10:19 PM, Carl Steinbach wrote: Hey, I'd like to reference Hive Releases page in the 0.6.0 release announcement email. Ed, can you please update this page? (http://hive.apache.org/releases.html) It looks like the links for the old releases need to be updated, and for the 0.6.0 release we should provide a link to the JIRA release notes page instead of to the CHANGES document, i.e. https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310843styleName=Htmlversion=12314524 Thanks. Carl On Tue, Oct 26, 2010 at 7:01 PM, Carl Steinbach c...@cloudera.com wrote: Carl, as release manager, can you send out the release announcement once everything is ready? I'll be at ApacheCon US next week in Atlanta and will be spreading the word on the release there. Will do!
[jira] Commented: (HIVE-1634) Allow access to Primitive types stored in binary format in HBase
[ https://issues.apache.org/jira/browse/HIVE-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12925915#action_12925915 ] John Sichi commented on HIVE-1634: -- Thanks Basab, I'm going to try to take a look at this one next week. Allow access to Primitive types stored in binary format in HBase Key: HIVE-1634 URL: https://issues.apache.org/jira/browse/HIVE-1634 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.7.0 Reporter: Basab Maulik Assignee: Basab Maulik Attachments: HIVE-1634.0.patch, TestHiveHBaseExternalTable.java This addresses HIVE-1245 in part, for atomic or primitive types. The serde property hbase.columns.storage.types = -,b,b,b,b,b,b,b,b is a specification of the storage option for the corresponding column in the serde property hbase.columns.mapping. Allowed values are '-' for table default, 's' for standard string storage, and 'b' for binary storage as would be obtained from o.a.h.hbase.utils.Bytes. Map types for HBase column families use a colon separated pair such as 's:b' for the key and value part specifiers respectively. See the test cases and queries for HBase handler for additional examples. There is also a table property hbase.table.default.storage.type = string to specify a table level default storage type. The other valid specification is binary. The table level default is overridden by a column level specification. This control is available for the boolean, tinyint, smallint, int, bigint, float, and double primitive types. The attached patch also relaxes the mapping of map types to HBase column families to allow any primitive type to be the map key. Attached is a program for creating a table and populating it in HBase. The external table in Hive can access the data as shown in the example below. hive create external table TestHiveHBaseExternalTable (key string, c_bool boolean, c_byte tinyint, c_short smallint, c_int int, c_long bigint, c_string string, c_float float, c_double double) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties (hbase.columns.mapping = :key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double) tblproperties (hbase.table.name = TestHiveHBaseExternalTable); OK Time taken: 0.691 seconds hive select * from TestHiveHBaseExternalTable; OK key-1 NULLNULLNULLNULLNULLTest-String NULLNULL Time taken: 0.346 seconds hive drop table TestHiveHBaseExternalTable; OK Time taken: 0.139 seconds hive create external table TestHiveHBaseExternalTable (key string, c_bool boolean, c_byte tinyint, c_short smallint, c_int int, c_long bigint, c_string string, c_float float, c_double double) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties ( hbase.columns.mapping = :key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double, hbase.columns.storage.types = -,b,b,b,b,b,b,b,b ) tblproperties ( hbase.table.name = TestHiveHBaseExternalTable, hbase.table.default.storage.type = string); OK Time taken: 0.139 seconds hive select * from TestHiveHBaseExternalTable; OK key-1 true-128-32768 -2147483648 -9223372036854775808 Test-String -2.1793132E-11 2.01345E291 Time taken: 0.151 seconds hive drop table TestHiveHBaseExternalTable; OK Time taken: 0.154 seconds hive create external table TestHiveHBaseExternalTable (key string, c_bool boolean, c_byte tinyint, c_short smallint, c_int int, c_long bigint, c_string string, c_float float, c_double double) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties ( hbase.columns.mapping = :key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double, hbase.columns.storage.types = -,b,b,b,b,b,-,b,b ) tblproperties (hbase.table.name = TestHiveHBaseExternalTable); OK Time taken: 0.347 seconds hive select * from TestHiveHBaseExternalTable; OK key-1 true-128-32768 -2147483648 -9223372036854775808 Test-String -2.1793132E-11 2.01345E291 Time taken: 0.245 seconds hive -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1498) support IDXPROPERTIES on CREATE INDEX
[ https://issues.apache.org/jira/browse/HIVE-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1498: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed. Thanks Marquis et al! support IDXPROPERTIES on CREATE INDEX - Key: HIVE-1498 URL: https://issues.apache.org/jira/browse/HIVE-1498 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Marquis Wang Fix For: 0.7.0 Attachments: 1498.2.patch, 1498.patch, hive-1498.prelim.patch It's partially there in the grammar but not hooked in; should work pretty much the same as TBLPROPERTIES. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1498) support IDXPROPERTIES on CREATE INDEX
[ https://issues.apache.org/jira/browse/HIVE-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12925529#action_12925529 ] John Sichi commented on HIVE-1498: -- +1. Will commit when tests pass. support IDXPROPERTIES on CREATE INDEX - Key: HIVE-1498 URL: https://issues.apache.org/jira/browse/HIVE-1498 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Marquis Wang Fix For: 0.7.0 Attachments: 1498.2.patch, 1498.patch, hive-1498.prelim.patch It's partially there in the grammar but not hooked in; should work pretty much the same as TBLPROPERTIES. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
test output just keeps getting noisier...
JVS ... [junit] rmr: cannot remove pfile:/data/users/jsichi/open/commit-trunk/build/ql/test/data/warehouse/clustergroupby/ds=103: No such file or directory. [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I lastUpdateTime -I lastAccessTime -I [Oo]wner -I CreateTime -I LastAccessTime -I Location -I transient_lastDdlTime -I last_modified_ -I java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I Caused by: -I [.][.][.] [0-9]* more /data/users/jsichi/open/commit-trunk/build/ql/test/logs/clientpositive/bucket_groupby.q.out /data/users/jsichi/open/commit-trunk/ql/src/test/results/clientpositive/bucket_groupby.q.out [junit] Done query: bucket_groupby.q [junit] Begin query: bucketmapjoin1.q [junit] Upload 1 JDBM File to: file:/tmp/jsichi/hive_2010-10-27_13-25-50_233_1471536439352402066/-local-10003/JDBM-Stage-1/-1-srcbucket20.txt.jdbm [junit] Upload 1 JDBM File to: file:/tmp/jsichi/hive_2010-10-27_13-25-50_233_1471536439352402066/-local-10003/JDBM-Stage-1/-1-srcbucket21.txt.jdbm [junit] Reloading existing hashtable... [junit] Reloading existing hashtable... [junit] Deleted file:/data/users/jsichi/open/commit-trunk/build/ql/test/data/warehouse/bucketmapjoin_tmp_result [junit] Deleted file:/data/users/jsichi/open/commit-trunk/build/ql/test/data/warehouse/bucketmapjoin_hash_result_1 [junit] Upload 1 JDBM File to: file:/tmp/jsichi/hive_2010-10-27_13-26-12_690_3816402881235870624/-local-10003/JDBM-Stage-1/-1--.jdbm [junit] Reloading existing hashtable... [junit] Reloading existing hashtable... [junit] Deleted file:/data/users/jsichi/open/commit-trunk/build/ql/test/data/warehouse/bucket
Re: svn move and INFRA-3036
I'm starting on the svn move in a little bit. Committers, please hold off on further commits until you see an update on this. JVS On Oct 7, 2010, at 10:45 AM, Edward Capriolo wrote: All, Part of the move to TLP will require us moving our SVN. https://issues.apache.org/jira/browse/INFRA-3036 Infra is going to tackle item #2 soon. After creates the new svn, we need to do the svn mv's into it. Users will have to run for their workspaces: 'svn switch https://svn.apache.org/repos/asf/hive/trunk .' @hive-dev. Once item #2 is completed we should schedule the SVN move. We can do this without any help from infra. So we should schedule this internally (hive-dev). Edward
svn repository move complete
If you have outstanding checkouts (including ones with changes) you can update them using svn switch: svn switch https://svn.apache.org/repos/asf/hive/trunk The above assumes you have trunk checked out (with https for committing). If you instead have a branch checked out, or are using http, then adjust the URL accordingly. I'll update the wiki etc with the new location. JVS
release 0.6.0 wrapup
Since the release vote passed, I've gone ahead and moved the tag from candidate to release, and then copied the release binaries to the distribution directory. According to the instructions, those will take about 24 hours to propagate to all mirrors. Ed, I think you mentioned in IRC that you are going to work on updating http://hive.apache.org?Besides the releases/news, we should get rid of references to Hive as a Hadoop subproject. (For the bar at the top, I checked Pig and HBase, and they still have Hadoop logo and nav menu, so I guess we can leave that for now.) We should also update the mailing list addresses (but leave the wiki URL alone for now). Let me know whether you're going to take care of these or you want me to. Carl, as release manager, can you send out the release announcement once everything is ready? I'll be at ApacheCon US next week in Atlanta and will be spreading the word on the release there. JVS
[jira] Commented: (HIVE-1750) Remove Partition Filtering Conditions when Possible
[ https://issues.apache.org/jira/browse/HIVE-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12925209#action_12925209 ] John Sichi commented on HIVE-1750: -- This is the same logic I have in IndexPredicateAnalyzer.analyzePredicate. And it can be configured with the specific set of columns to allow. So you might be able to reuse it as is. Remove Partition Filtering Conditions when Possible --- Key: HIVE-1750 URL: https://issues.apache.org/jira/browse/HIVE-1750 Project: Hive Issue Type: Improvement Reporter: Siying Dong Assignee: Siying Dong For some simple queries, partition filtering constraints take 8% of CPU time (now 16% since we filter twice) even if the result is always true. When possible, we should remove these constraints to save CPU times. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1575) get_json_object does not support JSON array at the root level
[ https://issues.apache.org/jira/browse/HIVE-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi reassigned HIVE-1575: Assignee: Mike Lewis get_json_object does not support JSON array at the root level - Key: HIVE-1575 URL: https://issues.apache.org/jira/browse/HIVE-1575 Project: Hive Issue Type: Improvement Components: UDF Affects Versions: 0.7.0 Reporter: Steven Wong Assignee: Mike Lewis Attachments: 0001-Updated-UDFJson-to-allow-arrays-as-a-root-object.patch Currently, get_json_object(json_txt, path) always returns null if json_txt is not a JSON object (e.g. is a JSON array) at the root level. I have a table column of JSON arrays at the root level, but I can't parse it because of that. get_json_object should accept any JSON value (string, number, object, array, true, false, null), not just object, at the root level. In other words, it should behave as if it were named get_json_value or simply get_json. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1740) support NOT IN and NOT LIKE syntax
support NOT IN and NOT LIKE syntax -- Key: HIVE-1740 URL: https://issues.apache.org/jira/browse/HIVE-1740 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi Fix For: 0.7.0 Hive should support standard SQL syntax x NOT LIKE p x NOT IN (...) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1740) support NOT IN and NOT LIKE syntax
[ https://issues.apache.org/jira/browse/HIVE-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12923641#action_12923641 ] John Sichi commented on HIVE-1740: -- This may apply to more operators such as RLIKE too. Workaround is currently to use prefix NOT NOT(x LIKE p) etc. support NOT IN and NOT LIKE syntax -- Key: HIVE-1740 URL: https://issues.apache.org/jira/browse/HIVE-1740 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi Fix For: 0.7.0 Hive should support standard SQL syntax x NOT LIKE p x NOT IN (...) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1376) Simple UDAFs with more than 1 parameter crash on empty row query
[
https://issues.apache.org/jira/browse/HIVE-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12923090#action_12923090
]
John Sichi commented on HIVE-1376:
--
This patch only did (1), not (3). I think we'll still need a followup to avoid
the problem for arbitrary UDAF's (unless we require them to avoid primitive
types).
Simple UDAFs with more than 1 parameter crash on empty row query
-
Key: HIVE-1376
URL: https://issues.apache.org/jira/browse/HIVE-1376
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.6.0
Reporter: Mayank Lahiri
Assignee: Ning Zhang
Fix For: 0.7.0
Attachments: HIVE-1376.2.patch, HIVE-1376.patch
Simple UDAFs with more than 1 parameter crash when the query returns no rows.
Currently, this only seems to affect the percentile() UDAF where the second
parameter is the percentile to be computed (of type double). I've also
verified the bug by adding a dummy parameter to ExampleMin in contrib.
On an empty query, Hive seems to be trying to resolve an iterate() method
with signature {null,null} instead of {null,double}. You can reproduce this
bug using:
CREATE TABLE pct_test ( val INT );
SELECT percentile(val, 0.5) FROM pct_test;
which produces a lot of errors like:
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
execute method public boolean
org.apache.hadoop.hive.ql.udf.UDAFPercentile$PercentileLongEvaluator.iterate(org.apache.hadoop.io.LongWritable,double)
on object
org.apache.hadoop.hive.ql.udf.udafpercentile$percentilelongevalua...@11d13272
of class org.apache.hadoop.hive.ql.udf.UDAFPercentile$PercentileLongEvaluator
with arguments {null, null} of size 2
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[VOTE] hive 0.6.0 release candidate 0
The tarballs are at http://people.apache.org/~jvs/hive-0.6.0-candidate-0 Carl did some sanity testing on it already, but any additional testing you can do before voting helps to ensure a quality release. JVS
[jira] Updated: (HIVE-1729) Satisfy ASF release management requirements
[ https://issues.apache.org/jira/browse/HIVE-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1729: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed to branch and trunk. Thanks Carl! Satisfy ASF release management requirements --- Key: HIVE-1729 URL: https://issues.apache.org/jira/browse/HIVE-1729 Project: Hive Issue Type: Task Components: Build Infrastructure, Documentation Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.6.0 Attachments: HIVE-1729-backport06.1.patch.txt, HIVE-1729.1.patch.txt We need to make sure we satisfy the ASF release requirements: * http://www.apache.org/dev/release.html * http://incubator.apache.org/guides/releasemanagement.html -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1736) Remove -dev suffix from release package name and generate MD5 checksum using Ant
[ https://issues.apache.org/jira/browse/HIVE-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1736: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) +1. Committed to branch and trunk. Thanks Carl! Remove -dev suffix from release package name and generate MD5 checksum using Ant -- Key: HIVE-1736 URL: https://issues.apache.org/jira/browse/HIVE-1736 Project: Hive Issue Type: Task Components: Build Infrastructure Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.6.0 Attachments: HIVE-1736-backport.1.patch.txt, HIVE-1736.1.patch.txt -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1725) Include metastore upgrade scripts in release tarball
[ https://issues.apache.org/jira/browse/HIVE-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1725: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed. Thanks Carl! Include metastore upgrade scripts in release tarball Key: HIVE-1725 URL: https://issues.apache.org/jira/browse/HIVE-1725 Project: Hadoop Hive Issue Type: Task Components: Build Infrastructure Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.6.0 Attachments: HIVE-1725-backport06.2.patch.txt, HIVE-1725.1.patch.txt, HIVE-1725.2.patch.txt Need to update the build so that the metastore upgrade scripts are included in the tar ball generated by the 'tar' ant target. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1726) Update README file for 0.6.0 release
[ https://issues.apache.org/jira/browse/HIVE-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1726: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed. Thanks Carl! Update README file for 0.6.0 release Key: HIVE-1726 URL: https://issues.apache.org/jira/browse/HIVE-1726 Project: Hadoop Hive Issue Type: Task Components: Build Infrastructure Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.6.0 Attachments: HIVE-1726.1.patch.txt Notest from John: * recommend Hadoop 0.20 instead of 0.17 * should we update the mailing lists to the new locations? * what about svn co location? mention both old and new? or just replace this section with a wiki link? * mention metastore upgrade * tone down the prototype only stuff? * get rid of everything starting from Using Hive and replace with corresponding wiki links? This stuff is all super stale. The assumption here is that we either leave around old stuff on the Hadoop wiki or replace with forwarding links. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1724) The query in join_filters.q will failed if the table size is larger than 25000
[ https://issues.apache.org/jira/browse/HIVE-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1724: - Fix Version/s: (was: 0.6.0) The query in join_filters.q will failed if the table size is larger than 25000 -- Key: HIVE-1724 URL: https://issues.apache.org/jira/browse/HIVE-1724 Project: Hadoop Hive Issue Type: Bug Components: Query Processor Reporter: Liyin Tang Assignee: Liyin Tang Fix For: 0.7.0 In the test case join_filter.q, there is a query: SELECT /*+ MAPJOIN(a) */ * FROM myinput1 a JOIN myinput1 b on a.key 40 AND a.value 50 AND a.key = a.value AND b.key 40 AND b.value 50 AND b.key = b.value; Right now the table of myinput is only 4 rows. I tried to create a new table and it has more than 25000 rows. Actually, this query will failed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1651) ScriptOperator should not forward any output to downstream operators if an exception is happened
[ https://issues.apache.org/jira/browse/HIVE-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12922168#action_12922168 ] John Sichi commented on HIVE-1651: -- Should we close this one as Won't Fix? ScriptOperator should not forward any output to downstream operators if an exception is happened Key: HIVE-1651 URL: https://issues.apache.org/jira/browse/HIVE-1651 Project: Hadoop Hive Issue Type: Bug Reporter: Ning Zhang Assignee: Ning Zhang Attachments: HIVE-1651.patch ScriptOperator spawns 2 threads for getting the stdout and stderr from the script and then forward the output from stdout to downstream operators. In case of any exceptions to the script (e.g., got killed), the ScriptOperator got an exception and throw it to upstream operators until MapOperator got it and call close(abort). Before the ScriptOperator.close() is called the script output stream can still forward output to downstream operators. We should terminate it immediately. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1731) Improve miscellaneous error messages
Improve miscellaneous error messages Key: HIVE-1731 URL: https://issues.apache.org/jira/browse/HIVE-1731 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: John Sichi Fix For: 0.7.0 This is a place for accumulating error message improvements so that we can update a bunch in batch. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1731) Improve miscellaneous error messages
[ https://issues.apache.org/jira/browse/HIVE-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12922379#action_12922379 ] John Sichi commented on HIVE-1731: -- From Andrew Ryan: Instead of FAILED: Error in semantic analysis: line 1:333 In strict mode, limit must be specified if ORDER BY is present make it FAILED: Error in semantic analysis: line 1:333 In strict mode, if ORDER BY is specified, LIMIT must also be specified Improve miscellaneous error messages Key: HIVE-1731 URL: https://issues.apache.org/jira/browse/HIVE-1731 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: John Sichi Fix For: 0.7.0 This is a place for accumulating error message improvements so that we can update a bunch in batch. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1720) hbase_stats.q is failing
[ https://issues.apache.org/jira/browse/HIVE-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12921744#action_12921744 ] John Sichi commented on HIVE-1720: -- +1. Will commit when tests pass. hbase_stats.q is failing Key: HIVE-1720 URL: https://issues.apache.org/jira/browse/HIVE-1720 Project: Hadoop Hive Issue Type: Bug Components: HBase Handler Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Ning Zhang Fix For: 0.7.0 Attachments: HIVE-1720.patch Saw this failure on Hudson and in my own sandbox. https://hudson.apache.org/hudson/job/Hive-trunk-h0.20/392/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1720) hbase_stats.q is failing
[ https://issues.apache.org/jira/browse/HIVE-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1720: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed. Thanks Ning! hbase_stats.q is failing Key: HIVE-1720 URL: https://issues.apache.org/jira/browse/HIVE-1720 Project: Hadoop Hive Issue Type: Bug Components: HBase Handler Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Ning Zhang Fix For: 0.7.0 Attachments: HIVE-1720.patch Saw this failure on Hudson and in my own sandbox. https://hudson.apache.org/hudson/job/Hive-trunk-h0.20/392/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: hive svn is up and hive.apache.org is up
On Oct 15, 2010, at 1:21 PM, Carl Steinbach wrote: So in theory, together these could give us what we were looking for: do a one-time conversion from MoinMoin to Confluence, and then a per-release snapshot. But we would have to do some testing to see how well they actually work. It sounds like you're suggesting that we continue to maintain the documentation on the wiki? I thought we decided to move it over to version control. Just throwing it out in case we ever wanted to revisit that approach. I'll definitely be looking into the MoinMoin-Confluence auto-conversion part regardless since there's so much to move even if some of it goes to svn. (But if anyone beats me to it in testing out the converter, that would be great too.) JVS
[jira] Created: (HIVE-1720) hbase_stats.q is failing
hbase_stats.q is failing Key: HIVE-1720 URL: https://issues.apache.org/jira/browse/HIVE-1720 Project: Hadoop Hive Issue Type: Bug Components: HBase Handler Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Ning Zhang Fix For: 0.7.0 Saw this failure on Hudson and in my own sandbox. https://hudson.apache.org/hudson/job/Hive-trunk-h0.20/392/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: turn off Hudson tests on pre-0.20 Hadoop?
I just pinged Johan in IRC, and he says they recently changed the permissions so he no longer has access either, so I'm going to file an INFRA ticket. I think we'll need privs for keeping Hudson in sync with the TLP svn switch as well. JVS On Oct 13, 2010, at 1:20 PM, Carl Steinbach wrote: Hi Ning, I logged in hudson but didn't see 'delete project'. I guess hudson has a different set of committers? I'll ask around FB committers. I remember John mentioned that Johan was the person who originally setup all of these jobs. It may be worth getting in touch with him and asking if he give the rest of the Hive committers the necessary privileges to modify Hive hudson jobs. Otherwise you should probably file an INFRA ticket. Also there is already a 0.20 project running https://hudson.apache.org/hudson/view/Hive/job/Hive-trunk-h0.20/ My mistake. I overlooked this job since it doesn't appear in the Hive group (https://hudson.apache.org/hudson/view/Hive/). Once you get the necessary Hudson privileges can you please add this job to the Hive group? Thanks. Carl
Re: [howldev] RE: Howl Authorization proposal
On Oct 13, 2010, at 9:22 AM, Alan Gates wrote: Our biggest concern is that HDFS already has a permissions model, why create a whole new one? It is a lot of duplication. And that duplication will flow through to things like logging and auditing, all of which Hive/Howl will now need in addition to HDFS. To justify this we needed to understand what additional benefits a traditional ACL model would get us. We were not able to come up with compelling use cases where we had to have this traditional model. Here are some you probably already considered, but I'm listing them for consideration anyway... * table A can only be queried by roles X and Y; table B can only be queried by roles Y and Z; managing different groups for all the possible role combinations isn't very practical given large numbers of tables and roles * finer-grained access control (e.g. column-level) may not be expressible in terms of HDFS permissions without doing things like creating dummy files (although in SQL, views can be used to avoid column-level permissions) * privileges beyond read/write (e.g. delete vs update vs append) * (Hive-specific): GRANT/REVOKE is the standard SQL approach and requires ACL's (it can't be implemented in terms of HDFS permissions) All that said, I see no problem with having two models for now, and seeing which turns out to better provide what users need and/or be easier to maintain. OK, let us know if the hooks turn out to be insufficient as the implementation mechanism. JVS
HIVE-1701
I'm going to start working on a patch for dropping pre-0.20 Hadoop. JVS
[jira] Updated: (HIVE-1701) drop support for pre-0.20 Hadoop versions
[ https://issues.apache.org/jira/browse/HIVE-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1701: - Status: Patch Available (was: Open) drop support for pre-0.20 Hadoop versions - Key: HIVE-1701 URL: https://issues.apache.org/jira/browse/HIVE-1701 Project: Hadoop Hive Issue Type: Improvement Components: Build Infrastructure Affects Versions: 0.7.0 Reporter: John Sichi Assignee: John Sichi Fix For: 0.7.0 Attachments: HIVE-1701.1.patch As discussed previously on the mailing lists, we're dropping support for pre-0.20 Hadoop versions starting with Hive 0.7. This JIRA issue is for deleting the corresponding build and shim implementations. The shim mechanism itself will be left in place (we already have 0.20 and 0.20S coexisting). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1701) drop support for pre-0.20 Hadoop versions
[ https://issues.apache.org/jira/browse/HIVE-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1701: - Attachment: HIVE-1701.1.patch Note that this patch deletes a lot of files; the committer needs to do the following explicitly: cd shims/src svn remove 0.17 0.18 0.19 drop support for pre-0.20 Hadoop versions - Key: HIVE-1701 URL: https://issues.apache.org/jira/browse/HIVE-1701 Project: Hadoop Hive Issue Type: Improvement Components: Build Infrastructure Affects Versions: 0.7.0 Reporter: John Sichi Assignee: John Sichi Fix For: 0.7.0 Attachments: HIVE-1701.1.patch As discussed previously on the mailing lists, we're dropping support for pre-0.20 Hadoop versions starting with Hive 0.7. This JIRA issue is for deleting the corresponding build and shim implementations. The shim mechanism itself will be left in place (we already have 0.20 and 0.20S coexisting). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
