[jira] [Created] (HIVE-5025) Column aliases for input argument of GenericUDFs
Navis created HIVE-5025: --- Summary: Column aliases for input argument of GenericUDFs Key: HIVE-5025 URL: https://issues.apache.org/jira/browse/HIVE-5025 Project: Hive Issue Type: Improvement Components: UDF Reporter: Navis Assignee: Navis Priority: Trivial In some cases, column aliases for input argument are very useful to know. But I cannot sure of this in the sense that UDFs should not be dependent to contextual information like column alias. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-5026) HIVE-3926 is committed in the state of not rebased to trunk
Navis created HIVE-5026: --- Summary: HIVE-3926 is committed in the state of not rebased to trunk Key: HIVE-5026 URL: https://issues.apache.org/jira/browse/HIVE-5026 Project: Hive Issue Type: Task Components: Tests Reporter: Navis Assignee: Navis Current trunk build fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5025) Column aliases for input argument of GenericUDFs
[ https://issues.apache.org/jira/browse/HIVE-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-5025: Status: Open (was: Patch Available) Column aliases for input argument of GenericUDFs - Key: HIVE-5025 URL: https://issues.apache.org/jira/browse/HIVE-5025 Project: Hive Issue Type: Improvement Components: UDF Reporter: Navis Assignee: Navis Priority: Trivial Attachments: HIVE-5025.D12093.1.patch In some cases, column aliases for input argument are very useful to know. But I cannot sure of this in the sense that UDFs should not be dependent to contextual information like column alias. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5026) HIVE-3926 is committed in the state of not rebased to trunk
[ https://issues.apache.org/jira/browse/HIVE-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-5026: Status: Patch Available (was: Open) Quick fix for test HIVE-3926 is committed in the state of not rebased to trunk --- Key: HIVE-5026 URL: https://issues.apache.org/jira/browse/HIVE-5026 Project: Hive Issue Type: Task Components: Tests Reporter: Navis Assignee: Navis Current trunk build fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5025) Column aliases for input argument of GenericUDFs
[ https://issues.apache.org/jira/browse/HIVE-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-5025: Status: Patch Available (was: Open) Quick fix for test Column aliases for input argument of GenericUDFs - Key: HIVE-5025 URL: https://issues.apache.org/jira/browse/HIVE-5025 Project: Hive Issue Type: Improvement Components: UDF Reporter: Navis Assignee: Navis Priority: Trivial Attachments: HIVE-5025.D12093.1.patch In some cases, column aliases for input argument are very useful to know. But I cannot sure of this in the sense that UDFs should not be dependent to contextual information like column alias. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Deleted] (HIVE-5025) Column aliases for input argument of GenericUDFs
[ https://issues.apache.org/jira/browse/HIVE-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-5025: Comment: was deleted (was: Quick fix for test) Column aliases for input argument of GenericUDFs - Key: HIVE-5025 URL: https://issues.apache.org/jira/browse/HIVE-5025 Project: Hive Issue Type: Improvement Components: UDF Reporter: Navis Assignee: Navis Priority: Trivial Attachments: HIVE-5025.D12093.1.patch In some cases, column aliases for input argument are very useful to know. But I cannot sure of this in the sense that UDFs should not be dependent to contextual information like column alias. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5026) HIVE-3926 is committed in the state of not rebased to trunk
[ https://issues.apache.org/jira/browse/HIVE-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-5026: -- Attachment: HIVE-5026.D12099.1.patch navis requested code review of HIVE-5026 [jira] HIVE-3926 is committed in the state of not rebased to trunk. Reviewers: JIRA HIVE-5026 HIVE-3926 is committed in the state of not rebased to trunk Current trunk build fails. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D12099 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/28887/ To: JIRA, navis HIVE-3926 is committed in the state of not rebased to trunk --- Key: HIVE-5026 URL: https://issues.apache.org/jira/browse/HIVE-5026 Project: Hive Issue Type: Task Components: Tests Reporter: Navis Assignee: Navis Attachments: HIVE-5026.D12099.1.patch Current trunk build fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4847) add rename database functionality
[ https://issues.apache.org/jira/browse/HIVE-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733205#comment-13733205 ] Teddy Choi commented on HIVE-4847: -- I wrote some code to modify a MDatabase name on ObjectStore, and it seemed like working properly. But it had a serious bug. When a user creates and manipulates a new database and tables with old names, data gets corrupted. Because old tables and new tables used same data directories. It was more complex than I thought first. I will modify the code to move directories containing actual data. There will be some exceptional cases; such as there already exist directories with same names, or external tables. When Hive meets those cases, it will throw an error to prevent data corruption. add rename database functionality - Key: HIVE-4847 URL: https://issues.apache.org/jira/browse/HIVE-4847 Project: Hive Issue Type: New Feature Affects Versions: 0.11.0 Reporter: Greg Rahn Assignee: Teddy Choi Priority: Minor There seems to be no way to rename a database in Hive, functionality to do so would be nice. Proposed syntax: ALTER DATABASE dbname RENAME TO newdbname; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4847) add rename database functionality
[ https://issues.apache.org/jira/browse/HIVE-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733205#comment-13733205 ] Teddy Choi commented on HIVE-4847: -- I wrote some code to modify a MDatabase name on ObjectStore, and it seemed like working properly. But it had a serious bug. When a user creates and manipulates a new database and tables with old names, data gets corrupted. Because old tables and new tables used same data directories. It was more complex than I thought first. I will modify the code to move directories containing actual data. There will be some exceptional cases; such as there already exist directories with same names, or external tables. When Hive meets those cases, it will throw an error to prevent data corruption. add rename database functionality - Key: HIVE-4847 URL: https://issues.apache.org/jira/browse/HIVE-4847 Project: Hive Issue Type: New Feature Affects Versions: 0.11.0 Reporter: Greg Rahn Assignee: Teddy Choi Priority: Minor There seems to be no way to rename a database in Hive, functionality to do so would be nice. Proposed syntax: ALTER DATABASE dbname RENAME TO newdbname; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5008) Reuse MapRedUtils to generate Map/ReduceWork
[ https://issues.apache.org/jira/browse/HIVE-5008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-5008: - Attachment: HIVE-5008.1.patch Reuse MapRedUtils to generate Map/ReduceWork Key: HIVE-5008 URL: https://issues.apache.org/jira/browse/HIVE-5008 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Fix For: tez-branch Attachments: HIVE-5008.1.patch GenTezWork needs to be updated to use the right MapRedUtils to generate and setup the Map/ReduceWork properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5008) Reuse MapRedUtils to generate Map/ReduceWork
[ https://issues.apache.org/jira/browse/HIVE-5008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-5008: - Assignee: Gunther Hagleitner Reuse MapRedUtils to generate Map/ReduceWork Key: HIVE-5008 URL: https://issues.apache.org/jira/browse/HIVE-5008 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: tez-branch Attachments: HIVE-5008.1.patch GenTezWork needs to be updated to use the right MapRedUtils to generate and setup the Map/ReduceWork properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5008) Reuse MapRedUtils to generate Map/ReduceWork
[ https://issues.apache.org/jira/browse/HIVE-5008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-5008: - Description: GenTezWork needs to be updated to use the right MapRedUtils to generate and setup the Map/ReduceWork properly. NO PRECOMMIT TESTS (this is wip for the tez branch) was:GenTezWork needs to be updated to use the right MapRedUtils to generate and setup the Map/ReduceWork properly. Reuse MapRedUtils to generate Map/ReduceWork Key: HIVE-5008 URL: https://issues.apache.org/jira/browse/HIVE-5008 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: tez-branch Attachments: HIVE-5008.1.patch GenTezWork needs to be updated to use the right MapRedUtils to generate and setup the Map/ReduceWork properly. NO PRECOMMIT TESTS (this is wip for the tez branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-5008) Reuse MapRedUtils to generate Map/ReduceWork
[ https://issues.apache.org/jira/browse/HIVE-5008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner resolved HIVE-5008. -- Resolution: Fixed Reuse MapRedUtils to generate Map/ReduceWork Key: HIVE-5008 URL: https://issues.apache.org/jira/browse/HIVE-5008 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: tez-branch Attachments: HIVE-5008.1.patch GenTezWork needs to be updated to use the right MapRedUtils to generate and setup the Map/ReduceWork properly. NO PRECOMMIT TESTS (this is wip for the tez branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5008) Reuse MapRedUtils to generate Map/ReduceWork
[ https://issues.apache.org/jira/browse/HIVE-5008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733237#comment-13733237 ] Gunther Hagleitner commented on HIVE-5008: -- Committed to tez branch. Reuse MapRedUtils to generate Map/ReduceWork Key: HIVE-5008 URL: https://issues.apache.org/jira/browse/HIVE-5008 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: tez-branch Attachments: HIVE-5008.1.patch GenTezWork needs to be updated to use the right MapRedUtils to generate and setup the Map/ReduceWork properly. NO PRECOMMIT TESTS (this is wip for the tez branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4943) An explode function that includes the item's position in the array
[ https://issues.apache.org/jira/browse/HIVE-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niko Stahl updated HIVE-4943: - Fix Version/s: 0.11.0 Affects Version/s: 0.11.0 Status: Patch Available (was: Open) Please review. An explode function that includes the item's position in the array -- Key: HIVE-4943 URL: https://issues.apache.org/jira/browse/HIVE-4943 Project: Hive Issue Type: New Feature Components: Query Processor Affects Versions: 0.11.0 Reporter: Niko Stahl Labels: patch Fix For: 0.11.0 Original Estimate: 8h Remaining Estimate: 8h A function that explodes an array and includes an output column with the position of each item in the original array. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4987) Javadoc can generate argument list too long error
[ https://issues.apache.org/jira/browse/HIVE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733286#comment-13733286 ] Hudson commented on HIVE-4987: -- ABORTED: Integrated in Hive-trunk-hadoop1-ptest #121 (See [https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/121/]) HIVE-4987 : Javadoc can generate argument list too long error (Brock Noland via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511375) * /hive/trunk/build.xml * /hive/trunk/hcatalog/webhcat/svr/build.xml Javadoc can generate argument list too long error - Key: HIVE-4987 URL: https://issues.apache.org/jira/browse/HIVE-4987 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4987.patch We just to add to useexternalfile=yes to the javadoc statements. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4886) beeline code should have apache license headers
[ https://issues.apache.org/jira/browse/HIVE-4886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733288#comment-13733288 ] Hudson commented on HIVE-4886: -- ABORTED: Integrated in Hive-trunk-hadoop1-ptest #121 (See [https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/121/]) HIVE-4886 Beeline code should have Apache license headers. (Thejas M Nair via omalley) (omalley: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511485) * /hive/trunk/beeline/src/java/org/apache/hive/beeline/AbstractCommandHandler.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/AbstractOutputFormat.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLine.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLineCommandCompletor.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLineCompletor.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLineSignalHandler.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/BufferedRows.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/ColorBuffer.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/CommandHandler.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/Commands.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/DatabaseConnection.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/DatabaseConnections.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/DriverInfo.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/IncrementalRows.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/OutputFile.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/OutputFormat.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/ReflectiveCommandHandler.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/Reflector.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/Rows.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/SQLCompletor.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/SeparatedValuesOutputFormat.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/SunSignalHandler.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/TableNameCompletor.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/TableOutputFormat.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/VerticalOutputFormat.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/XMLAttributeOutputFormat.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/XMLElementOutputFormat.java beeline code should have apache license headers --- Key: HIVE-4886 URL: https://issues.apache.org/jira/browse/HIVE-4886 Project: Hive Issue Type: Task Components: JDBC Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.12.0 Attachments: HIVE-4886.2.patch, HIVE-4886.patch The beeline jdbc client added as part of hive server2 changes is based on SQLLine. As beeline is modified version of SQLLine and further modifications are also under apache license, the license headers of these files need to be replaced with apache license headers. We already have the license text of SQLLine in LICENSE file . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4051) Hive's metastore suffers from 1+N queries when querying partitions is slow
[
https://issues.apache.org/jira/browse/HIVE-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733287#comment-13733287
]
Hudson commented on HIVE-4051:
--
ABORTED: Integrated in Hive-trunk-hadoop1-ptest #121 (See
[https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/121/])
HIVE-4051 : Hive's metastore suffers from 1+N queries when querying partitions
is slow (Sergey Shelukhin via Ashutosh Chauhan) (hashutosh:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511177)
* /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
*
/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
*
/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
*
/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Hive's metastore suffers from 1+N queries when querying partitions is slow
Key: HIVE-4051
URL: https://issues.apache.org/jira/browse/HIVE-4051
Project: Hive
Issue Type: Bug
Components: Clients, Metastore
Environment: RHEL 6.3 / EC2 C1.XL
Reporter: Gopal V
Assignee: Sergey Shelukhin
Fix For: 0.12.0
Attachments: HIVE-4051.D11805.1.patch, HIVE-4051.D11805.2.patch,
HIVE-4051.D11805.3.patch, HIVE-4051.D11805.4.patch, HIVE-4051.D11805.5.patch,
HIVE-4051.D11805.6.patch, HIVE-4051.D11805.7.patch, HIVE-4051.D11805.8.patch,
HIVE-4051.D11805.9.patch
Hive's query client takes a long time to initialize start planning queries
because of delays in creating all the MTable/MPartition objects.
For a hive db with 1800 partitions, the metastore took 6-7 seconds to
initialize - firing approximately 5900 queries to the mysql database.
Several of those queries fetch exactly one row to create a single object on
the client.
The following 12 queries were repeated for each partition, generating a storm
of SQL queries
{code}
4 Query SELECT
`A0`.`SD_ID`,`B0`.`INPUT_FORMAT`,`B0`.`IS_COMPRESSED`,`B0`.`IS_STOREDASSUBDIRECTORIES`,`B0`.`LOCATION`,`B0`.`NUM_BUCKETS`,`B0`.`OUTPUT_FORMAT`,`B0`.`SD_ID`
FROM `PARTITIONS` `A0` LEFT OUTER JOIN `SDS` `B0` ON `A0`.`SD_ID` =
`B0`.`SD_ID` WHERE `A0`.`PART_ID` = 3945
4 Query SELECT `A0`.`CD_ID`,`B0`.`CD_ID` FROM `SDS` `A0` LEFT OUTER JOIN
`CDS` `B0` ON `A0`.`CD_ID` = `B0`.`CD_ID` WHERE `A0`.`SD_ID` =4871
4 Query SELECT COUNT(*) FROM `COLUMNS_V2` THIS WHERE THIS.`CD_ID`=1546
AND THIS.`INTEGER_IDX`=0
4 Query SELECT
`A0`.`COMMENT`,`A0`.`COLUMN_NAME`,`A0`.`TYPE_NAME`,`A0`.`INTEGER_IDX` AS
NUCORDER0 FROM `COLUMNS_V2` `A0` WHERE `A0`.`CD_ID` = 1546 AND
`A0`.`INTEGER_IDX` = 0 ORDER BY NUCORDER0
4 Query SELECT `A0`.`SERDE_ID`,`B0`.`NAME`,`B0`.`SLIB`,`B0`.`SERDE_ID`
FROM `SDS` `A0` LEFT OUTER JOIN `SERDES` `B0` ON `A0`.`SERDE_ID` =
`B0`.`SERDE_ID` WHERE `A0`.`SD_ID` =4871
4 Query SELECT COUNT(*) FROM `SORT_COLS` THIS WHERE THIS.`SD_ID`=4871 AND
THIS.`INTEGER_IDX`=0
4 Query SELECT `A0`.`COLUMN_NAME`,`A0`.`ORDER`,`A0`.`INTEGER_IDX` AS
NUCORDER0 FROM `SORT_COLS` `A0` WHERE `A0`.`SD_ID` =4871 AND
`A0`.`INTEGER_IDX` = 0 ORDER BY NUCORDER0
4 Query SELECT COUNT(*) FROM `SKEWED_VALUES` THIS WHERE
THIS.`SD_ID_OID`=4871 AND THIS.`INTEGER_IDX`=0
4 Query SELECT 'org.apache.hadoop.hive.metastore.model.MStringList' AS
NUCLEUS_TYPE,`A1`.`STRING_LIST_ID`,`A0`.`INTEGER_IDX` AS NUCORDER0 FROM
`SKEWED_VALUES` `A0` INNER JOIN `SKEWED_STRING_LIST` `A1` ON
`A0`.`STRING_LIST_ID_EID` = `A1`.`STRING_LIST_ID` WHERE `A0`.`SD_ID_OID`
=4871 AND `A0`.`INTEGER_IDX` = 0 ORDER BY NUCORDER0
4 Query SELECT COUNT(*) FROM `SKEWED_COL_VALUE_LOC_MAP` WHERE `SD_ID`
=4871 AND `STRING_LIST_ID_KID` IS NOT NULL
4 Query SELECT 'org.apache.hadoop.hive.metastore.model.MStringList' AS
NUCLEUS_TYPE,`A0`.`STRING_LIST_ID` FROM `SKEWED_STRING_LIST` `A0` INNER JOIN
`SKEWED_COL_VALUE_LOC_MAP` `B0` ON `A0`.`STRING_LIST_ID` =
`B0`.`STRING_LIST_ID_KID` WHERE `B0`.`SD_ID` =4871
4 Query SELECT `A0`.`STRING_LIST_ID_KID`,`A0`.`LOCATION` FROM
`SKEWED_COL_VALUE_LOC_MAP` `A0` WHERE `A0`.`SD_ID` =4871 AND NOT
(`A0`.`STRING_LIST_ID_KID` IS NULL)
{code}
This data is not detached or cached, so this operation is performed during
every query plan for the partitions, even in the same hive client.
The queries are automatically generated by JDO/DataNucleus which makes it
nearly impossible to rewrite it into a single denormalized join operation
process it locally.
Attempts to optimize this with JDO fetch-groups did not bear fruit in
improving the query count.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please
[jira] [Commented] (HIVE-4967) Don't serialize unnecessary fields in query plan
[ https://issues.apache.org/jira/browse/HIVE-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733285#comment-13733285 ] Hudson commented on HIVE-4967: -- ABORTED: Integrated in Hive-trunk-hadoop1-ptest #121 (See [https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/121/]) HIVE-4967 : Don't serialize unnecessary fields in query plan (Ashutosh Chauhan. Reviewed by Brock Noland) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511377) * /hive/trunk/contrib/src/java/org/apache/hadoop/hive/contrib/genericudf/example/GenericUDFDBOutput.java * /hive/trunk/contrib/src/java/org/apache/hadoop/hive/contrib/udtf/example/GenericUDTFExplode2.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFEWAHBitmapBop.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFReflect.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFAverage.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFContextNGrams.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCorrelation.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovariance.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEWAHBitmap.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFHistogramNumeric.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFLeadLag.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMax.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMin.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFNTile.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVariance.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFArray.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFArrayContains.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseCompare.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCase.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCoalesce.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcatWS.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEWAHBitmapEmpty.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFElt.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFField.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFormatNumber.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFromUtcTimestamp.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFHash.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIf.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIn.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInstr.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLocate.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMap.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMapKeys.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMapValues.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFNvl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPAnd.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPNot.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPOr.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFPrintf.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect2.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSentences.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSize.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java *
[jira] [Commented] (HIVE-4992) add ability to skip javadoc during build
[ https://issues.apache.org/jira/browse/HIVE-4992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733284#comment-13733284 ] Hudson commented on HIVE-4992: -- ABORTED: Integrated in Hive-trunk-hadoop1-ptest #121 (See [https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/121/]) HIVE-4992 : add ability to skip javadoc during build (Sergey Shelukhin via Ashutosh h Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511374) * /hive/trunk/build.xml * /hive/trunk/hcatalog/build.xml add ability to skip javadoc during build Key: HIVE-4992 URL: https://issues.apache.org/jira/browse/HIVE-4992 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Trivial Fix For: 0.12.0 Attachments: HIVE-4992-addendum.patch, HIVE-4992.D11967.1.patch, HIVE-4992.D11967.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5006) Re-factor HiveServer2 JDBC PreparedStatement to avoid duplicate code
[ https://issues.apache.org/jira/browse/HIVE-5006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733294#comment-13733294 ] Gunther Hagleitner commented on HIVE-5006: -- Lots of red in the code + green in the tests. Very nice. LGTM +1. [~prasadm] can you take a look at [~thejas]'s comments? Seems like they are minor - will commit after you make an assessment. Re-factor HiveServer2 JDBC PreparedStatement to avoid duplicate code Key: HIVE-5006 URL: https://issues.apache.org/jira/browse/HIVE-5006 Project: Hive Issue Type: Improvement Components: HiveServer2, JDBC Affects Versions: 0.11.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.12.0 Attachments: HIVE-5006.1.patch The JDBC PreparedStatement interface extends Statement interface. The Hive JDBC driver implementations of HivePreparedStatement interface is disjoint from the HiveStatement implementation. As a result there's some duplicate code in these two classes for executing query. Also patches like implicit close on re-execute etc. are not there for PreparedStatement. The HivePreparedStatement should extend the HiveStatement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4863) Fix parallel order by on hadoop2
[ https://issues.apache.org/jira/browse/HIVE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733309#comment-13733309 ] Gunther Hagleitner commented on HIVE-4863: -- [~appodictic] I think you can drop the second parameter to the shim method (the Configuration). The shim should do be able to do the exact same call in both cases - the important thing is that we compile it separately against 20S and hadoop 23. Also, in the 23 version you're setting the file on HiveConf not JobConf which I don't think will work, will it? Fix parallel order by on hadoop2 Key: HIVE-4863 URL: https://issues.apache.org/jira/browse/HIVE-4863 Project: Hive Issue Type: Bug Reporter: Edward Capriolo Assignee: Edward Capriolo Attachments: HIVE-4863.1.patch.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4992) add ability to skip javadoc during build
[ https://issues.apache.org/jira/browse/HIVE-4992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733523#comment-13733523 ] Hudson commented on HIVE-4992: -- SUCCESS: Integrated in Hive-trunk-h0.21 #2251 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2251/]) HIVE-4992 : add ability to skip javadoc during build (Sergey Shelukhin via Ashutosh h Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511374) * /hive/trunk/build.xml * /hive/trunk/hcatalog/build.xml add ability to skip javadoc during build Key: HIVE-4992 URL: https://issues.apache.org/jira/browse/HIVE-4992 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Trivial Fix For: 0.12.0 Attachments: HIVE-4992-addendum.patch, HIVE-4992.D11967.1.patch, HIVE-4992.D11967.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4930) Classes of metastore should not be included MR-task
[ https://issues.apache.org/jira/browse/HIVE-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733527#comment-13733527 ] Hudson commented on HIVE-4930: -- SUCCESS: Integrated in Hive-trunk-h0.21 #2251 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2251/]) HIVE-4930 : Classes of metastore should not be included MR-task (Navis via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511541) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ListBucketingCtx.java Classes of metastore should not be included MR-task --- Key: HIVE-4930 URL: https://issues.apache.org/jira/browse/HIVE-4930 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Navis Assignee: Navis Fix For: 0.12.0 Attachments: HIVE-4930.D11817.1.patch metastore.api.SkewedValueList is included in FileSinkOperator, which should not be. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4886) beeline code should have apache license headers
[ https://issues.apache.org/jira/browse/HIVE-4886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733526#comment-13733526 ] Hudson commented on HIVE-4886: -- SUCCESS: Integrated in Hive-trunk-h0.21 #2251 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2251/]) HIVE-4886 Beeline code should have Apache license headers. (Thejas M Nair via omalley) (omalley: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511485) * /hive/trunk/beeline/src/java/org/apache/hive/beeline/AbstractCommandHandler.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/AbstractOutputFormat.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLine.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLineCommandCompletor.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLineCompletor.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLineSignalHandler.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/BufferedRows.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/ColorBuffer.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/CommandHandler.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/Commands.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/DatabaseConnection.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/DatabaseConnections.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/DriverInfo.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/IncrementalRows.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/OutputFile.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/OutputFormat.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/ReflectiveCommandHandler.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/Reflector.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/Rows.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/SQLCompletor.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/SeparatedValuesOutputFormat.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/SunSignalHandler.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/TableNameCompletor.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/TableOutputFormat.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/VerticalOutputFormat.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/XMLAttributeOutputFormat.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/XMLElementOutputFormat.java beeline code should have apache license headers --- Key: HIVE-4886 URL: https://issues.apache.org/jira/browse/HIVE-4886 Project: Hive Issue Type: Task Components: JDBC Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.12.0 Attachments: HIVE-4886.2.patch, HIVE-4886.patch The beeline jdbc client added as part of hive server2 changes is based on SQLLine. As beeline is modified version of SQLLine and further modifications are also under apache license, the license headers of these files need to be replaced with apache license headers. We already have the license text of SQLLine in LICENSE file . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4967) Don't serialize unnecessary fields in query plan
[ https://issues.apache.org/jira/browse/HIVE-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733524#comment-13733524 ] Hudson commented on HIVE-4967: -- SUCCESS: Integrated in Hive-trunk-h0.21 #2251 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2251/]) HIVE-4967 : Don't serialize unnecessary fields in query plan (Ashutosh Chauhan. Reviewed by Brock Noland) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511377) * /hive/trunk/contrib/src/java/org/apache/hadoop/hive/contrib/genericudf/example/GenericUDFDBOutput.java * /hive/trunk/contrib/src/java/org/apache/hadoop/hive/contrib/udtf/example/GenericUDTFExplode2.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFEWAHBitmapBop.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFReflect.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFAverage.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFContextNGrams.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCorrelation.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovariance.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEWAHBitmap.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFHistogramNumeric.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFLeadLag.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMax.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMin.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFNTile.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVariance.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFArray.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFArrayContains.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseCompare.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCase.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCoalesce.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcatWS.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEWAHBitmapEmpty.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFElt.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFField.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFormatNumber.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFromUtcTimestamp.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFHash.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIf.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIn.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInstr.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLocate.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMap.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMapKeys.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMapValues.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFNvl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPAnd.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPNot.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPOr.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFPrintf.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect2.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSentences.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSize.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java *
[jira] [Commented] (HIVE-4987) Javadoc can generate argument list too long error
[ https://issues.apache.org/jira/browse/HIVE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733525#comment-13733525 ] Hudson commented on HIVE-4987: -- SUCCESS: Integrated in Hive-trunk-h0.21 #2251 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2251/]) HIVE-4987 : Javadoc can generate argument list too long error (Brock Noland via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511375) * /hive/trunk/build.xml * /hive/trunk/hcatalog/webhcat/svr/build.xml Javadoc can generate argument list too long error - Key: HIVE-4987 URL: https://issues.apache.org/jira/browse/HIVE-4987 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4987.patch We just to add to useexternalfile=yes to the javadoc statements. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4233) The TGT gotten from class 'CLIService' should be renewed on time
[ https://issues.apache.org/jira/browse/HIVE-4233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733539#comment-13733539 ] Hudson commented on HIVE-4233: -- FAILURE: Integrated in Hive-trunk-h0.21 #2252 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2252/]) HIVE-4233: The TGT gotten from class 'CLIService' should be renewed on time (Thejas M Nair via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511574) * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java * /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java * /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java * /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java The TGT gotten from class 'CLIService' should be renewed on time - Key: HIVE-4233 URL: https://issues.apache.org/jira/browse/HIVE-4233 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.11.0 Environment: CentOS release 6.3 (Final) jdk1.6.0_31 HiveServer2 0.10.0-cdh4.2.0 Kerberos Security Reporter: Dongyong Wang Assignee: Thejas M Nair Priority: Critical Fix For: 0.12.0 Attachments: 0001-FIX-HIVE-4233.patch, HIVE-4233-2.patch, HIVE-4233-3.patch, HIVE-4233.4.patch, HIVE-4233.5.patch When the HIveServer2 have started more than 7 days, I use beeline shell to connect the HiveServer2,all operation failed. The log of HiveServer2 shows it was caused by the Kerberos auth failure,the exception stack trace is: 2013-03-26 11:55:20,932 ERROR hive.ql.metadata.Hive: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1084) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(RetryingMetaStoreClient.java:51) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:61) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2140) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2151) at org.apache.hadoop.hive.ql.metadata.Hive.getDelegationToken(Hive.java:2275) at org.apache.hive.service.cli.CLIService.getDelegationTokenFromMetaStore(CLIService.java:358) at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:127) at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1073) at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1058) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge20S.java:565) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedConstructorAccessor52.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1082) ... 16 more Caused by: java.lang.IllegalStateException: This ticket is no longer valid at javax.security.auth.kerberos.KerberosTicket.toString(KerberosTicket.java:601) at java.lang.String.valueOf(String.java:2826) at java.lang.StringBuilder.append(StringBuilder.java:115) at sun.security.jgss.krb5.SubjectComber.findAux(SubjectComber.java:120) at sun.security.jgss.krb5.SubjectComber.find(SubjectComber.java:41) at sun.security.jgss.krb5.Krb5Util.getTicket(Krb5Util.java:130) at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:328) at java.security.AccessController.doPrivileged(Native Method) at sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:325) at
[jira] [Commented] (HIVE-3926) PPD on virtual column of partitioned table is not working
[
https://issues.apache.org/jira/browse/HIVE-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733540#comment-13733540
]
Hudson commented on HIVE-3926:
--
FAILURE: Integrated in Hive-trunk-h0.21 #2252 (See
[https://builds.apache.org/job/Hive-trunk-h0.21/2252/])
HIVE-3926 PPD on virtual column of partitioned table is not working (Navis Ryu
and Gunther Hagleitner via egc) (ecapriolo:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511578)
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/VirtualColumn.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractBucketJoinProc.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractSMBJoinProc.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/AvgPartitionSizeBasedBigTableSelectorForAutoSMJ.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GlobalLimitOptimizer.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GroupByOptimizer.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/TableSizeBasedBigTableSelectorForAutoSMJ.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/listbucketingpruner/LBPartitionProcFactory.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcCtx.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrOpProcFactory.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartExprEvalUtils.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java
* /hive/trunk/ql/src/test/queries/clientpositive/ppd_vc.q
* /hive/trunk/ql/src/test/results/clientpositive/ppd_vc.q.out
PPD on virtual column of partitioned table is not working
-
Key: HIVE-3926
URL: https://issues.apache.org/jira/browse/HIVE-3926
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
Fix For: 0.12.0
Attachments: HIVE-3926.6.patch, HIVE-3926.D8121.1.patch,
HIVE-3926.D8121.2.patch, HIVE-3926.D8121.3.patch, HIVE-3926.D8121.4.patch,
HIVE-3926.D8121.5.patch
{code}
select * from src where BLOCK__OFFSET__INSIDE__FILE100;
{code}
is working, but
{code}
select * from srcpart where BLOCK__OFFSET__INSIDE__FILE100;
{code}
throws SemanticException. Disabling PPD makes it work.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4863) Fix parallel order by on hadoop2
[
https://issues.apache.org/jira/browse/HIVE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733560#comment-13733560
]
Edward Capriolo commented on HIVE-4863:
---
So the case is this in 0.20
TotalOrderPartitioner.setPartitionFile(JobConf, partitionFile);
in 0.23
TotalOrderPartitioner.setPartitionFile(Configuration, partitionFile);
JobConf is a child of Configuration
{quote}
Also, in the 23 version you're setting the file on HiveConf not JobConf which
I don't think will work, will it?
{quote}
^ I think this will not matter since as long as the conf can find hdfs we
should be ready do add the file.
{quote}
The shim should do be able to do the exact same call in both cases - the
important thing is that we compile it separately against 20S and hadoop 23
{quote}
Good point. This shim stuff hurts my head :) I will rebase as you have
suggested.
Fix parallel order by on hadoop2
Key: HIVE-4863
URL: https://issues.apache.org/jira/browse/HIVE-4863
Project: Hive
Issue Type: Bug
Reporter: Edward Capriolo
Assignee: Edward Capriolo
Attachments: HIVE-4863.1.patch.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5026) HIVE-3926 is committed in the state of not rebased to trunk
[ https://issues.apache.org/jira/browse/HIVE-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733573#comment-13733573 ] Edward Capriolo commented on HIVE-5026: --- My bad. I thought I took the latest patch from jira. HIVE-3926 is committed in the state of not rebased to trunk --- Key: HIVE-5026 URL: https://issues.apache.org/jira/browse/HIVE-5026 Project: Hive Issue Type: Task Components: Tests Reporter: Navis Assignee: Navis Attachments: HIVE-5026.D12099.1.patch Current trunk build fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-5027) Upgrade Ivy to 2.3
Brock Noland created HIVE-5027: -- Summary: Upgrade Ivy to 2.3 Key: HIVE-5027 URL: https://issues.apache.org/jira/browse/HIVE-5027 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Ivy version 2.1 doesn't include classifiers when creating pom files. Therefore our generated pom's are not correct. Version 2.3 fixes this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5010) HCatalog maven integration doesn't override mvn.local.repo in two locations
[
https://issues.apache.org/jira/browse/HIVE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-5010:
---
Resolution: Fixed
Fix Version/s: 0.12.0
Status: Resolved (was: Patch Available)
Thanks for the review Ashutosh! I have committed this to trunk.
HCatalog maven integration doesn't override mvn.local.repo in two locations
---
Key: HIVE-5010
URL: https://issues.apache.org/jira/browse/HIVE-5010
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Brock Noland
Assignee: Brock Noland
Fix For: 0.12.0
Attachments: HIVE-5010.patch
Most locations in hcatalog/build-support/ant/deploy.xml specify:
{noformat}
localRepository path=${mvn.local.repo}/
{noformat}
but there are two locations which do not. This causes issues with ptest2
since those two locations end up using $HOME/.m2
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5027) Upgrade Ivy to 2.3
[ https://issues.apache.org/jira/browse/HIVE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-5027: --- Attachment: HIVE-5027.patch Trivial patch attached. Upgrade Ivy to 2.3 -- Key: HIVE-5027 URL: https://issues.apache.org/jira/browse/HIVE-5027 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Attachments: HIVE-5027.patch Ivy version 2.1 doesn't include classifiers when creating pom files. Therefore our generated pom's are not correct. Version 2.3 fixes this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5027) Upgrade Ivy to 2.3
[ https://issues.apache.org/jira/browse/HIVE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-5027: --- Status: Patch Available (was: Open) Upgrade Ivy to 2.3 -- Key: HIVE-5027 URL: https://issues.apache.org/jira/browse/HIVE-5027 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Attachments: HIVE-5027.patch Ivy version 2.1 doesn't include classifiers when creating pom files. Therefore our generated pom's are not correct. Version 2.3 fixes this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5027) Upgrade Ivy to 2.3
[ https://issues.apache.org/jira/browse/HIVE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733647#comment-13733647 ] Ashutosh Chauhan commented on HIVE-5027: +1 Upgrade Ivy to 2.3 -- Key: HIVE-5027 URL: https://issues.apache.org/jira/browse/HIVE-5027 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Attachments: HIVE-5027.patch Ivy version 2.1 doesn't include classifiers when creating pom files. Therefore our generated pom's are not correct. Version 2.3 fixes this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5016) Local mode FileNotFoundException: File does not exist
[ https://issues.apache.org/jira/browse/HIVE-5016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733660#comment-13733660 ] Abin Shahab commented on HIVE-5016: --- Root cause of this issue is Not picking the YarnRunner as the job runner when hive.exec.mode.local.auto=true mapreduce.framework.name gets set to 'local' instead of 'yarn'. This results in the LocalJobRunner being used as the JobRunner, and messes up the path creation. Local mode FileNotFoundException: File does not exist - Key: HIVE-5016 URL: https://issues.apache.org/jira/browse/HIVE-5016 Project: Hive Issue Type: Bug Affects Versions: 0.10.0 Environment: Centos 6.3 (final) Hadoop 2.0.2-alpha Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Hive libs: ls -1 lib/ antlr-2.7.7.jar antlr-runtime-3.0.1.jar avro-1.7.1.jar avro-mapred-1.7.1.jar commons-cli-1.2.jar commons-codec-1.4.jar commons-collections-3.2.1.jar commons-compress-1.4.1.jar commons-configuration-1.6.jar commons-dbcp-1.4.jar commons-lang-2.4.jar commons-logging-1.0.4.jar commons-logging-api-1.0.4.jar commons-pool-1.5.4.jar datanucleus-connectionpool-2.0.3.jar datanucleus-core-2.0.3.jar datanucleus-enhancer-2.0.3.jar datanucleus-rdbms-2.0.3.jar derby-10.4.2.0.jar guava-r09.jar hbase-0.92.0.jar hbase-0.92.0-tests.jar hive-builtins-0.10.0.jar hive-cli-0.10.0.jar hive-common-0.10.0.jar hive-contrib-0.10.0.jar hive-exec-0.10.0.jar hive-hbase-handler-0.10.0.jar hive-hwi-0.10.0.jar hive-hwi-0.10.0.war hive-jdbc-0.10.0.jar hive-metastore-0.10.0.jar hive-pdk-0.10.0.jar hive-serde-0.10.0.jar hive-service-0.10.0.jar hive-shims-0.10.0.jar jackson-core-asl-1.8.8.jar jackson-jaxrs-1.8.8.jar jackson-mapper-asl-1.8.8.jar jackson-xc-1.8.8.jar JavaEWAH-0.3.2.jar javolution-5.5.1.jar jdo2-api-2.3-ec.jar jetty-6.1.26.jar jetty-util-6.1.26.jar jline-0.9.94.jar json-20090211.jar libfb303-0.9.0.jar libthrift-0.9.0.jar log4j-1.2.16.jar php py servlet-api-2.5-20081211.jar slf4j-api-1.6.1.jar slf4j-log4j12-1.6.1.jar sqlline-1_0_2.jar stringtemplate-3.1-b1.jar xz-1.0.jar zookeeper-3.4.3.jar Reporter: Abin Shahab Priority: Critical Hive jobs in local mode fail with the error posted below. The jar file that's not being found exists and has the following access: ls -l hive-0.10.0/lib/hive-builtins-0.10.0.jar rw-rw-r-- 1 ashahab ashahab 3914 Dec 18 2012 hive-0.10.0/lib/hive-builtins-0.10.0.jar Steps to reproduce: hive set hive.exec.mode.local.auto=true; hive set hive.exec.mode.local.auto; hive.exec.mode.local.auto=true hive select count(*) from abin_test_table; Automatically selecting local only mode for query Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=number In order to limit the maximum number of reducers: set hive.exec.reducers.max=number In order to set a constant number of reducers: set mapred.reduce.tasks=number 13/08/06 21:37:11 WARN conf.Configuration: file:/tmp/ashahab/hive_2013-08-06_21-37-09_046_3263640403676309186/-local-10002/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 13/08/06 21:37:11 WARN conf.Configuration: file:/tmp/ashahab/hive_2013-08-06_21-37-09_046_3263640403676309186/-local-10002/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files. Execution log at: /tmp/ashahab/ashahab_20130806213737_7d26b796-5f55-44ca-a755-8898153d963b.log java.io.FileNotFoundException: File does not exist: /home/ashahab/dev/hive-0.10.0/lib/hive-builtins-0.10.0.jar at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:782) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:208) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:71) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:252) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:290) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:361) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215) at java.security.AccessController.doPrivileged(Native Method) at
[jira] [Commented] (HIVE-5026) HIVE-3926 is committed in the state of not rebased to trunk
[ https://issues.apache.org/jira/browse/HIVE-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733664#comment-13733664 ] Ashutosh Chauhan commented on HIVE-5026: [~appodictic] Can you take a look and get this in asap? Trunk is not compiling at the moment, leaving other devs stranded. HIVE-3926 is committed in the state of not rebased to trunk --- Key: HIVE-5026 URL: https://issues.apache.org/jira/browse/HIVE-5026 Project: Hive Issue Type: Task Components: Tests Reporter: Navis Assignee: Navis Attachments: HIVE-5026.D12099.1.patch Current trunk build fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5010) HCatalog maven integration doesn't override mvn.local.repo in two locations
[
https://issues.apache.org/jira/browse/HIVE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733674#comment-13733674
]
Hudson commented on HIVE-5010:
--
FAILURE: Integrated in Hive-trunk-hadoop2 #343 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2/343/])
HIVE-5010: HCatalog maven integration doesn't override mvn.local.repo in two
locations (brock:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511847)
* /hive/trunk/hcatalog/build-support/ant/deploy.xml
HCatalog maven integration doesn't override mvn.local.repo in two locations
---
Key: HIVE-5010
URL: https://issues.apache.org/jira/browse/HIVE-5010
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Brock Noland
Assignee: Brock Noland
Fix For: 0.12.0
Attachments: HIVE-5010.patch
Most locations in hcatalog/build-support/ant/deploy.xml specify:
{noformat}
localRepository path=${mvn.local.repo}/
{noformat}
but there are two locations which do not. This causes issues with ptest2
since those two locations end up using $HOME/.m2
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5010) HCatalog maven integration doesn't override mvn.local.repo in two locations
[
https://issues.apache.org/jira/browse/HIVE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733705#comment-13733705
]
Hudson commented on HIVE-5010:
--
FAILURE: Integrated in Hive-trunk-h0.21 #2254 (See
[https://builds.apache.org/job/Hive-trunk-h0.21/2254/])
HIVE-5010: HCatalog maven integration doesn't override mvn.local.repo in two
locations (brock:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511847)
* /hive/trunk/hcatalog/build-support/ant/deploy.xml
HCatalog maven integration doesn't override mvn.local.repo in two locations
---
Key: HIVE-5010
URL: https://issues.apache.org/jira/browse/HIVE-5010
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Brock Noland
Assignee: Brock Noland
Fix For: 0.12.0
Attachments: HIVE-5010.patch
Most locations in hcatalog/build-support/ant/deploy.xml specify:
{noformat}
localRepository path=${mvn.local.repo}/
{noformat}
but there are two locations which do not. This causes issues with ptest2
since those two locations end up using $HOME/.m2
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[ https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733747#comment-13733747 ] Sergey Shelukhin commented on HIVE-4914: Entire test suite passed, which is suspicious. Let me double check. Preliminarily, the patch is ready for review. filtering via partition name should be done inside metastore server (implementation) Key: HIVE-4914 URL: https://issues.apache.org/jira/browse/HIVE-4914 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch, HIVE-4914.patch Currently, if the filter pushdown is impossible (which is most cases), the client gets all partition names from metastore, filters them, and asks for partitions by names for the filtered set. Metastore server code should do that instead; it should check if pushdown is possible and do it if so; otherwise it should do name-based filtering. Saves the roundtrip with all partition names from the server to client, and also removes the need to have pushdown viability checking on both sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5026) HIVE-3926 is committed in the state of not rebased to trunk
[ https://issues.apache.org/jira/browse/HIVE-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733769#comment-13733769 ] Edward Capriolo commented on HIVE-5026: --- I do not have time to review this now. If no one else wants to +1 and commit this patch, I suggest rolling back and re-opening HIVE-3926 . I will not be able to get to this until about 7:00 PM eastern tonight. HIVE-3926 is committed in the state of not rebased to trunk --- Key: HIVE-5026 URL: https://issues.apache.org/jira/browse/HIVE-5026 Project: Hive Issue Type: Task Components: Tests Reporter: Navis Assignee: Navis Attachments: HIVE-5026.D12099.1.patch Current trunk build fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4545) HS2 should return describe table results without space padding
[
https://issues.apache.org/jira/browse/HIVE-4545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733775#comment-13733775
]
Hive QA commented on HIVE-4545:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12596693/HIVE-4545.3.patch
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/332/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/332/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests failed with: NonZeroExitCodeException: Command 'bash
/data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-332/source-prep.txt
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
svn: Working copy '.' locked
svn: run 'svn cleanup' to remove locks (type 'svn help cleanup' for details)
+ exit 1
'
{noformat}
This message is automatically generated.
HS2 should return describe table results without space padding
--
Key: HIVE-4545
URL: https://issues.apache.org/jira/browse/HIVE-4545
Project: Hive
Issue Type: Bug
Components: HiveServer2
Reporter: Thejas M Nair
Assignee: Thejas M Nair
Attachments: HIVE-4545-1.patch, HIVE-4545.2.patch, HIVE-4545.3.patch
HIVE-3140 changed behavior of 'DESCRIBE table;' to be like 'DESCRIBE
FORMATTED table;'. HIVE-3140 introduced changes to not print header in
'DESCRIBE table;'. But jdbc/odbc calls still get fields padded with space for
the 'DESCRIBE table;' query.
As the jdbc/odbc results are not for direct human consumption the space
padding should not be done for hive server2.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4545) HS2 should return describe table results without space padding
[ https://issues.apache.org/jira/browse/HIVE-4545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733780#comment-13733780 ] Brock Noland commented on HIVE-4545: The build process was stuck on a build and it looks like I killed this one by accident. I'll kick off a build for this again. HS2 should return describe table results without space padding -- Key: HIVE-4545 URL: https://issues.apache.org/jira/browse/HIVE-4545 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-4545-1.patch, HIVE-4545.2.patch, HIVE-4545.3.patch HIVE-3140 changed behavior of 'DESCRIBE table;' to be like 'DESCRIBE FORMATTED table;'. HIVE-3140 introduced changes to not print header in 'DESCRIBE table;'. But jdbc/odbc calls still get fields padded with space for the 'DESCRIBE table;' query. As the jdbc/odbc results are not for direct human consumption the space padding should not be done for hive server2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2742) InvalidOperationException alter table is not possible when using LOAD DATA INPATH OVERWRITE with database and partition
[
https://issues.apache.org/jira/browse/HIVE-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733798#comment-13733798
]
Pierre-Luc Bertrand commented on HIVE-2742:
---
I can't run USE [mySchema]; ALTER TABLE [myTable] ...
with Hive JDBC so there is no way of doing alter tables on tables within
databases.
InvalidOperationException alter table is not possible when using LOAD DATA
INPATH OVERWRITE with database and partition
-
Key: HIVE-2742
URL: https://issues.apache.org/jira/browse/HIVE-2742
Project: Hive
Issue Type: Bug
Components: Database/Schema, Metastore, Query Processor
Affects Versions: 0.7.1
Environment: reproduced on cdh3u2 (haven't tried other versions)
Reporter: Maxime Brugidou
Here is a repeatable procedure:
{code}
$ echo test | hadoop fs -put - test.txt
$ echo test2 | hadoop fs -put - test2.txt
{code}
Then in hive:
{code}
create database catalog;
use catalog;
create table test_load (t string) partitioned by (p string);
use default;
{code}
Then the problem arises:
{code}
load data inpath 'test.txt' overwrite into table catalog.test_load
partition (p='test');
Loading data to table catalog.test_load partition (p=test)
OK
Time taken: 0.175 seconds
load data inpath 'test2.txt' overwrite into table catalog.test_load
partition (p='test');
Loading data to table catalog.test_load partition (p=test)
Moved to trash:
hdfs://mycluster/user/hive/warehouse/catalog.db/test_load/p=test
Failed with exception InvalidOperationException(message:alter is not possible)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5026) HIVE-3926 is committed in the state of not rebased to trunk
[ https://issues.apache.org/jira/browse/HIVE-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733840#comment-13733840 ] Gunther Hagleitner commented on HIVE-5026: -- I am reviewing this right now. HIVE-3926 is committed in the state of not rebased to trunk --- Key: HIVE-5026 URL: https://issues.apache.org/jira/browse/HIVE-5026 Project: Hive Issue Type: Task Components: Tests Reporter: Navis Assignee: Navis Attachments: HIVE-5026.D12099.1.patch Current trunk build fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5023) Hive get wrong result when partition has the same path but different schema or authority
[
https://issues.apache.org/jira/browse/HIVE-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733859#comment-13733859
]
Hive QA commented on HIVE-5023:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12596740/HIVE-5023.1.patch
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/338/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/338/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests failed with: NonZeroExitCodeException: Command 'bash
/data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-338/source-prep.txt
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted
'hcatalog/core/src/test/java/org/apache/hcatalog/mapreduce/HCatMapReduceTest.java'
Reverted
'hcatalog/core/src/main/java/org/apache/hcatalog/mapreduce/FosterStorageHandler.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf build hcatalog/build
hcatalog/core/src/test/java/org/apache/hcatalog/mapreduce/TestHCatExternalHCatNonPartitioned.java
hcatalog/core/src/test/java/org/apache/hcatalog/mapreduce/TestHCatExternalPartitioned.java
hcatalog/core/src/test/java/org/apache/hcatalog/mapreduce/TestHCatExternalDynamicPartitioned.java
common/src/gen
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1511965.
At revision 1511965.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0 to p2
+ exit 1
'
{noformat}
This message is automatically generated.
Hive get wrong result when partition has the same path but different schema
or authority
Key: HIVE-5023
URL: https://issues.apache.org/jira/browse/HIVE-5023
Project: Hive
Issue Type: Bug
Reporter: Shuaishuai Nie
Assignee: Shuaishuai Nie
Attachments: HIVE-5023.1.patch
Hive does not differentiate scheme and authority in file uris which cause
wrong result when partition has the same path but different schema or
authority. Here is a simple repro
partition file path:
asv://contain...@secondary1.blob.core.windows.net/2013-08-05/00/text1.txt
with content 2013-08-05 00:00:00
asv://contain...@secondary1.blob.core.windows.net/2013-08-05/00/text2.txt
with content 2013-08-05 00:00:20
{noformat}
CREATE EXTERNAL TABLE IF NOT EXISTS T1 (t STRING) PARTITIONED BY (ProcessDate
STRING, Hour STRING, ClusterName STRING) ROW FORMAT DELIMITED FIELDS
TERMINATED by '\t' STORED AS TEXTFILE;
ALTER TABLE T1 DROP IF EXISTS PARTITION(processDate='2013-08-05', Hour='00',
clusterName ='CLusterA');
ALTER TABLE T1 ADD IF NOT EXISTS PARTITION(processDate='2013-08-05',
Hour='00', clusterName ='ClusterA') LOCATION
'asv://contain...@secondary1.blob.core.windows.net/2013-08-05/00';
ALTER TABLE T1 DROP IF EXISTS PARTITION(processDate='2013-08-05', Hour='00',
clusterName ='ClusterB');
ALTER TABLE T1 ADD IF NOT EXISTS PARTITION(processDate='2013-08-05',
Hour='00', clusterName ='ClusterB') LOCATION
'asv://contain...@secondary1.blob.core.windows.net/2013-08-05/00';
{noformat}
the expect output of the hive query
{noformat}
SELECT ClusterName, t FROM T1 WHERE ProcessDate=’2013-08-05’ AND Hour=’00’;
{noformat}
should be
{noformat}
ClusterA2013-08-05 00:00:00
ClusterB2013-08-05 00:00:20
{noformat}
However it is
{noformat}
ClusterA2013-08-05 00:00:00
ClusterA2013-08-05 00:00:20
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5026) HIVE-3926 is committed in the state of not rebased to trunk
[ https://issues.apache.org/jira/browse/HIVE-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733862#comment-13733862 ] Gunther Hagleitner commented on HIVE-5026: -- Patch looks good. Running a few local tests. If they pass I will commit right away since the build is broken already. If anyone disagrees with this, let me know and I will roll back instead. HIVE-3926 is committed in the state of not rebased to trunk --- Key: HIVE-5026 URL: https://issues.apache.org/jira/browse/HIVE-5026 Project: Hive Issue Type: Task Components: Tests Reporter: Navis Assignee: Navis Attachments: HIVE-5026.D12099.1.patch Current trunk build fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5026) HIVE-3926 is committed in the state of not rebased to trunk
[ https://issues.apache.org/jira/browse/HIVE-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-5026: - Priority: Blocker (was: Major) HIVE-3926 is committed in the state of not rebased to trunk --- Key: HIVE-5026 URL: https://issues.apache.org/jira/browse/HIVE-5026 Project: Hive Issue Type: Task Components: Tests Reporter: Navis Assignee: Navis Priority: Blocker Attachments: HIVE-5026.D12099.1.patch Current trunk build fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5026) HIVE-3926 is committed in the state of not rebased to trunk
[ https://issues.apache.org/jira/browse/HIVE-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733863#comment-13733863 ] Phabricator commented on HIVE-5026: --- hagleitn has accepted the revision HIVE-5026 [jira] HIVE-3926 is committed in the state of not rebased to trunk. LGTM +1 REVISION DETAIL https://reviews.facebook.net/D12099 BRANCH HIVE-5026 ARCANIST PROJECT hive To: JIRA, hagleitn, navis HIVE-3926 is committed in the state of not rebased to trunk --- Key: HIVE-5026 URL: https://issues.apache.org/jira/browse/HIVE-5026 Project: Hive Issue Type: Task Components: Tests Reporter: Navis Assignee: Navis Attachments: HIVE-5026.D12099.1.patch Current trunk build fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5026) HIVE-3926 is committed in the state of not rebased to trunk
[ https://issues.apache.org/jira/browse/HIVE-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-5026: - Issue Type: Bug (was: Task) HIVE-3926 is committed in the state of not rebased to trunk --- Key: HIVE-5026 URL: https://issues.apache.org/jira/browse/HIVE-5026 Project: Hive Issue Type: Bug Components: Tests Reporter: Navis Assignee: Navis Priority: Blocker Attachments: HIVE-5026.D12099.1.patch Current trunk build fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[
https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733866#comment-13733866
]
Hive QA commented on HIVE-4914:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12596754/HIVE-4914.patch
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/339/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/339/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests failed with: NonZeroExitCodeException: Command 'bash
/data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-339/source-prep.txt
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1511968.
At revision 1511968.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0 to p2
+ exit 1
'
{noformat}
This message is automatically generated.
filtering via partition name should be done inside metastore server
(implementation)
Key: HIVE-4914
URL: https://issues.apache.org/jira/browse/HIVE-4914
Project: Hive
Issue Type: Improvement
Components: Metastore
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch,
HIVE-4914.patch
Currently, if the filter pushdown is impossible (which is most cases), the
client gets all partition names from metastore, filters them, and asks for
partitions by names for the filtered set.
Metastore server code should do that instead; it should check if pushdown is
possible and do it if so; otherwise it should do name-based filtering.
Saves the roundtrip with all partition names from the server to client, and
also removes the need to have pushdown viability checking on both sides.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4913) Put deterministic ordering in the top-K ngrams output of UDF context_ngrams()
[ https://issues.apache.org/jira/browse/HIVE-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-4913: -- Attachment: HIVE-4913.patch Patch is updated to address the comments above. Put deterministic ordering in the top-K ngrams output of UDF context_ngrams() - Key: HIVE-4913 URL: https://issues.apache.org/jira/browse/HIVE-4913 Project: Hive Issue Type: Improvement Components: UDF Affects Versions: 0.10.0, 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4913.patch, HIVE-4913.patch Currently UDF context_ngrams() output top-K ngrams in the order of descending frequency. When there are ties, i.e., phrases with same ngram value, the order is indeterministic. Tie breaker is needed to break the ties, so that the output is deterministic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5026) HIVE-3926 is committed in the state of not rebased to trunk
[ https://issues.apache.org/jira/browse/HIVE-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-5026: - Resolution: Fixed Fix Version/s: 0.12.0 Status: Resolved (was: Patch Available) HIVE-3926 is committed in the state of not rebased to trunk --- Key: HIVE-5026 URL: https://issues.apache.org/jira/browse/HIVE-5026 Project: Hive Issue Type: Bug Components: Tests Reporter: Navis Assignee: Navis Priority: Blocker Fix For: 0.12.0 Attachments: HIVE-5026.D12099.1.patch Current trunk build fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5026) HIVE-3926 is committed in the state of not rebased to trunk
[ https://issues.apache.org/jira/browse/HIVE-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733876#comment-13733876 ] Gunther Hagleitner commented on HIVE-5026: -- Committed to trunk. Thanks, Navis! HIVE-3926 is committed in the state of not rebased to trunk --- Key: HIVE-5026 URL: https://issues.apache.org/jira/browse/HIVE-5026 Project: Hive Issue Type: Bug Components: Tests Reporter: Navis Assignee: Navis Priority: Blocker Attachments: HIVE-5026.D12099.1.patch Current trunk build fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5001) [WebHCat] JobState is read/written with different user credentials
[ https://issues.apache.org/jira/browse/HIVE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-5001: - Description: JobState can be persisted to HDFS or Zookeeper. At various points in the lifecycle it's accessed with different user credentials thus may cause errors depending on how permissions are set. Example: When submitting a MR job, templeton.JarDelegator is used. It calls LauncherDelegator#queueAsUser() which runs TempletonControllerJob with UserGroupInformation.doAs(). TempletonControllerJob will in turn create JobState and persist it. LauncherDelegator.registerJob() also modifies JobState but w/o doing a doAs() So in the later case it's possible that the persisted state of JobState by a different user than one that created/owns the file. templeton.tool.HDFSCleanup tries to delete these files w/o doAs. 'childid' file, for example, is created with rw-r--r--. and it's parent directory (job_201308051224_0001) has rwxr-xr-x. HDFSStorage doesn't set file permissions explicitly so it must be using default permissions. So there is a potential issue here (depending on UMASK) especially once HIVE-4601 is addressed. Actually, even w/o HIVE-4601 the user that owns the WebHCat process is likely different than the one submitting a request. The default for templeton.storage.class is org.apache.hcatalog.templeton.toolHDFSStorage, but it's likely that most production environments change it to Zookeeper, which may explain why this issue is not commonly seen. was: JobState can be persisted to HDFS or Zookeeper. At various points in the lifecycle it's accessed with different user credentials thus may cause errors depending on how permissions are set. Example: When submitting a MR job, templeton.JarDelegator is used. It calls LauncherDelegator#queueAsUser() which runs TempletonControllerJob with UserGroupInformation.doAs(). TempletonControllerJob will in turn create JobState and persist it. LauncherDelegator.registerJob() also modifies JobState but w/o doing a doAs() So in the later case it's possible that the persisted state of JobState by a different user than one that created/owns the file. templeton.tool.HDFSCleanup tries to delete these files w/o doAs. 'childid' file, for example, is created with rw-r--r--. and it's parent directory (job_201308051224_0001) has rwxr-xr-x. HDFSStorage doesn't set file permissions explicitly so it must be using default permissions. So there is a potential issue here (depending on UMASK) especially once HIVE-4601 is addressed. Actually, even w/o HIVE-4601 the user that owns the WebHCat process is likely different than the one submitting a request. [WebHCat] JobState is read/written with different user credentials -- Key: HIVE-5001 URL: https://issues.apache.org/jira/browse/HIVE-5001 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.11.0 Reporter: Eugene Koifman Assignee: Eugene Koifman JobState can be persisted to HDFS or Zookeeper. At various points in the lifecycle it's accessed with different user credentials thus may cause errors depending on how permissions are set. Example: When submitting a MR job, templeton.JarDelegator is used. It calls LauncherDelegator#queueAsUser() which runs TempletonControllerJob with UserGroupInformation.doAs(). TempletonControllerJob will in turn create JobState and persist it. LauncherDelegator.registerJob() also modifies JobState but w/o doing a doAs() So in the later case it's possible that the persisted state of JobState by a different user than one that created/owns the file. templeton.tool.HDFSCleanup tries to delete these files w/o doAs. 'childid' file, for example, is created with rw-r--r--. and it's parent directory (job_201308051224_0001) has rwxr-xr-x. HDFSStorage doesn't set file permissions explicitly so it must be using default permissions. So there is a potential issue here (depending on UMASK) especially once HIVE-4601 is addressed. Actually, even w/o HIVE-4601 the user that owns the WebHCat process is likely different than the one submitting a request. The default for templeton.storage.class is org.apache.hcatalog.templeton.toolHDFSStorage, but it's likely that most production environments change it to Zookeeper, which may explain why this issue is not commonly seen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4881) hive local mode: java.io.FileNotFoundException: emptyFile
[
https://issues.apache.org/jira/browse/HIVE-4881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733895#comment-13733895
]
Abin Shahab commented on HIVE-4881:
---
I find that this bug only happens when you have an empty table.
hive local mode: java.io.FileNotFoundException: emptyFile
-
Key: HIVE-4881
URL: https://issues.apache.org/jira/browse/HIVE-4881
Project: Hive
Issue Type: Bug
Environment: hive 0.9.0+158-1.cdh4.1.3.p0.23~squeeze-cdh4.1.3
Reporter: Bartosz Cisek
Priority: Critical
Our hive jobs fail due to strange error pasted below. Strace showed that
process created this file, accessed it a few times and then it throwed
exception that it couldn't find file it just accessed. In next step it
unliked it. Yay.
Very similar problem was reported [in already closed
task|https://issues.apache.org/jira/browse/HIVE-1633?focusedCommentId=13598983page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13598983]
or left unresolved on [mailing
lists|http://mail-archives.apache.org/mod_mbox/hive-user/201307.mbox/%3c94f02eb368b740ebbcd94df4d5d1d...@amxpr03mb054.eurprd03.prod.outlook.com%3E].
I'll be happy to provide required additional details.
{code:title=Stack trace}
2013-07-18 12:49:46,109 ERROR security.UserGroupInformation
(UserGroupInformation.java:doAs(1335)) - PriviledgedActionException
as:username (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not
exist:
/tmp/username/hive_2013-07-18_12-49-45_218_605775464480014480/-mr-1/1/emptyFile
2013-07-18 12:49:46,113 ERROR exec.ExecDriver
(SessionState.java:printError(403)) - Job Submission failed with exception
'java.io.FileNotFoundException(File does not exist:
/tmp/username/hive_2013-07-18_12-49-45_218_605775464480014480/-mr-1/1/emptyFile)'
java.io.FileNotFoundException: File does not exist:
/tmp/username/hive_2013-07-18_12-49-45_218_605775464480014480/-mr-1/1/emptyFile
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:787)
at
org.apache.hadoop.mapred.lib.CombineFileInputFormat$OneFileInfo.init(CombineFileInputFormat.java:462)
at
org.apache.hadoop.mapred.lib.CombineFileInputFormat.getMoreSplits(CombineFileInputFormat.java:256)
at
org.apache.hadoop.mapred.lib.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:212)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getSplits(HadoopShimsSecure.java:392)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getSplits(HadoopShimsSecure.java:358)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:387)
at
org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:1040)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1032)
at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:172)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:895)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:895)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:869)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:435)
at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:677)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
{code}
{code:title=strace with grep emptyFile}
7385 14:48:02.808096
stat(/tmp/username/hive_2013-07-18_14-48-00_700_8005967322498387476/-mr-1/1/emptyFile,
{st_mode=S_IFREG|0755, st_size=0, ...}) = 0
7385 14:48:02.808201
stat(/tmp/username/hive_2013-07-18_14-48-00_700_8005967322498387476/-mr-1/1/emptyFile,
{st_mode=S_IFREG|0755, st_size=0, ...}) = 0
7385 14:48:02.808277
stat(/tmp/username/hive_2013-07-18_14-48-00_700_8005967322498387476/-mr-1/1/emptyFile,
{st_mode=S_IFREG|0755, st_size=0, ...}) = 0
7385 14:48:02.808348
stat(/tmp/username/hive_2013-07-18_14-48-00_700_8005967322498387476/-mr-1/1/emptyFile,
{st_mode=S_IFREG|0755,
[jira] [Commented] (HIVE-3926) PPD on virtual column of partitioned table is not working
[
https://issues.apache.org/jira/browse/HIVE-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733893#comment-13733893
]
Hudson commented on HIVE-3926:
--
FAILURE: Integrated in Hive-trunk-hadoop2 #344 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2/344/])
HIVE-5026: HIVE-3926 is committed in the state of not rebased to trunk (Navis
Ryu via Gunther Hagleitner) (gunther:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511977)
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java
PPD on virtual column of partitioned table is not working
-
Key: HIVE-3926
URL: https://issues.apache.org/jira/browse/HIVE-3926
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
Fix For: 0.12.0
Attachments: HIVE-3926.6.patch, HIVE-3926.D8121.1.patch,
HIVE-3926.D8121.2.patch, HIVE-3926.D8121.3.patch, HIVE-3926.D8121.4.patch,
HIVE-3926.D8121.5.patch
{code}
select * from src where BLOCK__OFFSET__INSIDE__FILE100;
{code}
is working, but
{code}
select * from srcpart where BLOCK__OFFSET__INSIDE__FILE100;
{code}
throws SemanticException. Disabling PPD makes it work.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5026) HIVE-3926 is committed in the state of not rebased to trunk
[ https://issues.apache.org/jira/browse/HIVE-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733892#comment-13733892 ] Hudson commented on HIVE-5026: -- FAILURE: Integrated in Hive-trunk-hadoop2 #344 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/344/]) HIVE-5026: HIVE-3926 is committed in the state of not rebased to trunk (Navis Ryu via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511977) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java HIVE-3926 is committed in the state of not rebased to trunk --- Key: HIVE-5026 URL: https://issues.apache.org/jira/browse/HIVE-5026 Project: Hive Issue Type: Bug Components: Tests Reporter: Navis Assignee: Navis Priority: Blocker Fix For: 0.12.0 Attachments: HIVE-5026.D12099.1.patch Current trunk build fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3191) timestamp - timestamp causes null pointer exception
[ https://issues.apache.org/jira/browse/HIVE-3191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-3191: - Attachment: HIVE-3191.3.patch timestamp - timestamp causes null pointer exception --- Key: HIVE-3191 URL: https://issues.apache.org/jira/browse/HIVE-3191 Project: Hive Issue Type: Bug Affects Versions: 0.8.0 Reporter: N Campbell Assignee: Jason Dere Attachments: HIVE-3191.1.patch, HIVE-3191.2.patch, HIVE-3191.3.patch select tts.rnum, tts.cts - tts.cts from cert.tts tts Error: Query returned non-zero code: 12, cause: FAILED: Hive Internal Error: java.lang.NullPointerException(null) SQLState: 42000 ErrorCode: 12 create table if not exists CERT.TTS ( RNUM int , CTS timestamp) stored as sequencefile; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3191) timestamp - timestamp causes null pointer exception
[ https://issues.apache.org/jira/browse/HIVE-3191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733906#comment-13733906 ] Gunther Hagleitner commented on HIVE-3191: -- Reuploading .2 as .3 to retrigger tests (failed because of unrelated build issue) timestamp - timestamp causes null pointer exception --- Key: HIVE-3191 URL: https://issues.apache.org/jira/browse/HIVE-3191 Project: Hive Issue Type: Bug Affects Versions: 0.8.0 Reporter: N Campbell Assignee: Jason Dere Attachments: HIVE-3191.1.patch, HIVE-3191.2.patch, HIVE-3191.3.patch select tts.rnum, tts.cts - tts.cts from cert.tts tts Error: Query returned non-zero code: 12, cause: FAILED: Hive Internal Error: java.lang.NullPointerException(null) SQLState: 42000 ErrorCode: 12 create table if not exists CERT.TTS ( RNUM int , CTS timestamp) stored as sequencefile; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4911) Enable QOP configuration for Hive Server 2 thrift transport
[ https://issues.apache.org/jira/browse/HIVE-4911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-4911: --- Resolution: Fixed Fix Version/s: 0.12.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Arup! Enable QOP configuration for Hive Server 2 thrift transport --- Key: HIVE-4911 URL: https://issues.apache.org/jira/browse/HIVE-4911 Project: Hive Issue Type: New Feature Reporter: Arup Malakar Assignee: Arup Malakar Fix For: 0.12.0 Attachments: 20-build-temp-change-1.patch, 20-build-temp-change.patch, HIVE-4911-trunk-0.patch, HIVE-4911-trunk-1.patch, HIVE-4911-trunk-2.patch, HIVE-4911-trunk-3.patch The QoP for hive server 2 should be configurable to enable encryption. A new configuration should be exposed hive.server2.thrift.rpc.protection. This would give greater control configuring hive server 2 service. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4992) add ability to skip javadoc during build
[ https://issues.apache.org/jira/browse/HIVE-4992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733915#comment-13733915 ] Ashutosh Chauhan commented on HIVE-4992: +1 for addendum patch. Ran ant package -Dskip.javadoc=true to verify there is no more javadoc generation with flag on. add ability to skip javadoc during build Key: HIVE-4992 URL: https://issues.apache.org/jira/browse/HIVE-4992 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Trivial Fix For: 0.12.0 Attachments: HIVE-4992-addendum.patch, HIVE-4992.D11967.1.patch, HIVE-4992.D11967.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4913) Put deterministic ordering in the top-K ngrams output of UDF context_ngrams()
[ https://issues.apache.org/jira/browse/HIVE-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733916#comment-13733916 ] Ashutosh Chauhan commented on HIVE-4913: +1 Put deterministic ordering in the top-K ngrams output of UDF context_ngrams() - Key: HIVE-4913 URL: https://issues.apache.org/jira/browse/HIVE-4913 Project: Hive Issue Type: Improvement Components: UDF Affects Versions: 0.10.0, 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4913.patch, HIVE-4913.patch Currently UDF context_ngrams() output top-K ngrams in the order of descending frequency. When there are ties, i.e., phrases with same ngram value, the order is indeterministic. Tie breaker is needed to break the ties, so that the output is deterministic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4985) refactor/clean up partition name pruning to be usable inside metastore server
[ https://issues.apache.org/jira/browse/HIVE-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733930#comment-13733930 ] Ashutosh Chauhan commented on HIVE-4985: I am still reviewing the patch. But, one high level comment I have is we should be using SetPartition instead of ListPartition everywhere. refactor/clean up partition name pruning to be usable inside metastore server -- Key: HIVE-4985 URL: https://issues.apache.org/jira/browse/HIVE-4985 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-4985.D11961.1.patch, HIVE-4985.D11961.2.patch Preliminary for HIVE-4914. The patch is going to be large already, so some refactoring and dead code removal that is non-controversial can be done in advance in a separate patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4911) Enable QOP configuration for Hive Server 2 thrift transport
[ https://issues.apache.org/jira/browse/HIVE-4911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733931#comment-13733931 ] Arup Malakar commented on HIVE-4911: Thanks [~ashutoshc]. Enable QOP configuration for Hive Server 2 thrift transport --- Key: HIVE-4911 URL: https://issues.apache.org/jira/browse/HIVE-4911 Project: Hive Issue Type: New Feature Reporter: Arup Malakar Assignee: Arup Malakar Fix For: 0.12.0 Attachments: 20-build-temp-change-1.patch, 20-build-temp-change.patch, HIVE-4911-trunk-0.patch, HIVE-4911-trunk-1.patch, HIVE-4911-trunk-2.patch, HIVE-4911-trunk-3.patch The QoP for hive server 2 should be configurable to enable encryption. A new configuration should be exposed hive.server2.thrift.rpc.protection. This would give greater control configuring hive server 2 service. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-5028) Some tests with fail OutOfMemoryError PermGen Space on Hadoop2
Brock Noland created HIVE-5028:
--
Summary: Some tests with fail OutOfMemoryError PermGen Space on
Hadoop2
Key: HIVE-5028
URL: https://issues.apache.org/jira/browse/HIVE-5028
Project: Hive
Issue Type: Sub-task
Reporter: Brock Noland
Assignee: Brock Noland
{noformat}
java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:181)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:430)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:349)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:978)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:890)
at org.apache.hcatalog.cli.HCatDriver.run(HCatDriver.java:43)
at
org.apache.hcatalog.hbase.TestHBaseBulkOutputFormat.bulkModeAbortTest(TestHBaseBulkOutputFormat.java:540)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4911) Enable QOP configuration for Hive Server 2 thrift transport
[ https://issues.apache.org/jira/browse/HIVE-4911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733966#comment-13733966 ] Hudson commented on HIVE-4911: -- FAILURE: Integrated in Hive-trunk-hadoop2 #345 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/345/]) HIVE-4911 : Enable QOP configuration for Hive Server 2 thrift transport (Arup Malakar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512010) * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/conf/hive-default.xml.template * /hive/trunk/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java * /hive/trunk/service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java * /hive/trunk/service/src/java/org/apache/hive/service/auth/KerberosSaslHelper.java * /hive/trunk/service/src/java/org/apache/hive/service/auth/SaslQOP.java * /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge20S.java * /hive/trunk/shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java * /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge.java Enable QOP configuration for Hive Server 2 thrift transport --- Key: HIVE-4911 URL: https://issues.apache.org/jira/browse/HIVE-4911 Project: Hive Issue Type: New Feature Reporter: Arup Malakar Assignee: Arup Malakar Fix For: 0.12.0 Attachments: 20-build-temp-change-1.patch, 20-build-temp-change.patch, HIVE-4911-trunk-0.patch, HIVE-4911-trunk-1.patch, HIVE-4911-trunk-2.patch, HIVE-4911-trunk-3.patch The QoP for hive server 2 should be configurable to enable encryption. A new configuration should be exposed hive.server2.thrift.rpc.protection. This would give greater control configuring hive server 2 service. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1545) Add a bunch of UDFs and UDAFs
[
https://issues.apache.org/jira/browse/HIVE-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733976#comment-13733976
]
Brenden Matthews commented on HIVE-1545:
Where's the rest of the source?
Add a bunch of UDFs and UDAFs
-
Key: HIVE-1545
URL: https://issues.apache.org/jira/browse/HIVE-1545
Project: Hive
Issue Type: New Feature
Components: UDF
Reporter: Jonathan Chang
Assignee: Jonathan Chang
Priority: Minor
Attachments: core.tar.gz, ext.tar.gz, UDFEndsWith.java,
UDFFindInString.java, UDFLtrim.java, UDFRtrim.java, udfs.tar.gz, udfs.tar.gz,
UDFStartsWith.java, UDFTrim.java
Here some UD(A)Fs which can be incorporated into the Hive distribution:
UDFArgMax - Find the 0-indexed index of the largest argument. e.g., ARGMAX(4,
5, 3) returns 1.
UDFBucket - Find the bucket in which the first argument belongs. e.g.,
BUCKET(x, b_1, b_2, b_3, ...), will return the smallest i such that x b_{i}
but = b_{i+1}. Returns 0 if x is smaller than all the buckets.
UDFFindInArray - Finds the 1-index of the first element in the array given as
the second argument. Returns 0 if not found. Returns NULL if either argument
is NULL. E.g., FIND_IN_ARRAY(5, array(1,2,5)) will return 3. FIND_IN_ARRAY(5,
array(1,2,3)) will return 0.
UDFGreatCircleDist - Finds the great circle distance (in km) between two
lat/long coordinates (in degrees).
UDFLDA - Performs LDA inference on a vector given fixed topics.
UDFNumberRows - Number successive rows starting from 1. Counter resets to 1
whenever any of its parameters changes.
UDFPmax - Finds the maximum of a set of columns. e.g., PMAX(4, 5, 3) returns
5.
UDFRegexpExtractAll - Like REGEXP_EXTRACT except that it returns all matches
in an array.
UDFUnescape - Returns the string unescaped (using C/Java style unescaping).
UDFWhich - Given a boolean array, return the indices which are TRUE.
UDFJaccard
UDAFCollect - Takes all the values associated with a row and converts it into
a list. Make sure to have: set hive.map.aggr = false;
UDAFCollectMap - Like collect except that it takes tuples and generates a map.
UDAFEntropy - Compute the entropy of a column.
UDAFPearson (BROKEN!!!) - Computes the pearson correlation between two
columns.
UDAFTop - TOP(KEY, VAL) - returns the KEY associated with the largest value
of VAL.
UDAFTopN (BROKEN!!!) - Like TOP except returns a list of the keys associated
with the N (passed as the third parameter) largest values of VAL.
UDAFHistogram
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1662) Add file pruning into Hive.
[
https://issues.apache.org/jira/browse/HIVE-1662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733981#comment-13733981
]
Hive QA commented on HIVE-1662:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12596758/HIVE-1662.D8391.6.patch
{color:red}ERROR:{color} -1 due to 292 failed/errored test(s), 2769 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_ppd
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_auto
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_quote1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_second
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input11_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_map_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_unquote_or
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_to_unix_timestamp
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join4
org.apache.hadoop.hive.ql.parse.TestParse.testParse_udf1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_lateralview
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_11
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_dyn_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters_overlap
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join34
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_view
org.apache.hadoop.hive.ql.parse.TestParse.testParse_cast1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_map_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_grouping_sets4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_unquote_and
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_notequal
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nestedvirtual
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join41
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cluster
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize
org.apache.hadoop.hive.ql.parse.TestParse.testParse_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nullsafe
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonmr_fetch
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union26
[jira] [Updated] (HIVE-5011) Dynamic partitioning in HCatalog broken on external tables
[
https://issues.apache.org/jira/browse/HIVE-5011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-5011:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for the review, Daniel. Committed to trunk.
(Ignoring comment by Hive QA bot as that seems to be an unrelated build error)
Dynamic partitioning in HCatalog broken on external tables
--
Key: HIVE-5011
URL: https://issues.apache.org/jira/browse/HIVE-5011
Project: Hive
Issue Type: Bug
Components: HCatalog
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan
Priority: Critical
Attachments: HIVE-5011.patch
Dynamic partitioning with HCatalog has been broken as a result of
HCATALOG-500 trying to support user-set paths for external tables.
The goal there was to be able to support other custom destinations apart from
the normal hive-style partitions. However, it is not currently possible for
users to set paths for dynamic ptn writes, since we don't support any way for
users to specify patterns(like, say $\{rootdir\}/$v1.$v2/) into which
writes happen, only locations, and the values for dyn. partitions are not
known ahead of time. Also, specifying a custom path messes with the way
dynamic ptn. code tries to determine what was written to where from the
output committer, which means that even if we supported patterned-writes
instead of location-writes, we still have to do some more deep diving into
the output committer code to support it.
Thus, my current proposal is that we honour writes to user-specified paths
for external tables *ONLY* for static partition writes - i.e., if we can
determine that the write is a dyn. ptn. write, we will ignore the user
specification. (Note that this does not mean we ignore the table's external
location - we honour that - we just don't honour any HCatStorer/etc provided
additional location - we stick to what metadata tells us the root location is.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4989) Consolidate and simplify vectorization code and test generation
[ https://issues.apache.org/jira/browse/HIVE-4989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-4989: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to branch. Thanks, Tony! Consolidate and simplify vectorization code and test generation --- Key: HIVE-4989 URL: https://issues.apache.org/jira/browse/HIVE-4989 Project: Hive Issue Type: Sub-task Affects Versions: vectorization-branch Reporter: Tony Murphy Assignee: Tony Murphy Fix For: vectorization-branch Attachments: HIVE-4989-vectorization.patch The current code generation is unwieldy to use and prone to errors. This change consolidates all the code and test generation into a single location, and removes the need to manually place files which can lead to missing or incomplete code or tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-5029) direct SQL perf optimization cannot be tested well (yet)
Sergey Shelukhin created HIVE-5029: -- Summary: direct SQL perf optimization cannot be tested well (yet) Key: HIVE-5029 URL: https://issues.apache.org/jira/browse/HIVE-5029 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Critical HIVE-4051 introduced perf optimization that involves getting partitions directly via SQL in metastore. Given that SQL queries might not work on all datastores (and will not work on non-SQL ones), JDO fallback is in place. Given that perf improvement is very large for short queries, it's on by default. However, there's a problem with tests with regard to that. If SQL code is broken, tests may fall back to JDO and pass. If JDO code is broken, SQL might allow tests to pass. We are going to disable SQL by default before the testing problem is resolved. There are several possible solultions: 1) Separate build for this setting. Seems like an overkill... 2) Enable by default; disable by default in tests, create a clone of TestCliDriver with a subset of queries that will exercise the SQL path. 3) Have some sort of test hook inside metastore that will run both ORM and SQL and compare. 3') Or make a subclass of ObjectStore that will do that. ObjectStore is already pluggable. 4) Write unit tests for one of the modes (JDO, as non-default?) and declare that they are sufficient; disable fallback in tests. 3' seems like the easiest. For now we will disable SQL by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5029) direct SQL perf optimization cannot be tested well (yet)
[ https://issues.apache.org/jira/browse/HIVE-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-5029: --- Issue Type: Test (was: Bug) direct SQL perf optimization cannot be tested well (yet) Key: HIVE-5029 URL: https://issues.apache.org/jira/browse/HIVE-5029 Project: Hive Issue Type: Test Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Critical HIVE-4051 introduced perf optimization that involves getting partitions directly via SQL in metastore. Given that SQL queries might not work on all datastores (and will not work on non-SQL ones), JDO fallback is in place. Given that perf improvement is very large for short queries, it's on by default. However, there's a problem with tests with regard to that. If SQL code is broken, tests may fall back to JDO and pass. If JDO code is broken, SQL might allow tests to pass. We are going to disable SQL by default before the testing problem is resolved. There are several possible solultions: 1) Separate build for this setting. Seems like an overkill... 2) Enable by default; disable by default in tests, create a clone of TestCliDriver with a subset of queries that will exercise the SQL path. 3) Have some sort of test hook inside metastore that will run both ORM and SQL and compare. 3') Or make a subclass of ObjectStore that will do that. ObjectStore is already pluggable. 4) Write unit tests for one of the modes (JDO, as non-default?) and declare that they are sufficient; disable fallback in tests. 3' seems like the easiest. For now we will disable SQL by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5029) direct SQL perf optimization cannot be tested well (yet)
[ https://issues.apache.org/jira/browse/HIVE-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-5029: --- Status: Patch Available (was: Open) small patch. I will file a separate JIRA for actual resolution direct SQL perf optimization cannot be tested well (yet) Key: HIVE-5029 URL: https://issues.apache.org/jira/browse/HIVE-5029 Project: Hive Issue Type: Test Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Critical Attachments: HIVE-5029.patch HIVE-4051 introduced perf optimization that involves getting partitions directly via SQL in metastore. Given that SQL queries might not work on all datastores (and will not work on non-SQL ones), JDO fallback is in place. Given that perf improvement is very large for short queries, it's on by default. However, there's a problem with tests with regard to that. If SQL code is broken, tests may fall back to JDO and pass. If JDO code is broken, SQL might allow tests to pass. We are going to disable SQL by default before the testing problem is resolved. There are several possible solultions: 1) Separate build for this setting. Seems like an overkill... 2) Enable by default; disable by default in tests, create a clone of TestCliDriver with a subset of queries that will exercise the SQL path. 3) Have some sort of test hook inside metastore that will run both ORM and SQL and compare. 3') Or make a subclass of ObjectStore that will do that. ObjectStore is already pluggable. 4) Write unit tests for one of the modes (JDO, as non-default?) and declare that they are sufficient; disable fallback in tests. 3' seems like the easiest. For now we will disable SQL by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5029) direct SQL perf optimization cannot be tested well (yet)
[ https://issues.apache.org/jira/browse/HIVE-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-5029: --- Attachment: HIVE-5029.patch direct SQL perf optimization cannot be tested well (yet) Key: HIVE-5029 URL: https://issues.apache.org/jira/browse/HIVE-5029 Project: Hive Issue Type: Test Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Critical Attachments: HIVE-5029.patch HIVE-4051 introduced perf optimization that involves getting partitions directly via SQL in metastore. Given that SQL queries might not work on all datastores (and will not work on non-SQL ones), JDO fallback is in place. Given that perf improvement is very large for short queries, it's on by default. However, there's a problem with tests with regard to that. If SQL code is broken, tests may fall back to JDO and pass. If JDO code is broken, SQL might allow tests to pass. We are going to disable SQL by default before the testing problem is resolved. There are several possible solultions: 1) Separate build for this setting. Seems like an overkill... 2) Enable by default; disable by default in tests, create a clone of TestCliDriver with a subset of queries that will exercise the SQL path. 3) Have some sort of test hook inside metastore that will run both ORM and SQL and compare. 3') Or make a subclass of ObjectStore that will do that. ObjectStore is already pluggable. 4) Write unit tests for one of the modes (JDO, as non-default?) and declare that they are sufficient; disable fallback in tests. 3' seems like the easiest. For now we will disable SQL by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-5030) have fetchoperator provide all of the partition directories to the input format
Owen O'Malley created HIVE-5030: --- Summary: have fetchoperator provide all of the partition directories to the input format Key: HIVE-5030 URL: https://issues.apache.org/jira/browse/HIVE-5030 Project: Hive Issue Type: Bug Reporter: Owen O'Malley Assignee: Owen O'Malley For some custom input formats, it is convenient to be able to see all of the partition directories at once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1545) Add a bunch of UDFs and UDAFs
[
https://issues.apache.org/jira/browse/HIVE-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13734046#comment-13734046
]
Jonathan Chang commented on HIVE-1545:
--
What are you looking for in particular?
Add a bunch of UDFs and UDAFs
-
Key: HIVE-1545
URL: https://issues.apache.org/jira/browse/HIVE-1545
Project: Hive
Issue Type: New Feature
Components: UDF
Reporter: Jonathan Chang
Assignee: Jonathan Chang
Priority: Minor
Attachments: core.tar.gz, ext.tar.gz, UDFEndsWith.java,
UDFFindInString.java, UDFLtrim.java, UDFRtrim.java, udfs.tar.gz, udfs.tar.gz,
UDFStartsWith.java, UDFTrim.java
Here some UD(A)Fs which can be incorporated into the Hive distribution:
UDFArgMax - Find the 0-indexed index of the largest argument. e.g., ARGMAX(4,
5, 3) returns 1.
UDFBucket - Find the bucket in which the first argument belongs. e.g.,
BUCKET(x, b_1, b_2, b_3, ...), will return the smallest i such that x b_{i}
but = b_{i+1}. Returns 0 if x is smaller than all the buckets.
UDFFindInArray - Finds the 1-index of the first element in the array given as
the second argument. Returns 0 if not found. Returns NULL if either argument
is NULL. E.g., FIND_IN_ARRAY(5, array(1,2,5)) will return 3. FIND_IN_ARRAY(5,
array(1,2,3)) will return 0.
UDFGreatCircleDist - Finds the great circle distance (in km) between two
lat/long coordinates (in degrees).
UDFLDA - Performs LDA inference on a vector given fixed topics.
UDFNumberRows - Number successive rows starting from 1. Counter resets to 1
whenever any of its parameters changes.
UDFPmax - Finds the maximum of a set of columns. e.g., PMAX(4, 5, 3) returns
5.
UDFRegexpExtractAll - Like REGEXP_EXTRACT except that it returns all matches
in an array.
UDFUnescape - Returns the string unescaped (using C/Java style unescaping).
UDFWhich - Given a boolean array, return the indices which are TRUE.
UDFJaccard
UDAFCollect - Takes all the values associated with a row and converts it into
a list. Make sure to have: set hive.map.aggr = false;
UDAFCollectMap - Like collect except that it takes tuples and generates a map.
UDAFEntropy - Compute the entropy of a column.
UDAFPearson (BROKEN!!!) - Computes the pearson correlation between two
columns.
UDAFTop - TOP(KEY, VAL) - returns the KEY associated with the largest value
of VAL.
UDAFTopN (BROKEN!!!) - Like TOP except returns a list of the keys associated
with the N (passed as the third parameter) largest values of VAL.
UDAFHistogram
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4324) ORC Turn off dictionary encoding when number of distinct keys is greater than threshold
[ https://issues.apache.org/jira/browse/HIVE-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13734050#comment-13734050 ] Phabricator commented on HIVE-4324: --- ashutoshc has accepted the revision HIVE-4324 [jira] ORC Turn off dictionary encoding when number of distinct keys is greater than threshold. +1 LGTM REVISION DETAIL https://reviews.facebook.net/D12045 BRANCH h-4324 ARCANIST PROJECT hive To: JIRA, ashutoshc, omalley ORC Turn off dictionary encoding when number of distinct keys is greater than threshold --- Key: HIVE-4324 URL: https://issues.apache.org/jira/browse/HIVE-4324 Project: Hive Issue Type: Sub-task Components: File Formats Affects Versions: 0.11.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Fix For: 0.12.0 Attachments: HIVE-4324.1.patch.txt, HIVE-4324.D12045.1.patch, HIVE-4324.D12045.2.patch Add a configurable threshold so that if the number of distinct values in a string column is greater than that fraction of non-null values, dictionary encoding is turned off. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4990) ORC seeks fails with non-zero offset or column projection
[ https://issues.apache.org/jira/browse/HIVE-4990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13734066#comment-13734066 ] Owen O'Malley commented on HIVE-4990: - I committed this to trunk and branch-0.11. ORC seeks fails with non-zero offset or column projection - Key: HIVE-4990 URL: https://issues.apache.org/jira/browse/HIVE-4990 Project: Hive Issue Type: Bug Components: File Formats Affects Versions: 0.11.0 Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.11.1 Attachments: HIVE-4990.D12009.1.patch The ORC reader gets exceptions when seeking with non-zero offsets or column projection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2914) HiveConnection constructor ignores passed-in properties object
[
https://issues.apache.org/jira/browse/HIVE-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13734078#comment-13734078
]
Hive QA commented on HIVE-2914:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12596764/HIVE-2914.D2691.2.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 2769 tests executed
*Failed tests:*
{noformat}
org.apache.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_vc
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/342/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/342/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
HiveConnection constructor ignores passed-in properties object
--
Key: HIVE-2914
URL: https://issues.apache.org/jira/browse/HIVE-2914
Project: Hive
Issue Type: Bug
Components: JDBC
Reporter: Ashutosh Chauhan
Assignee: Navis
Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2914.D2691.1.patch,
HIVE-2914.D2691.2.patch
In local mode HiveConf should initialize itself with passed in properties and
in remote mode, connection should execute series of {{set}} command for all
the properties.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4990) ORC seeks fails with non-zero offset or column projection
[ https://issues.apache.org/jira/browse/HIVE-4990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-4990: Resolution: Fixed Status: Resolved (was: Patch Available) ORC seeks fails with non-zero offset or column projection - Key: HIVE-4990 URL: https://issues.apache.org/jira/browse/HIVE-4990 Project: Hive Issue Type: Bug Components: File Formats Affects Versions: 0.11.0 Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.11.1 Attachments: HIVE-4990.D12009.1.patch The ORC reader gets exceptions when seeking with non-zero offsets or column projection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4990) ORC seeks fails with non-zero offset or column projection
[ https://issues.apache.org/jira/browse/HIVE-4990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-4990: Fix Version/s: 0.12.0 ORC seeks fails with non-zero offset or column projection - Key: HIVE-4990 URL: https://issues.apache.org/jira/browse/HIVE-4990 Project: Hive Issue Type: Bug Components: File Formats Affects Versions: 0.11.0 Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.11.1, 0.12.0 Attachments: HIVE-4990.D12009.1.patch The ORC reader gets exceptions when seeking with non-zero offsets or column projection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1545) Add a bunch of UDFs and UDAFs
[
https://issues.apache.org/jira/browse/HIVE-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13734088#comment-13734088
]
Brenden Matthews commented on HIVE-1545:
There's a bunch of code missing. Your code doesn't build without modifications.
I've made a copy of this which seems to work (minus the broken parts) here:
https://github.com/brndnmtthws/facebook-hive-udfs
Add a bunch of UDFs and UDAFs
-
Key: HIVE-1545
URL: https://issues.apache.org/jira/browse/HIVE-1545
Project: Hive
Issue Type: New Feature
Components: UDF
Reporter: Jonathan Chang
Assignee: Jonathan Chang
Priority: Minor
Attachments: core.tar.gz, ext.tar.gz, UDFEndsWith.java,
UDFFindInString.java, UDFLtrim.java, UDFRtrim.java, udfs.tar.gz, udfs.tar.gz,
UDFStartsWith.java, UDFTrim.java
Here some UD(A)Fs which can be incorporated into the Hive distribution:
UDFArgMax - Find the 0-indexed index of the largest argument. e.g., ARGMAX(4,
5, 3) returns 1.
UDFBucket - Find the bucket in which the first argument belongs. e.g.,
BUCKET(x, b_1, b_2, b_3, ...), will return the smallest i such that x b_{i}
but = b_{i+1}. Returns 0 if x is smaller than all the buckets.
UDFFindInArray - Finds the 1-index of the first element in the array given as
the second argument. Returns 0 if not found. Returns NULL if either argument
is NULL. E.g., FIND_IN_ARRAY(5, array(1,2,5)) will return 3. FIND_IN_ARRAY(5,
array(1,2,3)) will return 0.
UDFGreatCircleDist - Finds the great circle distance (in km) between two
lat/long coordinates (in degrees).
UDFLDA - Performs LDA inference on a vector given fixed topics.
UDFNumberRows - Number successive rows starting from 1. Counter resets to 1
whenever any of its parameters changes.
UDFPmax - Finds the maximum of a set of columns. e.g., PMAX(4, 5, 3) returns
5.
UDFRegexpExtractAll - Like REGEXP_EXTRACT except that it returns all matches
in an array.
UDFUnescape - Returns the string unescaped (using C/Java style unescaping).
UDFWhich - Given a boolean array, return the indices which are TRUE.
UDFJaccard
UDAFCollect - Takes all the values associated with a row and converts it into
a list. Make sure to have: set hive.map.aggr = false;
UDAFCollectMap - Like collect except that it takes tuples and generates a map.
UDAFEntropy - Compute the entropy of a column.
UDAFPearson (BROKEN!!!) - Computes the pearson correlation between two
columns.
UDAFTop - TOP(KEY, VAL) - returns the KEY associated with the largest value
of VAL.
UDAFTopN (BROKEN!!!) - Like TOP except returns a list of the keys associated
with the N (passed as the third parameter) largest values of VAL.
UDAFHistogram
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-5031) [WebHCat] GET job/:jobid to return all information for a given job in addtion to status information
Daniel Dai created HIVE-5031: Summary: [WebHCat] GET job/:jobid to return all information for a given job in addtion to status information Key: HIVE-5031 URL: https://issues.apache.org/jira/browse/HIVE-5031 Project: Hive Issue Type: Bug Components: HCatalog Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.12.0 Attachments: HIVE-5031-1.patch It would be nice to also have any information that was passed into job creation API including job type specific information (e.g. mapreduce libjars) It would also be really helpful to support an dictionary param which could be utilized by the job author for custom properties about the job. For example, to persist the template that was used to create the job, a friendly job description and name, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5031) [WebHCat] GET job/:jobid to return all information for a given job in addtion to status information
[ https://issues.apache.org/jira/browse/HIVE-5031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-5031: - Attachment: HIVE-5031-1.patch Attach HIVE-5031-1.patch. Tests will be in follow up Jira. [WebHCat] GET job/:jobid to return all information for a given job in addtion to status information --- Key: HIVE-5031 URL: https://issues.apache.org/jira/browse/HIVE-5031 Project: Hive Issue Type: Bug Components: HCatalog Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.12.0 Attachments: HIVE-5031-1.patch It would be nice to also have any information that was passed into job creation API including job type specific information (e.g. mapreduce libjars) It would also be really helpful to support an dictionary param which could be utilized by the job author for custom properties about the job. For example, to persist the template that was used to create the job, a friendly job description and name, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[ https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13734137#comment-13734137 ] Sergey Shelukhin commented on HIVE-4914: There are few minor bugs, I will update later today. There's also some code in JDO filter pushdown that never worked in the first place, but client didn't let it execute due to more restrictive pushdown checks. I will fix/remove it. filtering via partition name should be done inside metastore server (implementation) Key: HIVE-4914 URL: https://issues.apache.org/jira/browse/HIVE-4914 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch, HIVE-4914.patch Currently, if the filter pushdown is impossible (which is most cases), the client gets all partition names from metastore, filters them, and asks for partitions by names for the filtered set. Metastore server code should do that instead; it should check if pushdown is possible and do it if so; otherwise it should do name-based filtering. Saves the roundtrip with all partition names from the server to client, and also removes the need to have pushdown viability checking on both sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5030) have fetchoperator provide all of the partition directories to the input format
[ https://issues.apache.org/jira/browse/HIVE-5030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-5030: -- Attachment: HIVE-5030.D12105.1.patch omalley requested code review of HIVE-5030 [jira] have fetchoperator provide all of the partition directories to the input format. Reviewers: JIRA hive-5030 all directories from the fecth operator For some custom input formats, it is convenient to be able to see all of the partition directories at once. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D12105 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/28893/ To: JIRA, omalley have fetchoperator provide all of the partition directories to the input format --- Key: HIVE-5030 URL: https://issues.apache.org/jira/browse/HIVE-5030 Project: Hive Issue Type: Bug Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-5030.D12105.1.patch For some custom input formats, it is convenient to be able to see all of the partition directories at once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
documentation issue for RLIKE/REGEXP
Double checking before I try to edit. The page here: https://cwiki.apache.org/Hive/languagemanual-udf.html says: A RLIKE B if A or B is NULL, TRUE if any (possibly empty) substring of A matches the Java regular expression B, otherwise FALSE. E.g. 'foobar' RLIKE 'foo' evaluates to FALSE whereas 'foobar' RLIKE '^f.*r$' evaluates to TRUE. 1) if A or B is NULL seems like an unfinished part. 2) any (possibly empty) substring of A [that] matches the Java regular expression B should be foo at 0 for 'foobar' RLIKE 'foo', and result in TRUE, right?
[jira] [Commented] (HIVE-4985) refactor/clean up partition name pruning to be usable inside metastore server
[ https://issues.apache.org/jira/browse/HIVE-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13734176#comment-13734176 ] Phabricator commented on HIVE-4985: --- ashutoshc has accepted the revision HIVE-4985 [jira] refactor/clean up partition name pruning to be usable inside metastore server. Changes look good. Please revert to Set usage (which is the current case on trunk). REVISION DETAIL https://reviews.facebook.net/D11961 BRANCH HIVE-4985-2 ARCANIST PROJECT hive To: JIRA, ashutoshc, sershe refactor/clean up partition name pruning to be usable inside metastore server -- Key: HIVE-4985 URL: https://issues.apache.org/jira/browse/HIVE-4985 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-4985.D11961.1.patch, HIVE-4985.D11961.2.patch Preliminary for HIVE-4914. The patch is going to be large already, so some refactoring and dead code removal that is non-controversial can be done in advance in a separate patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-5032) Enable hive creating external table at the root directory of DFS
Shuaishuai Nie created HIVE-5032: Summary: Enable hive creating external table at the root directory of DFS Key: HIVE-5032 URL: https://issues.apache.org/jira/browse/HIVE-5032 Project: Hive Issue Type: Bug Reporter: Shuaishuai Nie Creating external table using HIVE with location point to the root directory of DFS will fail because the function HiveFileFormatUtils#doGetPartitionDescFromPath treat authority of the path the same as folder and cannot find a match in the pathToPartitionInfo table when doing prefix match. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5032) Enable hive creating external table at the root directory of DFS
[ https://issues.apache.org/jira/browse/HIVE-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuaishuai Nie updated HIVE-5032: - Status: Patch Available (was: Open) Enable hive creating external table at the root directory of DFS Key: HIVE-5032 URL: https://issues.apache.org/jira/browse/HIVE-5032 Project: Hive Issue Type: Bug Reporter: Shuaishuai Nie Attachments: HIVE-5032.1.patch Creating external table using HIVE with location point to the root directory of DFS will fail because the function HiveFileFormatUtils#doGetPartitionDescFromPath treat authority of the path the same as folder and cannot find a match in the pathToPartitionInfo table when doing prefix match. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5032) Enable hive creating external table at the root directory of DFS
[ https://issues.apache.org/jira/browse/HIVE-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuaishuai Nie updated HIVE-5032: - Attachment: HIVE-5032.1.patch Enable hive creating external table at the root directory of DFS Key: HIVE-5032 URL: https://issues.apache.org/jira/browse/HIVE-5032 Project: Hive Issue Type: Bug Reporter: Shuaishuai Nie Attachments: HIVE-5032.1.patch Creating external table using HIVE with location point to the root directory of DFS will fail because the function HiveFileFormatUtils#doGetPartitionDescFromPath treat authority of the path the same as folder and cannot find a match in the pathToPartitionInfo table when doing prefix match. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4123) The RLE encoding for ORC can be improved
[ https://issues.apache.org/jira/browse/HIVE-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13734231#comment-13734231 ] Prasanth J commented on HIVE-4123: -- Thanks for the review Owen. I have addressed the following issues with this patch - Date type handled for new encoding - Better encoding check added by overriding checkEncoding() for valid types - Created factories for reader and writer creation - Indentation fix - DIRECT_V2 encoding can be turned on/off by using hive.exec.orc.write.format configuration parameter. If this parameter value is 0.11 then old RLE encoding will be used else if undefined or for any other values new RLE encoding will be used. Also, HIVE-4324 patch will get affected by this patch. So this new patch is generated on top of HIVE-4324. The RLE encoding for ORC can be improved Key: HIVE-4123 URL: https://issues.apache.org/jira/browse/HIVE-4123 Project: Hive Issue Type: New Feature Components: File Formats Affects Versions: 0.12.0 Reporter: Owen O'Malley Assignee: Prasanth J Labels: orcfile Fix For: 0.12.0 Attachments: HIVE-4123.1.git.patch.txt, HIVE-4123.2.git.patch.txt, HIVE-4123.3.patch.txt, HIVE-4123.4.patch.txt, HIVE-4123.5.txt, HIVE-4123.6.txt, ORC-Compression-Ratio-Comparison.xlsx The run length encoding of integers can be improved: * tighter bit packing * allow delta encoding * allow longer runs -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4123) The RLE encoding for ORC can be improved
[ https://issues.apache.org/jira/browse/HIVE-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-4123: - Attachment: HIVE-4123.7.txt The RLE encoding for ORC can be improved Key: HIVE-4123 URL: https://issues.apache.org/jira/browse/HIVE-4123 Project: Hive Issue Type: New Feature Components: File Formats Affects Versions: 0.12.0 Reporter: Owen O'Malley Assignee: Prasanth J Labels: orcfile Fix For: 0.12.0 Attachments: HIVE-4123.1.git.patch.txt, HIVE-4123.2.git.patch.txt, HIVE-4123.3.patch.txt, HIVE-4123.4.patch.txt, HIVE-4123.5.txt, HIVE-4123.6.txt, HIVE-4123.7.txt, ORC-Compression-Ratio-Comparison.xlsx The run length encoding of integers can be improved: * tighter bit packing * allow delta encoding * allow longer runs -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5027) Upgrade Ivy to 2.3
[
https://issues.apache.org/jira/browse/HIVE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13734262#comment-13734262
]
Hive QA commented on HIVE-5027:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12596866/HIVE-5027.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 2772 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_vc
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/347/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/347/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
Upgrade Ivy to 2.3
--
Key: HIVE-5027
URL: https://issues.apache.org/jira/browse/HIVE-5027
Project: Hive
Issue Type: Improvement
Components: Build Infrastructure
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Minor
Attachments: HIVE-5027.patch
Ivy version 2.1 doesn't include classifiers when creating pom files.
Therefore our generated pom's are not correct. Version 2.3 fixes this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HIVE-4961) Create bridge for custom UDFs to operate in vectorized mode
[ https://issues.apache.org/jira/browse/HIVE-4961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-4961 started by Eric Hanson. Create bridge for custom UDFs to operate in vectorized mode --- Key: HIVE-4961 URL: https://issues.apache.org/jira/browse/HIVE-4961 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Assignee: Eric Hanson Suppose you have a custom UDF myUDF() that you've created to extend hive. The goal of this JIRA is to create a facility where if you run a query that uses myUDF() in an expression, the query will run in vectorized mode. This would be a general-purpose bridge for custom UDFs that users add to Hive. It would work with existing UDFs. I'm considering a separate JIRA for a new kind of custom UDF implementation that is vectorized from the beginning, to optimize performance. That is not covered by this JIRA. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1545) Add a bunch of UDFs and UDAFs
[
https://issues.apache.org/jira/browse/HIVE-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13734306#comment-13734306
]
Edward Capriolo commented on HIVE-1545:
---
The annotations and other things you are seeing are part of an internal testing
framework at FB that was never open sourced, the hive plugin developer kit had
similar annotations but they were removed. So the UDFS likely compilefine but
the test cases will not.
Add a bunch of UDFs and UDAFs
-
Key: HIVE-1545
URL: https://issues.apache.org/jira/browse/HIVE-1545
Project: Hive
Issue Type: New Feature
Components: UDF
Reporter: Jonathan Chang
Assignee: Jonathan Chang
Priority: Minor
Attachments: core.tar.gz, ext.tar.gz, UDFEndsWith.java,
UDFFindInString.java, UDFLtrim.java, UDFRtrim.java, udfs.tar.gz, udfs.tar.gz,
UDFStartsWith.java, UDFTrim.java
Here some UD(A)Fs which can be incorporated into the Hive distribution:
UDFArgMax - Find the 0-indexed index of the largest argument. e.g., ARGMAX(4,
5, 3) returns 1.
UDFBucket - Find the bucket in which the first argument belongs. e.g.,
BUCKET(x, b_1, b_2, b_3, ...), will return the smallest i such that x b_{i}
but = b_{i+1}. Returns 0 if x is smaller than all the buckets.
UDFFindInArray - Finds the 1-index of the first element in the array given as
the second argument. Returns 0 if not found. Returns NULL if either argument
is NULL. E.g., FIND_IN_ARRAY(5, array(1,2,5)) will return 3. FIND_IN_ARRAY(5,
array(1,2,3)) will return 0.
UDFGreatCircleDist - Finds the great circle distance (in km) between two
lat/long coordinates (in degrees).
UDFLDA - Performs LDA inference on a vector given fixed topics.
UDFNumberRows - Number successive rows starting from 1. Counter resets to 1
whenever any of its parameters changes.
UDFPmax - Finds the maximum of a set of columns. e.g., PMAX(4, 5, 3) returns
5.
UDFRegexpExtractAll - Like REGEXP_EXTRACT except that it returns all matches
in an array.
UDFUnescape - Returns the string unescaped (using C/Java style unescaping).
UDFWhich - Given a boolean array, return the indices which are TRUE.
UDFJaccard
UDAFCollect - Takes all the values associated with a row and converts it into
a list. Make sure to have: set hive.map.aggr = false;
UDAFCollectMap - Like collect except that it takes tuples and generates a map.
UDAFEntropy - Compute the entropy of a column.
UDAFPearson (BROKEN!!!) - Computes the pearson correlation between two
columns.
UDAFTop - TOP(KEY, VAL) - returns the KEY associated with the largest value
of VAL.
UDAFTopN (BROKEN!!!) - Like TOP except returns a list of the keys associated
with the N (passed as the third parameter) largest values of VAL.
UDAFHistogram
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2482) Convenience UDFs for binary data type
[ https://issues.apache.org/jira/browse/HIVE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Wagner updated HIVE-2482: -- Description: HIVE-2380 introduced binary data type in Hive. It will be good to have following udfs to make it more useful: * UDF's to convert to/from hex string * UDF's to convert to/from string using a specific encoding * UDF's to convert to/from base64 string was: HIVE-2380 introduced binary data type in Hive. It will be good to have following udfs to make it more useful: * UDF's to convert to/from hex string * UDF's to convert to/from string using a specific encoding * UDF's to convert to/from base64 string * UDF's to convert to/from non-string types using a particular serde Convenience UDFs for binary data type - Key: HIVE-2482 URL: https://issues.apache.org/jira/browse/HIVE-2482 Project: Hive Issue Type: New Feature Affects Versions: 0.9.0 Reporter: Ashutosh Chauhan Assignee: Mark Wagner Attachments: HIVE-2482.1.patch, HIVE-2482.2.patch, HIVE-2482.3.patch HIVE-2380 introduced binary data type in Hive. It will be good to have following udfs to make it more useful: * UDF's to convert to/from hex string * UDF's to convert to/from string using a specific encoding * UDF's to convert to/from base64 string -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2482) Convenience UDFs for binary data type
[ https://issues.apache.org/jira/browse/HIVE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Wagner updated HIVE-2482: -- Fix Version/s: 0.12.0 Affects Version/s: (was: 0.9.0) Release Note: This is an incompatible change because the return type of unhex has been changed from string to binary Hadoop Flags: Incompatible change Status: Patch Available (was: Open) Convenience UDFs for binary data type - Key: HIVE-2482 URL: https://issues.apache.org/jira/browse/HIVE-2482 Project: Hive Issue Type: New Feature Reporter: Ashutosh Chauhan Assignee: Mark Wagner Fix For: 0.12.0 Attachments: HIVE-2482.1.patch, HIVE-2482.2.patch, HIVE-2482.3.patch HIVE-2380 introduced binary data type in Hive. It will be good to have following udfs to make it more useful: * UDF's to convert to/from hex string * UDF's to convert to/from string using a specific encoding * UDF's to convert to/from base64 string -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
