[jira] [Created] (HIVE-6107) Introduce nvl and nvl2 UDF similar to Oracle
Kostiantyn Kudriavtsev created HIVE-6107: Summary: Introduce nvl and nvl2 UDF similar to Oracle Key: HIVE-6107 URL: https://issues.apache.org/jira/browse/HIVE-6107 Project: Hive Issue Type: New Feature Components: UDF Reporter: Kostiantyn Kudriavtsev Priority: Minor Introduce new UDF: nvl : substitute a value when a null value is encountered NVL( string, replace_with ) where string is the string to test for a null value nvl2 : extends the functionality found in the NVL function. It lets you substitutes a value when a null value is encountered as well as when a non-null value is encountered. NVL2( string1, value_if_NOT_null, value_if_null ) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6006) Add UDF to calculate distance between geographic coordinates
[ https://issues.apache.org/jira/browse/HIVE-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kostiantyn Kudriavtsev updated HIVE-6006: - Status: Open (was: Patch Available) Add UDF to calculate distance between geographic coordinates Key: HIVE-6006 URL: https://issues.apache.org/jira/browse/HIVE-6006 Project: Hive Issue Type: New Feature Components: UDF Affects Versions: 0.13.0 Reporter: Kostiantyn Kudriavtsev Priority: Minor Fix For: 0.13.0 Attachments: hive-6006.patch Original Estimate: 336h Remaining Estimate: 336h It would be nice to have Hive UDF to calculate distance between two points on Earth. Haversine formula seems to be good enough to overcome this issue The next function is proposed: HaversineDistance(lat1, lon1, lat2, lon2) - calculate Harvesine Distance between 2 points with coordinates (lat1, lon1) and (lat2, lon2) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HIVE-6108) Introduce Cryptographic hash UDFs
Kostiantyn Kudriavtsev created HIVE-6108: Summary: Introduce Cryptographic hash UDFs Key: HIVE-6108 URL: https://issues.apache.org/jira/browse/HIVE-6108 Project: Hive Issue Type: New Feature Components: UDF Reporter: Kostiantyn Kudriavtsev Priority: Minor Introduce new UDF to implement Cryptographic hash algorithms: MD5 and SHA-1 which is already available in Java: MD5(string) Calculates an MD5 checksum for the string, return HEX representation SHA1(string) Calculates an SHA-1 checksum for the string, return HEX representation -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
Satish Mittal created HIVE-6109: --- Summary: Support customized location for EXTERNAL tables created by Dynamic Partitioning Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[ https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13856862#comment-13856862 ] Satish Mittal commented on HIVE-6109: - As an example, suppose a table user_logs is partitioned by (year, month, day, hour, minute, country) and stored at location hdfs://hcat/data/user_logs. Currently dynamic partitions for user_logs would get created at HDFS locations in the fixed format hdfs://hcat/data/user_logs/year=2013/month=12/hour=06/minute=10/country=US. However in our use-case this breaks other data pipeline applications which expect the data to arrive in a different format: hdfs://hcat/data/user_logs/2013/12/06/10/US. There could be similar other use-cases possible. Support customized location for EXTERNAL tables created by Dynamic Partitioning --- Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5011) Dynamic partitioning in HCatalog broken on external tables
[
https://issues.apache.org/jira/browse/HIVE-5011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13856864#comment-13856864
]
Satish Mittal commented on HIVE-5011:
-
Created [HIVE-6109|https://issues.apache.org/jira/browse/HIVE-6109] to track
the original issue.
Dynamic partitioning in HCatalog broken on external tables
--
Key: HIVE-5011
URL: https://issues.apache.org/jira/browse/HIVE-5011
Project: Hive
Issue Type: Bug
Components: HCatalog
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan
Priority: Critical
Fix For: 0.12.0
Attachments: HIVE-5011.patch
Dynamic partitioning with HCatalog has been broken as a result of
HCATALOG-500 trying to support user-set paths for external tables.
The goal there was to be able to support other custom destinations apart from
the normal hive-style partitions. However, it is not currently possible for
users to set paths for dynamic ptn writes, since we don't support any way for
users to specify patterns(like, say $\{rootdir\}/$v1.$v2/) into which
writes happen, only locations, and the values for dyn. partitions are not
known ahead of time. Also, specifying a custom path messes with the way
dynamic ptn. code tries to determine what was written to where from the
output committer, which means that even if we supported patterned-writes
instead of location-writes, we still have to do some more deep diving into
the output committer code to support it.
Thus, my current proposal is that we honour writes to user-specified paths
for external tables *ONLY* for static partition writes - i.e., if we can
determine that the write is a dyn. ptn. write, we will ignore the user
specification. (Note that this does not mean we ignore the table's external
location - we honour that - we just don't honour any HCatStorer/etc provided
additional location - we stick to what metadata tells us the root location is.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-6082) Certain KeeperException should be ignored in ZooKeeperHiveLockManage.unlockPrimitive
[ https://issues.apache.org/jira/browse/HIVE-6082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13856925#comment-13856925 ] Brock Noland commented on HIVE-6082: +1 Certain KeeperException should be ignored in ZooKeeperHiveLockManage.unlockPrimitive Key: HIVE-6082 URL: https://issues.apache.org/jira/browse/HIVE-6082 Project: Hive Issue Type: Bug Affects Versions: 0.12.0 Reporter: Chaoyu Tang Assignee: Chaoyu Tang Attachments: HIVE-6082.patch, Hive-6082.patch KeeperException.NoNodeException and NotEmptyException should be ignored when deleting a zLock or its parent in ZooKeeperHiveLockManager unlockPrimitive. The exceptions can happen: 1) ZooKeeperHiveLockManager retries deleting a zLock after a failure but it has been deleted. 2) a race condition where another process adds a zLock just before it is about to be deleted. Otherwise, unlock may unnecessarily be retried for numRetriesForUnLock times. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Hive-trunk-hadoop2 - Build # 625 - Still Failing
Changes for Build #584 [thejas] HIVE-5550 : Import fails for tables created with default text, sequence and orc file formats using HCatalog API (Sushanth Sowmyan via Thejas Nair) [ehans] HIVE-5895: vectorization handles division by zero differently from normal execution (Sergey Shelukhin via Eric Hanson) [hashutosh] HIVE-5938 : Remove apache.mina dependency for test (Navis via Ashutosh Chauhan) [xuefu] HIVE-5912: Show partition command doesn't support db.table (Yu Zhao via Xuefu) [brock] HIVE-5906 - TestGenericUDFPower should use delta to compare doubles (Szehon Ho via Brock Noland) [brock] HIVE-5855 - Add deprecated methods back to ColumnProjectionUtils (Brock Noland reviewed by Navis) [brock] HIVE-5915 - Shade Kryo dependency (Brock Noland reviewed by Ashutosh Chauhan) Changes for Build #585 [hashutosh] HIVE-5916 : No need to aggregate statistics collected via counter mechanism (Ashutosh Chauhan via Navis) [xuefu] HIVE-5947: Fix test failure in decimal_udf.q (reviewed by Brock) Changes for Build #586 [hashutosh] HIVE-5935 : hive.query.string is not provided to FetchTask (Navis via Ashutosh Chauhan) [navis] HIVE-3455 : ANSI CORR(X,Y) is incorrect (Maxim Bolotin via Navis) [hashutosh] HIVE-5921 : Better heuristics for worst case statistics estimates for join, limit and filter operator (Prasanth J via Harish Butani) [rhbutani] HIVE-5899 NPE during explain extended with char/varchar columns (Jason Dere via Harish Butani) Changes for Build #587 [xuefu] HIVE-3181: getDatabaseMajor/Minor version does not return values (Szehon via Xuefu, reviewed by Navis) [brock] HIVE-5641 - BeeLineOpts ignores Throwable (Brock Noland reviewed by Prasad and Thejas) [hashutosh] HIVE-5909 : locate and instr throw java.nio.BufferUnderflowException when empty string as substring (Navis via Ashutosh Chauhan) [hashutosh] HIVE-5686 : partition column type validation doesn't quite work for dates (Sergey Shelukhin via Ashutosh Chauhan) [hashutosh] HIVE-5887 : metastore direct sql doesn't work with oracle (Sergey Shelukhin via Ashutosh Chauhan) Changes for Build #588 Changes for Build #589 Changes for Build #590 [brock] HIVE-5981 - Add hive-unit back to itests pom (Brock Noland reviewed by Prasad) Changes for Build #591 [xuefu] HIVE-5872: Make UDAFs such as GenericUDAFSum report accurate precision/scale for decimal types (reviewed by Sergey Shelukhin) [hashutosh] HIVE-5978 : Rollups not supported in vector mode. (Jitendra Nath Pandey via Ashutosh Chauhan) [hashutosh] HIVE-5830 : SubQuery: Not In subqueries should check if subquery contains nulls in matching column (Harish Butani via Ashutosh Chauhan) [hashutosh] HIVE-5598 : Remove dummy new line at the end of non-sql commands (Navis via Ashutosh Chauhan) Changes for Build #592 [hashutosh] HIVE-5982 : Remove redundant filesystem operations and methods in FileSink (Ashutosh Chauhan via Thejas Nair) [navis] HIVE-5955 : decimal_precision.q test case fails in trunk (Prasanth J via Navis) [brock] HIVE-5983 - Fix name of ColumnProjectionUtils.appendReadColumnIDs (Brock Noland reviewed by Navis) Changes for Build #593 [omalley] HIVE-5580. Predicate pushdown predicates with an and-operator between non-SARGable predicates cause a NPE. (omalley) Changes for Build #594 [gunther] HIVE-6000: Hive build broken on hadoop2 (Vikram Dixit K via Gunther Hagleitner [gunther] HIVE-2093: UPDATE - add two missing files from previous commit (Gunther Hagleitner) [thejas] HIVE-2093 : create/drop database should populate inputs/outputs and check concurrency and user permission (Navis via Thejas Nair) [hashutosh] HIVE-6016 : Hadoop23Shims has a bug in listLocatedStatus impl. (Prasanth J via Ashutosh Chauhan) [hashutosh] HIVE-5994 : ORC RLEv2 encodes wrongly for large negative BIGINTs (64 bits ) (Prasanth J via Owen Omalley) [hashutosh] HIVE-5991 : ORC RLEv2 fails with ArrayIndexOutOfBounds exception for PATCHED_BLOB encoding (Prasanth J via Owen Omalley) [prasadm] HIVE-4395: Support TFetchOrientation.FIRST for HiveServer2 FetchResults (Prasad Mujumdar reviewed by Thejas Nair) [ehans] HIVE-5756: Implement vectorized support for IF conditional expression (Eric Hanson) [hashutosh] HIVE-6018 : FetchTask should not reference metastore classes (Navis via Prasad Mujumdar) [hashutosh] HIVE-5979. Failure in cast to timestamps. (Jitendra Pandey) [hashutosh] HIVE-5897 : Fix hadoop2 execution environment Milestone 2 (Vikram Dixit via Brock Noland) Changes for Build #595 Changes for Build #596 [hashutosh] HIVE-6027 : non-vectorized log10 has rounding issue (Sergey Shelukhin via Ashutosh Chauhan) [prasadm] HIVE-5993: JDBC Driver should not hard-code the database name (Szehon Ho via Prasad Mujumdar) Changes for Build #597 [hashutosh] HIVE-6004 : Fix statistics annotation related test failures in hadoop2 (Prasanth J via Ashutosh Chauhan) Changes for Build #598 [navis] HIVE-5985 : Make qfile_regex to accept multiple patterns (Navis
Hive-branch-0.12-hadoop2 - Build # 25 - Still Failing
Changes for Build #9 [thejas] HIVE-5087 : Rename npath UDF to matchpath (Edward Capriolo and Carl Steinbach via Ashutosh Chauhan) [thejas] HIVE-5402 : StorageBasedAuthorizationProvider is not correctly able to determine that it is running from client-side (Sushanth Sowmyan via Ashutosh Chauhan) [thejas] HIVE-5431 : PassthroughOutputFormat SH changes causes IllegalArgumentException (Sushanth Sowmyan via Ashutosh Chauhan) Changes for Build #10 [thejas] Updating RELEASE_NOTES.txt for 0.12 release Changes for Build #11 [thejas] Updating RELEASE_NOTES.txt HIVE-5477 inclusion [thejas] HIVE-5477 : maven-publish fails because it can't find hive-metastore-0.12.0.pom (Thejas M Nair via Ashutosh Chauhan) [thejas] Updating RELEASE_NOTES.txt for HIVE-4871 inclusion [thejas] HIVE-4871 : Apache builds fail with Target make-pom does not exist in the project hcatalog. (Eugene Koifman via Ashutosh Chauhan) Changes for Build #12 Changes for Build #13 [thejas] HIVE-5489 : NOTICE copyright dates are out of date, README needs update (Thejas M Nair via Carl Steinbach) [thejas] HIVE-5488 : some files are missing apache license headers (Thejas M Nair via Ashutosh Chauhan) [thejas] HIVE-5449 : Hive schematool info option incorrectly reports error for Postgres metastore (Prasad Mujumdar via Thejas Nair) [thejas] HIVE-5418 : Integer overflow bug in ConditionalResolverCommonJoin.AliasFileSizePair(add missing fiel) (Steven Wong via Ashutosh Chauhan) [thejas] HIVE-5418 : Integer overflow bug in ConditionalResolverCommonJoin.AliasFileSizePair (Steven Wong via Ashutosh Chauhan) Changes for Build #14 [thejas] Updating RELEASE_NOTES.txt for commits since 0.12.0 RC0 [thejas] HIVE-5497 : Hive trunk broken against hadoop 0.20.2 (Thejas M Nair via Brock Noland) [thejas] HIVE-5476: Authorization-provider tests fail in sequential run (Sushanth Sowmyan via Ashutosh Chauhan) [thejas] HIVE-5493 : duplicate jars with different versions for guava, commons-logging (Thejas Nair via Ashutosh Chauhan) Changes for Build #15 Changes for Build #17 Changes for Build #18 Changes for Build #19 Changes for Build #20 Changes for Build #21 Changes for Build #22 Changes for Build #23 [omalley] HIVE-5601: NPE in ORC's PPD when using select * from table with where predicate pushdown (Prasanth J via Owen O'Malley and Gunther Hagleitner) Changes for Build #24 Changes for Build #25 34 tests failed. REGRESSION: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 Error Message: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. Stack Trace: junit.framework.AssertionFailedError: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. at junit.framework.Assert.fail(Assert.java:50) at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:6710) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12(TestCliDriver.java:3931) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:422) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:931) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:785) REGRESSION: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_2 Error Message: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. Stack Trace: junit.framework.AssertionFailedError: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. at junit.framework.Assert.fail(Assert.java:50) at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:6710) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_2(TestCliDriver.java:3939) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at
Hive-trunk-h0.21 - Build # 2526 - Still Failing
Changes for Build #2487 [hashutosh] HIVE-5916 : No need to aggregate statistics collected via counter mechanism (Ashutosh Chauhan via Navis) [xuefu] HIVE-5947: Fix test failure in decimal_udf.q (reviewed by Brock) [thejas] HIVE-5550 : Import fails for tables created with default text, sequence and orc file formats using HCatalog API (Sushanth Sowmyan via Thejas Nair) Changes for Build #2488 [hashutosh] HIVE-5935 : hive.query.string is not provided to FetchTask (Navis via Ashutosh Chauhan) [navis] HIVE-3455 : ANSI CORR(X,Y) is incorrect (Maxim Bolotin via Navis) [hashutosh] HIVE-5921 : Better heuristics for worst case statistics estimates for join, limit and filter operator (Prasanth J via Harish Butani) [rhbutani] HIVE-5899 NPE during explain extended with char/varchar columns (Jason Dere via Harish Butani) Changes for Build #2489 [xuefu] HIVE-3181: getDatabaseMajor/Minor version does not return values (Szehon via Xuefu, reviewed by Navis) [brock] HIVE-5641 - BeeLineOpts ignores Throwable (Brock Noland reviewed by Prasad and Thejas) [hashutosh] HIVE-5909 : locate and instr throw java.nio.BufferUnderflowException when empty string as substring (Navis via Ashutosh Chauhan) [hashutosh] HIVE-5686 : partition column type validation doesn't quite work for dates (Sergey Shelukhin via Ashutosh Chauhan) [hashutosh] HIVE-5887 : metastore direct sql doesn't work with oracle (Sergey Shelukhin via Ashutosh Chauhan) Changes for Build #2490 Changes for Build #2491 Changes for Build #2492 [brock] HIVE-5981 - Add hive-unit back to itests pom (Brock Noland reviewed by Prasad) Changes for Build #2493 [xuefu] HIVE-5872: Make UDAFs such as GenericUDAFSum report accurate precision/scale for decimal types (reviewed by Sergey Shelukhin) [hashutosh] HIVE-5978 : Rollups not supported in vector mode. (Jitendra Nath Pandey via Ashutosh Chauhan) [hashutosh] HIVE-5830 : SubQuery: Not In subqueries should check if subquery contains nulls in matching column (Harish Butani via Ashutosh Chauhan) [hashutosh] HIVE-5598 : Remove dummy new line at the end of non-sql commands (Navis via Ashutosh Chauhan) Changes for Build #2494 [hashutosh] HIVE-5982 : Remove redundant filesystem operations and methods in FileSink (Ashutosh Chauhan via Thejas Nair) [navis] HIVE-5955 : decimal_precision.q test case fails in trunk (Prasanth J via Navis) [brock] HIVE-5983 - Fix name of ColumnProjectionUtils.appendReadColumnIDs (Brock Noland reviewed by Navis) Changes for Build #2495 [omalley] HIVE-5580. Predicate pushdown predicates with an and-operator between non-SARGable predicates cause a NPE. (omalley) Changes for Build #2496 [gunther] HIVE-6000: Hive build broken on hadoop2 (Vikram Dixit K via Gunther Hagleitner [gunther] HIVE-2093: UPDATE - add two missing files from previous commit (Gunther Hagleitner) [thejas] HIVE-2093 : create/drop database should populate inputs/outputs and check concurrency and user permission (Navis via Thejas Nair) [hashutosh] HIVE-6016 : Hadoop23Shims has a bug in listLocatedStatus impl. (Prasanth J via Ashutosh Chauhan) [hashutosh] HIVE-5994 : ORC RLEv2 encodes wrongly for large negative BIGINTs (64 bits ) (Prasanth J via Owen Omalley) [hashutosh] HIVE-5991 : ORC RLEv2 fails with ArrayIndexOutOfBounds exception for PATCHED_BLOB encoding (Prasanth J via Owen Omalley) [prasadm] HIVE-4395: Support TFetchOrientation.FIRST for HiveServer2 FetchResults (Prasad Mujumdar reviewed by Thejas Nair) [ehans] HIVE-5756: Implement vectorized support for IF conditional expression (Eric Hanson) [hashutosh] HIVE-6018 : FetchTask should not reference metastore classes (Navis via Prasad Mujumdar) [hashutosh] HIVE-5979. Failure in cast to timestamps. (Jitendra Pandey) [hashutosh] HIVE-5897 : Fix hadoop2 execution environment Milestone 2 (Vikram Dixit via Brock Noland) Changes for Build #2497 Changes for Build #2498 [hashutosh] HIVE-6004 : Fix statistics annotation related test failures in hadoop2 (Prasanth J via Ashutosh Chauhan) [hashutosh] HIVE-6027 : non-vectorized log10 has rounding issue (Sergey Shelukhin via Ashutosh Chauhan) [prasadm] HIVE-5993: JDBC Driver should not hard-code the database name (Szehon Ho via Prasad Mujumdar) Changes for Build #2499 [navis] HIVE-5985 : Make qfile_regex to accept multiple patterns (Navis reviewed by Ashutosh Chauhan) Changes for Build #2500 Changes for Build #2501 Changes for Build #2502 [navis] HIVE-5276 : Skip redundant string encoding/decoding for hiveserver2 (Navis Reviewed by Carl Steinbach) Changes for Build #2503 [xuefu] HIVE-6022: Load statements with incorrect order of partitions put input files to unreadable places (Teruyoshi Zenmyo via Xuefu) Changes for Build #2504 Changes for Build #2505 [thejas] HIVE-5975 : [WebHCat] templeton mapreduce job failed if provide define parameters (Shanyu Zhao via Thejas Nair) Changes for Build #2506 [prasadm] HIVE-1466: Add NULL DEFINED AS to ROW FORMAT specification
[jira] [Commented] (HIVE-6044) webhcat should be able to return detailed serde information when show table using fromat=extended
[ https://issues.apache.org/jira/browse/HIVE-6044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13856937#comment-13856937 ] Thejas M Nair commented on HIVE-6044: - +1 webhcat should be able to return detailed serde information when show table using fromat=extended --- Key: HIVE-6044 URL: https://issues.apache.org/jira/browse/HIVE-6044 Project: Hive Issue Type: Bug Reporter: Shuaishuai Nie Assignee: Shuaishuai Nie Attachments: HIVE-6044.1.patch Now in webhcat, when using GET ddl/database/:db/table/:table and format=extended, return value is based on query show table extended like. However, this query doesn't contains serde info like line.delim and filed.delim. In this case, user won't have enough information to reconstruct the exact same table based on the information from the json file. The descExtendedTable function in HcatDelegator should also return extra fields from query desc extended tablename which contains fields sd, retention, parameters parametersSize and tableType. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5958) SQL std auth - disable statements that work with paths
[ https://issues.apache.org/jira/browse/HIVE-5958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13856944#comment-13856944 ] Thejas M Nair commented on HIVE-5958: - I mean, in the first iteration, I am planning to add support for URI as an object to be authorized, but not supporting grant/revoke statements and authorization checks for it. So I am thinking of just returning access denied error for URI in first iteration. I understand it will limit the functionality available with the first iteration. What is available in first release would depend on when the next hive release happens. SQL std auth - disable statements that work with paths -- Key: HIVE-5958 URL: https://issues.apache.org/jira/browse/HIVE-5958 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Original Estimate: 24h Remaining Estimate: 24h In the first pass, statement such as create table, alter table that specify an path uri will get an authorization error under SQL std auth . -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HIVE-6110) schematool errors out when HIVE_OPTS is set
Venki Korukanti created HIVE-6110:
-
Summary: schematool errors out when HIVE_OPTS is set
Key: HIVE-6110
URL: https://issues.apache.org/jira/browse/HIVE-6110
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.12.0
Reporter: Venki Korukanti
Assignee: Venki Korukanti
Fix For: 0.13.0
If HIVE_OPTS is set schematool command fails to start. To repro this set
HIVE_AUX_JARS_PATH
{code}
$ export HIVE_AUX_JARS_PATH=/path/lib.jar
{code}
Above export adds a --hiveconf option to the commandline args to HiveSchemaTool
which doesn't recognize it.
{code}
$ hive-install-dir/bin/schematool -dbType derby -info
HiveSchemaTool:Parsing failed. Reason: Unrecognized option: -hiveconf
usage: schemaTool
-dbType databaseType Metastore database type
-dryRunlist SQL scripts (no execute)
-help print this message
-info Show config and schema details
-initSchemaSchema initialization
-initSchemaTo initTo Schema initialization to a version
-passWord password Override config file password
-upgradeSchema Schema upgrade
-upgradeSchemaFrom upgradeFrom Schema upgrade from a version
-userName user Override config file user name
-verbose only print SQL statements
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HIVE-6110) schematool errors out when HIVE_OPTS is set
[
https://issues.apache.org/jira/browse/HIVE-6110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venki Korukanti updated HIVE-6110:
--
Attachment: HIVE-6110.1.patch.txt
schematool errors out when HIVE_OPTS is set
---
Key: HIVE-6110
URL: https://issues.apache.org/jira/browse/HIVE-6110
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.12.0
Reporter: Venki Korukanti
Assignee: Venki Korukanti
Fix For: 0.13.0
Attachments: HIVE-6110.1.patch.txt
If HIVE_OPTS is set schematool command fails to start. To repro this set
HIVE_AUX_JARS_PATH
{code}
$ export HIVE_AUX_JARS_PATH=/path/lib.jar
{code}
Above export adds a --hiveconf option to the commandline args to
HiveSchemaTool which doesn't recognize it.
{code}
$ hive-install-dir/bin/schematool -dbType derby -info
HiveSchemaTool:Parsing failed. Reason: Unrecognized option: -hiveconf
usage: schemaTool
-dbType databaseType Metastore database type
-dryRunlist SQL scripts (no execute)
-help print this message
-info Show config and schema details
-initSchemaSchema initialization
-initSchemaTo initTo Schema initialization to a version
-passWord password Override config file password
-upgradeSchema Schema upgrade
-upgradeSchemaFrom upgradeFrom Schema upgrade from a version
-userName user Override config file user name
-verbose only print SQL statements
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-6110) schematool errors out when HIVE_OPTS is set
[
https://issues.apache.org/jira/browse/HIVE-6110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13856979#comment-13856979
]
Venki Korukanti commented on HIVE-6110:
---
Reset HIVE_OPTS before calling execHiveCmd
schematool errors out when HIVE_OPTS is set
---
Key: HIVE-6110
URL: https://issues.apache.org/jira/browse/HIVE-6110
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.12.0
Reporter: Venki Korukanti
Assignee: Venki Korukanti
Fix For: 0.13.0
Attachments: HIVE-6110.1.patch.txt
If HIVE_OPTS is set schematool command fails to start. To repro this set
HIVE_AUX_JARS_PATH
{code}
$ export HIVE_AUX_JARS_PATH=/path/lib.jar
{code}
Above export adds a --hiveconf option to the commandline args to
HiveSchemaTool which doesn't recognize it.
{code}
$ hive-install-dir/bin/schematool -dbType derby -info
HiveSchemaTool:Parsing failed. Reason: Unrecognized option: -hiveconf
usage: schemaTool
-dbType databaseType Metastore database type
-dryRunlist SQL scripts (no execute)
-help print this message
-info Show config and schema details
-initSchemaSchema initialization
-initSchemaTo initTo Schema initialization to a version
-passWord password Override config file password
-upgradeSchema Schema upgrade
-upgradeSchemaFrom upgradeFrom Schema upgrade from a version
-userName user Override config file user name
-verbose only print SQL statements
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HIVE-6110) schematool errors out when HIVE_OPTS is set
[
https://issues.apache.org/jira/browse/HIVE-6110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venki Korukanti updated HIVE-6110:
--
Status: Patch Available (was: Open)
schematool errors out when HIVE_OPTS is set
---
Key: HIVE-6110
URL: https://issues.apache.org/jira/browse/HIVE-6110
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.12.0
Reporter: Venki Korukanti
Assignee: Venki Korukanti
Fix For: 0.13.0
Attachments: HIVE-6110.1.patch.txt
If HIVE_OPTS is set schematool command fails to start. To repro this set
HIVE_AUX_JARS_PATH
{code}
$ export HIVE_AUX_JARS_PATH=/path/lib.jar
{code}
Above export adds a --hiveconf option to the commandline args to
HiveSchemaTool which doesn't recognize it.
{code}
$ hive-install-dir/bin/schematool -dbType derby -info
HiveSchemaTool:Parsing failed. Reason: Unrecognized option: -hiveconf
usage: schemaTool
-dbType databaseType Metastore database type
-dryRunlist SQL scripts (no execute)
-help print this message
-info Show config and schema details
-initSchemaSchema initialization
-initSchemaTo initTo Schema initialization to a version
-passWord password Override config file password
-upgradeSchema Schema upgrade
-upgradeSchemaFrom upgradeFrom Schema upgrade from a version
-userName user Override config file user name
-verbose only print SQL statements
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
Re: Review Request 16184: Hive should be able to skip header and footer rows when reading data file for a table (HIVE-5795)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16184/#review30872 --- ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java https://reviews.apache.org/r/16184/#comment59093 It would be better to have FooterBuffer as a top level class in this package (...ql.exec) ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java https://reviews.apache.org/r/16184/#comment59099 indentation needs fixing here ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java https://reviews.apache.org/r/16184/#comment59103 can you please change the variable name from ret to something more descriptive like notEOF ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java https://reviews.apache.org/r/16184/#comment59104 return false, is little more easy to read ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java https://reviews.apache.org/r/16184/#comment59100 indentation needs fixing here ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java https://reviews.apache.org/r/16184/#comment59106 can you please change the variable name from ret to something more descriptive like notEOF ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java https://reviews.apache.org/r/16184/#comment59107 this else block looks unnecessary (as mentioned in my previous review) ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java https://reviews.apache.org/r/16184/#comment59105 indentation needs fixing here - Thejas Nair On Dec. 19, 2013, 12:40 a.m., Shuaishuai Nie wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16184/ --- (Updated Dec. 19, 2013, 12:40 a.m.) Review request for hive, Eric Hanson and Thejas Nair. Bugs: hive-5795 https://issues.apache.org/jira/browse/hive-5795 Repository: hive-git Description --- Hive should be able to skip header and footer rows when reading data file for a table (follow up with review https://reviews.apache.org/r/15663/diff/#index_header) Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fa3e048 conf/hive-default.xml.template 1b30d19 data/files/header_footer_table_1/0001.txt PRE-CREATION data/files/header_footer_table_1/0002.txt PRE-CREATION data/files/header_footer_table_1/0003.txt PRE-CREATION data/files/header_footer_table_2/2012/01/01/0001.txt PRE-CREATION data/files/header_footer_table_2/2012/01/02/0002.txt PRE-CREATION data/files/header_footer_table_2/2012/01/03/0003.txt PRE-CREATION itests/qtest/pom.xml 971c5d3 ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java fc9b7e4 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 9afc80b ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java dd5cb6b ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 974a5d6 ql/src/test/org/apache/hadoop/hive/ql/io/TestHiveBinarySearchRecordReader.java 85dd975 ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java 0686d9b ql/src/test/queries/clientnegative/file_with_header_footer_negative.q PRE-CREATION ql/src/test/queries/clientpositive/file_with_header_footer.q PRE-CREATION ql/src/test/results/clientnegative/file_with_header_footer_negative.q.out PRE-CREATION ql/src/test/results/clientpositive/file_with_header_footer.q.out PRE-CREATION serde/if/serde.thrift 2ceb572 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/serdeConstants.java 22a6168 Diff: https://reviews.apache.org/r/16184/diff/ Testing --- Thanks, Shuaishuai Nie
Re: Review Request 16184: Hive should be able to skip header and footer rows when reading data file for a table (HIVE-5795)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16184/#review30876 --- ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java https://reviews.apache.org/r/16184/#comment59110 how about using the new utilitiy functions from here as well? - Thejas Nair On Dec. 19, 2013, 12:40 a.m., Shuaishuai Nie wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16184/ --- (Updated Dec. 19, 2013, 12:40 a.m.) Review request for hive, Eric Hanson and Thejas Nair. Bugs: hive-5795 https://issues.apache.org/jira/browse/hive-5795 Repository: hive-git Description --- Hive should be able to skip header and footer rows when reading data file for a table (follow up with review https://reviews.apache.org/r/15663/diff/#index_header) Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fa3e048 conf/hive-default.xml.template 1b30d19 data/files/header_footer_table_1/0001.txt PRE-CREATION data/files/header_footer_table_1/0002.txt PRE-CREATION data/files/header_footer_table_1/0003.txt PRE-CREATION data/files/header_footer_table_2/2012/01/01/0001.txt PRE-CREATION data/files/header_footer_table_2/2012/01/02/0002.txt PRE-CREATION data/files/header_footer_table_2/2012/01/03/0003.txt PRE-CREATION itests/qtest/pom.xml 971c5d3 ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java fc9b7e4 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 9afc80b ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java dd5cb6b ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 974a5d6 ql/src/test/org/apache/hadoop/hive/ql/io/TestHiveBinarySearchRecordReader.java 85dd975 ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java 0686d9b ql/src/test/queries/clientnegative/file_with_header_footer_negative.q PRE-CREATION ql/src/test/queries/clientpositive/file_with_header_footer.q PRE-CREATION ql/src/test/results/clientnegative/file_with_header_footer_negative.q.out PRE-CREATION ql/src/test/results/clientpositive/file_with_header_footer.q.out PRE-CREATION serde/if/serde.thrift 2ceb572 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/serdeConstants.java 22a6168 Diff: https://reviews.apache.org/r/16184/diff/ Testing --- Thanks, Shuaishuai Nie
[jira] [Updated] (HIVE-6095) Use paths consistently II
[ https://issues.apache.org/jira/browse/HIVE-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6095: --- Status: Open (was: Patch Available) Use paths consistently II - Key: HIVE-6095 URL: https://issues.apache.org/jira/browse/HIVE-6095 Project: Hive Issue Type: Improvement Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Priority: Minor Attachments: HIVE-6095.2.patch, HIVE-6095.patch This is follow-up of HIVE-3616. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6095) Use paths consistently II
[ https://issues.apache.org/jira/browse/HIVE-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6095: --- Attachment: HIVE-6095.2.patch Use paths consistently II - Key: HIVE-6095 URL: https://issues.apache.org/jira/browse/HIVE-6095 Project: Hive Issue Type: Improvement Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Priority: Minor Attachments: HIVE-6095.2.patch, HIVE-6095.patch This is follow-up of HIVE-3616. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6095) Use paths consistently II
[ https://issues.apache.org/jira/browse/HIVE-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6095: --- Status: Patch Available (was: Open) Use paths consistently II - Key: HIVE-6095 URL: https://issues.apache.org/jira/browse/HIVE-6095 Project: Hive Issue Type: Improvement Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Priority: Minor Attachments: HIVE-6095.2.patch, HIVE-6095.patch This is follow-up of HIVE-3616. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6110) schematool errors out when HIVE_OPTS is set
[
https://issues.apache.org/jira/browse/HIVE-6110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857021#comment-13857021
]
Hive QA commented on HIVE-6110:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12620521/HIVE-6110.1.patch.txt
{color:green}SUCCESS:{color} +1 4818 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/743/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/743/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12620521
schematool errors out when HIVE_OPTS is set
---
Key: HIVE-6110
URL: https://issues.apache.org/jira/browse/HIVE-6110
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.12.0
Reporter: Venki Korukanti
Assignee: Venki Korukanti
Fix For: 0.13.0
Attachments: HIVE-6110.1.patch.txt
If HIVE_OPTS is set schematool command fails to start. To repro this set
HIVE_AUX_JARS_PATH
{code}
$ export HIVE_AUX_JARS_PATH=/path/lib.jar
{code}
Above export adds a --hiveconf option to the commandline args to
HiveSchemaTool which doesn't recognize it.
{code}
$ hive-install-dir/bin/schematool -dbType derby -info
HiveSchemaTool:Parsing failed. Reason: Unrecognized option: -hiveconf
usage: schemaTool
-dbType databaseType Metastore database type
-dryRunlist SQL scripts (no execute)
-help print this message
-info Show config and schema details
-initSchemaSchema initialization
-initSchemaTo initTo Schema initialization to a version
-passWord password Override config file password
-upgradeSchema Schema upgrade
-upgradeSchemaFrom upgradeFrom Schema upgrade from a version
-userName user Override config file user name
-verbose only print SQL statements
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
Please make Rajesh Balamohan a contributor
Thanks!!
[jira] [Updated] (HIVE-5524) Unwanted delay in getting Hive metastore connection with METASTORE_CLIENT_CONNECT_RETRY_DELAY/
[
https://issues.apache.org/jira/browse/HIVE-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-5524:
---
Resolution: Fixed
Fix Version/s: 0.13.0
Status: Resolved (was: Patch Available)
I have committed this to trunk but I am unable to assign to Rajesh since he is
not a contributor in JIRA. I have requested Rajesh be added as a contributor
and will give proper credit when that is done.
Thank you for the contribution Rajesh!
Unwanted delay in getting Hive metastore connection with
METASTORE_CLIENT_CONNECT_RETRY_DELAY/
--
Key: HIVE-5524
URL: https://issues.apache.org/jira/browse/HIVE-5524
Project: Hive
Issue Type: Bug
Affects Versions: 0.10.0
Reporter: Rajesh Balamohan
Fix For: 0.13.0
Attachments: HIVE-5524.patch
Reference:
http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
for (URI store : metastoreUris) {
...
if (isConnected) {
break;
}
}
// Wait before launching the next round of connection retries.
if (retryDelaySeconds 0) {
try {
LOG.info(Waiting + retryDelaySeconds + seconds before next
connection attempt.);
Thread.sleep(retryDelaySeconds * 1000);
} catch (InterruptedException ignore) {}
}
By default hive.metastore.client.connect.retry.delay is set to 1 second.
If it is set to 10 seconds, this code will wait for 10 seconds even if a
successful connection is made in first attempt itself.
This can be avoided by changing to
if (!isConnected retryDelaySeconds 0) {
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HIVE-5524) Unwanted delay in getting Hive metastore connection with METASTORE_CLIENT_CONNECT_RETRY_DELAY/
[
https://issues.apache.org/jira/browse/HIVE-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-5524:
---
Assignee: Rajesh Balamohan
Unwanted delay in getting Hive metastore connection with
METASTORE_CLIENT_CONNECT_RETRY_DELAY/
--
Key: HIVE-5524
URL: https://issues.apache.org/jira/browse/HIVE-5524
Project: Hive
Issue Type: Bug
Affects Versions: 0.10.0
Reporter: Rajesh Balamohan
Assignee: Rajesh Balamohan
Fix For: 0.13.0
Attachments: HIVE-5524.patch
Reference:
http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
for (URI store : metastoreUris) {
...
if (isConnected) {
break;
}
}
// Wait before launching the next round of connection retries.
if (retryDelaySeconds 0) {
try {
LOG.info(Waiting + retryDelaySeconds + seconds before next
connection attempt.);
Thread.sleep(retryDelaySeconds * 1000);
} catch (InterruptedException ignore) {}
}
By default hive.metastore.client.connect.retry.delay is set to 1 second.
If it is set to 10 seconds, this code will wait for 10 seconds even if a
successful connection is made in first attempt itself.
This can be avoided by changing to
if (!isConnected retryDelaySeconds 0) {
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
Hive-trunk-h0.21 - Build # 2527 - Still Failing
Changes for Build #2488 [hashutosh] HIVE-5935 : hive.query.string is not provided to FetchTask (Navis via Ashutosh Chauhan) [navis] HIVE-3455 : ANSI CORR(X,Y) is incorrect (Maxim Bolotin via Navis) [hashutosh] HIVE-5921 : Better heuristics for worst case statistics estimates for join, limit and filter operator (Prasanth J via Harish Butani) [rhbutani] HIVE-5899 NPE during explain extended with char/varchar columns (Jason Dere via Harish Butani) Changes for Build #2489 [xuefu] HIVE-3181: getDatabaseMajor/Minor version does not return values (Szehon via Xuefu, reviewed by Navis) [brock] HIVE-5641 - BeeLineOpts ignores Throwable (Brock Noland reviewed by Prasad and Thejas) [hashutosh] HIVE-5909 : locate and instr throw java.nio.BufferUnderflowException when empty string as substring (Navis via Ashutosh Chauhan) [hashutosh] HIVE-5686 : partition column type validation doesn't quite work for dates (Sergey Shelukhin via Ashutosh Chauhan) [hashutosh] HIVE-5887 : metastore direct sql doesn't work with oracle (Sergey Shelukhin via Ashutosh Chauhan) Changes for Build #2490 Changes for Build #2491 Changes for Build #2492 [brock] HIVE-5981 - Add hive-unit back to itests pom (Brock Noland reviewed by Prasad) Changes for Build #2493 [xuefu] HIVE-5872: Make UDAFs such as GenericUDAFSum report accurate precision/scale for decimal types (reviewed by Sergey Shelukhin) [hashutosh] HIVE-5978 : Rollups not supported in vector mode. (Jitendra Nath Pandey via Ashutosh Chauhan) [hashutosh] HIVE-5830 : SubQuery: Not In subqueries should check if subquery contains nulls in matching column (Harish Butani via Ashutosh Chauhan) [hashutosh] HIVE-5598 : Remove dummy new line at the end of non-sql commands (Navis via Ashutosh Chauhan) Changes for Build #2494 [hashutosh] HIVE-5982 : Remove redundant filesystem operations and methods in FileSink (Ashutosh Chauhan via Thejas Nair) [navis] HIVE-5955 : decimal_precision.q test case fails in trunk (Prasanth J via Navis) [brock] HIVE-5983 - Fix name of ColumnProjectionUtils.appendReadColumnIDs (Brock Noland reviewed by Navis) Changes for Build #2495 [omalley] HIVE-5580. Predicate pushdown predicates with an and-operator between non-SARGable predicates cause a NPE. (omalley) Changes for Build #2496 [gunther] HIVE-6000: Hive build broken on hadoop2 (Vikram Dixit K via Gunther Hagleitner [gunther] HIVE-2093: UPDATE - add two missing files from previous commit (Gunther Hagleitner) [thejas] HIVE-2093 : create/drop database should populate inputs/outputs and check concurrency and user permission (Navis via Thejas Nair) [hashutosh] HIVE-6016 : Hadoop23Shims has a bug in listLocatedStatus impl. (Prasanth J via Ashutosh Chauhan) [hashutosh] HIVE-5994 : ORC RLEv2 encodes wrongly for large negative BIGINTs (64 bits ) (Prasanth J via Owen Omalley) [hashutosh] HIVE-5991 : ORC RLEv2 fails with ArrayIndexOutOfBounds exception for PATCHED_BLOB encoding (Prasanth J via Owen Omalley) [prasadm] HIVE-4395: Support TFetchOrientation.FIRST for HiveServer2 FetchResults (Prasad Mujumdar reviewed by Thejas Nair) [ehans] HIVE-5756: Implement vectorized support for IF conditional expression (Eric Hanson) [hashutosh] HIVE-6018 : FetchTask should not reference metastore classes (Navis via Prasad Mujumdar) [hashutosh] HIVE-5979. Failure in cast to timestamps. (Jitendra Pandey) [hashutosh] HIVE-5897 : Fix hadoop2 execution environment Milestone 2 (Vikram Dixit via Brock Noland) Changes for Build #2497 Changes for Build #2498 [hashutosh] HIVE-6004 : Fix statistics annotation related test failures in hadoop2 (Prasanth J via Ashutosh Chauhan) [hashutosh] HIVE-6027 : non-vectorized log10 has rounding issue (Sergey Shelukhin via Ashutosh Chauhan) [prasadm] HIVE-5993: JDBC Driver should not hard-code the database name (Szehon Ho via Prasad Mujumdar) Changes for Build #2499 [navis] HIVE-5985 : Make qfile_regex to accept multiple patterns (Navis reviewed by Ashutosh Chauhan) Changes for Build #2500 Changes for Build #2501 Changes for Build #2502 [navis] HIVE-5276 : Skip redundant string encoding/decoding for hiveserver2 (Navis Reviewed by Carl Steinbach) Changes for Build #2503 [xuefu] HIVE-6022: Load statements with incorrect order of partitions put input files to unreadable places (Teruyoshi Zenmyo via Xuefu) Changes for Build #2504 Changes for Build #2505 [thejas] HIVE-5975 : [WebHCat] templeton mapreduce job failed if provide define parameters (Shanyu Zhao via Thejas Nair) Changes for Build #2506 [prasadm] HIVE-1466: Add NULL DEFINED AS to ROW FORMAT specification (Prasad Mujumdar reviewed by Xuefu Zhang) Changes for Build #2507 [jitendra] HIVE-5521 : Remove CommonRCFileInputFormat. (hashutosh via jitendra) [rhbutani] HIVE-5973 SMB joins produce incorrect results with multiple partitions and buckets (Vikram Dixit via Harish Butani) [ehans] HIVE-6015: vectorized logarithm produces results for 0 that are different from a
[jira] [Commented] (HIVE-6095) Use paths consistently II
[
https://issues.apache.org/jira/browse/HIVE-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857054#comment-13857054
]
Hive QA commented on HIVE-6095:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12620530/HIVE-6095.2.patch
{color:red}ERROR:{color} -1 due to 119 failed/errored test(s), 4818 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_case_sensitivity
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cast1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_colname
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_hadoop20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_uses_database_location
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_file_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables_compact
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_multiple
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_self_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_update
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_auto_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_compression
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_compression
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input12_hadoop20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input34
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input35
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input36
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input38
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_dynamicserde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_testsequencefile
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_testxpath
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_testxpath2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_into4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_into5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_into6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join27
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join36
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join37
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join39
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lineage1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert_lateral_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert_move_tasks_share_dependencies
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonmr_fetch
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullformatCTAS
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_createas1
Hive-trunk-hadoop2 - Build # 626 - Still Failing
Changes for Build #586 [hashutosh] HIVE-5935 : hive.query.string is not provided to FetchTask (Navis via Ashutosh Chauhan) [navis] HIVE-3455 : ANSI CORR(X,Y) is incorrect (Maxim Bolotin via Navis) [hashutosh] HIVE-5921 : Better heuristics for worst case statistics estimates for join, limit and filter operator (Prasanth J via Harish Butani) [rhbutani] HIVE-5899 NPE during explain extended with char/varchar columns (Jason Dere via Harish Butani) Changes for Build #587 [xuefu] HIVE-3181: getDatabaseMajor/Minor version does not return values (Szehon via Xuefu, reviewed by Navis) [brock] HIVE-5641 - BeeLineOpts ignores Throwable (Brock Noland reviewed by Prasad and Thejas) [hashutosh] HIVE-5909 : locate and instr throw java.nio.BufferUnderflowException when empty string as substring (Navis via Ashutosh Chauhan) [hashutosh] HIVE-5686 : partition column type validation doesn't quite work for dates (Sergey Shelukhin via Ashutosh Chauhan) [hashutosh] HIVE-5887 : metastore direct sql doesn't work with oracle (Sergey Shelukhin via Ashutosh Chauhan) Changes for Build #588 Changes for Build #589 Changes for Build #590 [brock] HIVE-5981 - Add hive-unit back to itests pom (Brock Noland reviewed by Prasad) Changes for Build #591 [xuefu] HIVE-5872: Make UDAFs such as GenericUDAFSum report accurate precision/scale for decimal types (reviewed by Sergey Shelukhin) [hashutosh] HIVE-5978 : Rollups not supported in vector mode. (Jitendra Nath Pandey via Ashutosh Chauhan) [hashutosh] HIVE-5830 : SubQuery: Not In subqueries should check if subquery contains nulls in matching column (Harish Butani via Ashutosh Chauhan) [hashutosh] HIVE-5598 : Remove dummy new line at the end of non-sql commands (Navis via Ashutosh Chauhan) Changes for Build #592 [hashutosh] HIVE-5982 : Remove redundant filesystem operations and methods in FileSink (Ashutosh Chauhan via Thejas Nair) [navis] HIVE-5955 : decimal_precision.q test case fails in trunk (Prasanth J via Navis) [brock] HIVE-5983 - Fix name of ColumnProjectionUtils.appendReadColumnIDs (Brock Noland reviewed by Navis) Changes for Build #593 [omalley] HIVE-5580. Predicate pushdown predicates with an and-operator between non-SARGable predicates cause a NPE. (omalley) Changes for Build #594 [gunther] HIVE-6000: Hive build broken on hadoop2 (Vikram Dixit K via Gunther Hagleitner [gunther] HIVE-2093: UPDATE - add two missing files from previous commit (Gunther Hagleitner) [thejas] HIVE-2093 : create/drop database should populate inputs/outputs and check concurrency and user permission (Navis via Thejas Nair) [hashutosh] HIVE-6016 : Hadoop23Shims has a bug in listLocatedStatus impl. (Prasanth J via Ashutosh Chauhan) [hashutosh] HIVE-5994 : ORC RLEv2 encodes wrongly for large negative BIGINTs (64 bits ) (Prasanth J via Owen Omalley) [hashutosh] HIVE-5991 : ORC RLEv2 fails with ArrayIndexOutOfBounds exception for PATCHED_BLOB encoding (Prasanth J via Owen Omalley) [prasadm] HIVE-4395: Support TFetchOrientation.FIRST for HiveServer2 FetchResults (Prasad Mujumdar reviewed by Thejas Nair) [ehans] HIVE-5756: Implement vectorized support for IF conditional expression (Eric Hanson) [hashutosh] HIVE-6018 : FetchTask should not reference metastore classes (Navis via Prasad Mujumdar) [hashutosh] HIVE-5979. Failure in cast to timestamps. (Jitendra Pandey) [hashutosh] HIVE-5897 : Fix hadoop2 execution environment Milestone 2 (Vikram Dixit via Brock Noland) Changes for Build #595 Changes for Build #596 [hashutosh] HIVE-6027 : non-vectorized log10 has rounding issue (Sergey Shelukhin via Ashutosh Chauhan) [prasadm] HIVE-5993: JDBC Driver should not hard-code the database name (Szehon Ho via Prasad Mujumdar) Changes for Build #597 [hashutosh] HIVE-6004 : Fix statistics annotation related test failures in hadoop2 (Prasanth J via Ashutosh Chauhan) Changes for Build #598 [navis] HIVE-5985 : Make qfile_regex to accept multiple patterns (Navis reviewed by Ashutosh Chauhan) Changes for Build #599 Changes for Build #600 Changes for Build #601 [navis] HIVE-5276 : Skip redundant string encoding/decoding for hiveserver2 (Navis Reviewed by Carl Steinbach) Changes for Build #602 [xuefu] HIVE-6022: Load statements with incorrect order of partitions put input files to unreadable places (Teruyoshi Zenmyo via Xuefu) Changes for Build #603 Changes for Build #604 [thejas] HIVE-5975 : [WebHCat] templeton mapreduce job failed if provide define parameters (Shanyu Zhao via Thejas Nair) Changes for Build #605 [prasadm] HIVE-1466: Add NULL DEFINED AS to ROW FORMAT specification (Prasad Mujumdar reviewed by Xuefu Zhang) Changes for Build #606 [jitendra] HIVE-5521 : Remove CommonRCFileInputFormat. (hashutosh via jitendra) [rhbutani] HIVE-5973 SMB joins produce incorrect results with multiple partitions and buckets (Vikram Dixit via Harish Butani) [ehans] HIVE-6015: vectorized logarithm produces results for 0 that are different from a
[jira] [Commented] (HIVE-5941) SQL std auth - support 'show all roles'
[ https://issues.apache.org/jira/browse/HIVE-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857106#comment-13857106 ] Thejas M Nair commented on HIVE-5941: - [~navis] I have added some minor comments to HIVE-5414, will you be able to address those , so that we can get your changes for 'show all roles' in ? SQL std auth - support 'show all roles' --- Key: HIVE-5941 URL: https://issues.apache.org/jira/browse/HIVE-5941 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Original Estimate: 24h Remaining Estimate: 24h SHOW ALL ROLES - This will list all currently existing roles. This will be available only to the superuser. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Re: Please make Rajesh Balamohan a contributor
Hi Brock, I wasn't able to find Rajesh's JIRA Id, but you should now have the JIRA karma necessary to add him yourself. Thanks. Carl On Thu, Dec 26, 2013 at 11:45 AM, Brock Noland br...@cloudera.com wrote: Thanks!!
Re: Please make Rajesh Balamohan a contributor
Jeez...I do! Ashutosh assigned the JIRA to him so I think he took care of it. Thank you and sorry for the spam! On Thu, Dec 26, 2013 at 4:20 PM, Carl Steinbach cwsteinb...@gmail.comwrote: Hi Brock, I wasn't able to find Rajesh's JIRA Id, but you should now have the JIRA karma necessary to add him yourself. Thanks. Carl On Thu, Dec 26, 2013 at 11:45 AM, Brock Noland br...@cloudera.com wrote: Thanks!! -- Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
[jira] [Assigned] (HIVE-6088) Change StreamingDelegator to pass distributed cache files to jar delegator
[ https://issues.apache.org/jira/browse/HIVE-6088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan reassigned HIVE-6088: --- Assignee: Hari Sankar Sivarama Subramaniyan Change StreamingDelegator to pass distributed cache files to jar delegator -- Key: HIVE-6088 URL: https://issues.apache.org/jira/browse/HIVE-6088 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (HIVE-6075) Remove hive.metastore.warehouse.dir from webhcat config
[ https://issues.apache.org/jira/browse/HIVE-6075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan reassigned HIVE-6075: --- Assignee: Hari Sankar Sivarama Subramaniyan Remove hive.metastore.warehouse.dir from webhcat config - Key: HIVE-6075 URL: https://issues.apache.org/jira/browse/HIVE-6075 Project: Hive Issue Type: Task Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6104) Join-key logging in join operator
[ https://issues.apache.org/jira/browse/HIVE-6104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Wong updated HIVE-6104: -- Fix Version/s: 0.13.0 Join-key logging in join operator - Key: HIVE-6104 URL: https://issues.apache.org/jira/browse/HIVE-6104 Project: Hive Issue Type: Bug Components: Diagnosability Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Steven Wong Assignee: Steven Wong Priority: Minor Fix For: 0.13.0 Attachments: HIVE-6104.patch JoinOperator.processOp logs lines like table 0 has x rows for join key \[foo\]. It is supposed to log after x rows for x = i, 2i, 4i, However, it has never worked completely: * In 0.11.0 and before, it logs after i rows and not after i rows, because nextSz is not properly updated. * In 0.12.0, HIVE-4960 partially fixed that but x fails to be reset when the alias (tag) changes. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5970) ArrayIndexOutOfBoundsException in RunLengthIntegerReaderV2.java
[ https://issues.apache.org/jira/browse/HIVE-5970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857166#comment-13857166 ] Lefty Leverenz commented on HIVE-5970: -- The wiki does not document hive.exec.orc.write.format, nor any other ORC configuration parameters. I found these in HiveConf.java with an orc search: * hive.exec.orc.memory.pool * hive.exec.orc.write.format * hive.exec.orc.default.stripe.size Also, hive.default.fileformat can take the value ORC. Are there any others? ArrayIndexOutOfBoundsException in RunLengthIntegerReaderV2.java --- Key: HIVE-5970 URL: https://issues.apache.org/jira/browse/HIVE-5970 Project: Hive Issue Type: Bug Components: File Formats Affects Versions: 0.12.0 Reporter: Eric Chu Priority: Critical Labels: orcfile Attachments: test_data A workload involving ORC tables starts getting the following ArrayIndexOutOfBoundsException AFTER the upgrade to Hive 0.12. The file is added as part of HIVE-4123. 2013-12-04 14:42:08,537 ERROR cause:java.io.IOException: java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: 0 2013-12-04 14:42:08,537 WARN org.apache.hadoop.mapred.Child: Error running child java.io.IOException: java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: 0 at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:304) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:220) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:215) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:200) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by: java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: 0 at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77) at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:276) at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:101) at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:41) at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:108) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:302) ... 11 more Caused by: java.lang.ArrayIndexOutOfBoundsException: 0 at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.readPatchedBaseValues(RunLengthIntegerReaderV2.java:171) at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.readValues(RunLengthIntegerReaderV2.java:54) at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.next(RunLengthIntegerReaderV2.java:287) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$LongTreeReader.next(RecordReaderImpl.java:473) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StructTreeReader.next(RecordReaderImpl.java:1157) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:2196) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:129) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:80) at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:274) ... 15 more -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-5702) add logging on WebHCat startup to print all env var and Configuration
[
https://issues.apache.org/jira/browse/HIVE-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-5702:
-
Resolution: Fixed
Fix Version/s: 0.13.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
+1.
Patch committed to trunk.
add logging on WebHCat startup to print all env var and Configuration
-
Key: HIVE-5702
URL: https://issues.apache.org/jira/browse/HIVE-5702
Project: Hive
Issue Type: Bug
Components: WebHCat
Affects Versions: 0.13.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Priority: Minor
Fix For: 0.13.0
Attachments: HIVE-5702.patch
this helps debugging/diagnosing issues
Something like:
MapString, String env = System.getenv();
ListString propKeys = new ArrayListString(env.keySet());
Collections.sort(propKeys);
LOG.info(WebHCat environment:);
for(String propKey : propKeys) {
LOG.info(propKey + = + env.get(propKey));
}
LOG.info(Configuration props:);
IteratorMap.EntryString, String configProps = appConf.iterator();
while(configProps.hasNext()) {
Map.EntryString, String ent = configProps.next();
LOG.info(ent.getKey() + = + ent.getValue());
}
Could be useful to add a REST call to dump such info as well (is security a
concern here?)
Also, make sure to log what config files it attempted to load and which it
actually did load.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (HIVE-6111) SQL std auth - support granted-by in grant statements
Thejas M Nair created HIVE-6111: --- Summary: SQL std auth - support granted-by in grant statements Key: HIVE-6111 URL: https://issues.apache.org/jira/browse/HIVE-6111 Project: Hive Issue Type: Sub-task Reporter: Thejas M Nair sub-task includes parser changes -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HIVE-6112) SQL std auth - support new privileges INSERT, DELETE
Thejas M Nair created HIVE-6112: --- Summary: SQL std auth - support new privileges INSERT, DELETE Key: HIVE-6112 URL: https://issues.apache.org/jira/browse/HIVE-6112 Project: Hive Issue Type: Sub-task Reporter: Thejas M Nair Includes INSERT, DELETE privileges. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6112) SQL std auth - support new privileges INSERT, DELETE
[ https://issues.apache.org/jira/browse/HIVE-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-6112: Attachment: new-privs.patch SQL std auth - support new privileges INSERT, DELETE Key: HIVE-6112 URL: https://issues.apache.org/jira/browse/HIVE-6112 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Attachments: new-privs.patch Includes INSERT, DELETE privileges. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6112) SQL std auth - support new privileges INSERT, DELETE
[ https://issues.apache.org/jira/browse/HIVE-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-6112: Remaining Estimate: 12h Original Estimate: 12h SQL std auth - support new privileges INSERT, DELETE Key: HIVE-6112 URL: https://issues.apache.org/jira/browse/HIVE-6112 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Attachments: new-privs.patch Original Estimate: 12h Remaining Estimate: 12h Includes INSERT, DELETE privileges. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Hive-trunk-hadoop2 - Build # 627 - Still Failing
Changes for Build #586 [hashutosh] HIVE-5935 : hive.query.string is not provided to FetchTask (Navis via Ashutosh Chauhan) [navis] HIVE-3455 : ANSI CORR(X,Y) is incorrect (Maxim Bolotin via Navis) [hashutosh] HIVE-5921 : Better heuristics for worst case statistics estimates for join, limit and filter operator (Prasanth J via Harish Butani) [rhbutani] HIVE-5899 NPE during explain extended with char/varchar columns (Jason Dere via Harish Butani) Changes for Build #587 [xuefu] HIVE-3181: getDatabaseMajor/Minor version does not return values (Szehon via Xuefu, reviewed by Navis) [brock] HIVE-5641 - BeeLineOpts ignores Throwable (Brock Noland reviewed by Prasad and Thejas) [hashutosh] HIVE-5909 : locate and instr throw java.nio.BufferUnderflowException when empty string as substring (Navis via Ashutosh Chauhan) [hashutosh] HIVE-5686 : partition column type validation doesn't quite work for dates (Sergey Shelukhin via Ashutosh Chauhan) [hashutosh] HIVE-5887 : metastore direct sql doesn't work with oracle (Sergey Shelukhin via Ashutosh Chauhan) Changes for Build #588 Changes for Build #589 Changes for Build #590 [brock] HIVE-5981 - Add hive-unit back to itests pom (Brock Noland reviewed by Prasad) Changes for Build #591 [xuefu] HIVE-5872: Make UDAFs such as GenericUDAFSum report accurate precision/scale for decimal types (reviewed by Sergey Shelukhin) [hashutosh] HIVE-5978 : Rollups not supported in vector mode. (Jitendra Nath Pandey via Ashutosh Chauhan) [hashutosh] HIVE-5830 : SubQuery: Not In subqueries should check if subquery contains nulls in matching column (Harish Butani via Ashutosh Chauhan) [hashutosh] HIVE-5598 : Remove dummy new line at the end of non-sql commands (Navis via Ashutosh Chauhan) Changes for Build #592 [hashutosh] HIVE-5982 : Remove redundant filesystem operations and methods in FileSink (Ashutosh Chauhan via Thejas Nair) [navis] HIVE-5955 : decimal_precision.q test case fails in trunk (Prasanth J via Navis) [brock] HIVE-5983 - Fix name of ColumnProjectionUtils.appendReadColumnIDs (Brock Noland reviewed by Navis) Changes for Build #593 [omalley] HIVE-5580. Predicate pushdown predicates with an and-operator between non-SARGable predicates cause a NPE. (omalley) Changes for Build #594 [gunther] HIVE-6000: Hive build broken on hadoop2 (Vikram Dixit K via Gunther Hagleitner [gunther] HIVE-2093: UPDATE - add two missing files from previous commit (Gunther Hagleitner) [thejas] HIVE-2093 : create/drop database should populate inputs/outputs and check concurrency and user permission (Navis via Thejas Nair) [hashutosh] HIVE-6016 : Hadoop23Shims has a bug in listLocatedStatus impl. (Prasanth J via Ashutosh Chauhan) [hashutosh] HIVE-5994 : ORC RLEv2 encodes wrongly for large negative BIGINTs (64 bits ) (Prasanth J via Owen Omalley) [hashutosh] HIVE-5991 : ORC RLEv2 fails with ArrayIndexOutOfBounds exception for PATCHED_BLOB encoding (Prasanth J via Owen Omalley) [prasadm] HIVE-4395: Support TFetchOrientation.FIRST for HiveServer2 FetchResults (Prasad Mujumdar reviewed by Thejas Nair) [ehans] HIVE-5756: Implement vectorized support for IF conditional expression (Eric Hanson) [hashutosh] HIVE-6018 : FetchTask should not reference metastore classes (Navis via Prasad Mujumdar) [hashutosh] HIVE-5979. Failure in cast to timestamps. (Jitendra Pandey) [hashutosh] HIVE-5897 : Fix hadoop2 execution environment Milestone 2 (Vikram Dixit via Brock Noland) Changes for Build #595 Changes for Build #596 [hashutosh] HIVE-6027 : non-vectorized log10 has rounding issue (Sergey Shelukhin via Ashutosh Chauhan) [prasadm] HIVE-5993: JDBC Driver should not hard-code the database name (Szehon Ho via Prasad Mujumdar) Changes for Build #597 [hashutosh] HIVE-6004 : Fix statistics annotation related test failures in hadoop2 (Prasanth J via Ashutosh Chauhan) Changes for Build #598 [navis] HIVE-5985 : Make qfile_regex to accept multiple patterns (Navis reviewed by Ashutosh Chauhan) Changes for Build #599 Changes for Build #600 Changes for Build #601 [navis] HIVE-5276 : Skip redundant string encoding/decoding for hiveserver2 (Navis Reviewed by Carl Steinbach) Changes for Build #602 [xuefu] HIVE-6022: Load statements with incorrect order of partitions put input files to unreadable places (Teruyoshi Zenmyo via Xuefu) Changes for Build #603 Changes for Build #604 [thejas] HIVE-5975 : [WebHCat] templeton mapreduce job failed if provide define parameters (Shanyu Zhao via Thejas Nair) Changes for Build #605 [prasadm] HIVE-1466: Add NULL DEFINED AS to ROW FORMAT specification (Prasad Mujumdar reviewed by Xuefu Zhang) Changes for Build #606 [jitendra] HIVE-5521 : Remove CommonRCFileInputFormat. (hashutosh via jitendra) [rhbutani] HIVE-5973 SMB joins produce incorrect results with multiple partitions and buckets (Vikram Dixit via Harish Butani) [ehans] HIVE-6015: vectorized logarithm produces results for 0 that are different from a
[jira] [Updated] (HIVE-5923) SQL std auth - parser changes
[ https://issues.apache.org/jira/browse/HIVE-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-5923: Attachment: HIVE-5923.1.patch SQL std auth - parser changes - Key: HIVE-5923 URL: https://issues.apache.org/jira/browse/HIVE-5923 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Attachments: HIVE-5923.1.patch Original Estimate: 96h Remaining Estimate: 96h There are new access control statements proposed in the functional spec in HIVE-5837 . It also proposes some small changes to the existing query syntax (mostly extensions and some optional keywords). The syntax supported should depend on the current authorization mode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-5923) SQL std auth - parser changes
[ https://issues.apache.org/jira/browse/HIVE-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-5923: Assignee: Thejas M Nair Status: Patch Available (was: Open) SQL std auth - parser changes - Key: HIVE-5923 URL: https://issues.apache.org/jira/browse/HIVE-5923 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-5923.1.patch Original Estimate: 96h Remaining Estimate: 96h There are new access control statements proposed in the functional spec in HIVE-5837 . It also proposes some small changes to the existing query syntax (mostly extensions and some optional keywords). The syntax supported should depend on the current authorization mode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Work logged] (HIVE-5923) SQL std auth - parser changes
[ https://issues.apache.org/jira/browse/HIVE-5923?focusedWorklogId=15534page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-15534 ] Thejas M Nair logged work on HIVE-5923: --- Author: Thejas M Nair Created on: 27/Dec/13 00:44 Start Date: 27/Dec/13 00:44 Worklog Time Spent: 72h Issue Time Tracking --- Worklog Id: (was: 15534) Time Spent: 72h Remaining Estimate: 12h (was: 96h) SQL std auth - parser changes - Key: HIVE-5923 URL: https://issues.apache.org/jira/browse/HIVE-5923 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-5923.1.patch Original Estimate: 96h Time Spent: 72h Remaining Estimate: 12h There are new access control statements proposed in the functional spec in HIVE-5837 . It also proposes some small changes to the existing query syntax (mostly extensions and some optional keywords). The syntax supported should depend on the current authorization mode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5515) Writing to an HBase table throws IllegalArgumentException, failing job submission
[
https://issues.apache.org/jira/browse/HIVE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857209#comment-13857209
]
Nick Dimiduk commented on HIVE-5515:
Ping [~viraj]. Any word on the updated patch?
Writing to an HBase table throws IllegalArgumentException, failing job
submission
-
Key: HIVE-5515
URL: https://issues.apache.org/jira/browse/HIVE-5515
Project: Hive
Issue Type: Bug
Components: HBase Handler
Affects Versions: 0.12.0
Environment: Hadoop2, Hive 0.12.0, HBase-0.96RC
Reporter: Nick Dimiduk
Assignee: Viraj Bhat
Labels: hbase
Fix For: 0.13.0
Attachments: HIVE-5515.1.patch, HIVE-5515.patch
Inserting data into HBase table via hive query fails with the following
message:
{noformat}
$ hive -e FROM pgc INSERT OVERWRITE TABLE pagecounts_hbase SELECT pgc.*
WHERE rowkey LIKE 'en/q%' LIMIT 10;
...
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapred.reduce.tasks=number
java.lang.IllegalArgumentException: Property value must not be null
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:810)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:792)
at
org.apache.hadoop.hive.ql.exec.Utilities.copyTableJobPropertiesToConf(Utilities.java:2002)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.checkOutputSpecs(FileSinkOperator.java:947)
at
org.apache.hadoop.hive.ql.io.HiveOutputFormatImpl.checkOutputSpecs(HiveOutputFormatImpl.java:67)
at
org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:425)
at
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:136)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:731)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Job Submission failed with exception
'java.lang.IllegalArgumentException(Property value must
[jira] [Updated] (HIVE-6095) Use paths consistently II
[ https://issues.apache.org/jira/browse/HIVE-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6095: --- Status: Patch Available (was: Open) Use paths consistently II - Key: HIVE-6095 URL: https://issues.apache.org/jira/browse/HIVE-6095 Project: Hive Issue Type: Improvement Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Priority: Minor Attachments: HIVE-6095.2.patch, HIVE-6095.3.patch, HIVE-6095.patch This is follow-up of HIVE-3616. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6095) Use paths consistently II
[ https://issues.apache.org/jira/browse/HIVE-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6095: --- Status: Open (was: Patch Available) Use paths consistently II - Key: HIVE-6095 URL: https://issues.apache.org/jira/browse/HIVE-6095 Project: Hive Issue Type: Improvement Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Priority: Minor Attachments: HIVE-6095.2.patch, HIVE-6095.3.patch, HIVE-6095.patch This is follow-up of HIVE-3616. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-5941) SQL std auth - support 'show all roles'
[ https://issues.apache.org/jira/browse/HIVE-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-5941: Description: SHOW ALL ROLES - This will list all currently existing roles. This will be available only to the superuser. This task includes parser changes. was: SHOW ALL ROLES - This will list all currently existing roles. This will be available only to the superuser. SQL std auth - support 'show all roles' --- Key: HIVE-5941 URL: https://issues.apache.org/jira/browse/HIVE-5941 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Original Estimate: 24h Remaining Estimate: 24h SHOW ALL ROLES - This will list all currently existing roles. This will be available only to the superuser. This task includes parser changes. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6095) Use paths consistently II
[ https://issues.apache.org/jira/browse/HIVE-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6095: --- Attachment: HIVE-6095.3.patch Use paths consistently II - Key: HIVE-6095 URL: https://issues.apache.org/jira/browse/HIVE-6095 Project: Hive Issue Type: Improvement Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Priority: Minor Attachments: HIVE-6095.2.patch, HIVE-6095.3.patch, HIVE-6095.patch This is follow-up of HIVE-3616. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5923) SQL std auth - parser changes
[ https://issues.apache.org/jira/browse/HIVE-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857215#comment-13857215 ] Thejas M Nair commented on HIVE-5923: - Remaining work will be done in other jiras. Support for GRANTED BY needs to be added to all grant and revoke statements. - HIVE-6111 Privileges INSERT and DELETE need to be added to the language. - HIVE-6112 New statements SHOW ROLE and SHOW ROLE roleid need to be added to the language. - (describe role and not 'show role roleid' )- HIVE-5941,HIVE-5942 SQL std auth - parser changes - Key: HIVE-5923 URL: https://issues.apache.org/jira/browse/HIVE-5923 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-5923.1.patch Original Estimate: 96h Time Spent: 72h Remaining Estimate: 12h There are new access control statements proposed in the functional spec in HIVE-5837 . It also proposes some small changes to the existing query syntax (mostly extensions and some optional keywords). The syntax supported should depend on the current authorization mode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[ https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857216#comment-13857216 ] Lefty Leverenz commented on HIVE-6109: -- I updated HCatalog's wikidoc Dynamic Partitioning with information about this ticket, and elaborated on the information about HIVE-5011: * [External Tables|https://cwiki.apache.org/confluence/display/Hive/HCatalog+DynamicPartitions#HCatalogDynamicPartitions-ExternalTables] Corrections improvements are welcomed. Support customized location for EXTERNAL tables created by Dynamic Partitioning --- Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-5942) SQL std auth - support DESCRIBE ROLE
[ https://issues.apache.org/jira/browse/HIVE-5942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-5942: Description: DESCRIBE ROLE role must be supported. This will list all users and roles that participate in a role. This will be available to the superuser and any member of the role that has admin privileges on the role. This task includes parser changes. was: DESCRIBE ROLE role must be supported. This will list all users and roles that participate in a role. This will be available to the superuser and any member of the role that has admin privileges on the role. SQL std auth - support DESCRIBE ROLE Key: HIVE-5942 URL: https://issues.apache.org/jira/browse/HIVE-5942 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Original Estimate: 24h Remaining Estimate: 24h DESCRIBE ROLE role must be supported. This will list all users and roles that participate in a role. This will be available to the superuser and any member of the role that has admin privileges on the role. This task includes parser changes. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Hive-trunk-h0.21 - Build # 2528 - Still Failing
Changes for Build #2488 [hashutosh] HIVE-5935 : hive.query.string is not provided to FetchTask (Navis via Ashutosh Chauhan) [navis] HIVE-3455 : ANSI CORR(X,Y) is incorrect (Maxim Bolotin via Navis) [hashutosh] HIVE-5921 : Better heuristics for worst case statistics estimates for join, limit and filter operator (Prasanth J via Harish Butani) [rhbutani] HIVE-5899 NPE during explain extended with char/varchar columns (Jason Dere via Harish Butani) Changes for Build #2489 [xuefu] HIVE-3181: getDatabaseMajor/Minor version does not return values (Szehon via Xuefu, reviewed by Navis) [brock] HIVE-5641 - BeeLineOpts ignores Throwable (Brock Noland reviewed by Prasad and Thejas) [hashutosh] HIVE-5909 : locate and instr throw java.nio.BufferUnderflowException when empty string as substring (Navis via Ashutosh Chauhan) [hashutosh] HIVE-5686 : partition column type validation doesn't quite work for dates (Sergey Shelukhin via Ashutosh Chauhan) [hashutosh] HIVE-5887 : metastore direct sql doesn't work with oracle (Sergey Shelukhin via Ashutosh Chauhan) Changes for Build #2490 Changes for Build #2491 Changes for Build #2492 [brock] HIVE-5981 - Add hive-unit back to itests pom (Brock Noland reviewed by Prasad) Changes for Build #2493 [xuefu] HIVE-5872: Make UDAFs such as GenericUDAFSum report accurate precision/scale for decimal types (reviewed by Sergey Shelukhin) [hashutosh] HIVE-5978 : Rollups not supported in vector mode. (Jitendra Nath Pandey via Ashutosh Chauhan) [hashutosh] HIVE-5830 : SubQuery: Not In subqueries should check if subquery contains nulls in matching column (Harish Butani via Ashutosh Chauhan) [hashutosh] HIVE-5598 : Remove dummy new line at the end of non-sql commands (Navis via Ashutosh Chauhan) Changes for Build #2494 [hashutosh] HIVE-5982 : Remove redundant filesystem operations and methods in FileSink (Ashutosh Chauhan via Thejas Nair) [navis] HIVE-5955 : decimal_precision.q test case fails in trunk (Prasanth J via Navis) [brock] HIVE-5983 - Fix name of ColumnProjectionUtils.appendReadColumnIDs (Brock Noland reviewed by Navis) Changes for Build #2495 [omalley] HIVE-5580. Predicate pushdown predicates with an and-operator between non-SARGable predicates cause a NPE. (omalley) Changes for Build #2496 [gunther] HIVE-6000: Hive build broken on hadoop2 (Vikram Dixit K via Gunther Hagleitner [gunther] HIVE-2093: UPDATE - add two missing files from previous commit (Gunther Hagleitner) [thejas] HIVE-2093 : create/drop database should populate inputs/outputs and check concurrency and user permission (Navis via Thejas Nair) [hashutosh] HIVE-6016 : Hadoop23Shims has a bug in listLocatedStatus impl. (Prasanth J via Ashutosh Chauhan) [hashutosh] HIVE-5994 : ORC RLEv2 encodes wrongly for large negative BIGINTs (64 bits ) (Prasanth J via Owen Omalley) [hashutosh] HIVE-5991 : ORC RLEv2 fails with ArrayIndexOutOfBounds exception for PATCHED_BLOB encoding (Prasanth J via Owen Omalley) [prasadm] HIVE-4395: Support TFetchOrientation.FIRST for HiveServer2 FetchResults (Prasad Mujumdar reviewed by Thejas Nair) [ehans] HIVE-5756: Implement vectorized support for IF conditional expression (Eric Hanson) [hashutosh] HIVE-6018 : FetchTask should not reference metastore classes (Navis via Prasad Mujumdar) [hashutosh] HIVE-5979. Failure in cast to timestamps. (Jitendra Pandey) [hashutosh] HIVE-5897 : Fix hadoop2 execution environment Milestone 2 (Vikram Dixit via Brock Noland) Changes for Build #2497 Changes for Build #2498 [hashutosh] HIVE-6004 : Fix statistics annotation related test failures in hadoop2 (Prasanth J via Ashutosh Chauhan) [hashutosh] HIVE-6027 : non-vectorized log10 has rounding issue (Sergey Shelukhin via Ashutosh Chauhan) [prasadm] HIVE-5993: JDBC Driver should not hard-code the database name (Szehon Ho via Prasad Mujumdar) Changes for Build #2499 [navis] HIVE-5985 : Make qfile_regex to accept multiple patterns (Navis reviewed by Ashutosh Chauhan) Changes for Build #2500 Changes for Build #2501 Changes for Build #2502 [navis] HIVE-5276 : Skip redundant string encoding/decoding for hiveserver2 (Navis Reviewed by Carl Steinbach) Changes for Build #2503 [xuefu] HIVE-6022: Load statements with incorrect order of partitions put input files to unreadable places (Teruyoshi Zenmyo via Xuefu) Changes for Build #2504 Changes for Build #2505 [thejas] HIVE-5975 : [WebHCat] templeton mapreduce job failed if provide define parameters (Shanyu Zhao via Thejas Nair) Changes for Build #2506 [prasadm] HIVE-1466: Add NULL DEFINED AS to ROW FORMAT specification (Prasad Mujumdar reviewed by Xuefu Zhang) Changes for Build #2507 [jitendra] HIVE-5521 : Remove CommonRCFileInputFormat. (hashutosh via jitendra) [rhbutani] HIVE-5973 SMB joins produce incorrect results with multiple partitions and buckets (Vikram Dixit via Harish Butani) [ehans] HIVE-6015: vectorized logarithm produces results for 0 that are different from a
[jira] [Commented] (HIVE-5923) SQL std auth - parser changes
[
https://issues.apache.org/jira/browse/HIVE-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857223#comment-13857223
]
Hive QA commented on HIVE-5923:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12620570/HIVE-5923.1.patch
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/745/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/745/console
Messages:
{noformat}
This message was trimmed, see log for full details
/data/hive-ptest/working/maven/antlr/antlr/2.7.7/antlr-2.7.7.jar
/data/hive-ptest/working/maven/org/apache/thrift/libfb303/0.9.0/libfb303-0.9.0.jar
/data/hive-ptest/working/maven/org/apache/thrift/libthrift/0.9.0/libthrift-0.9.0.jar
/data/hive-ptest/working/maven/org/apache/httpcomponents/httpclient/4.2.5/httpclient-4.2.5.jar
/data/hive-ptest/working/maven/org/apache/httpcomponents/httpcore/4.2.4/httpcore-4.2.4.jar
/data/hive-ptest/working/maven/org/apache/hadoop/hadoop-core/1.2.1/hadoop-core-1.2.1.jar
/data/hive-ptest/working/maven/xmlenc/xmlenc/0.52/xmlenc-0.52.jar
/data/hive-ptest/working/maven/com/sun/jersey/jersey-core/1.14/jersey-core-1.14.jar
/data/hive-ptest/working/maven/com/sun/jersey/jersey-json/1.14/jersey-json-1.14.jar
/data/hive-ptest/working/maven/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar
/data/hive-ptest/working/maven/stax/stax-api/1.0.1/stax-api-1.0.1.jar
/data/hive-ptest/working/maven/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar
/data/hive-ptest/working/maven/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar
/data/hive-ptest/working/maven/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar
/data/hive-ptest/working/maven/javax/activation/activation/1.1/activation-1.1.jar
/data/hive-ptest/working/maven/org/codehaus/jackson/jackson-jaxrs/1.9.2/jackson-jaxrs-1.9.2.jar
/data/hive-ptest/working/maven/org/codehaus/jackson/jackson-xc/1.9.2/jackson-xc-1.9.2.jar
/data/hive-ptest/working/maven/com/sun/jersey/jersey-server/1.14/jersey-server-1.14.jar
/data/hive-ptest/working/maven/asm/asm/3.1/asm-3.1.jar
/data/hive-ptest/working/maven/commons-io/commons-io/2.4/commons-io-2.4.jar
/data/hive-ptest/working/maven/commons-httpclient/commons-httpclient/3.0.1/commons-httpclient-3.0.1.jar
/data/hive-ptest/working/maven/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
/data/hive-ptest/working/maven/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar
/data/hive-ptest/working/maven/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar
/data/hive-ptest/working/maven/commons-digester/commons-digester/1.8/commons-digester-1.8.jar
/data/hive-ptest/working/maven/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar
/data/hive-ptest/working/maven/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar
/data/hive-ptest/working/maven/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
/data/hive-ptest/working/maven/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar
/data/hive-ptest/working/maven/org/mortbay/jetty/servlet-api/2.5-20081211/servlet-api-2.5-20081211.jar
/data/hive-ptest/working/maven/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar
/data/hive-ptest/working/maven/tomcat/jasper-runtime/5.5.12/jasper-runtime-5.5.12.jar
/data/hive-ptest/working/maven/tomcat/jasper-compiler/5.5.12/jasper-compiler-5.5.12.jar
/data/hive-ptest/working/maven/org/mortbay/jetty/jsp-api-2.1/6.1.14/jsp-api-2.1-6.1.14.jar
/data/hive-ptest/working/maven/org/mortbay/jetty/servlet-api-2.5/6.1.14/servlet-api-2.5-6.1.14.jar
/data/hive-ptest/working/maven/org/mortbay/jetty/jsp-2.1/6.1.14/jsp-2.1-6.1.14.jar
/data/hive-ptest/working/maven/ant/ant/1.6.5/ant-1.6.5.jar
/data/hive-ptest/working/maven/commons-el/commons-el/1.0/commons-el-1.0.jar
/data/hive-ptest/working/maven/net/java/dev/jets3t/jets3t/0.6.1/jets3t-0.6.1.jar
/data/hive-ptest/working/maven/hsqldb/hsqldb/1.8.0.10/hsqldb-1.8.0.10.jar
/data/hive-ptest/working/maven/oro/oro/2.0.8/oro-2.0.8.jar
/data/hive-ptest/working/maven/org/eclipse/jdt/core/3.1.1/core-3.1.1.jar
/data/hive-ptest/working/maven/org/codehaus/jackson/jackson-mapper-asl/1.9.2/jackson-mapper-asl-1.9.2.jar
/data/hive-ptest/working/maven/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar
/data/hive-ptest/working/maven/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.jar
/data/hive-ptest/working/maven/log4j/log4j/1.2.16/log4j-1.2.16.jar
ENHANCED (PersistenceCapable) : org.apache.hadoop.hive.metastore.model.MDatabase
ENHANCED (PersistenceCapable) :
org.apache.hadoop.hive.metastore.model.MFieldSchema
ENHANCED (PersistenceCapable) : org.apache.hadoop.hive.metastore.model.MType
ENHANCED
[jira] [Commented] (HIVE-6095) Use paths consistently II
[
https://issues.apache.org/jira/browse/HIVE-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857245#comment-13857245
]
Hive QA commented on HIVE-6095:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12620571/HIVE-6095.3.patch
{color:green}SUCCESS:{color} +1 4818 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/746/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/746/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12620571
Use paths consistently II
-
Key: HIVE-6095
URL: https://issues.apache.org/jira/browse/HIVE-6095
Project: Hive
Issue Type: Improvement
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Priority: Minor
Attachments: HIVE-6095.2.patch, HIVE-6095.3.patch, HIVE-6095.patch
This is follow-up of HIVE-3616.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
Review Request 16480: Refactoring patch
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16480/ --- Review request for hive. Bugs: HIVE-6095 https://issues.apache.org/jira/browse/HIVE-6095 Repository: hive Description --- Refactoring patch, to use Paths instead of Strings. Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/MergeWork.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/AlterTablePartMergeFilesDesc.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/LoadFileDesc.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/LoadMultiFilesDesc.java 1553614 trunk/ql/src/test/results/compiler/plan/case_sensitivity.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/input1.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/input2.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/input3.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/input6.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/input7.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/input9.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/input_testsequencefile.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/sample2.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/sample3.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/sample4.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/sample5.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/sample6.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/sample7.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/subq.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/union.q.xml 1553614 Diff: https://reviews.apache.org/r/16480/diff/ Testing --- Refactoring patch. Existing regression suite suffices. Thanks, Ashutosh Chauhan
[jira] [Commented] (HIVE-6095) Use paths consistently II
[ https://issues.apache.org/jira/browse/HIVE-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857260#comment-13857260 ] Ashutosh Chauhan commented on HIVE-6095: RB request : https://reviews.apache.org/r/16480/ [~xuefuz] would you like to take a look? Use paths consistently II - Key: HIVE-6095 URL: https://issues.apache.org/jira/browse/HIVE-6095 Project: Hive Issue Type: Improvement Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Priority: Minor Attachments: HIVE-6095.2.patch, HIVE-6095.3.patch, HIVE-6095.patch This is follow-up of HIVE-3616. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[DISCUSS] Proposed Changes to the Apache Hive Project Bylaws
I think we should make several changes to the Apache Hive Project Bylaws. The proposed changes are available for review here: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=38568856 Most of the changes were directly inspired by provisions found in the Apache Hadoop Project Bylaws. Summary of proposed changes: * Add provisions for branch committers and speculative branches. * Define the responsibilities of a release manager. * PMC Chairs serve for one year and are elected by the PMC using Single Transferable Vote (STV) voting. * With the exception of code change votes, the minimum length of all voting periods is extended to seven days. Thanks. Carl
Re: Review Request 16172: ql.plan.ConditionalResolverCommonJoin.resolveMapJoinTask also sums those tables which are not used in the child of this conditional task.
On Dec. 18, 2013, 2:02 p.m., Yin Huai wrote:
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java, line 427
https://reviews.apache.org/r/16172/diff/2/?file=399281#file399281line427
Seems it is not an error? If so, let's not put it in the ErrorMsg.
done.
On Dec. 18, 2013, 2:02 p.m., Yin Huai wrote:
ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverCommonJoin.java,
line 262
https://reviews.apache.org/r/16172/diff/2/?file=399284#file399284line262
Is this one necessary?
changed to debug message
- Navis
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16172/#review30612
---
On Dec. 18, 2013, 5:04 a.m., Navis Ryu wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16172/
---
(Updated Dec. 18, 2013, 5:04 a.m.)
Review request for hive.
Bugs: HIVE-5945
https://issues.apache.org/jira/browse/HIVE-5945
Repository: hive-git
Description
---
Here is an example
{code}
select
i_item_id,
s_state,
avg(ss_quantity) agg1,
avg(ss_list_price) agg2,
avg(ss_coupon_amt) agg3,
avg(ss_sales_price) agg4
FROM store_sales
JOIN date_dim on (store_sales.ss_sold_date_sk = date_dim.d_date_sk)
JOIN item on (store_sales.ss_item_sk = item.i_item_sk)
JOIN customer_demographics on (store_sales.ss_cdemo_sk =
customer_demographics.cd_demo_sk)
JOIN store on (store_sales.ss_store_sk = store.s_store_sk)
where
cd_gender = 'F' and
cd_marital_status = 'U' and
cd_education_status = 'Primary' and
d_year = 2002 and
s_state in ('GA','PA', 'LA', 'SC', 'MI', 'AL')
group by
i_item_id,
s_state
order by
i_item_id,
s_state
limit 100;
{\code}
I turned off noconditionaltask. So, I expected that there will be 4 Map-only
jobs for this query. However, I got 1 Map-only job (joining strore_sales and
date_dim) and 3 MR job (for reduce joins.)
So, I checked the conditional task determining the plan of the join involving
item. In ql.plan.ConditionalResolverCommonJoin.resolveMapJoinTask,
aliasToFileSizeMap contains all input tables used in this query and the
intermediate table generated by joining store_sales and date_dim. So, when we
sum the size of all small tables, the size of store_sales (which is around
45GB in my test) will be also counted.
Diffs
-
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 45acc2b
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 9afc80b
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java
2efa7c2
ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverCommonJoin.java
faf2f9b
ql/src/test/org/apache/hadoop/hive/ql/plan/TestConditionalResolverCommonJoin.java
67203c9
ql/src/test/results/clientpositive/auto_join25.q.out 7427239
ql/src/test/results/clientpositive/infer_bucket_sort_convert_join.q.out
7d06739
ql/src/test/results/clientpositive/mapjoin_hook.q.out d60d16e
Diff: https://reviews.apache.org/r/16172/diff/
Testing
---
Thanks,
Navis Ryu
Re: Review Request 16172: ql.plan.ConditionalResolverCommonJoin.resolveMapJoinTask also sums those tables which are not used in the child of this conditional task.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16172/
---
(Updated Dec. 27, 2013, 3:13 a.m.)
Review request for hive.
Bugs: HIVE-5945
https://issues.apache.org/jira/browse/HIVE-5945
Repository: hive-git
Description
---
Here is an example
{code}
select
i_item_id,
s_state,
avg(ss_quantity) agg1,
avg(ss_list_price) agg2,
avg(ss_coupon_amt) agg3,
avg(ss_sales_price) agg4
FROM store_sales
JOIN date_dim on (store_sales.ss_sold_date_sk = date_dim.d_date_sk)
JOIN item on (store_sales.ss_item_sk = item.i_item_sk)
JOIN customer_demographics on (store_sales.ss_cdemo_sk =
customer_demographics.cd_demo_sk)
JOIN store on (store_sales.ss_store_sk = store.s_store_sk)
where
cd_gender = 'F' and
cd_marital_status = 'U' and
cd_education_status = 'Primary' and
d_year = 2002 and
s_state in ('GA','PA', 'LA', 'SC', 'MI', 'AL')
group by
i_item_id,
s_state
order by
i_item_id,
s_state
limit 100;
{\code}
I turned off noconditionaltask. So, I expected that there will be 4 Map-only
jobs for this query. However, I got 1 Map-only job (joining strore_sales and
date_dim) and 3 MR job (for reduce joins.)
So, I checked the conditional task determining the plan of the join involving
item. In ql.plan.ConditionalResolverCommonJoin.resolveMapJoinTask,
aliasToFileSizeMap contains all input tables used in this query and the
intermediate table generated by joining store_sales and date_dim. So, when we
sum the size of all small tables, the size of store_sales (which is around 45GB
in my test) will be also counted.
Diffs (updated)
-
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java daf4e4a
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java
37ed275
ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverCommonJoin.java
f75e366
ql/src/test/org/apache/hadoop/hive/ql/plan/TestConditionalResolverCommonJoin.java
67203c9
ql/src/test/results/clientpositive/auto_join25.q.out 7427239
ql/src/test/results/clientpositive/infer_bucket_sort_convert_join.q.out
7d06739
ql/src/test/results/clientpositive/mapjoin_hook.q.out d60d16e
Diff: https://reviews.apache.org/r/16172/diff/
Testing
---
Thanks,
Navis Ryu
[jira] [Updated] (HIVE-5945) ql.plan.ConditionalResolverCommonJoin.resolveMapJoinTask also sums those tables which are not used in the child of this conditional task.
[
https://issues.apache.org/jira/browse/HIVE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-5945:
Attachment: HIVE-5945.4.patch.txt
ql.plan.ConditionalResolverCommonJoin.resolveMapJoinTask also sums those
tables which are not used in the child of this conditional task.
-
Key: HIVE-5945
URL: https://issues.apache.org/jira/browse/HIVE-5945
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.8.0, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0
Reporter: Yin Huai
Assignee: Navis
Priority: Critical
Attachments: HIVE-5945.1.patch.txt, HIVE-5945.2.patch.txt,
HIVE-5945.3.patch.txt, HIVE-5945.4.patch.txt
Here is an example
{code}
select
i_item_id,
s_state,
avg(ss_quantity) agg1,
avg(ss_list_price) agg2,
avg(ss_coupon_amt) agg3,
avg(ss_sales_price) agg4
FROM store_sales
JOIN date_dim on (store_sales.ss_sold_date_sk = date_dim.d_date_sk)
JOIN item on (store_sales.ss_item_sk = item.i_item_sk)
JOIN customer_demographics on (store_sales.ss_cdemo_sk =
customer_demographics.cd_demo_sk)
JOIN store on (store_sales.ss_store_sk = store.s_store_sk)
where
cd_gender = 'F' and
cd_marital_status = 'U' and
cd_education_status = 'Primary' and
d_year = 2002 and
s_state in ('GA','PA', 'LA', 'SC', 'MI', 'AL')
group by
i_item_id,
s_state
order by
i_item_id,
s_state
limit 100;
{\code}
I turned off noconditionaltask. So, I expected that there will be 4 Map-only
jobs for this query. However, I got 1 Map-only job (joining strore_sales and
date_dim) and 3 MR job (for reduce joins.)
So, I checked the conditional task determining the plan of the join involving
item. In ql.plan.ConditionalResolverCommonJoin.resolveMapJoinTask,
aliasToFileSizeMap contains all input tables used in this query and the
intermediate table generated by joining store_sales and date_dim. So, when we
sum the size of all small tables, the size of store_sales (which is around
45GB in my test) will be also counted.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
Re: Review Request 16480: Refactoring patch
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16480/#review30897 --- Ship it! Ship It! - Xuefu Zhang On Dec. 27, 2013, 2:36 a.m., Ashutosh Chauhan wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16480/ --- (Updated Dec. 27, 2013, 2:36 a.m.) Review request for hive. Bugs: HIVE-6095 https://issues.apache.org/jira/browse/HIVE-6095 Repository: hive Description --- Refactoring patch, to use Paths instead of Strings. Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/MergeWork.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/AlterTablePartMergeFilesDesc.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/LoadFileDesc.java 1553614 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/LoadMultiFilesDesc.java 1553614 trunk/ql/src/test/results/compiler/plan/case_sensitivity.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/input1.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/input2.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/input3.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/input6.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/input7.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/input9.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/input_testsequencefile.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/sample2.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/sample3.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/sample4.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/sample5.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/sample6.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/sample7.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/subq.q.xml 1553614 trunk/ql/src/test/results/compiler/plan/union.q.xml 1553614 Diff: https://reviews.apache.org/r/16480/diff/ Testing --- Refactoring patch. Existing regression suite suffices. Thanks, Ashutosh Chauhan
[jira] [Commented] (HIVE-5945) ql.plan.ConditionalResolverCommonJoin.resolveMapJoinTask also sums those tables which are not used in the child of this conditional task.
[
https://issues.apache.org/jira/browse/HIVE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857297#comment-13857297
]
Hive QA commented on HIVE-5945:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12620580/HIVE-5945.4.patch.txt
{color:green}SUCCESS:{color} +1 4817 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/747/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/747/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12620580
ql.plan.ConditionalResolverCommonJoin.resolveMapJoinTask also sums those
tables which are not used in the child of this conditional task.
-
Key: HIVE-5945
URL: https://issues.apache.org/jira/browse/HIVE-5945
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.8.0, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0
Reporter: Yin Huai
Assignee: Navis
Priority: Critical
Attachments: HIVE-5945.1.patch.txt, HIVE-5945.2.patch.txt,
HIVE-5945.3.patch.txt, HIVE-5945.4.patch.txt
Here is an example
{code}
select
i_item_id,
s_state,
avg(ss_quantity) agg1,
avg(ss_list_price) agg2,
avg(ss_coupon_amt) agg3,
avg(ss_sales_price) agg4
FROM store_sales
JOIN date_dim on (store_sales.ss_sold_date_sk = date_dim.d_date_sk)
JOIN item on (store_sales.ss_item_sk = item.i_item_sk)
JOIN customer_demographics on (store_sales.ss_cdemo_sk =
customer_demographics.cd_demo_sk)
JOIN store on (store_sales.ss_store_sk = store.s_store_sk)
where
cd_gender = 'F' and
cd_marital_status = 'U' and
cd_education_status = 'Primary' and
d_year = 2002 and
s_state in ('GA','PA', 'LA', 'SC', 'MI', 'AL')
group by
i_item_id,
s_state
order by
i_item_id,
s_state
limit 100;
{\code}
I turned off noconditionaltask. So, I expected that there will be 4 Map-only
jobs for this query. However, I got 1 Map-only job (joining strore_sales and
date_dim) and 3 MR job (for reduce joins.)
So, I checked the conditional task determining the plan of the join involving
item. In ql.plan.ConditionalResolverCommonJoin.resolveMapJoinTask,
aliasToFileSizeMap contains all input tables used in this query and the
intermediate table generated by joining store_sales and date_dim. So, when we
sum the size of all small tables, the size of store_sales (which is around
45GB in my test) will be also counted.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[ANNOUNCE] New Hive PMC Member - Gunther Hagleitner
I am pleased to announce that Gunther Hagleitner has been elected to the Hive Project Management Committee. Please join me in congratulating Gunther! Thanks. Carl
Re: [ANNOUNCE] New Hive PMC Member - Gunther Hagleitner
Congrats Gunther!! On Dec 26, 2013, at 8:59 PM, Carl Steinbach c...@apache.org wrote: I am pleased to announce that Gunther Hagleitner has been elected to the Hive Project Management Committee. Please join me in congratulating Gunther! Thanks. Carl -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Updated] (HIVE-6101) Classpath is incorrect for hadoop-1 tests on tez
[ https://issues.apache.org/jira/browse/HIVE-6101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-6101: - Attachment: HIVE-6101.2.patch_part4 Classpath is incorrect for hadoop-1 tests on tez Key: HIVE-6101 URL: https://issues.apache.org/jira/browse/HIVE-6101 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: tez-branch Attachments: HIVE-6101.1.patch, HIVE-6101.1.patch.part2, HIVE-6101.2.patch_part2, HIVE-6101.2.patch_part3, HIVE-6101.2.patch_part4 shim 23 pom file needs updating. Otherwise it'll pull in unwanted dependencies for the q tests. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Re: [DISCUSS] Proposed Changes to the Apache Hive Project Bylaws
I like the changes. I believe rotating the PMC chair will keep the project fresh. Work and life events come in spurts, and it's hard to step down when your at the top ( http://disinfo.com/2013/02/ratzinger-resigns-first-pope-to-quit-since-1415/ :) . On Thu, Dec 26, 2013 at 10:08 PM, Carl Steinbach c...@apache.org wrote: I think we should make several changes to the Apache Hive Project Bylaws. The proposed changes are available for review here: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=38568856 Most of the changes were directly inspired by provisions found in the Apache Hadoop Project Bylaws. Summary of proposed changes: * Add provisions for branch committers and speculative branches. * Define the responsibilities of a release manager. * PMC Chairs serve for one year and are elected by the PMC using Single Transferable Vote (STV) voting. * With the exception of code change votes, the minimum length of all voting periods is extended to seven days. Thanks. Carl
[jira] [Updated] (HIVE-6098) Merge Tez branch into trunk
[ https://issues.apache.org/jira/browse/HIVE-6098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-6098: - Status: Open (was: Patch Available) Merge Tez branch into trunk --- Key: HIVE-6098 URL: https://issues.apache.org/jira/browse/HIVE-6098 Project: Hive Issue Type: New Feature Affects Versions: 0.12.0 Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-6098.1.patch, HIVE-6098.2.patch, HIVE-6098.3.patch, HIVE-6098.4.patch, HIVE-6098.5.patch, hive-on-tez-conf.txt I think the Tez branch is at a point where we can consider merging it back into trunk after review. Tez itself has had its first release, most hive features are available on Tez and the test coverage is decent. There are a few known limitations, all of which can be handled in trunk as far as I can tell (i.e.: None of them are large disruptive changes that still require a branch.) Limitations: - Union all is not yet supported on Tez - SMB is not yet supported on Tez - Bucketed map-join is executed as broadcast join (bucketing is ignored) Since the user is free to toggle hive.optimize.tez, it's obviously possible to just run these on MR. I am hoping to follow the approach that was taken with vectorization and shoot for a merge instead of single commit. This would retain history of the branch. Also in vectorization we required at least three +1s before merge, I'm hoping to go with that as well. I will add a combined patch to this ticket for review purposes (not for commit). I'll also attach instructions to run on a cluster if anyone wants to try. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6098) Merge Tez branch into trunk
[ https://issues.apache.org/jira/browse/HIVE-6098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-6098: - Attachment: HIVE-6098.6.patch Merge Tez branch into trunk --- Key: HIVE-6098 URL: https://issues.apache.org/jira/browse/HIVE-6098 Project: Hive Issue Type: New Feature Affects Versions: 0.12.0 Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-6098.1.patch, HIVE-6098.2.patch, HIVE-6098.3.patch, HIVE-6098.4.patch, HIVE-6098.5.patch, HIVE-6098.6.patch, hive-on-tez-conf.txt I think the Tez branch is at a point where we can consider merging it back into trunk after review. Tez itself has had its first release, most hive features are available on Tez and the test coverage is decent. There are a few known limitations, all of which can be handled in trunk as far as I can tell (i.e.: None of them are large disruptive changes that still require a branch.) Limitations: - Union all is not yet supported on Tez - SMB is not yet supported on Tez - Bucketed map-join is executed as broadcast join (bucketing is ignored) Since the user is free to toggle hive.optimize.tez, it's obviously possible to just run these on MR. I am hoping to follow the approach that was taken with vectorization and shoot for a merge instead of single commit. This would retain history of the branch. Also in vectorization we required at least three +1s before merge, I'm hoping to go with that as well. I will add a combined patch to this ticket for review purposes (not for commit). I'll also attach instructions to run on a cluster if anyone wants to try. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6098) Merge Tez branch into trunk
[ https://issues.apache.org/jira/browse/HIVE-6098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-6098: - Status: Patch Available (was: Open) Merge Tez branch into trunk --- Key: HIVE-6098 URL: https://issues.apache.org/jira/browse/HIVE-6098 Project: Hive Issue Type: New Feature Affects Versions: 0.12.0 Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-6098.1.patch, HIVE-6098.2.patch, HIVE-6098.3.patch, HIVE-6098.4.patch, HIVE-6098.5.patch, HIVE-6098.6.patch, hive-on-tez-conf.txt I think the Tez branch is at a point where we can consider merging it back into trunk after review. Tez itself has had its first release, most hive features are available on Tez and the test coverage is decent. There are a few known limitations, all of which can be handled in trunk as far as I can tell (i.e.: None of them are large disruptive changes that still require a branch.) Limitations: - Union all is not yet supported on Tez - SMB is not yet supported on Tez - Bucketed map-join is executed as broadcast join (bucketing is ignored) Since the user is free to toggle hive.optimize.tez, it's obviously possible to just run these on MR. I am hoping to follow the approach that was taken with vectorization and shoot for a merge instead of single commit. This would retain history of the branch. Also in vectorization we required at least three +1s before merge, I'm hoping to go with that as well. I will add a combined patch to this ticket for review purposes (not for commit). I'll also attach instructions to run on a cluster if anyone wants to try. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Re: Review Request 16440: HIVE-6098: Merge Tez branch into trunk
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16440/ --- (Updated Dec. 27, 2013, 6:38 a.m.) Review request for hive. Changes --- Updated patch contains the latest changes (rebase + class path stuff) Repository: hive Description --- I think the Tez branch is at a point where we can consider merging it back into trunk after review. Tez itself has had its first release, most hive features are available on Tez and the test coverage is decent. There are a few known limitations, all of which can be handled in trunk as far as I can tell (i.e.: None of them are large disruptive changes that still require a branch.) Limitations: Union all is not yet supported on Tez SMB is not yet supported on Tez Bucketed map-join is executed as broadcast join (bucketing is ignored) Since the user is free to toggle hive.optimize.tez, it's obviously possible to just run these on MR. I am hoping to follow the approach that was taken with vectorization and shoot for a merge instead of single commit. This would retain history of the branch. Also in vectorization we required at least three +1s before merge, I'm hoping to go with that as well. I will add a combined patch to this ticket for review purposes (not for commit). I'll also attach instructions to run on a cluster if anyone wants to try. Diffs (updated) - trunk/ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 1553631 trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1553631 trunk/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 1553631 trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1553631 trunk/conf/hive-default.xml.template 1553631 trunk/data/conf/tez/hive-site.xml UNKNOWN trunk/hbase-handler/src/test/templates/TestHBaseCliDriver.vm 1553631 trunk/hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm 1553631 trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java 1553631 trunk/itests/qtest/pom.xml 1553631 trunk/itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 1553631 trunk/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 1553631 trunk/pom.xml 1553631 trunk/ql/pom.xml 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/HashTableLoaderFactory.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableLoader.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapredContext.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MuxOperator.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCache.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapper.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecReducer.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HashTableLoader.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ObjectCache.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapTezProcessor.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java PRE-CREATION
[jira] [Created] (HIVE-6113) Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
William Stone created HIVE-6113: --- Summary: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient Key: HIVE-6113 URL: https://issues.apache.org/jira/browse/HIVE-6113 Project: Hive Issue Type: Bug Components: Database/Schema Affects Versions: 0.12.0 Environment: hadoop-0.20.2-cdh3u3,hive-0.12.0 Reporter: William Stone Priority: Critical 2013-12-25 03:01:32,290 ERROR exec.DDLTask (DDLTask.java:execute(435)) - org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1143) at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1128) at org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:3479) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:237) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:875) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:769) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:708) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1217) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(RetryingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383) at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1139) ... 20 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210) ... 25 more Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes to datastore NestedThrowables: java.sql.BatchUpdateException: Duplicate entry 'default' for key 'UNIQUE_DATABASE' at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165) at org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:358) at org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:404) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:124) at $Proxy9.createDatabase(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB_core(HiveMetaStore.java:422) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:441) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:326) at
[jira] [Updated] (HIVE-6113) Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
[ https://issues.apache.org/jira/browse/HIVE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Stone updated HIVE-6113: Description: When I exccute SQL use fdm; desc formatted fdm.tableName; in python, throw Error as followed. but when I tryit again , It will success. 2013-12-25 03:01:32,290 ERROR exec.DDLTask (DDLTask.java:execute(435)) - org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1143) at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1128) at org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:3479) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:237) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:875) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:769) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:708) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1217) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(RetryingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383) at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1139) ... 20 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210) ... 25 more Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes to datastore NestedThrowables: java.sql.BatchUpdateException: Duplicate entry 'default' for key 'UNIQUE_DATABASE' at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165) at org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:358) at org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:404) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:124) at $Proxy9.createDatabase(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB_core(HiveMetaStore.java:422) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:441) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:326) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:286) at
[jira] [Updated] (HIVE-6113) Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
[ https://issues.apache.org/jira/browse/HIVE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Stone updated HIVE-6113: Priority: Blocker (was: Critical) Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient -- Key: HIVE-6113 URL: https://issues.apache.org/jira/browse/HIVE-6113 Project: Hive Issue Type: Bug Components: Database/Schema Affects Versions: 0.12.0 Environment: hadoop-0.20.2-cdh3u3,hive-0.12.0 Reporter: William Stone Priority: Blocker When I exccute SQL use fdm; desc formatted fdm.tableName; in python, throw Error as followed. but when I tryit again , It will success. 2013-12-25 03:01:32,290 ERROR exec.DDLTask (DDLTask.java:execute(435)) - org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1143) at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1128) at org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:3479) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:237) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:875) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:769) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:708) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1217) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(RetryingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383) at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1139) ... 20 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210) ... 25 more Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes to datastore NestedThrowables: java.sql.BatchUpdateException: Duplicate entry 'default' for key 'UNIQUE_DATABASE' at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165) at org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:358) at org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:404) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:124) at $Proxy9.createDatabase(Unknown Source)
[jira] [Created] (HIVE-6114) NoSuchObjectException(message:bdm.tableName table not found)
William Stone created HIVE-6114: --- Summary: NoSuchObjectException(message:bdm.tableName table not found) Key: HIVE-6114 URL: https://issues.apache.org/jira/browse/HIVE-6114 Project: Hive Issue Type: Bug Components: Database/Schema Affects Versions: 0.12.0 Environment: hadoop-0.20.2-cdh3u3, hive-0.12.0 Reporter: William Stone Priority: Blocker When executing an Query USE bdm;DESC FORMATTED bdm_las_waybill_revoke_run_car_task; But when I try it again, It will success. This is a random phenomenon. Throw Exceptions like this 2013-12-26 04:20:38,849 ERROR metadata.Hive (Hive.java:getTable(953)) - NoSuchObjectException(message:bdm.bdm_las_waybill_revoke_run_car_task table not found) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1373) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:103) at $Proxy10.get_table(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:854) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89) at $Proxy11.getTable(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:950) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:892) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer$tableSpec.init(BaseSemanticAnalyzer.java:730) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer$tableSpec.init(BaseSemanticAnalyzer.java:707) at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:196) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:441) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:342) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:977) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:442) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:825) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:769) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:708) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) 2013-12-26 04:20:38,904 ERROR ql.Driver (SessionState.java:printError(419)) - FAILED: SemanticException [Error 10001]: Line 1:114 Table not found 'bdm_las_waybill_revoke_run_car_task' org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:114 Table not found 'bdm_las_waybill_revoke_run_car_task' at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer$tableSpec.init(BaseSemanticAnalyzer.java:733) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer$tableSpec.init(BaseSemanticAnalyzer.java:707) at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:196) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:441) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:342) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:977) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217) at
[jira] [Updated] (HIVE-6113) Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
[ https://issues.apache.org/jira/browse/HIVE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Stone updated HIVE-6113: Labels: HiveMetaStoreClient metastore unable_instantiate (was: ) Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient -- Key: HIVE-6113 URL: https://issues.apache.org/jira/browse/HIVE-6113 Project: Hive Issue Type: Bug Components: Database/Schema Affects Versions: 0.12.0 Environment: hadoop-0.20.2-cdh3u3,hive-0.12.0 Reporter: William Stone Priority: Blocker Labels: HiveMetaStoreClient, metastore, unable_instantiate When I exccute SQL use fdm; desc formatted fdm.tableName; in python, throw Error as followed. but when I tryit again , It will success. 2013-12-25 03:01:32,290 ERROR exec.DDLTask (DDLTask.java:execute(435)) - org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1143) at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1128) at org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:3479) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:237) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:875) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:769) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:708) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1217) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(RetryingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383) at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1139) ... 20 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210) ... 25 more Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes to datastore NestedThrowables: java.sql.BatchUpdateException: Duplicate entry 'default' for key 'UNIQUE_DATABASE' at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165) at org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:358) at org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:404) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at
[jira] [Updated] (HIVE-6098) Merge Tez branch into trunk
[ https://issues.apache.org/jira/browse/HIVE-6098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-6098: - Status: Patch Available (was: Open) Merge Tez branch into trunk --- Key: HIVE-6098 URL: https://issues.apache.org/jira/browse/HIVE-6098 Project: Hive Issue Type: New Feature Affects Versions: 0.12.0 Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-6098.1.patch, HIVE-6098.2.patch, HIVE-6098.3.patch, HIVE-6098.4.patch, HIVE-6098.5.patch, HIVE-6098.6.patch, HIVE-6098.7.patch, hive-on-tez-conf.txt I think the Tez branch is at a point where we can consider merging it back into trunk after review. Tez itself has had its first release, most hive features are available on Tez and the test coverage is decent. There are a few known limitations, all of which can be handled in trunk as far as I can tell (i.e.: None of them are large disruptive changes that still require a branch.) Limitations: - Union all is not yet supported on Tez - SMB is not yet supported on Tez - Bucketed map-join is executed as broadcast join (bucketing is ignored) Since the user is free to toggle hive.optimize.tez, it's obviously possible to just run these on MR. I am hoping to follow the approach that was taken with vectorization and shoot for a merge instead of single commit. This would retain history of the branch. Also in vectorization we required at least three +1s before merge, I'm hoping to go with that as well. I will add a combined patch to this ticket for review purposes (not for commit). I'll also attach instructions to run on a cluster if anyone wants to try. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6098) Merge Tez branch into trunk
[ https://issues.apache.org/jira/browse/HIVE-6098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-6098: - Attachment: HIVE-6098.7.patch re-uploading .6 as .7 for precommit tests Merge Tez branch into trunk --- Key: HIVE-6098 URL: https://issues.apache.org/jira/browse/HIVE-6098 Project: Hive Issue Type: New Feature Affects Versions: 0.12.0 Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-6098.1.patch, HIVE-6098.2.patch, HIVE-6098.3.patch, HIVE-6098.4.patch, HIVE-6098.5.patch, HIVE-6098.6.patch, HIVE-6098.7.patch, hive-on-tez-conf.txt I think the Tez branch is at a point where we can consider merging it back into trunk after review. Tez itself has had its first release, most hive features are available on Tez and the test coverage is decent. There are a few known limitations, all of which can be handled in trunk as far as I can tell (i.e.: None of them are large disruptive changes that still require a branch.) Limitations: - Union all is not yet supported on Tez - SMB is not yet supported on Tez - Bucketed map-join is executed as broadcast join (bucketing is ignored) Since the user is free to toggle hive.optimize.tez, it's obviously possible to just run these on MR. I am hoping to follow the approach that was taken with vectorization and shoot for a merge instead of single commit. This would retain history of the branch. Also in vectorization we required at least three +1s before merge, I'm hoping to go with that as well. I will add a combined patch to this ticket for review purposes (not for commit). I'll also attach instructions to run on a cluster if anyone wants to try. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6098) Merge Tez branch into trunk
[ https://issues.apache.org/jira/browse/HIVE-6098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-6098: - Status: Open (was: Patch Available) Merge Tez branch into trunk --- Key: HIVE-6098 URL: https://issues.apache.org/jira/browse/HIVE-6098 Project: Hive Issue Type: New Feature Affects Versions: 0.12.0 Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-6098.1.patch, HIVE-6098.2.patch, HIVE-6098.3.patch, HIVE-6098.4.patch, HIVE-6098.5.patch, HIVE-6098.6.patch, HIVE-6098.7.patch, hive-on-tez-conf.txt I think the Tez branch is at a point where we can consider merging it back into trunk after review. Tez itself has had its first release, most hive features are available on Tez and the test coverage is decent. There are a few known limitations, all of which can be handled in trunk as far as I can tell (i.e.: None of them are large disruptive changes that still require a branch.) Limitations: - Union all is not yet supported on Tez - SMB is not yet supported on Tez - Bucketed map-join is executed as broadcast join (bucketing is ignored) Since the user is free to toggle hive.optimize.tez, it's obviously possible to just run these on MR. I am hoping to follow the approach that was taken with vectorization and shoot for a merge instead of single commit. This would retain history of the branch. Also in vectorization we required at least three +1s before merge, I'm hoping to go with that as well. I will add a combined patch to this ticket for review purposes (not for commit). I'll also attach instructions to run on a cluster if anyone wants to try. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
