Hive-trunk-h0.21 - Build # 2179 - Still Failing
Changes for Build #2142
[hashutosh] HIVE-4717 : Fix non-deterministic TestCliDriver on 0.23 (Brock
Noland via Ashutosh Chauhan)
Changes for Build #2143
Changes for Build #2144
[hashutosh] HIVE-4713 : Fix TestCliDriver.skewjoin_union_remove_{1,2}.q on 0.23
(Brock Noland via Ashutosh Chauhan)
[hashutosh] HIVE-4715 : Fix
TestCliDriver.{recursive_dir.q,sample_islocalmode_hook.q,input12.q,input39.q,auto_join14.q}
on 0.23 (Brock Noland via Ashutosh Chauhan)
Changes for Build #2145
Changes for Build #2146
[ecapriolo] Submitted by: Brock Noland Jonathon Chang
Reviewed by: Edward Capriolo
Approved by: Edward Capriolo
Changes for Build #2147
Changes for Build #2148
Changes for Build #2149
[navis] HIVE-4708 : Temporary files are not closed in PTFPersistence on jvm
reuse (Navis reviewed by Harish Butani)
Changes for Build #2150
[hashutosh] HIVE-4711 : Fix TestCliDriver.list_bucket_query_oneskew_{1,2,3}.q
on 0.23 (Brock Noland via Ashutosh Chauhan)
Changes for Build #2151
[hashutosh] HIVE-4707 : Support configurable domain name for HiveServer2 LDAP
authentication using Active Directory (Prasad Mujumdar via Ashutosh Chauhan)
[gates] HIVE-4969 WebHCat e2e test framework is missing files and instructions
(ekoifman via gates)
[hashutosh] HIVE-4593 : ErrorMsg has several messages that reuse the same error
code (Navis via Ashutosh Chauhan)
Changes for Build #2152
Changes for Build #2153
[hashutosh] HIVE-4746 : Fix TestCliDriver.list_bucket_dml_{2,4,5,9,12,13}.q on
0.23 (Brock Noland via Ashutosh Chauhan)
Changes for Build #2154
[gates] HIVE-4679 WebHCat can deadlock Hadoop if the number of concurrently
running tasks if higher or equal than the number of mappers (ekoifman via gates)
Changes for Build #2155
[ecapriolo] HIVE-4616: Support outer lateral view
Submitted by: Navis
Reviewed by:Edward Capriolo
Approved by:Edward Capriolo
Changes for Build #2156
Changes for Build #2157
[hashutosh] HIVE-3953 : Reading of partitioned Avro data fails because of
missing properties (Mark Wagner via Ashutosh Chauhan)
Changes for Build #2158
[hashutosh] HIVE-4609 : Allow hive tests to specify an alternative to /tmp
(Brock Noland via Ashutosh Chauhan)
Changes for Build #2159
[hashutosh] HIVE-4761 : ZooKeeperHiveLockManage.unlockPrimitive has race
condition with threads (Brock Noland via Ashutosh Chauhan)
Changes for Build #2160
[hashutosh] HIVE-4172 : JDBC2 does not support VOID type (Navis via Ashutosh
Chauhan)
Changes for Build #2161
[hashutosh] HIVE-4767 : ObjectStore.getPMF has concurrency problems (Brock
Noland via Ashutosh Chauhan)
[hashutosh] HIVE-4743 : Improve test coverage of package
org.apache.hadoop.hive.ql.io (Ivan Veselovsky via Ashutosh Chauhan)
Changes for Build #2162
Changes for Build #2163
Changes for Build #2164
Changes for Build #2165
Changes for Build #2166
Changes for Build #2167
Changes for Build #2168
[omalley] Undoing the accidental update of the 0.11 release notes.
[omalley] HIVE-4478. In ORC remove ispresent stream from columns that contain
no null
values in a stripe. (Prasanth Jayachandran via omalley)
[daijy] HIVE-4591: Making changes to webhcat-site.xml have no effect
[daijy] HIVE-4784:ant testreport doesn't include any HCatalog tests
Changes for Build #2169
Changes for Build #2170
[hashutosh] HIVE-4740 : HIVE2379 is missing hbase.jar itself (Navis via
Ashutosh Chauhan)
[hashutosh] HIVE-4756 : Upgrade Hadoop 0.23 profile to 2.0.5-alpha (Brock
Noland via Ashutosh Chauhan)
[hashutosh] HIVE-4791 : improve test coverage of package
org.apache.hadoop.hive.ql.udf.xml (Ivan Veselovsky via Ashutosh Chauhan)
Changes for Build #2171
[hashutosh] HIVE-4618 : show create table creating unusable DDL when field
delimiter is \001 (Navis via Ashutosh Chauhan)
[hashutosh] HIVE-4559 : hcatalog/webhcat scripts in tar.gz don't have execute
permissions set (Eugene Koifman via Ashutosh Chauhan)
[hashutosh] HIVE-4798 : NPE when we call isSame from an instance of
ExprNodeConstantDesc with null value (Yin Huai via Ashutosh Chauhan)
[hashutosh] HIVE-4781 : LEFT SEMI JOIN generates wrong results when the number
of rows belonging to a single key of the right table exceed
hive.join.emit.interval (Yin Huai via Ashutosh Chauhan)
[hashutosh] HIVE-4647 : RetryingHMSHandler logs too many error messages (Navis
via Ashutosh Chauhan)
[hashutosh] HIVE-4692 : Constant agg parameters will be replaced by
ExprNodeColumnDesc with single-sourced multi-gby cases (Navis via Ashutosh
Chauhan)
Changes for Build #2172
[hashutosh] HIVE-4781 : Adding new data files for tests. Missed in original
commit.
Changes for Build #2173
Changes for Build #2174
[navis] HIVE-2517 : Support group by on struct type (Ashutosh Chauhan via Navis)
[hashutosh] HIVE-4406 : Missing / or / in hs2 jdbc uri switches mode to
embedded mode(Anandha Ranganathan via Ashutosh Chauhan)
[hashutosh] HIVE-4430 : Semantic analysis fails in presence of certain literals
in on clause
[jira] [Commented] (HIVE-4144) Add "select database()" command to show the current database
[
https://issues.apache.org/jira/browse/HIVE-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13700401#comment-13700401
]
Navis commented on HIVE-4144:
-
Yes, I've made generic dummy n-row input format and OneNullRowIF (for
metadata-only query) to extend that. Quite a fun patch.
> Add "select database()" command to show the current database
>
>
> Key: HIVE-4144
> URL: https://issues.apache.org/jira/browse/HIVE-4144
> Project: Hive
> Issue Type: Bug
> Components: SQL
>Reporter: Mark Grover
>Assignee: Navis
> Attachments: HIVE-4144.D9597.1.patch, HIVE-4144.D9597.2.patch
>
>
> A recent hive-user mailing list conversation asked about having a command to
> show the current database.
> http://mail-archives.apache.org/mod_mbox/hive-user/201303.mbox/%3CCAMGr+0i+CRY69m3id=DxthmUCWLf0NxpKMCtROb=uauh2va...@mail.gmail.com%3E
> MySQL seems to have a command to do so:
> {code}
> select database();
> {code}
> http://dev.mysql.com/doc/refman/5.0/en/information-functions.html#function_database
> We should look into having something similar in Hive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4222) Timestamp type constants cannot be deserialized in JDK 1.6 or less
[
https://issues.apache.org/jira/browse/HIVE-4222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13700396#comment-13700396
]
Navis commented on HIVE-4222:
-
It's copied from Oracle JDK7 and I admit I don't know about license things. hm..
> Timestamp type constants cannot be deserialized in JDK 1.6 or less
> --
>
> Key: HIVE-4222
> URL: https://issues.apache.org/jira/browse/HIVE-4222
> Project: Hive
> Issue Type: Bug
> Components: Types
>Reporter: Navis
>Assignee: Navis
> Attachments: HIVE-4222.D9681.1.patch
>
>
> For example,
> {noformat}
> ExprNodeConstantDesc constant = new
> ExprNodeConstantDesc(TypeInfoFactory.timestampTypeInfo, new Timestamp(100));
> String serialized = Utilities.serializeExpression(constant);
> ExprNodeConstantDesc deserilized = (ExprNodeConstantDesc)
> Utilities.deserializeExpression(serialized, new Configuration());
> {noformat}
> logs error message
> {noformat}
> java.lang.InstantiationException: java.sql.Timestamp
> Continuing ...
> java.lang.RuntimeException: failed to evaluate: =Class.new();
> Continuing ...
> {noformat}
> and makes NPE in final.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4815) Create cloud hosting option for ptest2
[ https://issues.apache.org/jira/browse/HIVE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4815: --- Attachment: HIVE-4815.patch Add's tests for new host allocation retry logic (allocating spot instances), retries RSync on exit with 255, and cleans up log4j config. > Create cloud hosting option for ptest2 > -- > > Key: HIVE-4815 > URL: https://issues.apache.org/jira/browse/HIVE-4815 > Project: Hive > Issue Type: New Feature >Reporter: Brock Noland >Assignee: Brock Noland > Fix For: 0.12.0 > > Attachments: HIVE-4815.patch, HIVE-4815.patch, HIVE-4815.patch > > > HIVE-4675 creates a parallel testing environment. To support HIVE-4739 we > should allow this environment to run in a cloud environment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3534) Add left Lateral view syntax
[ https://issues.apache.org/jira/browse/HIVE-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo updated HIVE-3534: -- Resolution: Won't Fix Status: Resolved (was: Patch Available) OUTER LATERAL VIEW added to the language. > Add left Lateral view syntax > > > Key: HIVE-3534 > URL: https://issues.apache.org/jira/browse/HIVE-3534 > Project: Hive > Issue Type: New Feature >Reporter: Edward Capriolo >Assignee: Edward Capriolo > Attachments: hive-3534.diff.txt > > > A lateral view is like a join and if the function in question (like > explode()) produces zero rows then you lose the row > whereas conceivably you could keep the row and just fill all entry with nulls. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4144) Add "select database()" command to show the current database
[
https://issues.apache.org/jira/browse/HIVE-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13700265#comment-13700265
]
Edward Capriolo commented on HIVE-4144:
---
Navis. This is cool. I will review this soon. I did notice that there is
already a Dummy input format like yours that was added to hive a while back. I
think it was added to support queries only on meta-data. In any case we
probably do not need two dummy Input formats in the project.
> Add "select database()" command to show the current database
>
>
> Key: HIVE-4144
> URL: https://issues.apache.org/jira/browse/HIVE-4144
> Project: Hive
> Issue Type: Bug
> Components: SQL
>Reporter: Mark Grover
>Assignee: Navis
> Attachments: HIVE-4144.D9597.1.patch, HIVE-4144.D9597.2.patch
>
>
> A recent hive-user mailing list conversation asked about having a command to
> show the current database.
> http://mail-archives.apache.org/mod_mbox/hive-user/201303.mbox/%3CCAMGr+0i+CRY69m3id=DxthmUCWLf0NxpKMCtROb=uauh2va...@mail.gmail.com%3E
> MySQL seems to have a command to do so:
> {code}
> select database();
> {code}
> http://dev.mysql.com/doc/refman/5.0/en/information-functions.html#function_database
> We should look into having something similar in Hive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4779) Enhance coverage of package org.apache.hadoop.hive.ql.udf
[ https://issues.apache.org/jira/browse/HIVE-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13700260#comment-13700260 ] Edward Capriolo commented on HIVE-4779: --- As long as something else implements I am cool. "unix_timestamp" > Enhance coverage of package org.apache.hadoop.hive.ql.udf > - > > Key: HIVE-4779 > URL: https://issues.apache.org/jira/browse/HIVE-4779 > Project: Hive > Issue Type: Test >Affects Versions: 0.12.0 >Reporter: Ivan A. Veselovsky >Assignee: Ivan A. Veselovsky > Attachments: HIVE-4779-trunk--N1.patch > > > Enhance coverage of package org.apache.hadoop.hive.ql.udf up to 80%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4804) parallel_orderby.q in trunk fails consistently
[
https://issues.apache.org/jira/browse/HIVE-4804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13700228#comment-13700228
]
Edward Capriolo commented on HIVE-4804:
---
+1 testing now.
> parallel_orderby.q in trunk fails consistently
> --
>
> Key: HIVE-4804
> URL: https://issues.apache.org/jira/browse/HIVE-4804
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Navis
>Assignee: Navis
> Attachments: HIVE-4804.D11571.1.patch
>
>
> {noformat}
> java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at
> org.apache.hadoop.mapred.MapTask$OldOutputCollector.(MapTask.java:481)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:390)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:324)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
> at org.apache.hadoop.mapred.Child.main(Child.java:260)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 10 more
> Caused by: java.lang.IllegalArgumentException: Can't read partitions file
> at
> org.apache.hadoop.mapred.lib.TotalOrderPartitioner.configure(TotalOrderPartitioner.java:91)
> at
> org.apache.hadoop.hive.ql.exec.HiveTotalOrderPartitioner.configure(HiveTotalOrderPartitioner.java:37)
> ... 15 more
> Caused by: java.io.IOException: Split points are out of order
> at
> org.apache.hadoop.mapred.lib.TotalOrderPartitioner.configure(TotalOrderPartitioner.java:78)
> ... 16 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4222) Timestamp type constants cannot be deserialized in JDK 1.6 or less
[
https://issues.apache.org/jira/browse/HIVE-4222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13700224#comment-13700224
]
Edward Capriolo commented on HIVE-4222:
---
[~navis] we can not include author tags. Since the code in your patch is not
yours we need to establish if the code if/how the code is licensed. Is it
public domain?
> Timestamp type constants cannot be deserialized in JDK 1.6 or less
> --
>
> Key: HIVE-4222
> URL: https://issues.apache.org/jira/browse/HIVE-4222
> Project: Hive
> Issue Type: Bug
> Components: Types
>Reporter: Navis
>Assignee: Navis
> Attachments: HIVE-4222.D9681.1.patch
>
>
> For example,
> {noformat}
> ExprNodeConstantDesc constant = new
> ExprNodeConstantDesc(TypeInfoFactory.timestampTypeInfo, new Timestamp(100));
> String serialized = Utilities.serializeExpression(constant);
> ExprNodeConstantDesc deserilized = (ExprNodeConstantDesc)
> Utilities.deserializeExpression(serialized, new Configuration());
> {noformat}
> logs error message
> {noformat}
> java.lang.InstantiationException: java.sql.Timestamp
> Continuing ...
> java.lang.RuntimeException: failed to evaluate: =Class.new();
> Continuing ...
> {noformat}
> and makes NPE in final.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2333) LazySimpleSerDe does not properly handle arrays / escape control characters
[
https://issues.apache.org/jira/browse/HIVE-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13700215#comment-13700215
]
Edward Capriolo commented on HIVE-2333:
---
Interesting. I am not sure what the semantics should be. Protobuf for example
does not support null arrays and null arrays are empty. What serde' support
what complex types is an interesting question I do not know the answerto. It
would be great to have a table of Lazy, thrift, avro, ocr, rcfile and determine
exactly what is supported by each.
> LazySimpleSerDe does not properly handle arrays / escape control characters
> ---
>
> Key: HIVE-2333
> URL: https://issues.apache.org/jira/browse/HIVE-2333
> Project: Hive
> Issue Type: Bug
>Reporter: Jonathan Chang
>Priority: Critical
>
> LazySimpleSerDe, the default SerDe for Hive is severely broken:
> * Empty arrays are serialized as an empty string. Hence an array(array()) is
> indistinguishable from array(array(array())) from array().
> * Similarly, empty strings are serialized as an empty string. Hence array('')
> is also indistinguishable from an empty array.
> * if the serialized string equals the null sequence, then it is ambiguous as
> to whether it is an array with a single null element or a null array.
> It also does not do well with control characters:
> > select array('foo\002bar') from tmp;
> ...
> ["foo","bar"]
> > select array('foo\001bar') from tmp;
> ...
> ["foo"]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4807) Hive metastore hangs
[
https://issues.apache.org/jira/browse/HIVE-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-4807:
---
Status: Open (was: Patch Available)
All 34 & 22 tests in TestJdbcDriver & TestJdbcDriver2 failed with following
stacktrace:
{code}
org.apache.hive.service.ServiceException: Unable to connect to MetaStore!
at org.apache.hive.service.cli.CLIService.start(CLIService.java:85)
at
org.apache.hive.service.cli.thrift.EmbeddedThriftCLIService.(EmbeddedThriftCLIService.java:35)
at
org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:87)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:111)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:185)
at org.apache.hive.jdbc.TestJdbcDriver2.setUp(TestJdbcDriver2.java:90)
at junit.framework.TestCase.runBare(TestCase.java:125)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:118)
at junit.framework.TestSuite.runTest(TestSuite.java:208)
at junit.framework.TestSuite.run(TestSuite.java:203)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional
connection factory
NestedThrowables:
java.lang.reflect.InvocationTargetException
at
org.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:425)
at
org.datanucleus.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:601)
at
org.datanucleus.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:286)
at
org.datanucleus.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:182)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at javax.jdo.JDOHelper$16.run(JDOHelper.java:1958)
at java.security.AccessController.doPrivileged(Native Method)
at javax.jdo.JDOHelper.invoke(JDOHelper.java:1953)
at
javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1159)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:698)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:263)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:292)
at
org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:225)
at
org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:200)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at
org.apache.hadoop.hive.metastore.RetryingRawStore.(RetryingRawStore.java:62)
at
org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:71)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:414)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:402)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:440)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:326)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.(HiveMetaStore.java:286)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:54)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
at
org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4183)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:121)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:104)
at org.apache.hive.service.cli.CLIService.start(CLIService.java:82)
... 16 more
{code}
> Hive metastore hangs
>
>
> Key:
[jira] [Comment Edited] (HIVE-4807) Hive metastore hangs
[
https://issues.apache.org/jira/browse/HIVE-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13700189#comment-13700189
]
Ashutosh Chauhan edited comment on HIVE-4807 at 7/4/13 4:35 PM:
All 34 & 22 tests in TestJdbcDriver & TestJdbcDriver2 failed with following
stacktrace:
{code}
org.apache.hive.service.ServiceException: Unable to connect to MetaStore!
at org.apache.hive.service.cli.CLIService.start(CLIService.java:85)
at
org.apache.hive.service.cli.thrift.EmbeddedThriftCLIService.(EmbeddedThriftCLIService.java:35)
at
org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:87)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:111)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:185)
at org.apache.hive.jdbc.TestJdbcDriver2.setUp(TestJdbcDriver2.java:90)
at junit.framework.TestCase.runBare(TestCase.java:125)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:118)
at junit.framework.TestSuite.runTest(TestSuite.java:208)
at junit.framework.TestSuite.run(TestSuite.java:203)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional
connection factory
NestedThrowables:
java.lang.reflect.InvocationTargetException
at
org.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:425)
at
org.datanucleus.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:601)
at
org.datanucleus.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:286)
at
org.datanucleus.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:182)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at javax.jdo.JDOHelper$16.run(JDOHelper.java:1958)
at java.security.AccessController.doPrivileged(Native Method)
at javax.jdo.JDOHelper.invoke(JDOHelper.java:1953)
at
javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1159)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:698)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:263)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:292)
at
org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:225)
at
org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:200)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at
org.apache.hadoop.hive.metastore.RetryingRawStore.(RetryingRawStore.java:62)
at
org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:71)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:414)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:402)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:440)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:326)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.(HiveMetaStore.java:286)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:54)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
at
org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4183)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:121)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:104)
at org.apache.hive.service.cli.CLIService.start(CLIService.java:82)
... 16 more
Caused by: java.lang.reflect.Invo
Hive-trunk-hadoop2 - Build # 271 - Still Failing
Changes for Build #238
Changes for Build #239
[hashutosh] HIVE-4713 : Fix TestCliDriver.skewjoin_union_remove_{1,2}.q on 0.23
(Brock Noland via Ashutosh Chauhan)
[hashutosh] HIVE-4715 : Fix
TestCliDriver.{recursive_dir.q,sample_islocalmode_hook.q,input12.q,input39.q,auto_join14.q}
on 0.23 (Brock Noland via Ashutosh Chauhan)
Changes for Build #240
Changes for Build #241
[ecapriolo] Submitted by: Brock Noland Jonathon Chang
Reviewed by: Edward Capriolo
Approved by: Edward Capriolo
Changes for Build #242
Changes for Build #243
Changes for Build #244
[navis] HIVE-4708 : Temporary files are not closed in PTFPersistence on jvm
reuse (Navis reviewed by Harish Butani)
Changes for Build #245
[hashutosh] HIVE-4593 : ErrorMsg has several messages that reuse the same error
code (Navis via Ashutosh Chauhan)
[hashutosh] HIVE-4711 : Fix TestCliDriver.list_bucket_query_oneskew_{1,2,3}.q
on 0.23 (Brock Noland via Ashutosh Chauhan)
Changes for Build #246
[hashutosh] HIVE-4707 : Support configurable domain name for HiveServer2 LDAP
authentication using Active Directory (Prasad Mujumdar via Ashutosh Chauhan)
[gates] HIVE-4969 WebHCat e2e test framework is missing files and instructions
(ekoifman via gates)
Changes for Build #247
[hashutosh] HIVE-4746 : Fix TestCliDriver.list_bucket_dml_{2,4,5,9,12,13}.q on
0.23 (Brock Noland via Ashutosh Chauhan)
Changes for Build #248
[ecapriolo] HIVE-4616: Support outer lateral view
Submitted by: Navis
Reviewed by:Edward Capriolo
Approved by:Edward Capriolo
[gates] HIVE-4679 WebHCat can deadlock Hadoop if the number of concurrently
running tasks if higher or equal than the number of mappers (ekoifman via gates)
Changes for Build #249
[hashutosh] HIVE-3953 : Reading of partitioned Avro data fails because of
missing properties (Mark Wagner via Ashutosh Chauhan)
Changes for Build #250
[hashutosh] HIVE-4761 : ZooKeeperHiveLockManage.unlockPrimitive has race
condition with threads (Brock Noland via Ashutosh Chauhan)
[hashutosh] HIVE-4609 : Allow hive tests to specify an alternative to /tmp
(Brock Noland via Ashutosh Chauhan)
Changes for Build #251
[hashutosh] HIVE-4172 : JDBC2 does not support VOID type (Navis via Ashutosh
Chauhan)
Changes for Build #252
[hashutosh] HIVE-4767 : ObjectStore.getPMF has concurrency problems (Brock
Noland via Ashutosh Chauhan)
[hashutosh] HIVE-4743 : Improve test coverage of package
org.apache.hadoop.hive.ql.io (Ivan Veselovsky via Ashutosh Chauhan)
Changes for Build #253
Changes for Build #254
Changes for Build #255
[ecapriolo] HIVE-1402 Add parallel order by to hive (Navis Ryu and Jeff Zhang
via egc)
Submitted by: Navis Ryu
Reviewed by:Edward Capriolo
Changes for Build #256
Changes for Build #257
[navis] HIVE-4496 : JDBC2 won't compile with JDK7 (Chris Drome via Navis)
Changes for Build #258
Changes for Build #259
Changes for Build #260
Changes for Build #261
Changes for Build #262
[daijy] HIVE-4591: Making changes to webhcat-site.xml have no effect
[daijy] HIVE-4784:ant testreport doesn't include any HCatalog tests
Changes for Build #263
[omalley] Undoing the accidental update of the 0.11 release notes.
[omalley] HIVE-4478. In ORC remove ispresent stream from columns that contain
no null
values in a stripe. (Prasanth Jayachandran via omalley)
Changes for Build #264
Changes for Build #265
[hashutosh] HIVE-4740 : HIVE2379 is missing hbase.jar itself (Navis via
Ashutosh Chauhan)
[hashutosh] HIVE-4756 : Upgrade Hadoop 0.23 profile to 2.0.5-alpha (Brock
Noland via Ashutosh Chauhan)
[hashutosh] HIVE-4791 : improve test coverage of package
org.apache.hadoop.hive.ql.udf.xml (Ivan Veselovsky via Ashutosh Chauhan)
Changes for Build #266
[hashutosh] HIVE-4618 : show create table creating unusable DDL when field
delimiter is \001 (Navis via Ashutosh Chauhan)
[hashutosh] HIVE-4559 : hcatalog/webhcat scripts in tar.gz don't have execute
permissions set (Eugene Koifman via Ashutosh Chauhan)
[hashutosh] HIVE-4798 : NPE when we call isSame from an instance of
ExprNodeConstantDesc with null value (Yin Huai via Ashutosh Chauhan)
[hashutosh] HIVE-4781 : LEFT SEMI JOIN generates wrong results when the number
of rows belonging to a single key of the right table exceed
hive.join.emit.interval (Yin Huai via Ashutosh Chauhan)
[hashutosh] HIVE-4647 : RetryingHMSHandler logs too many error messages (Navis
via Ashutosh Chauhan)
[hashutosh] HIVE-4692 : Constant agg parameters will be replaced by
ExprNodeColumnDesc with single-sourced multi-gby cases (Navis via Ashutosh
Chauhan)
Changes for Build #267
[hashutosh] HIVE-4781 : Adding new data files for tests. Missed in original
commit.
Changes for Build #268
[navis] HIVE-2517 : Support group by on struct type (Ashutosh Chauhan via Navis)
[hashutosh] HIVE-4406 : Missing / or / in hs2 jdbc uri switches mode to
embedded mode(Anandha Ranganathan via Ashutosh Chauhan)
[hashutosh] HIVE-4430 : Semantic analysis fails in presenc
[jira] [Commented] (HIVE-4388) HBase tests fail against Hadoop 2
[ https://issues.apache.org/jira/browse/HIVE-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13700157#comment-13700157 ] Brock Noland commented on HIVE-4388: It doesn't sound like HBASE is going to publish a 0.94 hadoop2 artifact. I think to get our tests running we'll have to publish a hbase 0.94 hadoop2 artifact. Perhaps on people.apache.org. > HBase tests fail against Hadoop 2 > - > > Key: HIVE-4388 > URL: https://issues.apache.org/jira/browse/HIVE-4388 > Project: Hive > Issue Type: Bug >Reporter: Gunther Hagleitner > > Currently we're building by default against 0.92. When you run against hadoop > 2 (-Dhadoop.mr.rev=23) builds fail because of: HBASE-5963. > HIVE-3861 upgrades the version of hbase used. This will get you past the > problem in HBASE-5963 (which was fixed in 0.94.1) but fails with: HBASE-6396. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1918) Add export/import facilities to the hive system
[ https://issues.apache.org/jira/browse/HIVE-1918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13699829#comment-13699829 ] Gelesh commented on HIVE-1918: -- Wish:- Can We have a option to copy meta information alone, UseCase:- So that, during DistCp with out copying the Hive files, (with partition folder and clustered file structure) to a temp location, we can create a _meta file alone. Then, DistCp the hive files (the partioned and clusteded file structure) as such and load re create a hive table in the new cluster. > Add export/import facilities to the hive system > --- > > Key: HIVE-1918 > URL: https://issues.apache.org/jira/browse/HIVE-1918 > Project: Hive > Issue Type: New Feature > Components: Metastore, Query Processor >Reporter: Krishna Kumar >Assignee: Krishna Kumar > Fix For: 0.8.0 > > Attachments: HIVE-1918.patch.1.txt, HIVE-1918.patch.2.txt, > HIVE-1918.patch.3.txt, HIVE-1918.patch.4.txt, HIVE-1918.patch.5.txt, > HIVE-1918.patch.txt, hive-metastore-er.pdf > > > This is an enhancement request to add export/import features to hive. > With this language extension, the user can export the data of the table - > which may be located in different hdfs locations in case of a partitioned > table - as well as the metadata of the table into a specified output > location. This output location can then be moved over to another different > hadoop/hive instance and imported there. > This should work independent of the source and target metastore dbms used; > for instance, between derby and mysql. > For partitioned tables, the ability to export/import a subset of the > partition must be supported. > Howl will add more features on top of this: The ability to create/use the > exported data even in the absence of hive, using MR or Pig. Please see > http://wiki.apache.org/pig/Howl/HowlImportExport for these details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4160) Vectorized Query Execution in Hive
[ https://issues.apache.org/jira/browse/HIVE-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13699820#comment-13699820 ] Vinod Kumar Vavilapalli commented on HIVE-4160: --- A huge +1 to that. Having a common set of operators will be a huge win. That said, I already see that the current branch follows Hive's operator base classes, uses HiveConf etc. I believe with little effort, this can be cleaned and pulled apart into one separate maven module that everyone can use. Some points to think about: - The target location of the module. The dependency graph can become un-wieldly. - Given the use of base Operator, OperatorDesc etc from Hive, if at all there is interest and commitment, we should do this ASAP when we only have a handful of operators. - Make one other project demonstrate how it can be reused across ecosystem projects, PIG will be great - just a few operators will be a great start Thoughts? > Vectorized Query Execution in Hive > -- > > Key: HIVE-4160 > URL: https://issues.apache.org/jira/browse/HIVE-4160 > Project: Hive > Issue Type: New Feature >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Attachments: Hive-Vectorized-Query-Execution-Design.docx, > Hive-Vectorized-Query-Execution-Design-rev2.docx, > Hive-Vectorized-Query-Execution-Design-rev3.docx, > Hive-Vectorized-Query-Execution-Design-rev3.docx, > Hive-Vectorized-Query-Execution-Design-rev3.pdf, > Hive-Vectorized-Query-Execution-Design-rev4.docx, > Hive-Vectorized-Query-Execution-Design-rev4.pdf, > Hive-Vectorized-Query-Execution-Design-rev5.docx, > Hive-Vectorized-Query-Execution-Design-rev5.pdf, > Hive-Vectorized-Query-Execution-Design-rev6.docx, > Hive-Vectorized-Query-Execution-Design-rev6.pdf, > Hive-Vectorized-Query-Execution-Design-rev7.docx, > Hive-Vectorized-Query-Execution-Design-rev8.docx, > Hive-Vectorized-Query-Execution-Design-rev8.pdf, > Hive-Vectorized-Query-Execution-Design-rev9.docx, > Hive-Vectorized-Query-Execution-Design-rev9.pdf > > > The Hive query execution engine currently processes one row at a time. A > single row of data goes through all the operators before the next row can be > processed. This mode of processing is very inefficient in terms of CPU usage. > Research has demonstrated that this yields very low instructions per cycle > [MonetDB X100]. Also currently Hive heavily relies on lazy deserialization > and data columns go through a layer of object inspectors that identify column > type, deserialize data and determine appropriate expression routines in the > inner loop. These layers of virtual method calls further slow down the > processing. > This work will add support for vectorized query execution to Hive, where, > instead of individual rows, batches of about a thousand rows at a time are > processed. Each column in the batch is represented as a vector of a primitive > data type. The inner loop of execution scans these vectors very fast, > avoiding method calls, deserialization, unnecessary if-then-else, etc. This > substantially reduces CPU time used, and gives excellent instructions per > cycle (i.e. improved processor pipeline utilization). See the attached design > specification for more details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
