[jira] [Commented] (HIVE-9500) Support nested structs over 24 levels.
[ https://issues.apache.org/jira/browse/HIVE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314265#comment-14314265 ] Xuefu Zhang commented on HIVE-9500: --- +1 Support nested structs over 24 levels. -- Key: HIVE-9500 URL: https://issues.apache.org/jira/browse/HIVE-9500 Project: Hive Issue Type: Improvement Reporter: Aihua Xu Assignee: Aihua Xu Labels: SerDe Fix For: 1.2.0 Attachments: HIVE-9500.1.patch, HIVE-9500.2.patch, HIVE-9500.3.patch Customer has deeply nested avro structure and is receiving the following error when performing queries. 15/01/09 20:59:29 ERROR ql.Driver: FAILED: SemanticException org.apache.hadoop.hive.serde2.SerDeException: Number of levels of nesting supported for LazySimpleSerde is 23 Unable to work with level 24 Currently we support up to 24 levels of nested structs when hive.serialization.extend.nesting.levels is set to true, while the customers have the requirement to support more than that. It would be better to make the supported levels configurable or completely removed (i.e., we can support any number of levels). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9612) Turn off DEBUG logging for Lazy Objects for tests
[
https://issues.apache.org/jira/browse/HIVE-9612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-9612:
--
Attachment: HIVE-9612.1.patch
Turn off DEBUG logging for Lazy Objects for tests
-
Key: HIVE-9612
URL: https://issues.apache.org/jira/browse/HIVE-9612
Project: Hive
Issue Type: Bug
Reporter: Brock Noland
Assignee: Sergio Peña
Attachments: HIVE-9612.1.patch
Our tests are collecting a tremendous amounts of logs:
{noformat}
[root@ip-10-152-185-204 TestRCFile]# pwd
/data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-2709/succeeded/TestRCFile
[root@ip-10-152-185-204 TestRCFile]# ls -lh hive.log
-rw-r--r-- 1 hiveptest hiveptest 143M Feb 8 03:54 hive.log
{noformat}
Much of this logging is due to stack traces printed at DEBUG.
{noformat}
2015-02-08 00:54:07,942 DEBUG [main]: lazy.LazyDouble
(LazyDouble.java:init(55)) - Data not in the Double data type range so
converted to null. Given data is :
java.lang.NumberFormatException: empty String
at
sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1011)
at java.lang.Double.parseDouble(Double.java:540)
at
org.apache.hadoop.hive.serde2.lazy.LazyDouble.init(LazyDouble.java:51)
at
org.apache.hadoop.hive.serde2.columnar.ColumnarStructBase$FieldInfo.uncheckedGetField(ColumnarStructBase.java:111)
at
org.apache.hadoop.hive.serde2.columnar.ColumnarStructBase.getFieldsAsList(ColumnarStructBase.java:224)
at
org.apache.hadoop.hive.serde2.objectinspector.ColumnarStructObjectInspector.getStructFieldsDataAsList(ColumnarStructObjectInspector.java:76)
at
org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe.serialize(ColumnarSerDe.java:144)
at
org.apache.hadoop.hive.ql.io.TestRCFile.partialReadTest(TestRCFile.java:598)
at
org.apache.hadoop.hive.ql.io.TestRCFile.testWriteAndPartialRead(TestRCFile.java:417)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
2015-02-08 00:54:17,992 DEBUG [main]: lazy.LazyPrimitive
(LazyPrimitive.java:logExceptionMessage(81)) - Data not in the INT data type
range so converted to null. Given data is :
java.lang.Exception: For debugging purposes
at
org.apache.hadoop.hive.serde2.lazy.LazyPrimitive.logExceptionMessage(LazyPrimitive.java:81)
at
org.apache.hadoop.hive.serde2.lazy.LazyInteger.init(LazyInteger.java:59)
at
org.apache.hadoop.hive.serde2.columnar.ColumnarStructBase$FieldInfo.uncheckedGetField(ColumnarStructBase.java:111)
at
org.apache.hadoop.hive.serde2.columnar.ColumnarStructBase.getField(ColumnarStructBase.java:172)
at
[jira] [Updated] (HIVE-9612) Turn off DEBUG logging for Lazy Objects for tests
[
https://issues.apache.org/jira/browse/HIVE-9612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-9612:
--
Status: Patch Available (was: Open)
Turn off DEBUG logging for Lazy Objects for tests
-
Key: HIVE-9612
URL: https://issues.apache.org/jira/browse/HIVE-9612
Project: Hive
Issue Type: Bug
Reporter: Brock Noland
Assignee: Sergio Peña
Attachments: HIVE-9612.1.patch
Our tests are collecting a tremendous amounts of logs:
{noformat}
[root@ip-10-152-185-204 TestRCFile]# pwd
/data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-2709/succeeded/TestRCFile
[root@ip-10-152-185-204 TestRCFile]# ls -lh hive.log
-rw-r--r-- 1 hiveptest hiveptest 143M Feb 8 03:54 hive.log
{noformat}
Much of this logging is due to stack traces printed at DEBUG.
{noformat}
2015-02-08 00:54:07,942 DEBUG [main]: lazy.LazyDouble
(LazyDouble.java:init(55)) - Data not in the Double data type range so
converted to null. Given data is :
java.lang.NumberFormatException: empty String
at
sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1011)
at java.lang.Double.parseDouble(Double.java:540)
at
org.apache.hadoop.hive.serde2.lazy.LazyDouble.init(LazyDouble.java:51)
at
org.apache.hadoop.hive.serde2.columnar.ColumnarStructBase$FieldInfo.uncheckedGetField(ColumnarStructBase.java:111)
at
org.apache.hadoop.hive.serde2.columnar.ColumnarStructBase.getFieldsAsList(ColumnarStructBase.java:224)
at
org.apache.hadoop.hive.serde2.objectinspector.ColumnarStructObjectInspector.getStructFieldsDataAsList(ColumnarStructObjectInspector.java:76)
at
org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe.serialize(ColumnarSerDe.java:144)
at
org.apache.hadoop.hive.ql.io.TestRCFile.partialReadTest(TestRCFile.java:598)
at
org.apache.hadoop.hive.ql.io.TestRCFile.testWriteAndPartialRead(TestRCFile.java:417)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
2015-02-08 00:54:17,992 DEBUG [main]: lazy.LazyPrimitive
(LazyPrimitive.java:logExceptionMessage(81)) - Data not in the INT data type
range so converted to null. Given data is :
java.lang.Exception: For debugging purposes
at
org.apache.hadoop.hive.serde2.lazy.LazyPrimitive.logExceptionMessage(LazyPrimitive.java:81)
at
org.apache.hadoop.hive.serde2.lazy.LazyInteger.init(LazyInteger.java:59)
at
org.apache.hadoop.hive.serde2.columnar.ColumnarStructBase$FieldInfo.uncheckedGetField(ColumnarStructBase.java:111)
at
org.apache.hadoop.hive.serde2.columnar.ColumnarStructBase.getField(ColumnarStructBase.java:172)
at
[jira] [Commented] (HIVE-9626) Hive CREATE table failed using regular experssion serde
[
https://issues.apache.org/jira/browse/HIVE-9626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314358#comment-14314358
]
Will Du commented on HIVE-9626:
---
The workround found is backslash ; as follows
CREATE TABLE test_serde_rex(
name string,
sex string,
age string
)
ROW FORMAT SERDE
'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
WITH SERDEPROPERTIES(
'input.regex' = '([^\;]*)\;([^\;]*)\;([^\;]*)',
'output.format.string' = '%1$s %2$s %3$s'
)
STORED AS TEXTFILE;
Hive CREATE table failed using regular experssion serde
---
Key: HIVE-9626
URL: https://issues.apache.org/jira/browse/HIVE-9626
Project: Hive
Issue Type: Bug
Components: Parser
Affects Versions: 1.0.0
Environment: Apache Hive 1.0.0
Reporter: Will Du
Run the below CREATE table statement is successful in Hive 0.13.0 and 0.14.0.
However, it is failed in the 1.0.0 of Hive.
CREATE TABLE test_serde_rex(
name string,
sex string,
age string
)
ROW FORMAT SERDE
'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
WITH SERDEPROPERTIES(
'input.regex' = '([^;]*);([^;]*);([^;]*)',
'output.format.string' = '%1$s %2$s %3$s'
)
STORED AS TEXTFILE;
The exception in Hive 1.0.0 is as follows
MismatchedTokenException(283!=300)
at
org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115)
at
org.apache.hadoop.hive.ql.parse.HiveParser.keyValueProperty(HiveParser.java:33488)
at
org.apache.hadoop.hive.ql.parse.HiveParser.tablePropertiesList(HiveParser.java:33258)
at
org.apache.hadoop.hive.ql.parse.HiveParser.tableProperties(HiveParser.java:33121)
at
org.apache.hadoop.hive.ql.parse.HiveParser.rowFormatSerde(HiveParser.java:32497)
at
org.apache.hadoop.hive.ql.parse.HiveParser.tableRowFormat(HiveParser.java:32962)
at
org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:4916)
at
org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:2350)
at
org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1574)
at
org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1052)
at
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:199)
at
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:389)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:303)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1067)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1129)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:994)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:201)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:153)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:364)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:712)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:631)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:570)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
FAILED: ParseException line 9:17 mismatched input '(' expecting StringLiteral
near '=' in specifying key/value property
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9612) Turn off DEBUG logging for Lazy Objects for tests
[
https://issues.apache.org/jira/browse/HIVE-9612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314395#comment-14314395
]
Szehon Ho commented on HIVE-9612:
-
Appreciate it, will be a big help. Though for the lazy objects debug log, that
might be needed in hive-exec as well as Brock indicated?
Turn off DEBUG logging for Lazy Objects for tests
-
Key: HIVE-9612
URL: https://issues.apache.org/jira/browse/HIVE-9612
Project: Hive
Issue Type: Bug
Reporter: Brock Noland
Assignee: Sergio Peña
Attachments: HIVE-9612.1.patch
Our tests are collecting a tremendous amounts of logs:
{noformat}
[root@ip-10-152-185-204 TestRCFile]# pwd
/data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-2709/succeeded/TestRCFile
[root@ip-10-152-185-204 TestRCFile]# ls -lh hive.log
-rw-r--r-- 1 hiveptest hiveptest 143M Feb 8 03:54 hive.log
{noformat}
Much of this logging is due to stack traces printed at DEBUG.
{noformat}
2015-02-08 00:54:07,942 DEBUG [main]: lazy.LazyDouble
(LazyDouble.java:init(55)) - Data not in the Double data type range so
converted to null. Given data is :
java.lang.NumberFormatException: empty String
at
sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1011)
at java.lang.Double.parseDouble(Double.java:540)
at
org.apache.hadoop.hive.serde2.lazy.LazyDouble.init(LazyDouble.java:51)
at
org.apache.hadoop.hive.serde2.columnar.ColumnarStructBase$FieldInfo.uncheckedGetField(ColumnarStructBase.java:111)
at
org.apache.hadoop.hive.serde2.columnar.ColumnarStructBase.getFieldsAsList(ColumnarStructBase.java:224)
at
org.apache.hadoop.hive.serde2.objectinspector.ColumnarStructObjectInspector.getStructFieldsDataAsList(ColumnarStructObjectInspector.java:76)
at
org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe.serialize(ColumnarSerDe.java:144)
at
org.apache.hadoop.hive.ql.io.TestRCFile.partialReadTest(TestRCFile.java:598)
at
org.apache.hadoop.hive.ql.io.TestRCFile.testWriteAndPartialRead(TestRCFile.java:417)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
2015-02-08 00:54:17,992 DEBUG [main]: lazy.LazyPrimitive
(LazyPrimitive.java:logExceptionMessage(81)) - Data not in the INT data type
range so converted to null. Given data is :
java.lang.Exception: For debugging purposes
at
org.apache.hadoop.hive.serde2.lazy.LazyPrimitive.logExceptionMessage(LazyPrimitive.java:81)
at
org.apache.hadoop.hive.serde2.lazy.LazyInteger.init(LazyInteger.java:59)
at
[jira] [Updated] (HIVE-9622) Getting NPE when trying to restart HS2 when metastore is configured to use org.apache.hadoop.hive.thrift.DBTokenStore
[
https://issues.apache.org/jira/browse/HIVE-9622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-9622:
---
Status: In Progress (was: Patch Available)
Getting NPE when trying to restart HS2 when metastore is configured to use
org.apache.hadoop.hive.thrift.DBTokenStore
-
Key: HIVE-9622
URL: https://issues.apache.org/jira/browse/HIVE-9622
Project: Hive
Issue Type: Bug
Reporter: Aihua Xu
Assignee: Aihua Xu
Labels: HiveServer2, Security
Fix For: 1.2.0
Attachments: HIVE-9622.1.patch
# Configure the cluster to use kerberos for HS2 and Metastore.
##
http://www.cloudera.com/content/cloudera/en/documentation/cdh4/v4-3-0/CDH4-Security-Guide/cdh4sg_topic_9_1.html
##
http://www.cloudera.com/content/cloudera/en/documentation/cdh4/v4-6-0/CDH4-Security-Guide/cdh4sg_topic_9_2.html
# Set hive metastore delegation token to
org.apache.hadoop.hive.thrift.DBTokenStore in hive-site.xml
{code}
property
namehive.cluster.delegation.token.store.class/name
valueorg.apache.hadoop.hive.thrift.DBTokenStore/value
/property
{code}
# Then trying to restart hive service, HS2 fails to start the NPE below:
{code}
9:43:10.711 AMERROR
org.apache.hive.service.cli.thrift.ThriftCLIService
Error:
org.apache.thrift.transport.TTransportException: Failed to start token manager
at
org.apache.hive.service.auth.HiveAuthFactory.init(HiveAuthFactory.java:107)
at
org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:51)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed to initialize master key
at
org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:223)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server.startDelegationTokenSecretManager(HadoopThriftAuthBridge20S.java:438)
at
org.apache.hive.service.auth.HiveAuthFactory.init(HiveAuthFactory.java:105)
... 2 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at
org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:221)
... 4 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnRawStore(DBTokenStore.java:145)
at
org.apache.hadoop.hive.thrift.DBTokenStore.addMasterKey(DBTokenStore.java:41)
at
org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.logUpdateMasterKey(TokenStoreDelegationTokenSecretManager.java:203)
at
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.updateCurrentKey(AbstractDelegationTokenSecretManager.java:339)
... 9 more
9:43:10.719 AMINFOorg.apache.hive.service.server.HiveServer2
SHUTDOWN_MSG:
/
SHUTDOWN_MSG: Shutting down HiveServer2 at
a1909.halxg.cloudera.com/10.20.202.109
/
{code}
The problem appears that we didn't pass a {{RawStore}} object in the
following:
https://github.com/apache/hive/blob/trunk/service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java#L111
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9228) Problem with subquery using windowing functions
[ https://issues.apache.org/jira/browse/HIVE-9228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314227#comment-14314227 ] Aihua Xu commented on HIVE-9228: Thanks for your contribution. Navis. Problem with subquery using windowing functions --- Key: HIVE-9228 URL: https://issues.apache.org/jira/browse/HIVE-9228 Project: Hive Issue Type: Bug Components: PTF-Windowing Affects Versions: 0.14.0, 0.13.1, 1.0.0 Reporter: Aihua Xu Assignee: Navis Fix For: 1.2.0 Attachments: HIVE-9228.1.patch.txt, HIVE-9228.2.patch.txt, HIVE-9228.3.patch.txt, create_table_tab1.sql, tab1.csv Original Estimate: 96h Remaining Estimate: 96h The following query with window functions failed. The internal query works fine. select col1, col2, col3 from (select col1,col2, col3, count(case when col4=1 then 1 end ) over (partition by col1, col2) as col5, row_number() over (partition by col1, col2 order by col4) as col6 from tab1) t; HIVE generates an execution plan with 2 jobs. 1. The first job is to basically calculate window function for col5. 2. The second job is to calculate window function for col6 and output. The plan says the first job outputs the columns (col1, col2, col3, col4) to a tmp file since only these columns are used in later stage. While, the PTF operator for the first job outputs (_wcol0, col1, col2, col3, col4) with _wcol0 as the result of the window function even it's not used. In the second job, the map operator still reads the 4 columns (col1, col2, col3, col4) from the temp file using the plan. That causes the exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9626) Hive CREATE table failed using regular experssion serde
[
https://issues.apache.org/jira/browse/HIVE-9626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314353#comment-14314353
]
Will Du commented on HIVE-9626:
---
The further finding is because it uses ; as field separator. If we use other
separator, such as |, the query can run successful. But it is still a bug since
in 0.13, 0.14, it can create such table and parser data correctly.
Hive CREATE table failed using regular experssion serde
---
Key: HIVE-9626
URL: https://issues.apache.org/jira/browse/HIVE-9626
Project: Hive
Issue Type: Bug
Components: Parser
Affects Versions: 1.0.0
Environment: Apache Hive 1.0.0
Reporter: Will Du
Run the below CREATE table statement is successful in Hive 0.13.0 and 0.14.0.
However, it is failed in the 1.0.0 of Hive.
CREATE TABLE test_serde_rex(
name string,
sex string,
age string
)
ROW FORMAT SERDE
'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
WITH SERDEPROPERTIES(
'input.regex' = '([^;]*);([^;]*);([^;]*)',
'output.format.string' = '%1$s %2$s %3$s'
)
STORED AS TEXTFILE;
The exception in Hive 1.0.0 is as follows
MismatchedTokenException(283!=300)
at
org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115)
at
org.apache.hadoop.hive.ql.parse.HiveParser.keyValueProperty(HiveParser.java:33488)
at
org.apache.hadoop.hive.ql.parse.HiveParser.tablePropertiesList(HiveParser.java:33258)
at
org.apache.hadoop.hive.ql.parse.HiveParser.tableProperties(HiveParser.java:33121)
at
org.apache.hadoop.hive.ql.parse.HiveParser.rowFormatSerde(HiveParser.java:32497)
at
org.apache.hadoop.hive.ql.parse.HiveParser.tableRowFormat(HiveParser.java:32962)
at
org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:4916)
at
org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:2350)
at
org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1574)
at
org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1052)
at
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:199)
at
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:389)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:303)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1067)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1129)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:994)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:201)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:153)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:364)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:712)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:631)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:570)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
FAILED: ParseException line 9:17 mismatched input '(' expecting StringLiteral
near '=' in specifying key/value property
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9596) move standard getDisplayString impl to GenericUDF
[
https://issues.apache.org/jira/browse/HIVE-9596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314362#comment-14314362
]
Hive QA commented on HIVE-9596:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12697623/HIVE-9596.4.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 7540 tests executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchAbort
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2738/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2738/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2738/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12697623 - PreCommit-HIVE-TRUNK-Build
move standard getDisplayString impl to GenericUDF
-
Key: HIVE-9596
URL: https://issues.apache.org/jira/browse/HIVE-9596
Project: Hive
Issue Type: Improvement
Components: UDF
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov
Priority: Minor
Attachments: HIVE-9596.1.patch, HIVE-9596.2.patch, HIVE-9596.3.patch,
HIVE-9596.4.patch
54 GenericUDF derived classes have very similar getDisplayString impl which
returns fname(child1, child2, childn)
instr() and locate() have bugs in their implementation (no comma btw children)
Instead of having 54 implementations of the same method it's better to move
standard implementation to the base class.
affected UDF classes:
{code}
contrib/src/java/org/apache/hadoop/hive/contrib/genericudf/example/GenericUDFDBOutput.java
itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEvaluateNPE.java
itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTestGetJavaBoolean.java
itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTestGetJavaString.java
itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTestTranslate.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFEWAHBitmapBop.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFReflect.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAbs.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAddMonths.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFArray.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrue.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseNumeric.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBasePad.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseTrim.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCoalesce.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcat.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcatWS.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDate.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDateAdd.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDateDiff.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDateSub.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDecode.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEWAHBitmapEmpty.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFElt.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEncode.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFField.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFloorCeilBase.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFormatNumber.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFGreatest.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFHash.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIf.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInitCap.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInstr.java
Re: VOTE Bylaw for having branch committers in hive
+1 On Tue, Feb 10, 2015 at 4:28 PM, Prasanth Jayachandran pjayachand...@hortonworks.com wrote: +1 for the proposal. On Feb 9, 2015, at 2:50 PM, Lefty Leverenz leftylever...@gmail.com wrote: +1 cc'ing the PMC mailing list -- Lefty On Mon, Feb 9, 2015 at 2:48 PM, Gunther Hagleitner ghagleit...@hortonworks.com wrote: Very useful: +1 Thanks, Gunther. From: Vikram Dixit K vikram.di...@gmail.com Sent: Monday, February 09, 2015 2:43 PM To: dev@hive.apache.org Subject: VOTE Bylaw for having branch committers in hive Hi Folks, We seem to have quite a few projects going around and in the interest of time and the project as a whole, it seems good to have branch committers much like what is there in the Hadoop project. I am proposing an addition to the committer bylaws as follows ( taken from the hadoop project bylaws http://hadoop.apache.org/bylaws.html ) Significant, pervasive features are often developed in a speculative branch of the repository. The PMC may grant commit rights on the branch to its consistent contributors, while the initiative is active. Branch committers are responsible for shepherding their feature into an active release and do not cast binding votes or vetoes in the project. Actions: New Branch Committer Description: When a new branch committer is proposed for the project. Approval: Lazy Consensus Binding Votes: Active PMC members Minimum Length: 3 days Mailing List: priv...@hive.apache.org Actions: Removal of Branch Committer Description: When a branch committer is removed from the project. Approval: Consensus Binding Votes: Active PMC members excluding the committer in question if they are PMC members too. Minimum Length: 6 days Mailing List: priv...@hive.apache.org This vote will run for 6 days. PMC members please vote. Thanks Vikram.
[jira] [Commented] (HIVE-9500) Support nested structs over 24 levels.
[
https://issues.apache.org/jira/browse/HIVE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14315494#comment-14315494
]
Hive QA commented on HIVE-9500:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12697892/HIVE-9500.4.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 7540 tests executed
*Failed tests:*
{noformat}
TestHs2Hooks - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_unionDistinct_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_unionDistinct_1
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2748/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2748/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2748/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12697892 - PreCommit-HIVE-TRUNK-Build
Support nested structs over 24 levels.
--
Key: HIVE-9500
URL: https://issues.apache.org/jira/browse/HIVE-9500
Project: Hive
Issue Type: Improvement
Reporter: Aihua Xu
Assignee: Aihua Xu
Labels: SerDe
Fix For: 1.2.0
Attachments: HIVE-9500.1.patch, HIVE-9500.2.patch, HIVE-9500.3.patch,
HIVE-9500.4.patch, TClass.java
Customer has deeply nested avro structure and is receiving the following
error when performing queries.
15/01/09 20:59:29 ERROR ql.Driver: FAILED: SemanticException
org.apache.hadoop.hive.serde2.SerDeException: Number of levels of nesting
supported for LazySimpleSerde is 23 Unable to work with level 24
Currently we support up to 24 levels of nested structs when
hive.serialization.extend.nesting.levels is set to true, while the customers
have the requirement to support more than that.
It would be better to make the supported levels configurable or completely
removed (i.e., we can support any number of levels).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
RE: [ANNOUNCE] New Hive Committers -- Chao Sun, Chengxiang Li, and Rui Li
Thanks guys. It's a great honor! Cheers, Rui Li -Original Message- From: Vaibhav Gumashta [mailto:vgumas...@hortonworks.com] Sent: Tuesday, February 10, 2015 6:12 AM To: u...@hive.apache.org; dev@hive.apache.org Subject: Re: [ANNOUNCE] New Hive Committers -- Chao Sun, Chengxiang Li, and Rui Li Congratulations to all. Vaibhav On 2/9/15, 2:06 PM, Prasanth Jayachandran pjayachand...@hortonworks.com wrote: Congratulations! On Feb 9, 2015, at 1:57 PM, Na Yang ny...@maprtech.com wrote: Congratulations! On Mon, Feb 9, 2015 at 1:06 PM, Vikram Dixit K vikram.di...@gmail.com wrote: Congrats guys! On Mon, Feb 9, 2015 at 12:42 PM, Szehon Ho sze...@cloudera.com wrote: Congratulations guys ! On Mon, Feb 9, 2015 at 3:38 PM, Jimmy Xiang jxi...@cloudera.com wrote: Congrats!! On Mon, Feb 9, 2015 at 12:36 PM, Alexander Pivovarov apivova...@gmail.com wrote: Congrats! On Mon, Feb 9, 2015 at 12:31 PM, Carl Steinbach c...@apache.org wrote: The Apache Hive PMC has voted to make Chao Sun, Chengxiang Li, and Rui Li committers on the Apache Hive Project. Please join me in congratulating Chao, Chengxiang, and Rui! Thanks. - Carl -- Nothing better than when appreciated for hard work. -Mark
[jira] [Updated] (HIVE-6617) Reduce ambiguity in grammar
[ https://issues.apache.org/jira/browse/HIVE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-6617: -- Status: Patch Available (was: Open) Reduce ambiguity in grammar --- Key: HIVE-6617 URL: https://issues.apache.org/jira/browse/HIVE-6617 Project: Hive Issue Type: Task Reporter: Ashutosh Chauhan Assignee: Pengcheng Xiong Attachments: HIVE-6617.01.patch, HIVE-6617.02.patch, HIVE-6617.03.patch, HIVE-6617.04.patch, HIVE-6617.05.patch, HIVE-6617.06.patch, HIVE-6617.07.patch, HIVE-6617.08.patch, HIVE-6617.09.patch, HIVE-6617.10.patch, HIVE-6617.11.patch, HIVE-6617.12.patch CLEAR LIBRARY CACHE As of today, antlr reports 214 warnings. Need to bring down this number, ideally to 0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6617) Reduce ambiguity in grammar
[ https://issues.apache.org/jira/browse/HIVE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-6617: -- Status: Open (was: Patch Available) Reduce ambiguity in grammar --- Key: HIVE-6617 URL: https://issues.apache.org/jira/browse/HIVE-6617 Project: Hive Issue Type: Task Reporter: Ashutosh Chauhan Assignee: Pengcheng Xiong Attachments: HIVE-6617.01.patch, HIVE-6617.02.patch, HIVE-6617.03.patch, HIVE-6617.04.patch, HIVE-6617.05.patch, HIVE-6617.06.patch, HIVE-6617.07.patch, HIVE-6617.08.patch, HIVE-6617.09.patch, HIVE-6617.10.patch, HIVE-6617.11.patch, HIVE-6617.12.patch CLEAR LIBRARY CACHE As of today, antlr reports 214 warnings. Need to bring down this number, ideally to 0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6617) Reduce ambiguity in grammar
[ https://issues.apache.org/jira/browse/HIVE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-6617: -- Attachment: HIVE-6617.12.patch Now, zero warnings. Need to wait for QA. Reduce ambiguity in grammar --- Key: HIVE-6617 URL: https://issues.apache.org/jira/browse/HIVE-6617 Project: Hive Issue Type: Task Reporter: Ashutosh Chauhan Assignee: Pengcheng Xiong Attachments: HIVE-6617.01.patch, HIVE-6617.02.patch, HIVE-6617.03.patch, HIVE-6617.04.patch, HIVE-6617.05.patch, HIVE-6617.06.patch, HIVE-6617.07.patch, HIVE-6617.08.patch, HIVE-6617.09.patch, HIVE-6617.10.patch, HIVE-6617.11.patch, HIVE-6617.12.patch CLEAR LIBRARY CACHE As of today, antlr reports 214 warnings. Need to bring down this number, ideally to 0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9648) Null check key provider before doing set
Brock Noland created HIVE-9648: -- Summary: Null check key provider before doing set Key: HIVE-9648 URL: https://issues.apache.org/jira/browse/HIVE-9648 Project: Hive Issue Type: Bug Components: Encryption Reporter: Brock Noland Some versions of hdfs do not like DFSClient.setkeyProvider(null). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
RE: [ANNOUNCE] New Hive Committers -- Chao Sun, Chengxiang Li, and Rui Li
Congrats! -Original Message- From: Li, Rui [mailto:rui...@intel.com] Sent: Wednesday, February 11, 2015 10:26 AM To: u...@hive.apache.org; dev@hive.apache.org Subject: RE: [ANNOUNCE] New Hive Committers -- Chao Sun, Chengxiang Li, and Rui Li Thanks guys. It's a great honor! Cheers, Rui Li -Original Message- From: Vaibhav Gumashta [mailto:vgumas...@hortonworks.com] Sent: Tuesday, February 10, 2015 6:12 AM To: u...@hive.apache.org; dev@hive.apache.org Subject: Re: [ANNOUNCE] New Hive Committers -- Chao Sun, Chengxiang Li, and Rui Li Congratulations to all. Vaibhav On 2/9/15, 2:06 PM, Prasanth Jayachandran pjayachand...@hortonworks.com wrote: Congratulations! On Feb 9, 2015, at 1:57 PM, Na Yang ny...@maprtech.com wrote: Congratulations! On Mon, Feb 9, 2015 at 1:06 PM, Vikram Dixit K vikram.di...@gmail.com wrote: Congrats guys! On Mon, Feb 9, 2015 at 12:42 PM, Szehon Ho sze...@cloudera.com wrote: Congratulations guys ! On Mon, Feb 9, 2015 at 3:38 PM, Jimmy Xiang jxi...@cloudera.com wrote: Congrats!! On Mon, Feb 9, 2015 at 12:36 PM, Alexander Pivovarov apivova...@gmail.com wrote: Congrats! On Mon, Feb 9, 2015 at 12:31 PM, Carl Steinbach c...@apache.org wrote: The Apache Hive PMC has voted to make Chao Sun, Chengxiang Li, and Rui Li committers on the Apache Hive Project. Please join me in congratulating Chao, Chengxiang, and Rui! Thanks. - Carl -- Nothing better than when appreciated for hard work. -Mark
[jira] [Updated] (HIVE-9648) Null check key provider before doing set
[ https://issues.apache.org/jira/browse/HIVE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-9648: --- Attachment: HIVE-9648.patch Null check key provider before doing set Key: HIVE-9648 URL: https://issues.apache.org/jira/browse/HIVE-9648 Project: Hive Issue Type: Bug Components: Encryption Affects Versions: 1.1.0 Reporter: Brock Noland Attachments: HIVE-9648.patch Some versions of hdfs do not like DFSClient.setkeyProvider(null). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9648) Null check key provider before doing set
[ https://issues.apache.org/jira/browse/HIVE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-9648: --- Affects Version/s: 1.1.0 Status: Patch Available (was: Open) Null check key provider before doing set Key: HIVE-9648 URL: https://issues.apache.org/jira/browse/HIVE-9648 Project: Hive Issue Type: Bug Components: Encryption Affects Versions: 1.1.0 Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-9648.patch Some versions of hdfs do not like DFSClient.setkeyProvider(null). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9646) Beeline doesn't show Spark job progress info [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-9646: -- Attachment: HIVE-9646.1-spark.patch Beeline doesn't show Spark job progress info [Spark Branch] --- Key: HIVE-9646 URL: https://issues.apache.org/jira/browse/HIVE-9646 Project: Hive Issue Type: Bug Affects Versions: spark-branch Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: HIVE-9646.1-spark.patch Beeline can show MR job progress info, but can't show that of Spark job. CLI doesn't have this problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9646) Beeline doesn't show Spark job progress info [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-9646: -- Status: Patch Available (was: Open) Beeline doesn't show Spark job progress info [Spark Branch] --- Key: HIVE-9646 URL: https://issues.apache.org/jira/browse/HIVE-9646 Project: Hive Issue Type: Bug Affects Versions: spark-branch Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: HIVE-9646.1-spark.patch Beeline can show MR job progress info, but can't show that of Spark job. CLI doesn't have this problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-9601) New Beeline queries will hang If Beeline terminates in-properly [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho resolved HIVE-9601. - Resolution: Invalid New Beeline queries will hang If Beeline terminates in-properly [Spark Branch] -- Key: HIVE-9601 URL: https://issues.apache.org/jira/browse/HIVE-9601 Project: Hive Issue Type: Bug Components: Spark Reporter: Szehon Ho User session's Spark application seems to stay around if beeline is not quit properly (!quit) because the user is not disconnected. If Beeline is started, it will create a new Spark application which will hang waiting for the first one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9574) Lazy computing in HiveBaseFunctionResultList may hurt performance [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314491#comment-14314491
]
Jimmy Xiang commented on HIVE-9574:
---
Cool, thanks. Attached v6 that addressed more minor review comments.
Lazy computing in HiveBaseFunctionResultList may hurt performance [Spark
Branch]
Key: HIVE-9574
URL: https://issues.apache.org/jira/browse/HIVE-9574
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Rui Li
Assignee: Jimmy Xiang
Fix For: spark-branch
Attachments: HIVE-9574.1-spark.patch, HIVE-9574.2-spark.patch,
HIVE-9574.3-spark.patch, HIVE-9574.4-spark.patch, HIVE-9574.5-spark.patch,
HIVE-9574.6-spark.patch
{{RowContainer.first}} may call {{InputFormat.getSplits}}, which is
expensive. If we switch {{container}} and {{backupContainer}} frequently in
{{HiveKVResultCache}}, it will downgrade performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9618) Deduplicate RS keys for ptf/windowing
[
https://issues.apache.org/jira/browse/HIVE-9618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314497#comment-14314497
]
Hive QA commented on HIVE-9618:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12697668/HIVE-9618.2.patch.txt
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2740/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2740/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2740/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-2740/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'ql/src/test/org/apache/hadoop/hive/ql/parse/TestIUD.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/QBMetaData.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java'
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20S/target
shims/0.23/target shims/aggregator/target shims/common/target
shims/scheduler/target packaging/target hbase-handler/target testutils/target
jdbc/target metastore/target itests/target itests/thirdparty
itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target
itests/hive-unit-hadoop2/target itests/hive-minikdc/target
itests/hive-jmh/target itests/hive-unit/target itests/custom-serde/target
itests/util/target itests/qtest-spark/target hcatalog/target
hcatalog/core/target hcatalog/streaming/target
hcatalog/server-extensions/target hcatalog/hcatalog-pig-adapter/target
hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target
accumulo-handler/target hwi/target common/target common/src/gen
spark-client/target contrib/target service/target serde/target beeline/target
odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
ql/src/test/results/clientpositive/insert_into_with_schema.q.out
ql/src/test/queries/clientpositive/insert_into_with_schema.q
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1658765.
At revision 1658765.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12697668 - PreCommit-HIVE-TRUNK-Build
Deduplicate RS keys for ptf/windowing
-
Key: HIVE-9618
URL: https://issues.apache.org/jira/browse/HIVE-9618
Project: Hive
Issue Type: Improvement
Components: PTF-Windowing
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-9618.1.patch.txt, HIVE-9618.2.patch.txt
Currently, partition spec containing same column for partition-by and
order-by makes duplicated key column for RS. For example,
{noformat}
explain
select p_mfgr, p_name,
[jira] [Updated] (HIVE-9569) Enable more unit tests for UNION ALL [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao updated HIVE-9569: --- Attachment: HIVE-9569.4.patch Updated golden files. Enable more unit tests for UNION ALL [Spark Branch] --- Key: HIVE-9569 URL: https://issues.apache.org/jira/browse/HIVE-9569 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chao Assignee: Chao Attachments: HIVE-9569.1-spark.patch, HIVE-9569.1.patch, HIVE-9569.2.patch, HIVE-9569.3.patch, HIVE-9569.4.patch Currently, we only enabled a subset of all the union tests. We should try to enable the rest, and see if there's any issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9638) Drop Index does not check Index or Table exisit or not
Will Du created HIVE-9638: - Summary: Drop Index does not check Index or Table exisit or not Key: HIVE-9638 URL: https://issues.apache.org/jira/browse/HIVE-9638 Project: Hive Issue Type: Bug Components: Parser Affects Versions: 0.14.0, 0.13.0, 0.11.0, 1.0.0 Reporter: Will Du DROP INDEX index_name ON table_name; statement will be always successful no matter the index_name or table_name exsit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 30739: HIVE-9574 Lazy computing in HiveBaseFunctionResultList may hurt performance [Spark Branch]
On Feb. 10, 2015, 3:24 p.m., Xuefu Zhang wrote: ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveKVResultCache.java, line 56 https://reviews.apache.org/r/30739/diff/5/?file=858858#file858858line56 This one is also better to be private, if not used outside this class. It is used in the unit test. On Feb. 10, 2015, 3:24 p.m., Xuefu Zhang wrote: ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveKVResultCache.java, line 107 https://reviews.apache.org/r/30739/diff/5/?file=858858#file858858line107 Is it possible to have fd leak, if new Output() fails? Right, fixed. On Feb. 10, 2015, 3:24 p.m., Xuefu Zhang wrote: ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveKVResultCache.java, line 135 https://reviews.apache.org/r/30739/diff/5/?file=858858#file858858line135 Nit: could we move the constructor to the top, after the member variables? Done. - Jimmy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30739/#review71790 --- On Feb. 9, 2015, 7:41 p.m., Jimmy Xiang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30739/ --- (Updated Feb. 9, 2015, 7:41 p.m.) Review request for hive, Rui Li and Xuefu Zhang. Bugs: HIVE-9574 https://issues.apache.org/jira/browse/HIVE-9574 Repository: hive-git Description --- Result KV cache doesn't use RowContainer any more since it has logic we don't need, which is some overhead. We don't do lazy computing right away, instead we wait a little till the cache is close to spill. Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java 78ab680 ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveKVResultCache.java 8ead0cb ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 7a09b4d ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunctionResultList.java e92e299 ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 070ea4d ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunctionResultList.java d4ff37c ql/src/java/org/apache/hadoop/hive/ql/exec/spark/KryoSerializer.java 286816b ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestHiveKVResultCache.java 0df4598 Diff: https://reviews.apache.org/r/30739/diff/ Testing --- Unit test, test on cluster Thanks, Jimmy Xiang
[jira] [Updated] (HIVE-9454) Test failures due to new Calcite version
[
https://issues.apache.org/jira/browse/HIVE-9454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-9454:
---
Resolution: Fixed
Fix Version/s: 1.1.0
Status: Resolved (was: Patch Available)
Committed to trunk and branch-1.1! Thank you to Laljo and everyone who worked
on this!
Test failures due to new Calcite version
Key: HIVE-9454
URL: https://issues.apache.org/jira/browse/HIVE-9454
Project: Hive
Issue Type: Bug
Reporter: Brock Noland
Assignee: Laljo John Pullokkaran
Fix For: 1.1.0
Attachments: HIVE-9454.02.patch, HIVE-9454.03.patch,
HIVE-9454.04.patch, HIVE-9454.05.patch, HIVE-9454.06.patch,
HIVE-9454.07.patch, HIVE-9454.1.patch
A bunch of failures have started appearing in patches which seen unrelated. I
am thinking we've picked up a new version of Calcite. E.g.:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2488/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_auto_join12/
{noformat}
Running: diff -a
/home/hiveptest/54.147.202.89-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../itests/qtest/target/qfile-results/clientpositive/auto_join12.q.out
/home/hiveptest/54.147.202.89-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../ql/src/test/results/clientpositive/auto_join12.q.out
32c32
$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src
---
$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src
35c35
$hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:src
---
$hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:$hdt$_1:src
39c39
$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src
---
$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src
54c54
$hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:src
---
$hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:$hdt$_1:src
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9622) Getting NPE when trying to restart HS2 when metastore is configured to use org.apache.hadoop.hive.thrift.DBTokenStore
[
https://issues.apache.org/jira/browse/HIVE-9622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-9622:
---
Attachment: HIVE-9622.2.patch
Unit tests added. The previous 5 failures all passed locally.
Getting NPE when trying to restart HS2 when metastore is configured to use
org.apache.hadoop.hive.thrift.DBTokenStore
-
Key: HIVE-9622
URL: https://issues.apache.org/jira/browse/HIVE-9622
Project: Hive
Issue Type: Bug
Reporter: Aihua Xu
Assignee: Aihua Xu
Labels: HiveServer2, Security
Fix For: 1.2.0
Attachments: HIVE-9622.1.patch, HIVE-9622.2.patch
# Configure the cluster to use kerberos for HS2 and Metastore.
##
http://www.cloudera.com/content/cloudera/en/documentation/cdh4/v4-3-0/CDH4-Security-Guide/cdh4sg_topic_9_1.html
##
http://www.cloudera.com/content/cloudera/en/documentation/cdh4/v4-6-0/CDH4-Security-Guide/cdh4sg_topic_9_2.html
# Set hive metastore delegation token to
org.apache.hadoop.hive.thrift.DBTokenStore in hive-site.xml
{code}
property
namehive.cluster.delegation.token.store.class/name
valueorg.apache.hadoop.hive.thrift.DBTokenStore/value
/property
{code}
# Then trying to restart hive service, HS2 fails to start the NPE below:
{code}
9:43:10.711 AMERROR
org.apache.hive.service.cli.thrift.ThriftCLIService
Error:
org.apache.thrift.transport.TTransportException: Failed to start token manager
at
org.apache.hive.service.auth.HiveAuthFactory.init(HiveAuthFactory.java:107)
at
org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:51)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed to initialize master key
at
org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:223)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server.startDelegationTokenSecretManager(HadoopThriftAuthBridge20S.java:438)
at
org.apache.hive.service.auth.HiveAuthFactory.init(HiveAuthFactory.java:105)
... 2 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at
org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:221)
... 4 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnRawStore(DBTokenStore.java:145)
at
org.apache.hadoop.hive.thrift.DBTokenStore.addMasterKey(DBTokenStore.java:41)
at
org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.logUpdateMasterKey(TokenStoreDelegationTokenSecretManager.java:203)
at
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.updateCurrentKey(AbstractDelegationTokenSecretManager.java:339)
... 9 more
9:43:10.719 AMINFOorg.apache.hive.service.server.HiveServer2
SHUTDOWN_MSG:
/
SHUTDOWN_MSG: Shutting down HiveServer2 at
a1909.halxg.cloudera.com/10.20.202.109
/
{code}
The problem appears that we didn't pass a {{RawStore}} object in the
following:
https://github.com/apache/hive/blob/trunk/service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java#L111
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Fwd: Hive Index without WITH DEFERRED REBUILD
Can we create Hive index without specifying WITH DEFERRED REBUILD? Right now, it reports following errors. CREATE INDEX table01_index ON TABLE table01 (column2) AS 'COMPACT'; FAILED: Error in metadata: java.lang.RuntimeException: Please specify deferred rebuild using WITH DEFERRED REBUILD . FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask According to the design here https://cwiki.apache.org/confluence/display/Hive/IndexDev WITH DEFERRED REBUILD is optional. Is this a bug? In addition, I found drop index never check metadata and log here https://issues.apache.org/jira/browse/HIVE-9638 And issues of index in Hive 1.0.0 here https://issues.apache.org/jira/browse/HIVE-9639 Can someone verify this and provide workaround? Thanks, Will
[jira] [Commented] (HIVE-9601) New Beeline queries will hang If Beeline terminates in-properly [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314489#comment-14314489 ] Szehon Ho commented on HIVE-9601: - Some comments, this problem seems to be about orphan sessions leaving resources open, in this case the Remote Spark Application. The orphan sessions take up resources as specified by 'spark.cores.max', 'spark.executor.memory', leaving no room for others. The properties 'hive.server2.session.check.interval, hive.server2.idle.session.timeout' seem to help this case. I couldn't find any other way to identify orphan sessions via Thrift and close them. New Beeline queries will hang If Beeline terminates in-properly [Spark Branch] -- Key: HIVE-9601 URL: https://issues.apache.org/jira/browse/HIVE-9601 Project: Hive Issue Type: Bug Components: Spark Reporter: Szehon Ho User session's Spark application seems to stay around if beeline is not quit properly (!quit) because the user is not disconnected. If Beeline is started, it will create a new Spark application which will hang waiting for the first one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9574) Lazy computing in HiveBaseFunctionResultList may hurt performance [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HIVE-9574:
--
Attachment: HIVE-9574.6-spark.patch
Lazy computing in HiveBaseFunctionResultList may hurt performance [Spark
Branch]
Key: HIVE-9574
URL: https://issues.apache.org/jira/browse/HIVE-9574
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Rui Li
Assignee: Jimmy Xiang
Fix For: spark-branch
Attachments: HIVE-9574.1-spark.patch, HIVE-9574.2-spark.patch,
HIVE-9574.3-spark.patch, HIVE-9574.4-spark.patch, HIVE-9574.5-spark.patch,
HIVE-9574.6-spark.patch
{{RowContainer.first}} may call {{InputFormat.getSplits}}, which is
expensive. If we switch {{container}} and {{backupContainer}} frequently in
{{HiveKVResultCache}}, it will downgrade performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9023) HiveHistoryImpl relies on removed counters to print num rows
[
https://issues.apache.org/jira/browse/HIVE-9023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-9023:
Attachment: HIVE-9023.2.patch
I have revised the fix earlier to account for a single query inserting tables
with same names but across different databases.
{code}
.
Table default.booleantest2 stats: [numFiles=1, numRows=0, totalSize=67,
rawDataSize=0]
Table myDB.booleantest4 stats: [numFiles=1, numRows=0, totalSize=67,
rawDataSize=0]
Table default.booleantest3 stats: [numFiles=1, numRows=0, totalSize=16,
rawDataSize=0]
Table default.booleantest4 stats: [numFiles=1, numRows=0, totalSize=15,
rawDataSize=0]
12 Rows loaded to myDB.booleantest4
12 Rows loaded to default.booleantest2
3 Rows loaded to default.booleantest3
3 Rows loaded to default.booleantest4
MapReduce Jobs Launched:
Stage-Stage-4: Map: 1 Cumulative CPU: 3.23 sec HDFS Read: 6610 HDFS Write:
468 SUCCESS
Total MapReduce CPU Time Spent: 3 seconds 230 msec
OK
{code}
Thank you in advance
HiveHistoryImpl relies on removed counters to print num rows
Key: HIVE-9023
URL: https://issues.apache.org/jira/browse/HIVE-9023
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.13.0, 0.14.0, 0.13.1, 0.14.1
Reporter: Slava Markeyev
Assignee: Naveen Gangam
Priority: Minor
Labels: regression
Attachments: HIVE-9023.2.patch, HIVE-9023.patch
HiveHistoryImpl still relies on the counters that were removed in HIVE-5982
to determine the number of rows loaded. This results in regression of
functionality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9639) Create Index failed in Hive 1.0
Will Du created HIVE-9639: - Summary: Create Index failed in Hive 1.0 Key: HIVE-9639 URL: https://issues.apache.org/jira/browse/HIVE-9639 Project: Hive Issue Type: Bug Components: Indexing, Parser Affects Versions: 1.0.0 Reporter: Will Du Create Index in either type COMPACT or BITMAP does not work on version 1.0. The same query can run successfully in 0.13.0 and 0.14.0 CREATE INDEX idx_id_employee_id ON TABLE employee_id (employee_id) AS 'COMPACT' WITH DEFERRED REBUILD; FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Table not found default__employee_id_idx_id_employee_id__ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314474#comment-14314474 ] Brock Noland commented on HIVE-3454: Seems reasonable to me! [~spena] could you look as you have more experience here than I do. Problem with CAST(BIGINT as TIMESTAMP) -- Key: HIVE-3454 URL: https://issues.apache.org/jira/browse/HIVE-3454 Project: Hive Issue Type: Bug Components: Types, UDF Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, 0.13.1 Reporter: Ryan Harris Assignee: Aihua Xu Labels: newbie, newdev, patch Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, HIVE-3454.3.patch, HIVE-3454.patch Ran into an issue while working with timestamp conversion. CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current time from the BIGINT returned by unix_timestamp() Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 30739: HIVE-9574 Lazy computing in HiveBaseFunctionResultList may hurt performance [Spark Branch]
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30739/ --- (Updated Feb. 10, 2015, 5:24 p.m.) Review request for hive, Rui Li and Xuefu Zhang. Bugs: HIVE-9574 https://issues.apache.org/jira/browse/HIVE-9574 Repository: hive-git Description --- Result KV cache doesn't use RowContainer any more since it has logic we don't need, which is some overhead. We don't do lazy computing right away, instead we wait a little till the cache is close to spill. Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java 78ab680 ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveKVResultCache.java 8ead0cb ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 7a09b4d ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunctionResultList.java e92e299 ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 070ea4d ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunctionResultList.java d4ff37c ql/src/java/org/apache/hadoop/hive/ql/exec/spark/KryoSerializer.java 286816b ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestHiveKVResultCache.java 0df4598 Diff: https://reviews.apache.org/r/30739/diff/ Testing --- Unit test, test on cluster Thanks, Jimmy Xiang
[jira] [Commented] (HIVE-9481) allow column list specification in INSERT statement
[
https://issues.apache.org/jira/browse/HIVE-9481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314494#comment-14314494
]
Hive QA commented on HIVE-9481:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12697667/HIVE-9481.4.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 7543 tests executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2739/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2739/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2739/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12697667 - PreCommit-HIVE-TRUNK-Build
allow column list specification in INSERT statement
---
Key: HIVE-9481
URL: https://issues.apache.org/jira/browse/HIVE-9481
Project: Hive
Issue Type: Bug
Components: Parser, Query Processor, SQL
Affects Versions: 0.14.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Attachments: HIVE-9481.2.patch, HIVE-9481.4.patch, HIVE-9481.patch
Given a table FOO(a int, b int, c int), ANSI SQL supports insert into
FOO(c,b) select x,y from T. The expectation is that 'x' is written to column
'c' and 'y' is written column 'b' and 'a' is set to NULL, assuming column 'a'
is NULLABLE.
Hive does not support this. In Hive one has to ensure that the data
producing statement has a schema that matches target table schema.
Since Hive doesn't support DEFAULT value for columns in CREATE TABLE, when
target schema is explicitly provided, missing columns will be set to NULL if
they are NULLABLE, otherwise an error will be raised.
If/when DEFAULT clause is supported, this can be enhanced to set default
value rather than NULL.
Thus, given {noformat}
create table source (a int, b int);
create table target (x int, y int, z int);
create table target2 (x int, y int, z int);
{noformat}
{noformat}insert into target(y,z) select * from source;{noformat}
will mean
{noformat}insert into target select null as x, a, b from source;{noformat}
and
{noformat}insert into target(z,y) select * from source;{noformat}
will meant
{noformat}insert into target select null as x, b, a from source;{noformat}
Also,
{noformat}
from source
insert into target(y,z) select null as x, *
insert into target2(y,z) select null as x, source.*;
{noformat}
and for partitioned tables, given
{noformat}
Given:
CREATE TABLE pageviews (userid VARCHAR(64), link STRING, from STRING)
PARTITIONED BY (datestamp STRING) CLUSTERED BY (userid) INTO 256 BUCKETS
STORED AS ORC;
INSERT INTO TABLE pageviews PARTITION (datestamp = '2014-09-23')(userid,link)
VALUES ('jsmith', 'mail.com');
{noformat}
And dynamic partitioning
{noformat}
INSERT INTO TABLE pageviews PARTITION (datestamp)(userid,datestamp,link)
VALUES ('jsmith', '2014-09-23', 'mail.com');
{noformat}
In all cases, the schema specification contains columns of the target table
which are matched by position to the values produced by VALUES clause/SELECT
statement. If the producer side provides values for a dynamic partition
column, the column should be in the specified schema. Static partition
values are part of the partition spec and thus are not produced by the
producer and should not be part of the schema specification.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9023) HiveHistoryImpl relies on removed counters to print num rows
[ https://issues.apache.org/jira/browse/HIVE-9023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314538#comment-14314538 ] Naveen Gangam commented on HIVE-9023: - The above test failures do not seem to be result of my patch. They failed in the earlier build too. HiveHistoryImpl relies on removed counters to print num rows Key: HIVE-9023 URL: https://issues.apache.org/jira/browse/HIVE-9023 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.13.0, 0.14.0, 0.13.1, 0.14.1 Reporter: Slava Markeyev Assignee: Naveen Gangam Priority: Minor Labels: regression Attachments: HIVE-9023.2.patch, HIVE-9023.patch HiveHistoryImpl still relies on the counters that were removed in HIVE-5982 to determine the number of rows loaded. This results in regression of functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9635) LLAP: I'm the decider
[ https://issues.apache.org/jira/browse/HIVE-9635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-9635: - Attachment: HIVE-9635.1.patch first draft - doesn't handle udfs yet. LLAP: I'm the decider - Key: HIVE-9635 URL: https://issues.apache.org/jira/browse/HIVE-9635 Project: Hive Issue Type: Sub-task Affects Versions: llap Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-9635.1.patch https://www.youtube.com/watch?v=r8VbzrZ9yHQ Physical optimizer to choose what to run inside/outside llap. Tests first whether user code has to be shipped then if the specific query fragment is suitable to run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 30638: HIVE-9432
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30638/ --- (Updated Feb. 10, 2015, 9:15 a.m.) Review request for hive. Bugs: HIVE-9432 https://issues.apache.org/jira/browse/HIVE-9432 Repository: hive-git Description --- CBO (Calcite Return Path): Removing QB from ParseContext Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java 5dab17148c5d63fdb5f65e9ddf31536852e00a68 ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java 868cf04a5a5b51044b6b6690ff81f3b3906c7b1b ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java 6922f89112283303e340e9ec41189266a2ac ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 1342afeade74c5e49bcb0b35ad4cb037d913ffd6 ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 69a454581609cb88d29fef37d3841f3e43f8b75e ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 0ea81abe5b7ee38f02468ff71f4795b81fe4c49b ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java fad582a6970040a7f5ae633fd3ead4ba4a241733 ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchAggregation.java 31c83f2ba693e15dc74f4e663300898f1bd3e49e ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java 986e33bdb09cd0f3455ae2364877b8719676a3c1 ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java 0d19be671ffb9c46f7e4a4130f2042cc14ddfceb ql/src/java/org/apache/hadoop/hive/ql/optimizer/StatsOptimizer.java f69b7aabb9b6e035849368fcaead68f1a14a42f4 ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java a0459c48225359df04de3e50575147bfd5773677 ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 8979b4c3472476799f0b0ce7c87f88b692a8b106 ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 7ac0551d4946c62fdf2b7a66f2ba060ecf1a0ccb ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java e5e6a550c21cdc4f7a73a8fe3636f1dc5c3cf79a ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 7efad0ce2672dd7036deb8bdea827c5657a45338 ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 6c1ab07cbf7334030608f04a76bc926cecc5939c ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java 61592c1e1b982cb6e9d93f9a1ca2014370f6f1c6 ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 3e511884713579b7a36b8b09a02036ab78be305d ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java bdb92041756dbebcddf2bfad0c3cc2569b873de1 ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 6ef09cdcbad01f30423a2dd7e7dac7cb59ecad2d ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 0a037609b572a4589e08f095c398040bc431da61 ql/src/java/org/apache/hadoop/hive/ql/plan/StatsNoJobWork.java 3e5a6074c7432b49b91def865c09a86d16955832 ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java 66d4d4ac02618f8adaa51e2a620005004dceca84 ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 0e85990f5abc4aa3072ed53f8397e858f5b4062d Diff: https://reviews.apache.org/r/30638/diff/ Testing --- Existing tests. Thanks, Jesús Camacho Rodríguez
[jira] [Commented] (HIVE-9432) CBO (Calcite Return Path): Removing QB from ParseContext
[
https://issues.apache.org/jira/browse/HIVE-9432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313815#comment-14313815
]
Hive QA commented on HIVE-9432:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12697575/HIVE-9432.04.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 7540 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_serde
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2734/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2734/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2734/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12697575 - PreCommit-HIVE-TRUNK-Build
CBO (Calcite Return Path): Removing QB from ParseContext
Key: HIVE-9432
URL: https://issues.apache.org/jira/browse/HIVE-9432
Project: Hive
Issue Type: Sub-task
Components: CBO
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez
Fix For: 0.15.0
Attachments: HIVE-9432.01.patch, HIVE-9432.02.patch,
HIVE-9432.03.patch, HIVE-9432.04.patch, HIVE-9432.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9635) LLAP: I'm the decider
Gunther Hagleitner created HIVE-9635: Summary: LLAP: I'm the decider Key: HIVE-9635 URL: https://issues.apache.org/jira/browse/HIVE-9635 Project: Hive Issue Type: Sub-task Affects Versions: llap Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner https://www.youtube.com/watch?v=r8VbzrZ9yHQ Physical optimizer to choose what to run inside/outside llap. Tests first whether user code has to be shipped then if the specific query fragment is suitable to run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9432) CBO (Calcite Return Path): Removing QB from ParseContext
[ https://issues.apache.org/jira/browse/HIVE-9432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9432: -- Attachment: HIVE-9432.05.patch Rebasing patch. CBO (Calcite Return Path): Removing QB from ParseContext Key: HIVE-9432 URL: https://issues.apache.org/jira/browse/HIVE-9432 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 0.15.0 Attachments: HIVE-9432.01.patch, HIVE-9432.02.patch, HIVE-9432.03.patch, HIVE-9432.04.patch, HIVE-9432.05.patch, HIVE-9432.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9594) Add qtests for LAST_DAY udf
[
https://issues.apache.org/jira/browse/HIVE-9594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313930#comment-14313930
]
Hive QA commented on HIVE-9594:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12697577/HIVE-9594.2.patch
{color:green}SUCCESS:{color} +1 7542 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2735/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2735/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2735/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12697577 - PreCommit-HIVE-TRUNK-Build
Add qtests for LAST_DAY udf
---
Key: HIVE-9594
URL: https://issues.apache.org/jira/browse/HIVE-9594
Project: Hive
Issue Type: Improvement
Components: UDF
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov
Priority: Minor
Attachments: HIVE-9594.1.patch, HIVE-9594.2.patch
currently udf_last_day.q contains only
{code}
DESCRIBE FUNCTION last_day;
DESCRIBE FUNCTION EXTENDED last_day;
{code}
Better to add several function executions to the file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9636) Apply IdentityProjectRemover after other optimizations
[ https://issues.apache.org/jira/browse/HIVE-9636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9636: -- Attachment: HIVE-9636.patch Apply IdentityProjectRemover after other optimizations -- Key: HIVE-9636 URL: https://issues.apache.org/jira/browse/HIVE-9636 Project: Hive Issue Type: Bug Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Attachments: HIVE-9636.patch IdentityProjectRemove optimizer may remove some Project operators that change the table alias for the input columns; this might cause optimizations such as StatsOptimizer, that rely on colExprMap and consequently store table aliases, to behave incorrectly. To avoid this, we could apply IdentityProjectRemove optimization after all the rest have been previously applied, and document it accordingly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9625) Delegation tokens for HMS are not renewed
[
https://issues.apache.org/jira/browse/HIVE-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314028#comment-14314028
]
Hive QA commented on HIVE-9625:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12697589/HIVE-9625.1.patch
{color:green}SUCCESS:{color} +1 7540 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2736/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2736/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2736/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12697589 - PreCommit-HIVE-TRUNK-Build
Delegation tokens for HMS are not renewed
-
Key: HIVE-9625
URL: https://issues.apache.org/jira/browse/HIVE-9625
Project: Hive
Issue Type: Bug
Components: HiveServer2
Reporter: Brock Noland
Assignee: Brock Noland
Attachments: HIVE-9625.1.patch
AFAICT the delegation tokens stored in [HiveSessionImplwithUGI
|https://github.com/apache/hive/blob/trunk/service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java#L45]
for HMS + Impersonation are never renewed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9636) Apply IdentityProjectRemover after other optimizations
[ https://issues.apache.org/jira/browse/HIVE-9636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9636: -- Status: Patch Available (was: Open) Apply IdentityProjectRemover after other optimizations -- Key: HIVE-9636 URL: https://issues.apache.org/jira/browse/HIVE-9636 Project: Hive Issue Type: Bug Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez IdentityProjectRemove optimizer may remove some Project operators that change the table alias for the input columns; this might cause optimizations such as StatsOptimizer, that rely on colExprMap and consequently store table aliases, to behave incorrectly. To avoid this, we could apply IdentityProjectRemove optimization after all the rest have been previously applied, and document it accordingly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9636) Apply IdentityProjectRemover after other optimizations
Jesus Camacho Rodriguez created HIVE-9636: - Summary: Apply IdentityProjectRemover after other optimizations Key: HIVE-9636 URL: https://issues.apache.org/jira/browse/HIVE-9636 Project: Hive Issue Type: Bug Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez IdentityProjectRemove optimizer may remove some Project operators that change the table alias for the input columns; this might cause optimizations such as StatsOptimizer, that rely on colExprMap and consequently store table aliases, to behave incorrectly. To avoid this, we could apply IdentityProjectRemove optimization after all the rest have been previously applied, and document it accordingly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9637) Hive Snapshots - Metadata recovery (tightly coupled with HDFS data recovery)
Hari Sekhon created HIVE-9637: - Summary: Hive Snapshots - Metadata recovery (tightly coupled with HDFS data recovery) Key: HIVE-9637 URL: https://issues.apache.org/jira/browse/HIVE-9637 Project: Hive Issue Type: New Feature Components: Database/Schema, SQL Affects Versions: 0.14.0 Environment: HDP 2.2 Reporter: Hari Sekhon Feature request to add Hive HQL Snapshots against tables / databases that save the Hive metadata, analagous to HDFS snapshots for storage. Currently HDFS snapshots do not address the loss of Hive Metadata if a Hive user were to drop a table - it would need to be re-entered manually from HQL scripts as well as HDFS snapshot data restored. Hive Snapshots should ideally be tied together with HDFS snapshots so a single HQL command would trigger a simulatenous snapshot of the table HDFS data and Hive Metadata These snapshots should be visible from Hive shell and able to restore both HDFS data and Hive Metadata in a single Hive command also. Regards, Hari Sekhon http://www.linkedin.com/in/harisekhon -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9637) Hive Snapshots - Metadata recovery (tightly coupled with HDFS data recovery)
[ https://issues.apache.org/jira/browse/HIVE-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated HIVE-9637: -- Description: Feature request to add Hive HQL Snapshots against tables / databases that save the Hive metadata, analagous to HDFS snapshots for storage. Currently HDFS snapshots do not address the loss of Hive Metadata if a Hive user were to drop a table - it would need to be re-entered manually from HQL scripts as well as HDFS snapshot data restored. Hive Snapshots should ideally be tightly coupled with HDFS snapshots so a single HQL command would trigger a simulatenous snapshot of the table HDFS data and Hive Metadata. These snapshots should be visible from Hive shell / beeline session and able to restore both HDFS data and Hive Metadata in a single Hive command. Regards, Hari Sekhon http://www.linkedin.com/in/harisekhon was: Feature request to add Hive HQL Snapshots against tables / databases that save the Hive metadata, analagous to HDFS snapshots for storage. Currently HDFS snapshots do not address the loss of Hive Metadata if a Hive user were to drop a table - it would need to be re-entered manually from HQL scripts as well as HDFS snapshot data restored. Hive Snapshots should ideally be tightly coupled with HDFS snapshots so a single HQL command would trigger a simulatenous snapshot of the table HDFS data and Hive Metadata. These snapshots should be visible from Hive shell / beeline session and able to restore both HDFS data and Hive Metadata in a single Hive command also. Regards, Hari Sekhon http://www.linkedin.com/in/harisekhon Hive Snapshots - Metadata recovery (tightly coupled with HDFS data recovery) Key: HIVE-9637 URL: https://issues.apache.org/jira/browse/HIVE-9637 Project: Hive Issue Type: New Feature Components: Database/Schema, SQL Affects Versions: 0.14.0 Environment: HDP 2.2 Reporter: Hari Sekhon Feature request to add Hive HQL Snapshots against tables / databases that save the Hive metadata, analagous to HDFS snapshots for storage. Currently HDFS snapshots do not address the loss of Hive Metadata if a Hive user were to drop a table - it would need to be re-entered manually from HQL scripts as well as HDFS snapshot data restored. Hive Snapshots should ideally be tightly coupled with HDFS snapshots so a single HQL command would trigger a simulatenous snapshot of the table HDFS data and Hive Metadata. These snapshots should be visible from Hive shell / beeline session and able to restore both HDFS data and Hive Metadata in a single Hive command. Regards, Hari Sekhon http://www.linkedin.com/in/harisekhon -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9637) Hive Snapshots - Metadata recovery (tightly coupled with HDFS data recovery)
[ https://issues.apache.org/jira/browse/HIVE-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated HIVE-9637: -- Description: Feature request to add Hive HQL Snapshots against tables / databases that save the Hive metadata, analagous to HDFS snapshots for storage. Currently HDFS snapshots do not address the loss of Hive Metadata if a Hive user were to drop a table - it would need to be re-entered manually from HQL scripts as well as HDFS snapshot data restored. Hive Snapshots should ideally be tightly coupled with HDFS snapshots so a single HQL command would trigger a simulatenous snapshot of the table HDFS data and Hive Metadata. These snapshots should be visible from Hive shell / beeline session and able to restore both HDFS data and Hive Metadata in a single Hive command also. Regards, Hari Sekhon http://www.linkedin.com/in/harisekhon was: Feature request to add Hive HQL Snapshots against tables / databases that save the Hive metadata, analagous to HDFS snapshots for storage. Currently HDFS snapshots do not address the loss of Hive Metadata if a Hive user were to drop a table - it would need to be re-entered manually from HQL scripts as well as HDFS snapshot data restored. Hive Snapshots should ideally be tied together with HDFS snapshots so a single HQL command would trigger a simulatenous snapshot of the table HDFS data and Hive Metadata These snapshots should be visible from Hive shell and able to restore both HDFS data and Hive Metadata in a single Hive command also. Regards, Hari Sekhon http://www.linkedin.com/in/harisekhon Hive Snapshots - Metadata recovery (tightly coupled with HDFS data recovery) Key: HIVE-9637 URL: https://issues.apache.org/jira/browse/HIVE-9637 Project: Hive Issue Type: New Feature Components: Database/Schema, SQL Affects Versions: 0.14.0 Environment: HDP 2.2 Reporter: Hari Sekhon Feature request to add Hive HQL Snapshots against tables / databases that save the Hive metadata, analagous to HDFS snapshots for storage. Currently HDFS snapshots do not address the loss of Hive Metadata if a Hive user were to drop a table - it would need to be re-entered manually from HQL scripts as well as HDFS snapshot data restored. Hive Snapshots should ideally be tightly coupled with HDFS snapshots so a single HQL command would trigger a simulatenous snapshot of the table HDFS data and Hive Metadata. These snapshots should be visible from Hive shell / beeline session and able to restore both HDFS data and Hive Metadata in a single Hive command also. Regards, Hari Sekhon http://www.linkedin.com/in/harisekhon -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9023) HiveHistoryImpl relies on removed counters to print num rows
[ https://issues.apache.org/jira/browse/HIVE-9023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314709#comment-14314709 ] Naveen Gangam commented on HIVE-9023: - In FileSinkOperator, I now used a [full] table-name based suffix for the Counter keys (similar to the earlier description) instead of just table IDs from the initial patch, for which HiveHistoryImpl looked up the name from the IdToTableMap (basically a map of integer ids-tablenames) generated by query plan. However, these table names were short table names without the database prefix. So to get a full table name including the dbname, I am proposing the use of the value returned by the plan's Table descriptor API. getTableInfo().getTableName() as part of the Counter map's key (suffix) in the FileSinkOperator. Does this help? HiveHistoryImpl relies on removed counters to print num rows Key: HIVE-9023 URL: https://issues.apache.org/jira/browse/HIVE-9023 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.13.0, 0.14.0, 0.13.1, 0.14.1 Reporter: Slava Markeyev Assignee: Naveen Gangam Priority: Minor Labels: regression Attachments: HIVE-9023.2.patch, HIVE-9023.patch HiveHistoryImpl still relies on the counters that were removed in HIVE-5982 to determine the number of rows loaded. This results in regression of functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Propose to put JIRA traffic on separate hive list
I just got around to creating the ticket for this: https://issues.apache.org/jira/browse/INFRA-9156 Sorry to be so slow on it. Alan. Lefty Leverenz mailto:leftylever...@gmail.com February 9, 2015 at 22:02 Nudge, nudge. (Or is there an INFRA ticket pending?) -- Lefty Alan Gates mailto:ga...@hortonworks.com January 12, 2015 at 17:12 Yeah, I need to follow up on this. I'll file a JIRA to get infra to switch it. Alan. Lefty Leverenz mailto:leftylever...@gmail.com January 12, 2015 at 16:15 A public mailing list was created, iss...@hive.apache.org, and I joined it with the usual subscription message (issues-subscr...@hive.apache.org). But so far it doesn't have any traffic. -- Lefty On Mon, Jan 12, 2015 at 12:30 AM, Lars Francke lars.fran...@gmail.com Lars Francke mailto:lars.fran...@gmail.com January 12, 2015 at 0:30 Hi Alan, did you get around to creating a ticket for this? Cheers, Lars Alan Gates mailto:ga...@hortonworks.com December 15, 2014 at 14:20 Sorry, I dropped the ball on this. I seemed to get enough +1s and no major objections, so I'll get an INFRA ticket filed to make this happen. Alan.
[jira] [Commented] (HIVE-9574) Lazy computing in HiveBaseFunctionResultList may hurt performance [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314697#comment-14314697
]
Hive QA commented on HIVE-9574:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12697814/HIVE-9574.6-spark.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 7471 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cbo_gby_empty
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchEmptyCommit
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/720/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/720/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-720/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12697814 - PreCommit-HIVE-SPARK-Build
Lazy computing in HiveBaseFunctionResultList may hurt performance [Spark
Branch]
Key: HIVE-9574
URL: https://issues.apache.org/jira/browse/HIVE-9574
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Rui Li
Assignee: Jimmy Xiang
Fix For: spark-branch
Attachments: HIVE-9574.1-spark.patch, HIVE-9574.2-spark.patch,
HIVE-9574.3-spark.patch, HIVE-9574.4-spark.patch, HIVE-9574.5-spark.patch,
HIVE-9574.6-spark.patch
{{RowContainer.first}} may call {{InputFormat.getSplits}}, which is
expensive. If we switch {{container}} and {{backupContainer}} frequently in
{{HiveKVResultCache}}, it will downgrade performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9634) Hive
[ https://issues.apache.org/jira/browse/HIVE-9634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314719#comment-14314719 ] Alan Gates commented on HIVE-9634: -- As Sergey said, the dev list is the place to ask these questions, but I'll answer them here for now as I don't know how else to respond. Update and delete, along with insert...values were added in Hive 0.14 (released last December). [~ekoifman] is working on insert for selected columns right now and applying his changes to trunk, which I assume means they will go out in Hive 1.2 (as Hive 1.1 has already branched). Hive Key: HIVE-9634 URL: https://issues.apache.org/jira/browse/HIVE-9634 Project: Hive Issue Type: New Feature Reporter: srinivas Hi, In which version of Hive we will have all operations like Insert with selected columns, Update records and delete records, -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-3454: --- Release Note: The behaviors of converting from BOOLEAN/BYTE/SHORT/INT/BIGINT and converting from FLOAT/DOUBLE to TIMESTAMP have been inconsistent. The value of a BOOLEAN/BYTE/SHORT/INT/BIGINT is treated as the time in milliseconds while the value of a FLOAT/DOUBLE is treated as the time in seconds. With the change of HIVE-3454, the value of all the data types is treated as the time in seconds during the conversion. Hadoop Flags: Incompatible change Problem with CAST(BIGINT as TIMESTAMP) -- Key: HIVE-3454 URL: https://issues.apache.org/jira/browse/HIVE-3454 Project: Hive Issue Type: Bug Components: Types, UDF Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, 0.13.1 Reporter: Ryan Harris Assignee: Aihua Xu Labels: newbie, newdev, patch Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, HIVE-3454.3.patch, HIVE-3454.patch Ran into an issue while working with timestamp conversion. CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current time from the BIGINT returned by unix_timestamp() Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9023) HiveHistoryImpl relies on removed counters to print num rows
[ https://issues.apache.org/jira/browse/HIVE-9023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314638#comment-14314638 ] Szehon Ho commented on HIVE-9023: - Patch looks better, but can you explain how that case is taken care of (as I'm not terribly familiar)? HiveHistoryImpl relies on removed counters to print num rows Key: HIVE-9023 URL: https://issues.apache.org/jira/browse/HIVE-9023 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.13.0, 0.14.0, 0.13.1, 0.14.1 Reporter: Slava Markeyev Assignee: Naveen Gangam Priority: Minor Labels: regression Attachments: HIVE-9023.2.patch, HIVE-9023.patch HiveHistoryImpl still relies on the counters that were removed in HIVE-5982 to determine the number of rows loaded. This results in regression of functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-9634) Hive
[ https://issues.apache.org/jira/browse/HIVE-9634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin resolved HIVE-9634. Resolution: Invalid Hi. Please use dev/user list for questions... [~owen.omalley] and [~alangates] might know more Hive Key: HIVE-9634 URL: https://issues.apache.org/jira/browse/HIVE-9634 Project: Hive Issue Type: New Feature Reporter: srinivas Hi, In which version of Hive we will have all operations like Insert with selected columns, Update records and delete records, -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9432) CBO (Calcite Return Path): Removing QB from ParseContext
[
https://issues.apache.org/jira/browse/HIVE-9432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314681#comment-14314681
]
Hive QA commented on HIVE-9432:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12697694/HIVE-9432.05.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 7540 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchAbortAndCommit
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2741/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2741/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2741/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12697694 - PreCommit-HIVE-TRUNK-Build
CBO (Calcite Return Path): Removing QB from ParseContext
Key: HIVE-9432
URL: https://issues.apache.org/jira/browse/HIVE-9432
Project: Hive
Issue Type: Sub-task
Components: CBO
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez
Fix For: 0.15.0
Attachments: HIVE-9432.01.patch, HIVE-9432.02.patch,
HIVE-9432.03.patch, HIVE-9432.04.patch, HIVE-9432.05.patch, HIVE-9432.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314699#comment-14314699 ] Jason Dere commented on HIVE-3454: -- We may need to mark this as an incompatible change - folks have been using this behavior (inconsistent as it is) for a while now. Also, HIVE-9298 added TimestampParser capable of interpreting numeric text input as milliseconds since Unix epoch. Should we change this to seconds to make it consistent with the changes done here? Problem with CAST(BIGINT as TIMESTAMP) -- Key: HIVE-3454 URL: https://issues.apache.org/jira/browse/HIVE-3454 Project: Hive Issue Type: Bug Components: Types, UDF Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, 0.13.1 Reporter: Ryan Harris Assignee: Aihua Xu Labels: newbie, newdev, patch Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, HIVE-3454.3.patch, HIVE-3454.patch Ran into an issue while working with timestamp conversion. CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current time from the BIGINT returned by unix_timestamp() Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9574) Lazy computing in HiveBaseFunctionResultList may hurt performance [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314722#comment-14314722
]
Jimmy Xiang commented on HIVE-9574:
---
Test index_auto_mult_tables is ok for me on my box.
Lazy computing in HiveBaseFunctionResultList may hurt performance [Spark
Branch]
Key: HIVE-9574
URL: https://issues.apache.org/jira/browse/HIVE-9574
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Rui Li
Assignee: Jimmy Xiang
Fix For: spark-branch
Attachments: HIVE-9574.1-spark.patch, HIVE-9574.2-spark.patch,
HIVE-9574.3-spark.patch, HIVE-9574.4-spark.patch, HIVE-9574.5-spark.patch,
HIVE-9574.6-spark.patch
{{RowContainer.first}} may call {{InputFormat.getSplits}}, which is
expensive. If we switch {{container}} and {{backupContainer}} frequently in
{{HiveKVResultCache}}, it will downgrade performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9635) LLAP: I'm the decider
[ https://issues.apache.org/jira/browse/HIVE-9635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314706#comment-14314706 ] Sergey Shelukhin commented on HIVE-9635: do you need review? LLAP: I'm the decider - Key: HIVE-9635 URL: https://issues.apache.org/jira/browse/HIVE-9635 Project: Hive Issue Type: Sub-task Affects Versions: llap Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-9635.1.patch https://www.youtube.com/watch?v=r8VbzrZ9yHQ Physical optimizer to choose what to run inside/outside llap. Tests first whether user code has to be shipped then if the specific query fragment is suitable to run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9611) Allow SPARK_HOME as well as spark.home to define sparks location
[
https://issues.apache.org/jira/browse/HIVE-9611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314551#comment-14314551
]
Brock Noland commented on HIVE-9611:
Thank you Lefty! I added this to the getting started guide..
Allow SPARK_HOME as well as spark.home to define sparks location
Key: HIVE-9611
URL: https://issues.apache.org/jira/browse/HIVE-9611
Project: Hive
Issue Type: Bug
Components: Spark
Affects Versions: spark-branch, 1.1.0
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Minor
Fix For: 1.1.0
Attachments: HIVE-9611.patch
Right now {{SparkClientImpl}} requires {{spark.home}} to be defined. We
should allow {{SPARK_HOME}} as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9614) Encrypt mapjoin tables
[ https://issues.apache.org/jira/browse/HIVE-9614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314621#comment-14314621 ] Gunther Hagleitner commented on HIVE-9614: -- 3 months sprints? i believe that's called a marathon. Encrypt mapjoin tables -- Key: HIVE-9614 URL: https://issues.apache.org/jira/browse/HIVE-9614 Project: Hive Issue Type: Sub-task Reporter: Brock Noland When performing a MapJoin we store data which is put in the distributed cache which is stored on local disk. Ideally we would encrypt these tables to the same degree that the they are encrypted in HDFS or find some other way to ensure they are encrypted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7653) Hive AvroSerDe does not support circular references in Schema
[
https://issues.apache.org/jira/browse/HIVE-7653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314673#comment-14314673
]
Szehon Ho commented on HIVE-7653:
-
I almost forgot this one by now :)
Thanks, pending tests on the new patch. +1
Hive AvroSerDe does not support circular references in Schema
-
Key: HIVE-7653
URL: https://issues.apache.org/jira/browse/HIVE-7653
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Sachin Goyal
Assignee: Sachin Goyal
Attachments: HIVE-7653.1.patch, HIVE-7653.2.patch, HIVE-7653.3.patch,
HIVE-7653.4.patch, HIVE-7653.5.patch, HIVE-7653.6.patch
Avro allows nullable circular references but Hive AvroSerDe does not.
Example of circular references (passing in Avro but failing in AvroSerDe):
{code}
class AvroCycleParent {
AvroCycleChild child;
public AvroCycleChild getChild () {return child;}
public void setChild (AvroCycleChild child) {this.child = child;}
}
class AvroCycleChild {
AvroCycleParent parent;
public AvroCycleParent getParent () {return parent;}
public void setParent (AvroCycleParent parent) {this.parent = parent;}
}
{code}
Due to this discrepancy, Hive is unable to read Avro records having
circular-references. For some third-party code with such references, it
becomes very hard to directly serialize it with Avro and use in Hive.
I have a patch for this with a unit-test and I will submit it shortly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 30739: HIVE-9574 Lazy computing in HiveBaseFunctionResultList may hurt performance [Spark Branch]
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30739/#review71790 --- ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveKVResultCache.java https://reviews.apache.org/r/30739/#comment117653 This one is also better to be private, if not used outside this class. ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveKVResultCache.java https://reviews.apache.org/r/30739/#comment117654 Is it possible to have fd leak, if new Output() fails? ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveKVResultCache.java https://reviews.apache.org/r/30739/#comment117655 Nit: could we move the constructor to the top, after the member variables? ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveKVResultCache.java https://reviews.apache.org/r/30739/#comment117656 Same as above. - Xuefu Zhang On Feb. 9, 2015, 7:41 p.m., Jimmy Xiang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30739/ --- (Updated Feb. 9, 2015, 7:41 p.m.) Review request for hive, Rui Li and Xuefu Zhang. Bugs: HIVE-9574 https://issues.apache.org/jira/browse/HIVE-9574 Repository: hive-git Description --- Result KV cache doesn't use RowContainer any more since it has logic we don't need, which is some overhead. We don't do lazy computing right away, instead we wait a little till the cache is close to spill. Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java 78ab680 ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveKVResultCache.java 8ead0cb ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 7a09b4d ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunctionResultList.java e92e299 ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 070ea4d ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunctionResultList.java d4ff37c ql/src/java/org/apache/hadoop/hive/ql/exec/spark/KryoSerializer.java 286816b ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestHiveKVResultCache.java 0df4598 Diff: https://reviews.apache.org/r/30739/diff/ Testing --- Unit test, test on cluster Thanks, Jimmy Xiang
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314342#comment-14314342 ] Aihua Xu commented on HIVE-3454: +[~brocknoland] Can you take a look at the code? Problem with CAST(BIGINT as TIMESTAMP) -- Key: HIVE-3454 URL: https://issues.apache.org/jira/browse/HIVE-3454 Project: Hive Issue Type: Bug Components: Types, UDF Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, 0.13.1 Reporter: Ryan Harris Assignee: Aihua Xu Labels: newbie, newdev, patch Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, HIVE-3454.3.patch, HIVE-3454.patch Ran into an issue while working with timestamp conversion. CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current time from the BIGINT returned by unix_timestamp() Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9640) refactor ORC metadata and index reading out of RecordReaderImpl
[ https://issues.apache.org/jira/browse/HIVE-9640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314926#comment-14314926 ] Sergey Shelukhin commented on HIVE-9640: I have a patch for LLAP branch which is very small (moves reading footer and indexes into separate class that RRImpl uses), but on trunk bloom filters re-coupled SARG with everything again... Will figure it out after HIVE-9555 refactor ORC metadata and index reading out of RecordReaderImpl --- Key: HIVE-9640 URL: https://issues.apache.org/jira/browse/HIVE-9640 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin For LLAP and general ease of use of the code -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9578) Add support for getDatabases and alterDatabase calls [hbase-metastore branch]
[ https://issues.apache.org/jira/browse/HIVE-9578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314955#comment-14314955 ] Thejas M Nair commented on HIVE-9578: - +1 Add support for getDatabases and alterDatabase calls [hbase-metastore branch] - Key: HIVE-9578 URL: https://issues.apache.org/jira/browse/HIVE-9578 Project: Hive Issue Type: Sub-task Components: Metastore Reporter: Alan Gates Assignee: Alan Gates Attachments: HIVE-9578.2.patch, HIVE-9578.patch The initial patch only supporting getting a single database, add database, and drop database. Support needs to be added for alter database, getting all the databases, and getting database names by pattern. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9641) Fill out remaining partition functions in HBaseStore
Alan Gates created HIVE-9641: Summary: Fill out remaining partition functions in HBaseStore Key: HIVE-9641 URL: https://issues.apache.org/jira/browse/HIVE-9641 Project: Hive Issue Type: Sub-task Reporter: Alan Gates Assignee: Alan Gates A number of the listPartition and getPartition methods are not implemented. The rest need to be implemented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 30750: HIVE-9605 Remove parquet nested objects from wrapper writable objects
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30750/ --- (Updated Feb. 10, 2015, 9:25 p.m.) Review request for hive, Ryan Blue, cheng xu, and Dong Chen. Changes --- Fix broken unit-tests Bugs: HIVE-9605 https://issues.apache.org/jira/browse/HIVE-9605 Repository: hive-git Description --- Remove wrapper object from parquet nested types (map/array) Diffs (updated) - itests/hive-jmh/src/main/java/org/apache/hive/benchmark/storage/ColumnarStorageBench.java 61c2eb418e4ad317a603e87240295f9396d1d5b5 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/HiveCollectionConverter.java 6621a8768953a9bef54e7a144ae045abcc32f458 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/HiveGroupConverter.java 4809f9b5882ae409159b422c08c665aa24f796d8 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/Repeated.java fdea782167d63593f6cbde5e7154d771761757f7 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/AbstractParquetMapInspector.java 62c61fc7502f24e6a032076f384b5a946c1cc9a6 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/DeepParquetHiveMapInspector.java d38c64192e01371c0c98b339113348d2e52cedc3 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveArrayInspector.java 53ca31d0b516c4a941e048e98e7f8f763752c436 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/StandardParquetHiveMapInspector.java 5aa14482899fed5711b40c5554b056d07818afb5 ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestArrayCompatibility.java f7f3e57c1607dec3e8dc3a29e819d3aadc65155a ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestMapStructures.java ca4805082fd717d15ed41ca15a730e19da267c8a ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetSerDe.java 007a665529857bcec612f638a157aa5043562a15 ql/src/test/org/apache/hadoop/hive/ql/io/parquet/serde/TestAbstractParquetMapInspector.java ef05150494027ddd70790dcf26b772ebc4cd2b8b ql/src/test/org/apache/hadoop/hive/ql/io/parquet/serde/TestDeepParquetHiveMapInspector.java 8646ff4d3413d7d642e2559e1a485d77472b156a ql/src/test/org/apache/hadoop/hive/ql/io/parquet/serde/TestParquetHiveArrayInspector.java f3a24af2e5f4eeb24e1e286ada19fc9592daacb6 ql/src/test/org/apache/hadoop/hive/ql/io/parquet/serde/TestStandardParquetHiveMapInspector.java 278419f73b311322dcf3c70abb340bf63d8a4337 Diff: https://reviews.apache.org/r/30750/diff/ Testing --- Thanks, Sergio Pena
[jira] [Updated] (HIVE-9579) Support all get tables [hbase-metastore branch]
[ https://issues.apache.org/jira/browse/HIVE-9579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-9579: - Description: The initial patch only supported getting a single table, creating a table, altering a table, and dropping a table. Support is needed for getting multiple tables at a time. (was: The initial patch only supported getting a single table, creating a table, and dropping a table. Support is needed for getting multiple tables at a time and altering tables.) Support all get tables [hbase-metastore branch] --- Key: HIVE-9579 URL: https://issues.apache.org/jira/browse/HIVE-9579 Project: Hive Issue Type: Sub-task Components: Metastore Reporter: Alan Gates Assignee: Alan Gates The initial patch only supported getting a single table, creating a table, altering a table, and dropping a table. Support is needed for getting multiple tables at a time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9579) Support all get tables [hbase-metastore branch]
[ https://issues.apache.org/jira/browse/HIVE-9579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-9579: - Summary: Support all get tables [hbase-metastore branch] (was: Support for alter table and get tables [hbase-metastore branch]) Support all get tables [hbase-metastore branch] --- Key: HIVE-9579 URL: https://issues.apache.org/jira/browse/HIVE-9579 Project: Hive Issue Type: Sub-task Components: Metastore Reporter: Alan Gates Assignee: Alan Gates The initial patch only supported getting a single table, creating a table, and dropping a table. Support is needed for getting multiple tables at a time and altering tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9578) Add support for getDatabases and alterDatabase calls [hbase-metastore branch]
[ https://issues.apache.org/jira/browse/HIVE-9578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-9578: - Attachment: HIVE-9578.2.patch New patch that addresses Thejas' comments on the regular expression patterns not being correct for show databases like. One thing to note is that I did not disable other regular expression syntax, so it will work (ie show databases like db[12]; would show db1 and db2, even though it wouldn't work in the RDBMS case. Add support for getDatabases and alterDatabase calls [hbase-metastore branch] - Key: HIVE-9578 URL: https://issues.apache.org/jira/browse/HIVE-9578 Project: Hive Issue Type: Sub-task Components: Metastore Reporter: Alan Gates Assignee: Alan Gates Attachments: HIVE-9578.2.patch, HIVE-9578.patch The initial patch only supporting getting a single database, add database, and drop database. Support needs to be added for alter database, getting all the databases, and getting database names by pattern. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9639) Create Index failed in Hive 1.0
[
https://issues.apache.org/jira/browse/HIVE-9639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damien Carol updated HIVE-9639:
---
Description:
Create Index in either type COMPACT or BITMAP does not work on version 1.0. The
same query can run successfully in 0.13.0 and 0.14.0
{code:sql}
CREATE INDEX idx_id_employee_id
ON TABLE employee_id (employee_id)
AS 'COMPACT'
WITH DEFERRED REBUILD;
{code}
{noformat}
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. Table not found
default__employee_id_idx_id_employee_id__
{noformat}
was:
Create Index in either type COMPACT or BITMAP does not work on version 1.0. The
same query can run successfully in 0.13.0 and 0.14.0
CREATE INDEX idx_id_employee_id
ON TABLE employee_id (employee_id)
AS 'COMPACT'
WITH DEFERRED REBUILD;
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. Table not found
default__employee_id_idx_id_employee_id__
Create Index failed in Hive 1.0
---
Key: HIVE-9639
URL: https://issues.apache.org/jira/browse/HIVE-9639
Project: Hive
Issue Type: Bug
Components: Indexing, Parser
Affects Versions: 1.0.0
Reporter: Will Du
Create Index in either type COMPACT or BITMAP does not work on version 1.0.
The same query can run successfully in 0.13.0 and 0.14.0
{code:sql}
CREATE INDEX idx_id_employee_id
ON TABLE employee_id (employee_id)
AS 'COMPACT'
WITH DEFERRED REBUILD;
{code}
{noformat}
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. Table not found
default__employee_id_idx_id_employee_id__
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9578) Add support for getDatabases and alterDatabase calls [hbase-metastore branch]
[ https://issues.apache.org/jira/browse/HIVE-9578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-9578: - Resolution: Fixed Fix Version/s: 1.2.0 Status: Resolved (was: Patch Available) Patch 2 checked in. Thanks Thejas for all the good feedback. Add support for getDatabases and alterDatabase calls [hbase-metastore branch] - Key: HIVE-9578 URL: https://issues.apache.org/jira/browse/HIVE-9578 Project: Hive Issue Type: Sub-task Components: Metastore Reporter: Alan Gates Assignee: Alan Gates Fix For: 1.2.0 Attachments: HIVE-9578.2.patch, HIVE-9578.patch The initial patch only supporting getting a single database, add database, and drop database. Support needs to be added for alter database, getting all the databases, and getting database names by pattern. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9639) Create Index failed in Hive 1.0
[
https://issues.apache.org/jira/browse/HIVE-9639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14315000#comment-14315000
]
Damien Carol commented on HIVE-9639:
[~willddy] Could you provide the stack trace?
Create Index failed in Hive 1.0
---
Key: HIVE-9639
URL: https://issues.apache.org/jira/browse/HIVE-9639
Project: Hive
Issue Type: Bug
Components: Indexing, Parser
Affects Versions: 1.0.0
Reporter: Will Du
Create Index in either type COMPACT or BITMAP does not work on version 1.0.
The same query can run successfully in 0.13.0 and 0.14.0
{code:sql}
CREATE INDEX idx_id_employee_id
ON TABLE employee_id (employee_id)
AS 'COMPACT'
WITH DEFERRED REBUILD;
{code}
{noformat}
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. Table not found
default__employee_id_idx_id_employee_id__
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9640) refactor ORC metadata and index reading out of RecordReaderImpl
Sergey Shelukhin created HIVE-9640: -- Summary: refactor ORC metadata and index reading out of RecordReaderImpl Key: HIVE-9640 URL: https://issues.apache.org/jira/browse/HIVE-9640 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin For LLAP and general ease of use of the code -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9023) HiveHistoryImpl relies on removed counters to print num rows
[ https://issues.apache.org/jira/browse/HIVE-9023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314956#comment-14314956 ] Szehon Ho commented on HIVE-9023: - OK makes sense, good to know getTableInfo().getTableName() is the fully qualified name, +1 pending tests on latest patch HiveHistoryImpl relies on removed counters to print num rows Key: HIVE-9023 URL: https://issues.apache.org/jira/browse/HIVE-9023 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.13.0, 0.14.0, 0.13.1, 0.14.1 Reporter: Slava Markeyev Assignee: Naveen Gangam Priority: Minor Labels: regression Attachments: HIVE-9023.2.patch, HIVE-9023.patch HiveHistoryImpl still relies on the counters that were removed in HIVE-5982 to determine the number of rows loaded. This results in regression of functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9642) Hive metastore client retries don't happen consistently for all api calls
Xiaobing Zhou created HIVE-9642: --- Summary: Hive metastore client retries don't happen consistently for all api calls Key: HIVE-9642 URL: https://issues.apache.org/jira/browse/HIVE-9642 Project: Hive Issue Type: Bug Affects Versions: 1.0.0 Reporter: Xiaobing Zhou Assignee: Xiaobing Zhou When org.apache.thrift.transport.TTransportException is thrown for issues like socket timeout, the retry via RetryingMetaStoreClient happens only in certain cases. Retry happens for the getDatabase call in but not for getAllDatabases(). The reason is RetryingMetaStoreClient checks for TTransportException being the cause for InvocationTargetException. But in case of some calls such as getAllDatabases in HiveMetastoreClient, all exceptions get wrapped in a MetaException. We should remove this unnecessary wrapping of exceptions for certain functions in HMC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8119) Implement Date in ParquetSerde
[ https://issues.apache.org/jira/browse/HIVE-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-8119: -- Assignee: Dong Chen (was: Mohit Sabharwal) Implement Date in ParquetSerde -- Key: HIVE-8119 URL: https://issues.apache.org/jira/browse/HIVE-8119 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Dong Chen Attachments: HIVE-8119.1.patch, HIVE-8119.patch Date type in Parquet is discussed here: http://mail-archives.apache.org/mod_mbox/incubator-parquet-dev/201406.mbox/%3CCAKa9qDkp7xn+H8fNZC7ms3ckd=xr8gdpe7gqgj5o+pybdem...@mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314773#comment-14314773 ] Aihua Xu commented on HIVE-3454: [~jdere] Thanks for pointing that out. I marked it as incompatible change (of course technically it's not incompatible). Regarding the TimestampParser, since MillisDateFormatParser is just one of the parser to support, I think it should be good without any change. Probably we can support SecondsDateFormatParser in the future if it's necessary. Problem with CAST(BIGINT as TIMESTAMP) -- Key: HIVE-3454 URL: https://issues.apache.org/jira/browse/HIVE-3454 Project: Hive Issue Type: Bug Components: Types, UDF Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, 0.13.1 Reporter: Ryan Harris Assignee: Aihua Xu Labels: newbie, newdev, patch Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, HIVE-3454.3.patch, HIVE-3454.patch Ran into an issue while working with timestamp conversion. CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current time from the BIGINT returned by unix_timestamp() Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9627) Add cbo_gby_empty.q.out for Spark [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-9627: -- Resolution: Fixed Fix Version/s: spark-branch Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Jimmy and Chao. Add cbo_gby_empty.q.out for Spark [Spark Branch] Key: HIVE-9627 URL: https://issues.apache.org/jira/browse/HIVE-9627 Project: Hive Issue Type: Test Affects Versions: spark-branch Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Trivial Fix For: spark-branch Attachments: HIVE-9627.1-spark.patch The golden file cbo_gby_empty.q.out for Spark is missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 30717: HIVE-8119: Implement Date in ParquetSerde
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30717/#review71844 --- One minor thing, but this looks good otherwise. Sergio knows more about the compatibility between this and his recent performance work and the object inspector code, so I'll leave that to him for review. ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/HiveSchemaConverter.java https://reviews.apache.org/r/30717/#comment117710 For primitive types, this should be using the Types API (like the line above) because we're going to remove the constructors from the public API in favor of the bulider. This is to avoid invalid types, like an INT64 with a DATE annotation. This should be: ```java Types.primitive(repetition, INT32).as(DATE).named(name); ``` - Ryan Blue On Feb. 5, 2015, 11:51 p.m., Dong Chen wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30717/ --- (Updated Feb. 5, 2015, 11:51 p.m.) Review request for hive. Repository: hive-git Description --- HIVE-8119: Implement Date in ParquetSerde This patch map the Date in Hive to INT32 in Parquet, based on the Parquet Logical Type Definitions in https://github.com/apache/incubator-parquet-format/blob/master/LogicalTypes.md Diffs - data/files/parquet_types.txt 31a10c9 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ETypeConverter.java 377e362 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/HiveSchemaConverter.java e5bd70c ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ArrayWritableObjectInspector.java bb066af ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 9199127 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriter.java 1d83bf3 ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestHiveSchemaConverter.java f232c57 ql/src/test/queries/clientnegative/parquet_date.q 89d3602 ql/src/test/queries/clientpositive/parquet_types.q 806db24 ql/src/test/results/clientnegative/parquet_date.q.out d1c38d6 ql/src/test/results/clientpositive/parquet_types.q.out dc5ceb0 Diff: https://reviews.apache.org/r/30717/diff/ Testing --- UT passed. 2 tests are added Thanks, Dong Chen
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314804#comment-14314804 ] Aihua Xu commented on HIVE-3454: Me either and I feel a new configuration may make things more complicated. From the comments above, actually the users are more using workarounds (converting int/bigint to float/double first before interpreting it as timestamp). My opinion is not to add a new configuration and keep things simple. Problem with CAST(BIGINT as TIMESTAMP) -- Key: HIVE-3454 URL: https://issues.apache.org/jira/browse/HIVE-3454 Project: Hive Issue Type: Bug Components: Types, UDF Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, 0.13.1 Reporter: Ryan Harris Assignee: Aihua Xu Labels: newbie, newdev, patch Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, HIVE-3454.3.patch, HIVE-3454.patch Ran into an issue while working with timestamp conversion. CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current time from the BIGINT returned by unix_timestamp() Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9500) Support nested structs over 24 levels.
[
https://issues.apache.org/jira/browse/HIVE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-9500:
Attachment: TClass.java
If the theory is not convincing, attaching TClass.java that compares the
performance.
The results -
{noformat}
$ java -server testpkg.TClass
Array escape time(ms) 5919
Num chars escaped 2
Map escape time(ms) 23275
Num chars escaped 2
Perf diff factor = 3.9322520696063523
{noformat}
Support nested structs over 24 levels.
--
Key: HIVE-9500
URL: https://issues.apache.org/jira/browse/HIVE-9500
Project: Hive
Issue Type: Improvement
Reporter: Aihua Xu
Assignee: Aihua Xu
Labels: SerDe
Fix For: 1.2.0
Attachments: HIVE-9500.1.patch, HIVE-9500.2.patch, HIVE-9500.3.patch,
TClass.java
Customer has deeply nested avro structure and is receiving the following
error when performing queries.
15/01/09 20:59:29 ERROR ql.Driver: FAILED: SemanticException
org.apache.hadoop.hive.serde2.SerDeException: Number of levels of nesting
supported for LazySimpleSerde is 23 Unable to work with level 24
Currently we support up to 24 levels of nested structs when
hive.serialization.extend.nesting.levels is set to true, while the customers
have the requirement to support more than that.
It would be better to make the supported levels configurable or completely
removed (i.e., we can support any number of levels).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9605) Remove parquet nested objects from wrapper writable objects
[
https://issues.apache.org/jira/browse/HIVE-9605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-9605:
--
Status: Open (was: Patch Available)
Remove parquet nested objects from wrapper writable objects
---
Key: HIVE-9605
URL: https://issues.apache.org/jira/browse/HIVE-9605
Project: Hive
Issue Type: Sub-task
Affects Versions: 0.14.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-9605.2.patch
Parquet nested types are using an extra wrapper object (ArrayWritable) as a
wrapper of map and list elements. This extra object is not needed and causing
unnecessary memory allocations.
An example of code is on HiveCollectionConverter.java:
{noformat}
public void end() {
parent.set(index, wrapList(new ArrayWritable(
Writable.class, list.toArray(new Writable[list.size()];
}
{noformat}
This object is later unwrapped on AbstractParquetMapInspector, i.e.:
{noformat}
final Writable[] mapContainer = ((ArrayWritable) data).get();
final Writable[] mapArray = ((ArrayWritable) mapContainer[0]).get();
for (final Writable obj : mapArray) {
...
}
{noformat}
We should get rid of this wrapper object to save time and memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-9500) Support nested structs over 24 levels.
[
https://issues.apache.org/jira/browse/HIVE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314862#comment-14314862
]
Thejas M Nair edited comment on HIVE-9500 at 2/10/15 8:31 PM:
--
If the theory is not convincing, attaching TClass.java that compares the
performance. Use of HashMap is slower by around 4 times.
The results -
{noformat}
$ java -server testpkg.TClass
Array escape time(ms) 5919
Num chars escaped 2
Map escape time(ms) 23275
Num chars escaped 2
Perf diff factor = 3.9322520696063523
{noformat}
was (Author: thejas):
If the theory is not convincing, attaching TClass.java that compares the
performance.
The results -
{noformat}
$ java -server testpkg.TClass
Array escape time(ms) 5919
Num chars escaped 2
Map escape time(ms) 23275
Num chars escaped 2
Perf diff factor = 3.9322520696063523
{noformat}
Support nested structs over 24 levels.
--
Key: HIVE-9500
URL: https://issues.apache.org/jira/browse/HIVE-9500
Project: Hive
Issue Type: Improvement
Reporter: Aihua Xu
Assignee: Aihua Xu
Labels: SerDe
Fix For: 1.2.0
Attachments: HIVE-9500.1.patch, HIVE-9500.2.patch, HIVE-9500.3.patch,
TClass.java
Customer has deeply nested avro structure and is receiving the following
error when performing queries.
15/01/09 20:59:29 ERROR ql.Driver: FAILED: SemanticException
org.apache.hadoop.hive.serde2.SerDeException: Number of levels of nesting
supported for LazySimpleSerde is 23 Unable to work with level 24
Currently we support up to 24 levels of nested structs when
hive.serialization.extend.nesting.levels is set to true, while the customers
have the requirement to support more than that.
It would be better to make the supported levels configurable or completely
removed (i.e., we can support any number of levels).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9500) Support nested structs over 24 levels.
[ https://issues.apache.org/jira/browse/HIVE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314871#comment-14314871 ] Aihua Xu commented on HIVE-9500: Thanks for the comparison result. I will revert some of the changes. Support nested structs over 24 levels. -- Key: HIVE-9500 URL: https://issues.apache.org/jira/browse/HIVE-9500 Project: Hive Issue Type: Improvement Reporter: Aihua Xu Assignee: Aihua Xu Labels: SerDe Fix For: 1.2.0 Attachments: HIVE-9500.1.patch, HIVE-9500.2.patch, HIVE-9500.3.patch, TClass.java Customer has deeply nested avro structure and is receiving the following error when performing queries. 15/01/09 20:59:29 ERROR ql.Driver: FAILED: SemanticException org.apache.hadoop.hive.serde2.SerDeException: Number of levels of nesting supported for LazySimpleSerde is 23 Unable to work with level 24 Currently we support up to 24 levels of nested structs when hive.serialization.extend.nesting.levels is set to true, while the customers have the requirement to support more than that. It would be better to make the supported levels configurable or completely removed (i.e., we can support any number of levels). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9500) Support nested structs over 24 levels.
[ https://issues.apache.org/jira/browse/HIVE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314872#comment-14314872 ] Aihua Xu commented on HIVE-9500: Thanks for the comparison result. I will revert some of the changes. Support nested structs over 24 levels. -- Key: HIVE-9500 URL: https://issues.apache.org/jira/browse/HIVE-9500 Project: Hive Issue Type: Improvement Reporter: Aihua Xu Assignee: Aihua Xu Labels: SerDe Fix For: 1.2.0 Attachments: HIVE-9500.1.patch, HIVE-9500.2.patch, HIVE-9500.3.patch, TClass.java Customer has deeply nested avro structure and is receiving the following error when performing queries. 15/01/09 20:59:29 ERROR ql.Driver: FAILED: SemanticException org.apache.hadoop.hive.serde2.SerDeException: Number of levels of nesting supported for LazySimpleSerde is 23 Unable to work with level 24 Currently we support up to 24 levels of nested structs when hive.serialization.extend.nesting.levels is set to true, while the customers have the requirement to support more than that. It would be better to make the supported levels configurable or completely removed (i.e., we can support any number of levels). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9500) Support nested structs over 24 levels.
[ https://issues.apache.org/jira/browse/HIVE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314869#comment-14314869 ] Aihua Xu commented on HIVE-9500: Thanks for the comparison result. I will revert some of the changes. Support nested structs over 24 levels. -- Key: HIVE-9500 URL: https://issues.apache.org/jira/browse/HIVE-9500 Project: Hive Issue Type: Improvement Reporter: Aihua Xu Assignee: Aihua Xu Labels: SerDe Fix For: 1.2.0 Attachments: HIVE-9500.1.patch, HIVE-9500.2.patch, HIVE-9500.3.patch, TClass.java Customer has deeply nested avro structure and is receiving the following error when performing queries. 15/01/09 20:59:29 ERROR ql.Driver: FAILED: SemanticException org.apache.hadoop.hive.serde2.SerDeException: Number of levels of nesting supported for LazySimpleSerde is 23 Unable to work with level 24 Currently we support up to 24 levels of nested structs when hive.serialization.extend.nesting.levels is set to true, while the customers have the requirement to support more than that. It would be better to make the supported levels configurable or completely removed (i.e., we can support any number of levels). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9500) Support nested structs over 24 levels.
[ https://issues.apache.org/jira/browse/HIVE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314873#comment-14314873 ] Aihua Xu commented on HIVE-9500: Thanks for the comparison result. I will revert some of the changes. Support nested structs over 24 levels. -- Key: HIVE-9500 URL: https://issues.apache.org/jira/browse/HIVE-9500 Project: Hive Issue Type: Improvement Reporter: Aihua Xu Assignee: Aihua Xu Labels: SerDe Fix For: 1.2.0 Attachments: HIVE-9500.1.patch, HIVE-9500.2.patch, HIVE-9500.3.patch, TClass.java Customer has deeply nested avro structure and is receiving the following error when performing queries. 15/01/09 20:59:29 ERROR ql.Driver: FAILED: SemanticException org.apache.hadoop.hive.serde2.SerDeException: Number of levels of nesting supported for LazySimpleSerde is 23 Unable to work with level 24 Currently we support up to 24 levels of nested structs when hive.serialization.extend.nesting.levels is set to true, while the customers have the requirement to support more than that. It would be better to make the supported levels configurable or completely removed (i.e., we can support any number of levels). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9500) Support nested structs over 24 levels.
[ https://issues.apache.org/jira/browse/HIVE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314870#comment-14314870 ] Aihua Xu commented on HIVE-9500: Thanks for the comparison result. I will revert some of the changes. Support nested structs over 24 levels. -- Key: HIVE-9500 URL: https://issues.apache.org/jira/browse/HIVE-9500 Project: Hive Issue Type: Improvement Reporter: Aihua Xu Assignee: Aihua Xu Labels: SerDe Fix For: 1.2.0 Attachments: HIVE-9500.1.patch, HIVE-9500.2.patch, HIVE-9500.3.patch, TClass.java Customer has deeply nested avro structure and is receiving the following error when performing queries. 15/01/09 20:59:29 ERROR ql.Driver: FAILED: SemanticException org.apache.hadoop.hive.serde2.SerDeException: Number of levels of nesting supported for LazySimpleSerde is 23 Unable to work with level 24 Currently we support up to 24 levels of nested structs when hive.serialization.extend.nesting.levels is set to true, while the customers have the requirement to support more than that. It would be better to make the supported levels configurable or completely removed (i.e., we can support any number of levels). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9500) Support nested structs over 24 levels.
[ https://issues.apache.org/jira/browse/HIVE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314874#comment-14314874 ] Aihua Xu commented on HIVE-9500: Sorry for the multiple comments. The site is slow. Support nested structs over 24 levels. -- Key: HIVE-9500 URL: https://issues.apache.org/jira/browse/HIVE-9500 Project: Hive Issue Type: Improvement Reporter: Aihua Xu Assignee: Aihua Xu Labels: SerDe Fix For: 1.2.0 Attachments: HIVE-9500.1.patch, HIVE-9500.2.patch, HIVE-9500.3.patch, TClass.java Customer has deeply nested avro structure and is receiving the following error when performing queries. 15/01/09 20:59:29 ERROR ql.Driver: FAILED: SemanticException org.apache.hadoop.hive.serde2.SerDeException: Number of levels of nesting supported for LazySimpleSerde is 23 Unable to work with level 24 Currently we support up to 24 levels of nested structs when hive.serialization.extend.nesting.levels is set to true, while the customers have the requirement to support more than that. It would be better to make the supported levels configurable or completely removed (i.e., we can support any number of levels). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9574) Lazy computing in HiveBaseFunctionResultList may hurt performance [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314742#comment-14314742
]
Xuefu Zhang commented on HIVE-9574:
---
+1
Lazy computing in HiveBaseFunctionResultList may hurt performance [Spark
Branch]
Key: HIVE-9574
URL: https://issues.apache.org/jira/browse/HIVE-9574
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Rui Li
Assignee: Jimmy Xiang
Fix For: spark-branch
Attachments: HIVE-9574.1-spark.patch, HIVE-9574.2-spark.patch,
HIVE-9574.3-spark.patch, HIVE-9574.4-spark.patch, HIVE-9574.5-spark.patch,
HIVE-9574.6-spark.patch
{{RowContainer.first}} may call {{InputFormat.getSplits}}, which is
expensive. If we switch {{container}} and {{backupContainer}} frequently in
{{HiveKVResultCache}}, it will downgrade performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9574) Lazy computing in HiveBaseFunctionResultList may hurt performance [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-9574:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to Spark branch. Thanks, Jimmy.
Lazy computing in HiveBaseFunctionResultList may hurt performance [Spark
Branch]
Key: HIVE-9574
URL: https://issues.apache.org/jira/browse/HIVE-9574
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Rui Li
Assignee: Jimmy Xiang
Fix For: spark-branch
Attachments: HIVE-9574.1-spark.patch, HIVE-9574.2-spark.patch,
HIVE-9574.3-spark.patch, HIVE-9574.4-spark.patch, HIVE-9574.5-spark.patch,
HIVE-9574.6-spark.patch
{{RowContainer.first}} may call {{InputFormat.getSplits}}, which is
expensive. If we switch {{container}} and {{backupContainer}} frequently in
{{HiveKVResultCache}}, it will downgrade performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9605) Remove parquet nested objects from wrapper writable objects
[
https://issues.apache.org/jira/browse/HIVE-9605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-9605:
--
Attachment: (was: HIVE-9605.2.patch)
Remove parquet nested objects from wrapper writable objects
---
Key: HIVE-9605
URL: https://issues.apache.org/jira/browse/HIVE-9605
Project: Hive
Issue Type: Sub-task
Affects Versions: 0.14.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-9605.3.patch
Parquet nested types are using an extra wrapper object (ArrayWritable) as a
wrapper of map and list elements. This extra object is not needed and causing
unnecessary memory allocations.
An example of code is on HiveCollectionConverter.java:
{noformat}
public void end() {
parent.set(index, wrapList(new ArrayWritable(
Writable.class, list.toArray(new Writable[list.size()];
}
{noformat}
This object is later unwrapped on AbstractParquetMapInspector, i.e.:
{noformat}
final Writable[] mapContainer = ((ArrayWritable) data).get();
final Writable[] mapArray = ((ArrayWritable) mapContainer[0]).get();
for (final Writable obj : mapArray) {
...
}
{noformat}
We should get rid of this wrapper object to save time and memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9605) Remove parquet nested objects from wrapper writable objects
[
https://issues.apache.org/jira/browse/HIVE-9605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-9605:
--
Attachment: HIVE-9605.3.patch
Fixed unit tests.
This patch depends from HIVE-9333. Once it is merged, I will submit this patch.
Remove parquet nested objects from wrapper writable objects
---
Key: HIVE-9605
URL: https://issues.apache.org/jira/browse/HIVE-9605
Project: Hive
Issue Type: Sub-task
Affects Versions: 0.14.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-9605.3.patch
Parquet nested types are using an extra wrapper object (ArrayWritable) as a
wrapper of map and list elements. This extra object is not needed and causing
unnecessary memory allocations.
An example of code is on HiveCollectionConverter.java:
{noformat}
public void end() {
parent.set(index, wrapList(new ArrayWritable(
Writable.class, list.toArray(new Writable[list.size()];
}
{noformat}
This object is later unwrapped on AbstractParquetMapInspector, i.e.:
{noformat}
final Writable[] mapContainer = ((ArrayWritable) data).get();
final Writable[] mapArray = ((ArrayWritable) mapContainer[0]).get();
for (final Writable obj : mapArray) {
...
}
{noformat}
We should get rid of this wrapper object to save time and memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314805#comment-14314805 ] Aihua Xu commented on HIVE-3454: Me either and I feel a new configuration may make things more complicated. From the comments above, actually the users are more using workarounds (converting int/bigint to float/double first before interpreting it as timestamp). My opinion is not to add a new configuration and keep things simple. Problem with CAST(BIGINT as TIMESTAMP) -- Key: HIVE-3454 URL: https://issues.apache.org/jira/browse/HIVE-3454 Project: Hive Issue Type: Bug Components: Types, UDF Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, 0.13.1 Reporter: Ryan Harris Assignee: Aihua Xu Labels: newbie, newdev, patch Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, HIVE-3454.3.patch, HIVE-3454.patch Ran into an issue while working with timestamp conversion. CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current time from the BIGINT returned by unix_timestamp() Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9500) Support nested structs over 24 levels.
[ https://issues.apache.org/jira/browse/HIVE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314843#comment-14314843 ] Thejas M Nair commented on HIVE-9500: - Sorry about the delay in getting back. I don't think the replacement of java array for mapping with HashMap is reasonable, in terms of performance. This use is in a very tight loop. Something that gets called for every record is considered part of tight loop, and this is actually getting called for each char in with a loop on records. So its actually within a tight loop within a tight loop. We have to be sensitive about performance for this case. The performance overheads of using HashMap over native array should be obvious. For one, HashMap requires the use of Objects, instead of native types, so the memory footprint and over heads are going to be large. The memory of the datastructure is not contiguous, there will be several lookups needed to get to the answer of if this char needs to be escaped or not. This will result in CPU over head as well as CPU cache misses. Support nested structs over 24 levels. -- Key: HIVE-9500 URL: https://issues.apache.org/jira/browse/HIVE-9500 Project: Hive Issue Type: Improvement Reporter: Aihua Xu Assignee: Aihua Xu Labels: SerDe Fix For: 1.2.0 Attachments: HIVE-9500.1.patch, HIVE-9500.2.patch, HIVE-9500.3.patch Customer has deeply nested avro structure and is receiving the following error when performing queries. 15/01/09 20:59:29 ERROR ql.Driver: FAILED: SemanticException org.apache.hadoop.hive.serde2.SerDeException: Number of levels of nesting supported for LazySimpleSerde is 23 Unable to work with level 24 Currently we support up to 24 levels of nested structs when hive.serialization.extend.nesting.levels is set to true, while the customers have the requirement to support more than that. It would be better to make the supported levels configurable or completely removed (i.e., we can support any number of levels). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
