[jira] [Commented] (HIVE-6298) Add config flag to turn off fetching partition stats
[ https://issues.apache.org/jira/browse/HIVE-6298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944364#comment-13944364 ] Lefty Leverenz commented on HIVE-6298: -- This needs a fix version. Add config flag to turn off fetching partition stats Key: HIVE-6298 URL: https://issues.apache.org/jira/browse/HIVE-6298 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-6298.1.patch, HIVE-6298.2.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6229) Stats are missing sometimes (regression from HIVE-5936)
[
https://issues.apache.org/jira/browse/HIVE-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944371#comment-13944371
]
Lefty Leverenz commented on HIVE-6229:
--
*hive.stats.key.prefix.reserve.length* is now documented in the wiki, but its
description needs review and clarification:
{quote}
Reserved length for postfix of statistics key. Currently only meaningful for
counter type statistics which should keep the length of the full statistics key
smaller than the maximum length configured by hive.stats.key.prefix.max.length.
For counter type statistics, it should be bigger than the length of LB spec if
exists.
{quote}
What does LB spec mean? All I could find by googling was the Ljung–Box test.
Does if exists mean if the LB spec exists? And what is it in it should
be bigger -- this parameter's setting or the length of the full statistics key?
* [Configuration Properties: hive.stats.key.prefix.reserve.length
|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.stats.key.prefix.reserve.length]
Stats are missing sometimes (regression from HIVE-5936)
---
Key: HIVE-6229
URL: https://issues.apache.org/jira/browse/HIVE-6229
Project: Hive
Issue Type: Bug
Components: Statistics
Reporter: Navis
Assignee: Navis
Fix For: 0.13.0
Attachments: HIVE-6229.1.patch.txt, HIVE-6229.2.patch.txt
if prefix length is smaller than hive.stats.key.prefix.max.length but length
of prefix + postfix is bigger than that, stats are missed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6500) Stats collection via filesystem
[
https://issues.apache.org/jira/browse/HIVE-6500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944389#comment-13944389
]
Lefty Leverenz commented on HIVE-6500:
--
I updated the wiki for *hive.stats.dbclass* -- please review:
{quote}
Hive 0.13 and later: The storage that stores temporary Hive statistics. In FS
based statistics collection, each task writes statistics it has collected in a
file on the filesystem, which will be aggregated after the job has finished.
Supported values are fs (filesystem), jdbc(:.*), hbase, counter and custom
(HIVE-6500).
{quote}
* [Configuration Properties: hive.stats.dbclass
|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.stats.dbclass]
Stats collection via filesystem
---
Key: HIVE-6500
URL: https://issues.apache.org/jira/browse/HIVE-6500
Project: Hive
Issue Type: New Feature
Components: Statistics
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Fix For: 0.13.0
Attachments: HIVE-6500.2.patch, HIVE-6500.3.patch, HIVE-6500.patch
Recently, support for stats gathering via counter was [added |
https://issues.apache.org/jira/browse/HIVE-4632] Although, its useful it has
following issues:
* [Length of counter group name is limited |
https://github.com/apache/hadoop-common/blob/branch-2.3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java?source=c#L340]
* [Length of counter name is limited |
https://github.com/apache/hadoop-common/blob/branch-2.3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java?source=c#L337]
* [Number of distinct counter groups are limited |
https://github.com/apache/hadoop-common/blob/branch-2.3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java?source=c#L343]
* [Number of distinct counters are limited |
https://github.com/apache/hadoop-common/blob/branch-2.3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java?source=c#L334]
Although, these limits are configurable, but setting them to higher value
implies increased memory load on AM and job history server.
Now, whether these limits makes sense or not is [debatable |
https://issues.apache.org/jira/browse/MAPREDUCE-5680] it is desirable that
Hive doesn't make use of counters features of framework so that it we can
evolve this feature without relying on support from framework. Filesystem
based counter collection is a step in that direction.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6706) Tez queries fail when there are no input paths
[
https://issues.apache.org/jira/browse/HIVE-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944408#comment-13944408
]
Hive QA commented on HIVE-6706:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12635726/HIVE-6706.1.patch
{color:green}SUCCESS:{color} +1 5440 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1918/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1918/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12635726
Tez queries fail when there are no input paths
--
Key: HIVE-6706
URL: https://issues.apache.org/jira/browse/HIVE-6706
Project: Hive
Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
Attachments: HIVE-6706.1.patch
Need to disable check for empty input paths in the InputFormat in Tez. Tez
allows empty vertices to make it easier to handle empty tables, partitions,
buckets, etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6700) In some queries inputs are closed on Tez before the operator pipeline is flushed
[
https://issues.apache.org/jira/browse/HIVE-6700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944422#comment-13944422
]
Hive QA commented on HIVE-6700:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12635657/HIVE-6700.2.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5440 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_bucketed_table
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1919/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1919/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12635657
In some queries inputs are closed on Tez before the operator pipeline is
flushed
Key: HIVE-6700
URL: https://issues.apache.org/jira/browse/HIVE-6700
Project: Hive
Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
Attachments: HIVE-6700.1.patch, HIVE-6700.2.patch
Group by operators won't flush their last row until operator is closed. In
Tez it's possible that the input is already closed at this point.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6455) Scalable dynamic partitioning and bucketing optimization
[
https://issues.apache.org/jira/browse/HIVE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1394#comment-1394
]
Hive QA commented on HIVE-6455:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12636170/HIVE-6455.20.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 5441 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample10
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_dyn_part
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1920/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1920/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12636170
Scalable dynamic partitioning and bucketing optimization
Key: HIVE-6455
URL: https://issues.apache.org/jira/browse/HIVE-6455
Project: Hive
Issue Type: New Feature
Components: Query Processor
Affects Versions: 0.13.0
Reporter: Prasanth J
Assignee: Prasanth J
Labels: optimization
Attachments: HIVE-6455.1.patch, HIVE-6455.1.patch,
HIVE-6455.10.patch, HIVE-6455.10.patch, HIVE-6455.11.patch,
HIVE-6455.12.patch, HIVE-6455.13.patch, HIVE-6455.13.patch,
HIVE-6455.14.patch, HIVE-6455.15.patch, HIVE-6455.16.patch,
HIVE-6455.17.patch, HIVE-6455.17.patch.txt, HIVE-6455.18.patch,
HIVE-6455.19.patch, HIVE-6455.2.patch, HIVE-6455.20.patch, HIVE-6455.3.patch,
HIVE-6455.4.patch, HIVE-6455.4.patch, HIVE-6455.5.patch, HIVE-6455.6.patch,
HIVE-6455.7.patch, HIVE-6455.8.patch, HIVE-6455.9.patch, HIVE-6455.9.patch
The current implementation of dynamic partition works by keeping at least one
record writer open per dynamic partition directory. In case of bucketing
there can be multispray file writers which further adds up to the number of
open record writers. The record writers of column oriented file format (like
ORC, RCFile etc.) keeps some sort of in-memory buffers (value buffer or
compression buffers) open all the time to buffer up the rows and compress
them before flushing it to disk. Since these buffers are maintained per
column basis the amount of constant memory that will required at runtime
increases as the number of partitions and number of columns per partition
increases. This often leads to OutOfMemory (OOM) exception in mappers or
reducers depending on the number of open record writers. Users often tune the
JVM heapsize (runtime memory) to get over such OOM issues.
With this optimization, the dynamic partition columns and bucketing columns
(in case of bucketed tables) are sorted before being fed to the reducers.
Since the partitioning and bucketing columns are sorted, each reducers can
keep only one record writer open at any time thereby reducing the memory
pressure on the reducers. This optimization is highly scalable as the number
of partition and number of columns per partition increases at the cost of
sorting the columns.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6673) sql std auth - show grant statement for all principals throws NPE
[
https://issues.apache.org/jira/browse/HIVE-6673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-6673:
---
Resolution: Fixed
Fix Version/s: 0.13.0
Status: Resolved (was: Patch Available)
Committed to 0.13 trunk. Thanks, Thejas!
sql std auth - show grant statement for all principals throws NPE
-
Key: HIVE-6673
URL: https://issues.apache.org/jira/browse/HIVE-6673
Project: Hive
Issue Type: Sub-task
Components: Authorization
Affects Versions: 0.13.0
Reporter: Thejas M Nair
Assignee: Thejas M Nair
Fix For: 0.13.0
Attachments: HIVE-6673.1.patch, HIVE-6673.1.patch
{code}
show grant on table t1;
2014-03-14 12:45:46,573 ERROR exec.DDLTask (DDLTask.java:execute(461)) -
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.DDLTask.getHivePrincipal(DDLTask.java:893)
at
org.apache.hadoop.hive.ql.exec.DDLTask.showGrantsV2(DDLTask.java:648)
at org.apache.hadoop.hive.ql.exec.DDLTask.showGrants(DDLTask.java:553)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:426)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
...
...
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6447) Bucket map joins in hive-tez
[
https://issues.apache.org/jira/browse/HIVE-6447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944475#comment-13944475
]
Hive QA commented on HIVE-6447:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12635732/HIVE-6447.4.patch
{color:green}SUCCESS:{color} +1 5440 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1921/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1921/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12635732
Bucket map joins in hive-tez
Key: HIVE-6447
URL: https://issues.apache.org/jira/browse/HIVE-6447
Project: Hive
Issue Type: Bug
Components: Tez
Affects Versions: tez-branch
Reporter: Vikram Dixit K
Assignee: Vikram Dixit K
Attachments: HIVE-6447.1.patch, HIVE-6447.2.patch, HIVE-6447.3.patch,
HIVE-6447.4.patch, HIVE-6447.WIP.patch
Support bucket map joins in tez.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HIVE-6364) HiveServer2 - Request serving thread should get class loader from existing SessionState
[
https://issues.apache.org/jira/browse/HIVE-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan resolved HIVE-6364.
Resolution: Fixed
Fix Version/s: 0.13.0
Fixed via HIVE-3969
HiveServer2 - Request serving thread should get class loader from existing
SessionState
---
Key: HIVE-6364
URL: https://issues.apache.org/jira/browse/HIVE-6364
Project: Hive
Issue Type: Bug
Components: HiveServer2
Reporter: Jaideep Dhok
Fix For: 0.13.0
Attachments: HIVE-6364.1.patch
SessionState is created for each session in HS2. If we do any add jars, a
class loader is set in the SessionState's conf object. This class loader
should also be set in each thread that serves request of the same session.
Scenario (both requests are in the same session)-
{noformat}
// req 1
add jar foo.jar // Served by thread th1, this updates class loader and sets
in SessionState.conf
// req2 served by th2, such that th1 != th2
CREATE TEMPORARY FUNCTION foo_udf AS 'some class in foo.jar'
// This can throw class not found error, because although
// the new thread (th2) gets the same session state as th1,
// the class loader is different (Thread.currentThread.getContextClassLoader()
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-3969) Session state for hive server should be cleaned-up
[ https://issues.apache.org/jira/browse/HIVE-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-3969: --- Resolution: Fixed Fix Version/s: 0.13.0 Status: Resolved (was: Patch Available) Committed to trunk 0.13 Session state for hive server should be cleaned-up -- Key: HIVE-3969 URL: https://issues.apache.org/jira/browse/HIVE-3969 Project: Hive Issue Type: Bug Components: Server Infrastructure Reporter: Navis Assignee: Navis Priority: Trivial Fix For: 0.13.0 Attachments: HIVE-3969.1.patch.txt, HIVE-3969.2.patch.txt, HIVE-3969.D8325.1.patch Currently add jar command by clients are adding child ClassLoader to worker thread cumulatively, causing various problems. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6672) JARs loaded by permanent functions don't work properly with HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-6672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6672: --- Resolution: Fixed Fix Version/s: 0.13.0 Status: Resolved (was: Patch Available) Committed to trunk 0.13. Thanks, Jason! JARs loaded by permanent functions don't work properly with HiveServer2 --- Key: HIVE-6672 URL: https://issues.apache.org/jira/browse/HIVE-6672 Project: Hive Issue Type: Bug Components: HiveServer2, UDF Reporter: Jason Dere Assignee: Jason Dere Fix For: 0.13.0 Attachments: HIVE-6672.1.patch Permanent functions which specify JAR files to load do not seem to work properly with HiveServer2. While the function can be created and see with SHOW FUNCTIONS, doing DESCRIBE FUNCTION or using the UDF in a query results in java.lang.ClassNotFoundException. It looks like the JAR is only added to the classloader of the very first thread which references the UDF. Subsequent threads will have a different class loader which may not have loaded the JAR, but because the UDF is already in the FunctionRegistry these threads do not attempt to load the resources for the UDF. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6704) date_add()/date_sub()/datediff() fail with NPE with null input
[
https://issues.apache.org/jira/browse/HIVE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-6704:
---
Resolution: Fixed
Fix Version/s: 0.13.0
Status: Resolved (was: Patch Available)
Committed to trunk 0.13 Thanks, Jason!
date_add()/date_sub()/datediff() fail with NPE with null input
--
Key: HIVE-6704
URL: https://issues.apache.org/jira/browse/HIVE-6704
Project: Hive
Issue Type: Bug
Components: UDF
Reporter: Jason Dere
Assignee: Jason Dere
Fix For: 0.13.0
Attachments: HIVE-6704.1.patch
Similar to HIVE-6645, the following functions also get NPE error if the input
is null:
date_add
date_sub
datediff
{noformat}
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row
{t:null,si:null,i:null,b:null,f:null,d:null,dc:null,bo:null,s:null,s2:null,ts:null,ts2:null,dt:null}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:195)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row
{t:null,si:null,i:null,b:null,f:null,d:null,dc:null,bo:null,s:null,s2:null,ts:null,ts2:null,dt:null}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating
date_add(ts, 2)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:791)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:791)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524)
... 9 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDFDateAdd.evaluate(GenericUDFDateAdd.java:144)
at
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
at
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
at
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:79)
... 13 more
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: Review Request 19525: Clean up math based UDFs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19525/#review38254 --- Ship it! Ship It! - Xuefu Zhang On March 22, 2014, 1:28 p.m., Lars Francke wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19525/ --- (Updated March 22, 2014, 1:28 p.m.) Review request for hive. Bugs: HIVE-6510 https://issues.apache.org/jira/browse/HIVE-6510 Repository: hive-git Description --- HIVE-6327, HIVE-6246 and HIVE-6385 touched a lot of the math based UDFs. There are some code inconsistencies and warnings left. This cleans up all the problems I could find. Diffs - ql/src/java/org/apache/hadoop/hive/ql/udf/UDFAcos.java 18c79a7 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFAsin.java cfd5d38 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFAtan.java 641bba2 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFBaseBitOP.java f277ea2 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFCos.java bfa95ee ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDegrees.java bc5e1e2 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFExp.java cf6f53e ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLn.java eb5f646 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLog.java 7a4d8a7 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLog10.java 00dc319 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLog2.java 9202258 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFMath.java c1981af ql/src/java/org/apache/hadoop/hive/ql/udf/UDFRadians.java fd1f0e3 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSign.java 6e4bee0 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSin.java 8f757f2 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSqrt.java 17094c9 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFTan.java c286619 Diff: https://reviews.apache.org/r/19525/diff/ Testing --- Thanks, Lars Francke
[jira] [Updated] (HIVE-6697) HiveServer2 secure thrift/http authentication needs to support SPNego
[ https://issues.apache.org/jira/browse/HIVE-6697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6697: --- Assignee: Dilli Arumugam (was: Vaibhav Gumashta) HiveServer2 secure thrift/http authentication needs to support SPNego -- Key: HIVE-6697 URL: https://issues.apache.org/jira/browse/HIVE-6697 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: Dilli Arumugam Assignee: Dilli Arumugam Attachments: hive-6697-req-impl-verify.md Looking to integrating Apache Knox to work with HiveServer2 secure thrift/http. Found that thrift/http uses some form of Kerberos authentication that is not SPNego. Considering it is going over http protocol, expected it to use SPNego protocol. Apache Knox is already integrated with WebHDFS, WebHCat, Oozie and HBase Stargate using SPNego for authentication. Requesting that HiveServer2 secure thrift/http authentication support SPNego. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6510) Clean up math based UDFs
[ https://issues.apache.org/jira/browse/HIVE-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944483#comment-13944483 ] Xuefu Zhang commented on HIVE-6510: --- +1 pending on test result. Clean up math based UDFs Key: HIVE-6510 URL: https://issues.apache.org/jira/browse/HIVE-6510 Project: Hive Issue Type: Improvement Reporter: Lars Francke Assignee: Lars Francke Priority: Minor Attachments: HIVE-6510.1.patch, HIVE-6510.2.patch HIVE-6327, HIVE-6246 and HIVE-6385 touched a lot of the math based UDFs. There are some code inconsistencies and warnings left. This cleans up all the problems I could find. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6697) HiveServer2 secure thrift/http authentication needs to support SPNego
[ https://issues.apache.org/jira/browse/HIVE-6697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944485#comment-13944485 ] Ashutosh Chauhan commented on HIVE-6697: [~darumugam] Can you upload the patch and name it as per [convention | https://cwiki.apache.org/confluence/display/Hive/Hive+PreCommit+Patch+Testing] so that Hive QA picks it up. Also, create an [review board | https://reviews.apache.org/r/new/] entry to facilitate review. HiveServer2 secure thrift/http authentication needs to support SPNego -- Key: HIVE-6697 URL: https://issues.apache.org/jira/browse/HIVE-6697 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: Dilli Arumugam Assignee: Dilli Arumugam Attachments: hive-6697-req-impl-verify.md Looking to integrating Apache Knox to work with HiveServer2 secure thrift/http. Found that thrift/http uses some form of Kerberos authentication that is not SPNego. Considering it is going over http protocol, expected it to use SPNego protocol. Apache Knox is already integrated with WebHDFS, WebHCat, Oozie and HBase Stargate using SPNego for authentication. Requesting that HiveServer2 secure thrift/http authentication support SPNego. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6500) Stats collection via filesystem
[ https://issues.apache.org/jira/browse/HIVE-6500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944488#comment-13944488 ] Lefty Leverenz commented on HIVE-6500: -- The part I'm not sure of is jdbc(:.*) but plain jdbc didn't seem sufficient. So how about jdbc:database? What other values can it have for database besides derby and mysql? Stats collection via filesystem --- Key: HIVE-6500 URL: https://issues.apache.org/jira/browse/HIVE-6500 Project: Hive Issue Type: New Feature Components: Statistics Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.13.0 Attachments: HIVE-6500.2.patch, HIVE-6500.3.patch, HIVE-6500.patch Recently, support for stats gathering via counter was [added | https://issues.apache.org/jira/browse/HIVE-4632] Although, its useful it has following issues: * [Length of counter group name is limited | https://github.com/apache/hadoop-common/blob/branch-2.3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java?source=c#L340] * [Length of counter name is limited | https://github.com/apache/hadoop-common/blob/branch-2.3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java?source=c#L337] * [Number of distinct counter groups are limited | https://github.com/apache/hadoop-common/blob/branch-2.3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java?source=c#L343] * [Number of distinct counters are limited | https://github.com/apache/hadoop-common/blob/branch-2.3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java?source=c#L334] Although, these limits are configurable, but setting them to higher value implies increased memory load on AM and job history server. Now, whether these limits makes sense or not is [debatable | https://issues.apache.org/jira/browse/MAPREDUCE-5680] it is desirable that Hive doesn't make use of counters features of framework so that it we can evolve this feature without relying on support from framework. Filesystem based counter collection is a step in that direction. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6695) bin/hcat should include hbase jar and dependencies in the classpath [followup/clone of HCATALOG-621]
[
https://issues.apache.org/jira/browse/HIVE-6695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-6695:
---
Fix Version/s: 0.14.0
bin/hcat should include hbase jar and dependencies in the classpath
[followup/clone of HCATALOG-621]
Key: HIVE-6695
URL: https://issues.apache.org/jira/browse/HIVE-6695
Project: Hive
Issue Type: Bug
Reporter: Sushanth Sowmyan
Assignee: Nick Dimiduk
Fix For: 0.14.0
Attachments: HIVE-6695.patch
This is to address the addendum of HCATALOG-621, now that the HCatalog jira
seems to be in read-only mode. To quote Nick from the original bug:
I'm not sure how this fixes anything for the error listed above. The find
command in the script we merged is broken, at least on linux. Maybe it worked
with BSD find and we both tested on Macs?
From the patch we committed:
{noformat}
if [ -d ${HBASE_HOME} ] ; then
for jar in $(find $HBASE_HOME -name *.jar -not -name thrift\*.jar); do
HBASE_CLASSPATH=$HBASE_CLASSPATH:${jar}
done
export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${HBASE_CLASSPATH}
fi
{noformat}
The find command syntax is wrong – it returns no jars ever.
{noformat}
$ find /usr/lib/hbase -name *.jar
$ find /usr/lib/hbase -name *.jar -not -name thrift\*.jar
$
{noformat}
What we need is more like:
{noformat}
$ find /usr/lib/hbase -name '*.jar'
... // prints lots of jars
$ find /usr/lib/hbase -name '*.jar' | grep thrift
/usr/lib/hbase/lib/libthrift-0.9.0.jar
$ find /usr/lib/hbase -name '*.jar' -not -name '*thrift*' | grep thrift
$
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6298) Add config flag to turn off fetching partition stats
[ https://issues.apache.org/jira/browse/HIVE-6298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6298: --- Fix Version/s: 0.13.0 Add config flag to turn off fetching partition stats Key: HIVE-6298 URL: https://issues.apache.org/jira/browse/HIVE-6298 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: 0.13.0 Attachments: HIVE-6298.1.patch, HIVE-6298.2.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6303) VectorExpressionWriter for date and decimal datatypes.
[ https://issues.apache.org/jira/browse/HIVE-6303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6303: --- Fix Version/s: 0.13.0 VectorExpressionWriter for date and decimal datatypes. -- Key: HIVE-6303 URL: https://issues.apache.org/jira/browse/HIVE-6303 Project: Hive Issue Type: Sub-task Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Fix For: 0.13.0 Attachments: HIVE-6303.1.patch VectorExpressionWriter for date and decimal datatypes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6264) Unbalanced number of HiveParser msgs.push/msgs.pop calls when doing lookahead
[
https://issues.apache.org/jira/browse/HIVE-6264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-6264:
---
Fix Version/s: 0.13.0
Unbalanced number of HiveParser msgs.push/msgs.pop calls when doing lookahead
-
Key: HIVE-6264
URL: https://issues.apache.org/jira/browse/HIVE-6264
Project: Hive
Issue Type: Bug
Reporter: Jason Dere
Assignee: Jason Dere
Fix For: 0.13.0
Attachments: HIVE-6264.1.patch, HIVE-6264.2.patch, HIVE-6264.3.patch,
HIVE-6264.4.patch
HiveParser pushes/pops messages describing the current parse rule like so:
{noformat}
joinSource
@init { gParent.msgs.push(join source); }
@after { gParent.msgs.pop(); }
...
{noformat}
The ANTLR generated code for the init/after actions looks like this:
{noformat}
gParent.msgs.push(join source);
...
if ( state.backtracking==0 ) { gParent.msgs.pop(); }
{noformat}
If we have a parse rule that does some lookahead, the message is always
pushed onto the message stack since the init action has no check of
state.backtracking. But that message is never popped because the after
action does check state.backtracking. As a result there can be a bunch of
parser context messages added to the stack which are never taken off.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6287) batchSize computation in Vectorized ORC reader can cause BufferUnderFlowException when PPD is enabled
[
https://issues.apache.org/jira/browse/HIVE-6287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-6287:
---
Fix Version/s: 0.13.0
batchSize computation in Vectorized ORC reader can cause
BufferUnderFlowException when PPD is enabled
-
Key: HIVE-6287
URL: https://issues.apache.org/jira/browse/HIVE-6287
Project: Hive
Issue Type: Bug
Components: Vectorization
Affects Versions: 0.13.0
Reporter: Prasanth J
Assignee: Prasanth J
Labels: orcfile, vectorization
Fix For: 0.13.0
Attachments: HIVE-6287.1.patch, HIVE-6287.2.patch, HIVE-6287.3.patch,
HIVE-6287.3.patch, HIVE-6287.4.patch, HIVE-6287.WIP.patch
nextBatch() method that computes the batchSize is only aware of stripe
boundaries. This will not work when predicate pushdown (PPD) in ORC is
enabled as PPD works at row group level (stripe contains multiple row
groups). By default, row group stride is 1. When PPD is enabled, some row
groups may get eliminated. After row group elimination, disk ranges are
computed based on the selected row groups. If batchSize computation is not
aware of this, it will lead to BufferUnderFlowException (reading beyond disk
range). Following scenario should illustrate it more clearly
{code}
|- STRIPE 1
|
|-- row grp 1 --|-- row grp 2 --|-- row grp 3 --|-- row grp 4 --|-- row grp 5
--|
|- diskrange 1 -| |- diskrange
2 -|
^
(marker)
{code}
diskrange1 will have 2 rows and diskrange 2 will have 1 rows. Since
nextBatch() was not aware of row groups and hence the diskranges, it tries to
read 1024 values from the end of diskrange 1 where it should only read 2
% 1024 = 544 values. This will result in BufferUnderFlowException.
To fix this, a marker is placed at the end of each range and batchSize is
computed accordingly. {code}batchSize =
Math.min(VectorizedRowBatch.DEFAULT_SIZE, (markerPosition -
rowInStripe));{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6240) Update jetty to the latest stable (9.x) in the service module
[ https://issues.apache.org/jira/browse/HIVE-6240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6240: --- Fix Version/s: 0.13.0 Update jetty to the latest stable (9.x) in the service module - Key: HIVE-6240 URL: https://issues.apache.org/jira/browse/HIVE-6240 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Vaibhav Gumashta Fix For: 0.13.0 We're using a very old version of jetty (6.x) which has moved a lot: http://www.eclipse.org/jetty/documentation/current/what-jetty-version.html. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6224) Remove unneeded tez dependencies from hive
[ https://issues.apache.org/jira/browse/HIVE-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6224: --- Fix Version/s: 0.13.0 Remove unneeded tez dependencies from hive -- Key: HIVE-6224 URL: https://issues.apache.org/jira/browse/HIVE-6224 Project: Hive Issue Type: Bug Components: Build Infrastructure, Tez Affects Versions: 0.13.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Minor Fix For: 0.13.0 Attachments: HIVE-6224.1.patch After re-organization of some of the classes in tez, we no longer need to depend on certain packages. Removing these from the shims and from the tests dependencies. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6232) allow user to control out-of-range values in HCatStorer
[ https://issues.apache.org/jira/browse/HIVE-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6232: --- Fix Version/s: 0.13.0 allow user to control out-of-range values in HCatStorer --- Key: HIVE-6232 URL: https://issues.apache.org/jira/browse/HIVE-6232 Project: Hive Issue Type: Sub-task Components: HCatalog Affects Versions: 0.13.0 Reporter: Eugene Koifman Assignee: Eugene Koifman Fix For: 0.13.0 Pig values support wider range than Hive. e.g. Pig BIGDECIMAL vs Hive DECIMAL. When storing Pig data into Hive table, if the value is out of range there are 2 options: 1. throw an exception. 2. write NULL instead of the value The 1st has the drawback that it may kill the process that loads 100M rows after 90M rows have been loaded. But the 2nd may not be appropriate for all use cases. Should add support for additional parameters in HCatStorer where the user can specify an option to controll this. see org.apache.pig.backend.hadoop.hbase.HBaseStorage for examples -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6237) Webhcat e2e test JOBS_2 fail due to permission when hdfs umask setting is 022
[ https://issues.apache.org/jira/browse/HIVE-6237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6237: --- Fix Version/s: 0.13.0 Webhcat e2e test JOBS_2 fail due to permission when hdfs umask setting is 022 - Key: HIVE-6237 URL: https://issues.apache.org/jira/browse/HIVE-6237 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Fix For: 0.13.0 Webhcat e2e test JOBS_2 fail due to permission when hdfs umask setting is 022. We need to make sure that the test is deterministic. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6176) Beeline gives bogus error message if an unaccepted command line option is given
[
https://issues.apache.org/jira/browse/HIVE-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-6176:
---
Fix Version/s: 0.14.0
Beeline gives bogus error message if an unaccepted command line option is
given
---
Key: HIVE-6176
URL: https://issues.apache.org/jira/browse/HIVE-6176
Project: Hive
Issue Type: Bug
Components: CLI
Affects Versions: 0.12.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Fix For: 0.14.0
{code}
$ beeline -o
-o (No such file or directory)
Beeline version 0.13.0-SNAPSHOT by Apache Hive
beeline
{code}
The message suggests that beeline accepts a file (without -f option) while it
enters interactive mode any way.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6189) Support top level union all statements
[ https://issues.apache.org/jira/browse/HIVE-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6189: --- Fix Version/s: 0.13.0 Support top level union all statements -- Key: HIVE-6189 URL: https://issues.apache.org/jira/browse/HIVE-6189 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: 0.13.0 Attachments: HIVE-6189.1.patch, HIVE-6189.2.patch, HIVE-6189.3.patch I've always wondered why union all has to be in subqueries in hive. After looking at it, problems are: - Hive Parser: - Union happens at the wrong place (insert ... select ... union all select ...) is parsed as (insert select) union select. - There are many rewrite rules in the parser to force any query into the a from - insert -select form. No doubt for historical reasons. - Plan generation/semantic analysis assumes top level TOK_QUERY and not top level TOK_UNION. The rewrite rules don't work when we move the UNION ALL recursion into the select statements. However, it's not hard to do that in code. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6124) Support basic Decimal arithmetic in vector mode (+, -, *)
[ https://issues.apache.org/jira/browse/HIVE-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6124: --- Fix Version/s: 0.13.0 Support basic Decimal arithmetic in vector mode (+, -, *) - Key: HIVE-6124 URL: https://issues.apache.org/jira/browse/HIVE-6124 Project: Hive Issue Type: Sub-task Affects Versions: 0.13.0 Reporter: Eric Hanson Assignee: Eric Hanson Fix For: 0.13.0 Attachments: HIVE-6124.01.patch, HIVE-6124.02.patch, HIVE-6124.03.patch, HIVE-6124.04.patch Create support for basic decimal arithmetic (+, -, * but not /, %) based on templates for column-scalar, scalar-column, and column-column operations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6156) Implement vectorized reader for Date datatype for ORC format.
[ https://issues.apache.org/jira/browse/HIVE-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6156: --- Fix Version/s: 0.13.0 Implement vectorized reader for Date datatype for ORC format. - Key: HIVE-6156 URL: https://issues.apache.org/jira/browse/HIVE-6156 Project: Hive Issue Type: Sub-task Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Fix For: 0.13.0 Attachments: HIVE-6156.1.patch, HIVE-6156.2.patch We need to implement vectorized reader for Date datatype for ORC format. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6085) Tez changed test parse tests output
[ https://issues.apache.org/jira/browse/HIVE-6085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6085: --- Fix Version/s: 0.13.0 Tez changed test parse tests output --- Key: HIVE-6085 URL: https://issues.apache.org/jira/browse/HIVE-6085 Project: Hive Issue Type: Bug Components: Tez Affects Versions: tez-branch Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.13.0 Attachments: HIVE-6085.1.patch Plans generated in hive changed in the tez branch. We need to update the results of TestParse to reflect the same. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6005) BETWEEN is broken after using KRYO
[ https://issues.apache.org/jira/browse/HIVE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6005: --- Fix Version/s: 0.13.0 BETWEEN is broken after using KRYO -- Key: HIVE-6005 URL: https://issues.apache.org/jira/browse/HIVE-6005 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Eric Chu Fix For: 0.13.0 After taking in HIVE-1511, HIVE-5422, and HIVE-5257 on top of Hive 0.12 to use Kryo, queries with BETWEEN start to fail with the following exception: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableConstantBooleanObjectInspector Serialization trace: argumentOIs (org.apache.hadoop.hive.ql.udf.generic.GenericUDFBetween) genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) filters (org.apache.hadoop.hive.ql.plan.JoinDesc) conf (org.apache.hadoop.hive.ql.exec.JoinOperator) reducer (org.apache.hadoop.hive.ql.plan.ReduceWork) at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1097) at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1109) at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:526) ... A workaround is to replace BETWEEN with = and =, but I think this failure is a bug and not by design. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6003) bin/hive --debug should not append HIVE_CLIENT_OPTS to HADOOP_OPTS
[
https://issues.apache.org/jira/browse/HIVE-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-6003:
---
Fix Version/s: 0.13.0
bin/hive --debug should not append HIVE_CLIENT_OPTS to HADOOP_OPTS
---
Key: HIVE-6003
URL: https://issues.apache.org/jira/browse/HIVE-6003
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Thejas M Nair
Assignee: Thejas M Nair
Fix For: 0.13.0
Attachments: HIVE-6003.1.patch
hadoop (0.20.2, 1.x, 2.x) appends HADOOP_CLIENT_OPTS to HADOO_OPTS.
So it is unnecessary to have this statement in bin/hive, under debug mode -
export HADOOP_OPTS=$HADOOP_OPTS $HADOOP_CLIENT_OPTS
It results in the HADOOP_CLIENT_OPTS being appended twice, resulting in this
error in debug mode.
{code}
bin/hive --debug
ERROR: Cannot load this JVM TI agent twice, check your java command line for
duplicate jdwp options.
Error occurred during initialization of VM
agent library failed to init: jdwp
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-5995) decimal_precision.q test case fails in trunk
[ https://issues.apache.org/jira/browse/HIVE-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5995: --- Fix Version/s: 0.13.0 decimal_precision.q test case fails in trunk Key: HIVE-5995 URL: https://issues.apache.org/jira/browse/HIVE-5995 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Prasanth J Assignee: Prasanth J Labels: test Fix For: 0.13.0 Attachments: HIVE-5995.1.patch decimal_precision test case fails reporting diff of one additional blank line. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5839) BytesRefArrayWritable compareTo violates contract
[ https://issues.apache.org/jira/browse/HIVE-5839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5839: --- Fix Version/s: 0.13.0 BytesRefArrayWritable compareTo violates contract - Key: HIVE-5839 URL: https://issues.apache.org/jira/browse/HIVE-5839 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 0.11.0, 0.12.0 Reporter: Ian Robertson Assignee: Xuefu Zhang Fix For: 0.13.0 Attachments: HIVE-5839.1.patch, HIVE-5839.2.patch, HIVE-5839.patch, HIVE-5839.patch BytesRefArrayWritable's compareTo violates the compareTo contract from java.lang.Object. Specifically: * The implementor must ensure sgn(x.compareTo( y )) == -sgn(y.compareTo( x )) for all x and y. The compareTo implementation on BytesRefArrayWritable does a proper comparison of the sizes of the two instances. However, if the sizes are the same, it proceeds to do a check if both array's have the same constant. If not, it returns 1. This means that if x and y are two BytesRefArrayWritable instances with the same size, but different contents, then x.compareTo( y ) == 1 and y.compareTo( x ) == 1. Additionally, the comparison of contents is order agnostic. This seems wrong, since order of entries should matter. It is also very inefficient, running at O(n^2), where n is the number of entries. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5955) SQL std auth - metastore api support for get_privilege_set api that checks specific role
[ https://issues.apache.org/jira/browse/HIVE-5955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5955: --- Fix Version/s: 0.13.0 SQL std auth - metastore api support for get_privilege_set api that checks specific role Key: HIVE-5955 URL: https://issues.apache.org/jira/browse/HIVE-5955 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Fix For: 0.13.0 Original Estimate: 48h Time Spent: 24h Remaining Estimate: 0h If the user has a set a specific role using 'SET ROLE role', then the authorization check should be done for specific role. The authorization check should not check with all the roles the user belongs to. This would new/different method in metastore api . -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5943) SQL std auth - authorize DESCRIBE ROLE role
[ https://issues.apache.org/jira/browse/HIVE-5943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5943: --- Fix Version/s: 0.13.0 SQL std auth - authorize DESCRIBE ROLE role Key: HIVE-5943 URL: https://issues.apache.org/jira/browse/HIVE-5943 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.13.0 Original Estimate: 12h Remaining Estimate: 12h Should be allowed only for user who belongs to the role or superuser. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5950) ORC SARG creation fails with NPE for predicate conditions with decimal/date/char/varchar datatypes
[ https://issues.apache.org/jira/browse/HIVE-5950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5950: --- Fix Version/s: 0.13.0 ORC SARG creation fails with NPE for predicate conditions with decimal/date/char/varchar datatypes -- Key: HIVE-5950 URL: https://issues.apache.org/jira/browse/HIVE-5950 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Prasanth J Assignee: Prasanth J Labels: orcfile Fix For: 0.13.0 Attachments: HIVE-5950.1.patch, HIVE-5950.2.patch, HIVE-5950.3.patch, HIVE-5950.4.patch, HIVE-5950.5.patch When decimal or date column is used, the type field in PredicateLeafImpl will be set to null. This will result in NPE during predicate leaf generation because of null dereferencing in hashcode computation. SARG creation should be extended to support/handle decimal and date data types. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5942) SQL std auth - support DESCRIBE ROLE
[ https://issues.apache.org/jira/browse/HIVE-5942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5942: --- Fix Version/s: 0.13.0 SQL std auth - support DESCRIBE ROLE Key: HIVE-5942 URL: https://issues.apache.org/jira/browse/HIVE-5942 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.13.0 Original Estimate: 24h Remaining Estimate: 24h DESCRIBE ROLE role must be supported. This will list all users and roles that participate in a role. This will be available to the superuser and any member of the role that has admin privileges on the role. This task includes parser changes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5919) COALESCE does not vectorize
[ https://issues.apache.org/jira/browse/HIVE-5919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5919: --- Fix Version/s: 0.13.0 COALESCE does not vectorize --- Key: HIVE-5919 URL: https://issues.apache.org/jira/browse/HIVE-5919 Project: Hive Issue Type: Improvement Components: Vectorization Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.13.0 COALESCE does not vectorize -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5904) HiveServer2 JDBC connect to non-default database
[ https://issues.apache.org/jira/browse/HIVE-5904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5904: --- Fix Version/s: 0.13.0 HiveServer2 JDBC connect to non-default database Key: HIVE-5904 URL: https://issues.apache.org/jira/browse/HIVE-5904 Project: Hive Issue Type: Bug Components: HiveServer2, JDBC Affects Versions: 0.12.0 Reporter: Matt Tucker Assignee: Matt Tucker Fix For: 0.13.0 Attachments: HIVE-5904.patch When connecting to HiveServer to via the following URLs, the session uses the 'default' database, instead of the intended database. jdbc://localhost:1/customDb jdbc:///customDb -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5917) Hive packaging build is broken
[
https://issues.apache.org/jira/browse/HIVE-5917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-5917:
---
Fix Version/s: 0.13.0
Hive packaging build is broken
--
Key: HIVE-5917
URL: https://issues.apache.org/jira/browse/HIVE-5917
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Affects Versions: 0.13.0
Reporter: Xuefu Zhang
Fix For: 0.13.0
mvn package -DskipTests -Phadoop-1 -Pdist
yield the following error:
{code}[INFO] Hive Packaging FAILURE
[1.224s]
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 12.559s
[INFO] Finished at: Mon Dec 02 15:24:58 PST 2013
[INFO] Final Memory: 70M/2933M
[INFO]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-assembly-plugin:2.3:single (assemble) on
project hive-packaging: Assembly is incorrectly configured: bin: Assembly is
incorrectly configured: bin:
[ERROR] Assembly: bin is not configured correctly: One or more filters had
unmatched criteria. Check debug log for more information.
[ERROR] - [Help 1]
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-5920) Implement vectorized support for CASE WHEN a THEN b [WHEN c THEN d]* [ELSE e] END
[ https://issues.apache.org/jira/browse/HIVE-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5920: --- Fix Version/s: 0.13.0 Implement vectorized support for CASE WHEN a THEN b [WHEN c THEN d]* [ELSE e] END -- Key: HIVE-5920 URL: https://issues.apache.org/jira/browse/HIVE-5920 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Fix For: 0.13.0 Implement vectorized support for the second of the two types of CASE expressions: CASE WHEN a THEN b [WHEN c THEN d]* [ELSE e] END -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5880) Rename HCatalog HBase Storage Handler artifact id
[ https://issues.apache.org/jira/browse/HIVE-5880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5880: --- Fix Version/s: 0.13.0 Rename HCatalog HBase Storage Handler artifact id - Key: HIVE-5880 URL: https://issues.apache.org/jira/browse/HIVE-5880 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Fix For: 0.13.0 Attachments: HIVE-5880.patch Current the HBase storage handler is named hive-hbase-storage-handler. I think we should rename it to hive-hcatalog-hbase-storage-handler to match the other hcatalog artifacts and to differentiate it from the hive-hbase-handler. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5899) NPE during explain extended with char/varchar columns
[ https://issues.apache.org/jira/browse/HIVE-5899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5899: --- Fix Version/s: 0.13.0 NPE during explain extended with char/varchar columns - Key: HIVE-5899 URL: https://issues.apache.org/jira/browse/HIVE-5899 Project: Hive Issue Type: Bug Components: Statistics Reporter: Jason Dere Assignee: Jason Dere Fix For: 0.13.0 Attachments: HIVE-5899.1.patch Running analyze table .. for columns with char/varchar columns and subsequently trying to run explain extended will get a NullPointerException when Hive tries to annotate the operator tree with stats: 2013-11-26 01:53:06,682 ERROR ql.Driver (SessionState.java:printError(440)) - FAILED: NullPointerException null java.lang.NullPointerException at org.apache.hadoop.hive.metastore.api.ColumnStatisticsData.getFieldDesc(ColumnStatisticsData.java:367) at org.apache.hadoop.hive.metastore.api.ColumnStatisticsData.getStringStats(ColumnStatisticsData.java:444) at org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatistics(StatsUtils.java:414) at org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStatsForColumn(StatsUtils.java:369) at org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:465) at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:109) at org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:102) at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78) at org.apache.hadoop.hive.ql.lib.PreOrderWalker.walk(PreOrderWalker.java:54) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109) at org.apache.hadoop.hive.ql.optimizer.stats.annotation.AnnotateWithStatistics.transform(AnnotateWithStatistics.java:76) at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:136) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8913) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:292) at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:65) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:292) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:441) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:341) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:994) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:905) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:422) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:790) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:623) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5893) hive-schema-0.13.0.mysql.sql contains reference to nonexistent column
[ https://issues.apache.org/jira/browse/HIVE-5893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5893: --- Fix Version/s: 0.13.0 hive-schema-0.13.0.mysql.sql contains reference to nonexistent column - Key: HIVE-5893 URL: https://issues.apache.org/jira/browse/HIVE-5893 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.13.0 Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.13.0 Attachments: HIVE-5892.1.patch.txt, HIVE-5893.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5863) INSERT OVERWRITE TABLE fails in vectorized mode for ORC format target table
[ https://issues.apache.org/jira/browse/HIVE-5863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5863: --- Fix Version/s: 0.13.0 INSERT OVERWRITE TABLE fails in vectorized mode for ORC format target table --- Key: HIVE-5863 URL: https://issues.apache.org/jira/browse/HIVE-5863 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Eric Hanson Assignee: Remus Rusanu Fix For: 0.13.0 create table store(s_store_key int, s_city string) stored as orc; set hive.vectorized.execution.enabled = true; insert overwrite table store select cint, cstring1 from alltypesorc; Alltypesorc is a test table that is checked in to the Hive source. Expected result: data is added to store table. Actual result: Total MapReduce jobs = 3 Launching Job 1 out of 3 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201311191600_0007, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201311191600_0007 Kill Command = c:\Hadoop\hadoop-1.1.0-SNAPSHOT\bin\hadoop.cmd job -kill job_201311191600_0007 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0 2013-11-20 16:39:53,271 Stage-1 map = 0%, reduce = 0% 2013-11-20 16:40:20,375 Stage-1 map = 100%, reduce = 100% Ended Job = job_201311191600_0007 with errors Error during job, obtaining debugging information... Job Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201311191600_0007 Examining task ID: task_201311191600_0007_m_02 (and more) from job job_201311191600_0007 Task with the most failures(4): - Task ID: task_201311191600_0007_m_00 URL: http://localhost:50030/taskdetails.jsp?jobid=job_201311191600_0007tipid=task_201311191600_0007_m_00 - Diagnostic Messages for this Task: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:181) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366) at org.apache.hadoop.mapred.Child$4.run(Child.java:266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136) at org.apache.hadoop.mapred.Child.main(Child.java:260) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row at org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:45) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163) ... 8 more Caused by: java.lang.ClassCastException: org.apache.hadoop.hive.ql.io.orc.OrcStruct cannot be cast to [Ljava.lang.Object; at org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldData(StandardStructObjectInspec tor.java:173) at org.apache.hadoop.hive.ql.io.orc.WriterImpl$StructTreeWriter.write(WriterImpl.java:1349) at org.apache.hadoop.hive.ql.io.orc.WriterImpl.addRow(WriterImpl.java:1962) at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.write(OrcOutputFormat.java:78) at org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.processOp(VectorFileSinkOperator.java:159) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:489) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:827) at org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.processOp(VectorSelectOperator.java:129) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:489) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:827) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:489) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:827) at org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:43) ... 9 more -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5862) While running some queries on large data using tez, we OOM.
[ https://issues.apache.org/jira/browse/HIVE-5862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5862: --- Fix Version/s: 0.13.0 While running some queries on large data using tez, we OOM. --- Key: HIVE-5862 URL: https://issues.apache.org/jira/browse/HIVE-5862 Project: Hive Issue Type: Bug Components: Tez Affects Versions: tez-branch Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.13.0 Attachments: HIVE-5862.1.patch, HIVE-5862.2.patch Running out of memory while running map joins in tez on large data sets. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5858) Alter table fails with exception when fs.hdfs.impl.disable.cache is set false
[
https://issues.apache.org/jira/browse/HIVE-5858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-5858:
---
Fix Version/s: 0.13.0
Alter table fails with exception when fs.hdfs.impl.disable.cache is set false
-
Key: HIVE-5858
URL: https://issues.apache.org/jira/browse/HIVE-5858
Project: Hive
Issue Type: Bug
Components: Metastore
Reporter: Navis
Assignee: Navis
Priority: Minor
Fix For: 0.13.0
Attachments: HIVE-5858.1.patch.txt
http://www.mail-archive.com/user@hive.apache.org/msg10098.html
{noformat}
2013-11-20 10:07:05,759 ERROR metastore.RetryingHMSHandler
(RetryingHMSHandler.java:invoke(141)) -
InvalidOperationException(message:table new location
hdfs://localhost:9000/user/hive/warehouse/xa is on a different file system
than the old location hdfs://localhost:9000/user/hive/warehouse/zz. This
operation is not supported)
at
org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:146)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:2352)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:103)
at com.sun.proxy.$Proxy9.alter_table_with_environment_context(Unknown
Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table(HiveMetaStoreClient.java:216)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table(HiveMetaStoreClient.java:211)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:74)
at com.sun.proxy.$Proxy10.alter_table(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:385)
at
org.apache.hadoop.hive.ql.exec.DDLTask.alterTable(DDLTask.java:3432)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:306)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:149)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:87)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1485)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1256)
at org.apache.hadoop.hive.ql.Driver.executePlan(Driver.java:1054)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:969)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:764)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-5816) maven test failure shows ant message when running single qfile test
[ https://issues.apache.org/jira/browse/HIVE-5816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5816: --- Fix Version/s: 0.13.0 maven test failure shows ant message when running single qfile test --- Key: HIVE-5816 URL: https://issues.apache.org/jira/browse/HIVE-5816 Project: Hive Issue Type: Bug Components: Testing Infrastructure Reporter: Szehon Ho Assignee: Szehon Ho Fix For: 0.13.0 Attachments: HIVE-5816.patch HIVE-5790 fixes this issue for unit tests, but not when running a qtest. Need to change the positive/negative template files as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5828) Make shims dependecny on specific hadoop hive shims optional
[ https://issues.apache.org/jira/browse/HIVE-5828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5828: --- Fix Version/s: 0.13.0 Make shims dependecny on specific hadoop hive shims optional Key: HIVE-5828 URL: https://issues.apache.org/jira/browse/HIVE-5828 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.13.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.13.0 The issue now is that hive-shims depends on hive-shims-0.20, hive-shims-0.20S, and hive-shims-0.23. ql depends on hive-shims. When ql brings in hive-shims it brings it all transitive dependencies which include three different versions of hadoop. Since hive-shims should not bring any dependencies with it because we expect the end-user module to bring it's hadoop hadoop version. One way to do that is to mark all the hive-shims-* dependencies in hive-shims optional. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5807) Implement vectorization support for IF conditional expression for string inputs
[ https://issues.apache.org/jira/browse/HIVE-5807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5807: --- Fix Version/s: 0.13.0 Implement vectorization support for IF conditional expression for string inputs --- Key: HIVE-5807 URL: https://issues.apache.org/jira/browse/HIVE-5807 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Assignee: Eric Hanson Fix For: 0.13.0 Attachments: hive5807.string-IF-and-fixes.patch.txt -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5806) Implement vectorization support for IF conditional expression for boolean and timestamp inputs
[ https://issues.apache.org/jira/browse/HIVE-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5806: --- Fix Version/s: 0.13.0 Implement vectorization support for IF conditional expression for boolean and timestamp inputs -- Key: HIVE-5806 URL: https://issues.apache.org/jira/browse/HIVE-5806 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Assignee: Eric Hanson Fix For: 0.13.0 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5802) Hive UDF pow limits the second input to integer when the first input is a decimal
[
https://issues.apache.org/jira/browse/HIVE-5802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-5802:
---
Fix Version/s: 0.13.0
Hive UDF pow limits the second input to integer when the first input is a
decimal
-
Key: HIVE-5802
URL: https://issues.apache.org/jira/browse/HIVE-5802
Project: Hive
Issue Type: Improvement
Components: Types, UDF
Affects Versions: 0.12.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Fix For: 0.13.0
For example,
{code}
hive desc test;
OK
i int None
b boolean None
d double None
s string None
dec decimal(5,2)None
hive explain select pow(dec, 2.5) from test;
FAILED: SemanticException [Error 10014]: Line 1:15 Wrong arguments '2.5': No
matching method for class org.apache.hadoop.hive.ql.udf.UDFPower with
(decimal(5,2), double). Possible choices: _FUNC_(decimal(65,30), int)
_FUNC_(double, double) _FUNC_(double, int)
{code}
This seems too restrictive. MySQL, on the other hand, supports fractional
power as the second input.
{code}
mysql select pow(d, 2.5) from test;
mysql desc test;
+---+--+--+-+-+---+
| Field | Type | Null | Key | Default | Extra |
+---+--+--+-+-+---+
| i | int(11) | YES | | NULL| |
| b | tinyint(1) | YES | | NULL| |
| d | double | YES | | NULL| |
| s | varchar(5) | YES | | NULL| |
| dd| decimal(5,2) | YES | | NULL| |
+---+--+--+-+-+---+
5 rows in set (0.00 sec)
mysql select pow(dd, 2.5) from test;
+---+
| pow(dd, 2.5) |
+---+
| 2570.215713318881 |
+---+
1 row in set (0.00 sec)
{code}
Hive should also support this.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-5761) Implement vectorized support for the DATE data type
[ https://issues.apache.org/jira/browse/HIVE-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5761: --- Fix Version/s: 0.13.0 Implement vectorized support for the DATE data type --- Key: HIVE-5761 URL: https://issues.apache.org/jira/browse/HIVE-5761 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Assignee: Teddy Choi Fix For: 0.13.0 Attachments: HIVE-5761.1.patch, HIVE-5761.2.patch, HIVE-5761.3.patch, HIVE-5761.4.patch, HIVE-5761.5.patch, HIVE-5761.6.patch, HIVE-5761.6.patch Add support to allow queries referencing DATE columns and expression results to run efficiently in vectorized mode. This should re-use the code for the the integer/timestamp types to the extent possible and beneficial. Include unit tests and end-to-end tests. Consider re-using or extending existing end-to-end tests for vectorized integer and/or timestamp operations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5759) Implement vectorized support for COALESCE conditional expression
[ https://issues.apache.org/jira/browse/HIVE-5759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5759: --- Fix Version/s: 0.13.0 Implement vectorized support for COALESCE conditional expression Key: HIVE-5759 URL: https://issues.apache.org/jira/browse/HIVE-5759 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Assignee: Jitendra Nath Pandey Fix For: 0.13.0 Attachments: HIVE-5759.1.patch, HIVE-5759.2.patch Implement full, end-to-end support for COALESCE in vectorized mode, including new VectorExpression class(es), VectorizationContext translation to a VectorExpression, and unit tests for these, as well as end-to-end ad hoc testing. An end-to-end .q test is recommended. This is lower priority than IF and CASE but it is still a fairly popular expression. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5784) Group By Operator doesn't carry forward table aliases in its RowResolver
[
https://issues.apache.org/jira/browse/HIVE-5784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-5784:
---
Fix Version/s: 0.13.0
Group By Operator doesn't carry forward table aliases in its RowResolver
Key: HIVE-5784
URL: https://issues.apache.org/jira/browse/HIVE-5784
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Harish Butani
Assignee: Harish Butani
Fix For: 0.13.0
Attachments: HIVE-5784.1.patch
The following queries fails:
{code}
select b.key, count(*) from src b group by key
select key, count(*) from src b group by b.key
{code}
with a SemanticException; the select expression b.key (key in the 2nd query)
are not resolved by the GBy RowResolver.
This is because the GBy RowResolver only supports resolving based on an
AST.toStringTree match. Underlying issue is that a RowResolver doesn't allow
multiple mappings to the same ColumnInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-5777) PTest2 IllegalArgumentException when creating hosts
[
https://issues.apache.org/jira/browse/HIVE-5777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-5777:
---
Fix Version/s: 0.13.0
PTest2 IllegalArgumentException when creating hosts
---
Key: HIVE-5777
URL: https://issues.apache.org/jira/browse/HIVE-5777
Project: Hive
Issue Type: Bug
Reporter: Brock Noland
Assignee: Brock Noland
Fix For: 0.13.0
{noformat}
2013-11-07 16:17:00,918 ERROR PTest.run:173 Test run exited with an
unexpected error java.lang.IllegalArgumentException
at
java.util.concurrent.ThreadPoolExecutor.init(ThreadPoolExecutor.java:589)
at
java.util.concurrent.ThreadPoolExecutor.init(ThreadPoolExecutor.java:480)
at java.util.concurrent.Executors.newFixedThreadPool(Executors.java:59)
at
org.apache.hive.ptest.execution.context.CloudExecutionContextProvider.verifyHosts(CloudExecutionContextProvider.java:245)
at
org.apache.hive.ptest.execution.context.CloudExecutionContextProvider.createNodes(CloudExecutionContextProvider.java:208)
at
org.apache.hive.ptest.execution.context.CloudExecutionContextProvider.replaceBadHosts(CloudExecutionContextProvider.java:167)
at
org.apache.hive.ptest.execution.context.ExecutionContext.replaceBadHosts(ExecutionContext.java:68)
at
org.apache.hive.ptest.execution.ExecutionPhase.replaceBadHosts(ExecutionPhase.java:138)
at
org.apache.hive.ptest.execution.ExecutionPhase.execute(ExecutionPhase.java:93)
at org.apache.hive.ptest.execution.PTest.run(PTest.java:162)
at
org.apache.hive.ptest.api.server.TestExecutor.run(TestExecutor.java:118)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-5750) Queue name should not be hardcoded to 'default' when running Tez jobs
[ https://issues.apache.org/jira/browse/HIVE-5750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5750: --- Fix Version/s: 0.13.0 Queue name should not be hardcoded to 'default' when running Tez jobs - Key: HIVE-5750 URL: https://issues.apache.org/jira/browse/HIVE-5750 Project: Hive Issue Type: Improvement Affects Versions: tez-branch Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: 0.13.0 Attachments: HIVE-5750.txt -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5739) Cleanup transitive dependencies
[
https://issues.apache.org/jira/browse/HIVE-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-5739:
---
Fix Version/s: 0.13.0
Cleanup transitive dependencies
---
Key: HIVE-5739
URL: https://issues.apache.org/jira/browse/HIVE-5739
Project: Hive
Issue Type: Sub-task
Reporter: Brock Noland
Priority: Critical
Fix For: 0.13.0
As you can see below we have many duplicate depends from various
dependencies. We need to put the correct exclusions in place.
{noformat}
activation-1.1.jar
ant-1.6.5.jar
ant-1.9.1.jar
ant-launcher-1.9.1.jar
antlr-2.7.7.jar
antlr-runtime-3.4.jar
aopalliance-1.0.jar
asm-3.1.jar
asm-commons-3.1.jar
asm-tree-3.1.jar
avro-1.5.3.jar
avro-1.7.1.jar
avro-ipc-1.5.3.jar
avro-ipc-1.7.1.jar
avro-mapred-1.7.1.jar
bonecp-0.7.1.RELEASE.jar
commons-beanutils-1.7.0.jar
commons-beanutils-core-1.8.0.jar
commons-cli-1.2.jar
commons-codec-1.3.jar
commons-codec-1.4.jar
commons-collections-3.1.jar
commons-collections-3.2.1.jar
commons-compress-1.4.1.jar
commons-configuration-1.6.jar
commons-daemon-1.0.13.jar
commons-digester-1.8.jar
commons-el-1.0.jar
commons-exec-1.1.jar
commons-httpclient-3.0.1.jar
commons-httpclient-3.1.jar
commons-io-2.1.jar
commons-io-2.4.jar
commons-lang-2.4.jar
commons-lang-2.5.jar
commons-logging-1.0.4.jar
commons-math-2.1.jar
commons-net-1.4.1.jar
commons-net-2.0.jar
commons-net-3.1.jar
core-3.1.1.jar
datanucleus-api-jdo-3.2.1.jar
datanucleus-core-3.2.2.jar
datanucleus-rdbms-3.2.1.jar
derby-10.4.2.0.jar
ftplet-api-1.0.0.jar
ftpserver-core-1.0.0.jar
ftpserver-deprecated-1.0.0-M2.jar
geronimo-annotation_1.0_spec-1.1.1.jar
geronimo-jaspic_1.0_spec-1.0.jar
geronimo-jta_1.1_spec-1.1.1.jar
gmbal-api-only-3.0.0-b023.jar
grizzly-framework-2.1.1.jar
grizzly-framework-2.1.1-tests.jar
grizzly-http-2.1.1.jar
grizzly-http-server-2.1.1.jar
grizzly-http-servlet-2.1.1.jar
grizzly-rcm-2.1.1.jar
groovy-all-2.1.6.jar
guava-11.0.2.jar
guava-r08.jar
guice-3.0.jar
guice-servlet-3.0.jar
hamcrest-core-1.1.jar
hbase-0.94.6.1.jar
hbase-0.94.6.1-tests.jar
high-scale-lib-1.1.1.jar
hive-ant-0.13.0-SNAPSHOT.jar
hive-cli-0.13.0-SNAPSHOT.jar
hive-common-0.13.0-SNAPSHOT.jar
hive-exec-0.13.0-SNAPSHOT.jar
hive-hbase-handler-0.13.0-SNAPSHOT.jar
hive-hcatalog-core-0.13.0-SNAPSHOT.jar
hive-metastore-0.13.0-SNAPSHOT.jar
hive-serde-0.13.0-SNAPSHOT.jar
hive-service-0.13.0-SNAPSHOT.jar
hive-shims-0.13.0-SNAPSHOT-uberjar.jar
hive-shims-0.20-0.13.0-SNAPSHOT.jar
hive-shims-0.20S-0.13.0-SNAPSHOT.jar
hive-shims-0.23-0.13.0-SNAPSHOT.jar
hive-shims-common-0.13.0-SNAPSHOT.jar
hive-shims-common-secure-0.13.0-SNAPSHOT.jar
hsqldb-1.8.0.10.jar
httpclient-4.1.3.jar
httpcore-4.1.3.jar
jackson-core-asl-1.7.1.jar
jackson-core-asl-1.8.8.jar
jackson-core-asl-1.9.2.jar
jackson-jaxrs-1.7.1.jar
jackson-jaxrs-1.8.8.jar
jackson-jaxrs-1.9.2.jar
jackson-mapper-asl-1.8.8.jar
jackson-mapper-asl-1.9.2.jar
jackson-xc-1.7.1.jar
jackson-xc-1.8.8.jar
jackson-xc-1.9.2.jar
jamon-runtime-2.3.1.jar
jasper-compiler-5.5.12.jar
jasper-compiler-5.5.23.jar
jasper-runtime-5.5.12.jar
jasper-runtime-5.5.23.jar
JavaEWAH-0.3.2.jar
javax.inject-1.jar
javax.servlet-3.0.jar
javolution-5.5.1.jar
jaxb-api-2.1.jar
jaxb-api-2.2.2.jar
jaxb-impl-2.2.3-1.jar
jdk.tools-1.6.jar
jdo-api-3.0.1.jar
jersey-client-1.8.jar
jersey-core-1.14.jar
jersey-core-1.8.jar
jersey-grizzly2-1.8.jar
jersey-guice-1.8.jar
jersey-json-1.14.jar
jersey-json-1.8.jar
jersey-server-1.14.jar
jersey-server-1.8.jar
jersey-servlet-1.14.jar
jersey-test-framework-core-1.8.jar
jersey-test-framework-grizzly2-1.8.jar
jets3t-0.6.1.jar
jets3t-0.7.1.jar
jettison-1.1.jar
jetty-6.1.14.jar
jetty-6.1.26.jar
jetty-all-server-7.6.0.v20120127.jar
jetty-util-6.1.14.jar
jetty-util-6.1.26.jar
jline-0.9.94.jar
jms-1.1.jar
jmxri-1.2.1.jar
jmxtools-1.2.1.jar
jruby-complete-1.6.5.jar
jsch-0.1.42.jar
json-20090211.jar
jsp-2.1-6.1.14.jar
jsp-api-2.1-6.1.14.jar
jsp-api-2.1.jar
jsr305-1.3.9.jar
jta-1.1.jar
jul-to-slf4j-1.6.1.jar
junit-3.8.1.jar
junit-4.10.jar
junit-4.5.jar
junit-4.8.1.jar
kfs-0.3.jar
kryo-2.22.jar
libfb303-0.9.0.jar
libthrift-0.9.0.jar
log4j-1.2.15.jar
log4j-1.2.16.jar
log4j-1.2.17.jar
mail-1.4.1.jar
management-api-3.0.0-b012.jar
metrics-core-2.1.2.jar
mina-core-2.0.0-M5.jar
netty-3.2.2.Final.jar
netty-3.4.0.Final.jar
netty-3.5.11.Final.jar
oro-2.0.8.jar
paranamer-2.2.jar
paranamer-2.3.jar
paranamer-ant-2.2.jar
paranamer-generator-2.2.jar
pig-0.10.1.jar
protobuf-java-2.4.0a.jar
protobuf-java-2.5.0.jar
qdox-1.10.1.jar
servlet-api-2.5-20081211.jar
servlet-api-2.5-6.1.14.jar
servlet-api-2.5.jar
slf4j-api-1.6.1.jar
slf4j-log4j12-1.6.1.jar
snappy-0.2.jar
snappy-java-1.0.3.2.jar
snappy-java-1.0.4.1.jar
ST4-4.0.4.jar
[jira] [Updated] (HIVE-5703) While using tez, Qtest needs to close session before creating a new one
[ https://issues.apache.org/jira/browse/HIVE-5703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5703: --- Fix Version/s: 0.13.0 While using tez, Qtest needs to close session before creating a new one --- Key: HIVE-5703 URL: https://issues.apache.org/jira/browse/HIVE-5703 Project: Hive Issue Type: Bug Components: Testing Infrastructure, Tez Affects Versions: tez-branch Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.13.0 Attachments: HIVE-5703.1.patch, HIVE-5703.2.patch, HIVE-5703.3.patch While using the mini tez cluster, if we do not close the session, containers do not get freed up resulting in locking up of resources and hive times out. We need to ensure clean-up of session before new ones are launched in the Qtest framework. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5694) on hbase table, function's result like 'count' or 'sum' is multiplied by the number of map tasks.
[ https://issues.apache.org/jira/browse/HIVE-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5694: --- Fix Version/s: 0.13.0 on hbase table, function's result like 'count' or 'sum' is multiplied by the number of map tasks. - Key: HIVE-5694 URL: https://issues.apache.org/jira/browse/HIVE-5694 Project: Hive Issue Type: Bug Components: HBase Handler Affects Versions: 0.12.0 Reporter: Sam Chen Fix For: 0.13.0 Attachments: HiveHBaseTableInputFormat.java when calling getRecordReader() on HiveHBaseTableInputFormat, it calls convertFilter(), which leads to the scanner on each map being set to the full range of where's clause -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5673) Create profile to generate protobuf
[ https://issues.apache.org/jira/browse/HIVE-5673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5673: --- Fix Version/s: 0.13.0 Create profile to generate protobuf --- Key: HIVE-5673 URL: https://issues.apache.org/jira/browse/HIVE-5673 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Brock Noland Fix For: 0.13.0 Attachments: HIVE-5673.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5649) error when casting partition column to varchar in where clause
[ https://issues.apache.org/jira/browse/HIVE-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5649: --- Fix Version/s: 0.13.0 error when casting partition column to varchar in where clause -- Key: HIVE-5649 URL: https://issues.apache.org/jira/browse/HIVE-5649 Project: Hive Issue Type: Bug Reporter: Jason Dere Assignee: Jason Dere Fix For: 0.13.0 hive select * from partition_varchar_2 where cast(dt as varchar(10)) = '2000-01-01'; FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Internal error: Cannot find ObjectInspector for VARCHAR -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5646) Cannot compile tests on tez branch
[ https://issues.apache.org/jira/browse/HIVE-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5646: --- Fix Version/s: 0.13.0 Cannot compile tests on tez branch -- Key: HIVE-5646 URL: https://issues.apache.org/jira/browse/HIVE-5646 Project: Hive Issue Type: Bug Components: Tez Affects Versions: tez-branch Reporter: Vikram Dixit K Fix For: 0.13.0 Orc tests do not compile on the latest tez branch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5636) Introduce getPartitionColumns() functionality from HCatInputFormat
[ https://issues.apache.org/jira/browse/HIVE-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5636: --- Fix Version/s: 0.13.0 Introduce getPartitionColumns() functionality from HCatInputFormat -- Key: HIVE-5636 URL: https://issues.apache.org/jira/browse/HIVE-5636 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.12.0 Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Fix For: 0.13.0 Attachments: HIVE-5636.2.patch, HIVE-5636.patch As of HCat 0.5, we made the class InputJobInfo private for hcatalog use only, and we made it so that setInput would not modify the InputJobInfo being passed in. However, if a user of HCatInputFormat wants to get what Partitioning columns or Data columns exist for the job, they are not able to do so directly from HCatInputFormat and are forced to use InputJobInfo, which currently does not work. Thus, we need to expose this functionality. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5633) Perflogger broken due to HIVE-5403
[ https://issues.apache.org/jira/browse/HIVE-5633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5633: --- Fix Version/s: 0.13.0 Perflogger broken due to HIVE-5403 -- Key: HIVE-5633 URL: https://issues.apache.org/jira/browse/HIVE-5633 Project: Hive Issue Type: Bug Components: Logging Affects Versions: 0.13.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.13.0 Attachments: HIVE-5633.1.patch, HIVE-5633.2.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5624) Create script for removing ant artifacts after merge
[ https://issues.apache.org/jira/browse/HIVE-5624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5624: --- Fix Version/s: 0.13.0 Create script for removing ant artifacts after merge Key: HIVE-5624 URL: https://issues.apache.org/jira/browse/HIVE-5624 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Brock Noland Fix For: 0.13.0 Attachments: HIVE-5624.patch Before marking HIVE-5107 resolved we should remove the build.xml files and other ant artifacts. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5620) Tez job progress printing stops after a specific amount of time
[ https://issues.apache.org/jira/browse/HIVE-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5620: --- Fix Version/s: 0.13.0 Tez job progress printing stops after a specific amount of time --- Key: HIVE-5620 URL: https://issues.apache.org/jira/browse/HIVE-5620 Project: Hive Issue Type: Bug Affects Versions: tez-branch Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: 0.13.0 Attachments: HIVE-5620.2.patch, HIVE-5620.txt -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5616) fix saveVersion.sh to work on mac
[ https://issues.apache.org/jira/browse/HIVE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5616: --- Fix Version/s: 0.13.0 fix saveVersion.sh to work on mac - Key: HIVE-5616 URL: https://issues.apache.org/jira/browse/HIVE-5616 Project: Hive Issue Type: Sub-task Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.13.0 Attachments: h-5616.patch There is no reason to not support builds on macs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5612) Ability to compile odbc and re-generate generated code stored in source control
[ https://issues.apache.org/jira/browse/HIVE-5612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5612: --- Fix Version/s: 0.13.0 Ability to compile odbc and re-generate generated code stored in source control --- Key: HIVE-5612 URL: https://issues.apache.org/jira/browse/HIVE-5612 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Brock Noland Fix For: 0.13.0 Attachments: HIVE-5612.patch We need the ability to re-generate protocol buffers (and thrift?) via maven. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5563) Skip reading columns in ORC for count(*)
[ https://issues.apache.org/jira/browse/HIVE-5563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5563: --- Fix Version/s: 0.13.0 Skip reading columns in ORC for count(*) Key: HIVE-5563 URL: https://issues.apache.org/jira/browse/HIVE-5563 Project: Hive Issue Type: Bug Components: File Formats Reporter: Owen O'Malley Fix For: 0.13.0 With HIVE-4113, the semantics of ColumnProjectionUtils.getReadColumnIds was fixed so that an empty list means no columns instead of all columns. (Except the caveat of the override of ColumnProjectionUtils.isReadAllColumns.) However, ORC's reader wasn't updated so it still reads all columns. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5528) hive log file name in local is .log
[
https://issues.apache.org/jira/browse/HIVE-5528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-5528:
---
Fix Version/s: 0.13.0
hive log file name in local is .log
-
Key: HIVE-5528
URL: https://issues.apache.org/jira/browse/HIVE-5528
Project: Hive
Issue Type: Bug
Affects Versions: 0.11.0, 0.12.0
Reporter: Thejas M Nair
Fix For: 0.13.0
In local mode the log is getting written to /tmp/{user.name}/.log instead of
/tmp/{user.name}/hive.log
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-5539) remove hcatalog wiki tab from hive site
[ https://issues.apache.org/jira/browse/HIVE-5539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5539: --- Fix Version/s: 0.13.0 remove hcatalog wiki tab from hive site --- Key: HIVE-5539 URL: https://issues.apache.org/jira/browse/HIVE-5539 Project: Hive Issue Type: Bug Components: Website Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.13.0 Attachments: HIVE-5539.1.patch, HIVE-5539.generatedchanges.patch The hcatalog wiki tab points to the old hcatalog wiki, not the one hosted under hive wiki. Now that the documentation is part of hive wiki, we can just remove the tab. NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5543) Running the mini tez cluster for tez unit tests
[ https://issues.apache.org/jira/browse/HIVE-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5543: --- Fix Version/s: 0.13.0 Running the mini tez cluster for tez unit tests --- Key: HIVE-5543 URL: https://issues.apache.org/jira/browse/HIVE-5543 Project: Hive Issue Type: Bug Components: Tez Affects Versions: tez-branch Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.13.0 Attachments: HIVE-5543.1.patch, HIVE-5543.2.patch, HIVE-5543.3.patch In order to simulate the tez execution in hive tests, we need to work with MiniTezCluster. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5427) TestMetastoreVersion.testVersionRestriction fails on hive 0.12
[ https://issues.apache.org/jira/browse/HIVE-5427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5427: --- Fix Version/s: 0.12.0 TestMetastoreVersion.testVersionRestriction fails on hive 0.12 -- Key: HIVE-5427 URL: https://issues.apache.org/jira/browse/HIVE-5427 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.12.0 Reporter: Thejas M Nair Priority: Blocker Fix For: 0.12.0 TestMetastoreVersion.testVersionRestriction failed on hive 0.12 . See https://builds.apache.org/job/Hive-branch-0.12-hadoop1/lastCompletedBuild/testReport/org.apache.hadoop.hive.metastore/TestMetastoreVersion/testVersionRestriction/ It also failed in a test run on another machine I ran tests on. The error - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Hive Schema version 0.12.0 does not match metastore's schema version fooVersion Metastore is not upgraded or corrupt) It looks like the fooVersion set by one test is getting used by this failing test. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5404) Remove changes from HIVE-5184
[ https://issues.apache.org/jira/browse/HIVE-5404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5404: --- Fix Version/s: 0.13.0 Remove changes from HIVE-5184 - Key: HIVE-5404 URL: https://issues.apache.org/jira/browse/HIVE-5404 Project: Hive Issue Type: Bug Components: Tez Affects Versions: tez-branch Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.13.0 Attachments: HIVE-5404.1.patch This is a clean up work following HIVE-5403. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5420) HiveServer2 async thread timeout should be cast to long
[ https://issues.apache.org/jira/browse/HIVE-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5420: --- Fix Version/s: 0.13.0 HiveServer2 async thread timeout should be cast to long --- Key: HIVE-5420 URL: https://issues.apache.org/jira/browse/HIVE-5420 Project: Hive Issue Type: Bug Reporter: Vaibhav Gumashta Assignee: Vaibhav Gumashta Fix For: 0.13.0 SessionManager#stop throws an exception while getting hiveConf.getLongVar(ConfVars.HIVE_SERVER2_ASYNC_EXEC_SHUTDOWN_TIMEOUT) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5384) hcatloadstore.pig should be upgraded to org.apache.hive.hcatalog.pig.HCatLoader
[ https://issues.apache.org/jira/browse/HIVE-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5384: --- Fix Version/s: 0.13.0 hcatloadstore.pig should be upgraded to org.apache.hive.hcatalog.pig.HCatLoader --- Key: HIVE-5384 URL: https://issues.apache.org/jira/browse/HIVE-5384 Project: Hive Issue Type: Bug Components: WebHCat Affects Versions: 0.12.0 Reporter: Eugene Koifman Assignee: Eugene Koifman Fix For: 0.13.0 this was missed in HIVE-4869 actually we should probably clone the test and also keep a version that uses old package name until Hive 0.14 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5368) Changes to work creation for tez
[ https://issues.apache.org/jira/browse/HIVE-5368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5368: --- Fix Version/s: 0.13.0 Changes to work creation for tez Key: HIVE-5368 URL: https://issues.apache.org/jira/browse/HIVE-5368 Project: Hive Issue Type: Bug Components: Tez Affects Versions: tez-branch Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.13.0 Attachments: HIVE-5368.1.patch, HIVE-5368.3.patch, HIVE-5368.4.patch For map join to work in tez, we need changes in the work and task creation phase (as documented in HIVE-5270). -- This message was sent by Atlassian JIRA (v6.2#6252)
fix version
Committers, Please don't forget to update fix version of jiras when you are committing patches. All, Sorry about the deluge of emails updating fix version of jiras. That was me. Thanks, Ashutosh
[jira] [Commented] (HIVE-6711) ORC maps uses getMapSize() from MapOI which is unreliable
[
https://issues.apache.org/jira/browse/HIVE-6711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944518#comment-13944518
]
Hive QA commented on HIVE-6711:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12635836/HIVE-6711.1.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5440 tests executed
*Failed tests:*
{noformat}
org.apache.hive.service.cli.thrift.TestThriftBinaryCLIService.testExecuteStatementAsync
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1925/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1925/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12635836
ORC maps uses getMapSize() from MapOI which is unreliable
-
Key: HIVE-6711
URL: https://issues.apache.org/jira/browse/HIVE-6711
Project: Hive
Issue Type: Bug
Affects Versions: 0.11.0, 0.12.0, 0.13.0
Reporter: Prasanth J
Assignee: Prasanth J
Labels: orcfile
Fix For: 0.13.0, 0.14.0
Attachments: HIVE-6711.1.patch
HIVE-6707 had issues with map size. getMapSize() of LazyMap and LazyBinaryMap
does not deserialize the keys and count the number of unique keys. Since
getMapSize() may return non-distinct count of keys, the length of maps stored
using ORC's map tree writer will not be in sync with actual map size. As a
result of this RLE reader will try to read beyond the disk range expecting
more map entries and will throw exception.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6222) Make Vector Group By operator abandon grouping if too many distinct keys
[ https://issues.apache.org/jira/browse/HIVE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-6222: --- Fix Version/s: (was: 0.14.0) 0.13.0 Make Vector Group By operator abandon grouping if too many distinct keys Key: HIVE-6222 URL: https://issues.apache.org/jira/browse/HIVE-6222 Project: Hive Issue Type: Sub-task Components: Query Processor Affects Versions: 0.13.0 Reporter: Remus Rusanu Assignee: Remus Rusanu Priority: Minor Labels: vectorization Fix For: 0.13.0 Attachments: HIVE-6222.1.patch, HIVE-6222.2.patch, HIVE-6222.3.patch, HIVE-6222.4.patch, HIVE-6222.5.patch Row mode GBY is becoming a pass-through if not enough aggregation occurs on the map side, relying on the shuffle+reduce side to do the work. Have VGBY do the same. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6222) Make Vector Group By operator abandon grouping if too many distinct keys
[ https://issues.apache.org/jira/browse/HIVE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944520#comment-13944520 ] Jitendra Nath Pandey commented on HIVE-6222: I have committed this to branch-0.13 as well. Make Vector Group By operator abandon grouping if too many distinct keys Key: HIVE-6222 URL: https://issues.apache.org/jira/browse/HIVE-6222 Project: Hive Issue Type: Sub-task Components: Query Processor Affects Versions: 0.13.0 Reporter: Remus Rusanu Assignee: Remus Rusanu Priority: Minor Labels: vectorization Fix For: 0.13.0 Attachments: HIVE-6222.1.patch, HIVE-6222.2.patch, HIVE-6222.3.patch, HIVE-6222.4.patch, HIVE-6222.5.patch Row mode GBY is becoming a pass-through if not enough aggregation occurs on the map side, relying on the shuffle+reduce side to do the work. Have VGBY do the same. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6707) Lazy maps are broken (LazyMap and LazyBinaryMap)
[
https://issues.apache.org/jira/browse/HIVE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944559#comment-13944559
]
Hive QA commented on HIVE-6707:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12635736/HIVE-6707.1.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5443 tests executed
*Failed tests:*
{noformat}
org.apache.hive.service.cli.thrift.TestThriftBinaryCLIService.testExecuteStatementAsync
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1926/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1926/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12635736
Lazy maps are broken (LazyMap and LazyBinaryMap)
Key: HIVE-6707
URL: https://issues.apache.org/jira/browse/HIVE-6707
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.5.0, 0.6.0, 0.7.0, 0.8.0, 0.9.0, 0.10.0, 0.11.0,
0.12.0, 0.13.0
Reporter: Prasanth J
Assignee: Prasanth J
Priority: Critical
Labels: serde
Fix For: 0.13.0, 0.14.0
Attachments: HIVE-6707.1.patch
LazyPrimitive and LazyBinaryPrimitive overrides hashcode method in HIVE-949.
But it failed to override equals() method. As a result, LazyMap and
LazyBinaryMap will end up having multiple values for the same key. Both
LazyMap and LazyBinaryMap uses LinkedHashMap, so the expected behaviour is to
have a single value per unique key.
In the following code from LazyMap (LazyBinaryMap also has same code segment)
{code}
LazyPrimitive?, ? lazyKey = uncheckedGetKey(i);
if (lazyKey == null) {
continue;
}
Object key = lazyKey.getObject();
if (key != null !cachedMap.containsKey(key)) {
{code}
lazyKey.hashcode() returns the writable object's hashcode. The containsKeys()
method of hash map
(http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/HashMap.java#366)
checks if the hashcode are same, if so then it uses equals() method to
verify if the key already exists. Since LazyPrimitive does not override
equals() method it falls back to use Object equals(). Object equals() will
return true only if both object are exactly the same (this == obj).
So in the above code segment, even if the key already exists, the new value
will be inserted with hash collision resulting in more number of map entries.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: fix version
Thanks for doing all those updates, Ashutosh. -- Lefty On Sun, Mar 23, 2014 at 2:34 PM, Ashutosh Chauhan hashut...@apache.orgwrote: Committers, Please don't forget to update fix version of jiras when you are committing patches. All, Sorry about the deluge of emails updating fix version of jiras. That was me. Thanks, Ashutosh
[jira] [Updated] (HIVE-6349) Column name map is broken
[
https://issues.apache.org/jira/browse/HIVE-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HIVE-6349:
---
Attachment: HIVE-6349.1.patch
The attached patch makes VectorSelectOperator implement
VectorizedRegionContext. Therefore, VectorSelectOperator also gives out a
vectorization context with updated column map. However, VectorSelectOperator
doesn't create a new row batch, therefore it re-uses the same Output Column
Manager from its parent's vectorization context.This is important because then,
it doesn't have to allocate scratch columns.
Column name map is broken
--
Key: HIVE-6349
URL: https://issues.apache.org/jira/browse/HIVE-6349
Project: Hive
Issue Type: Sub-task
Reporter: Jitendra Nath Pandey
Assignee: Jitendra Nath Pandey
Attachments: HIVE-6349.1.patch
Following query results in exception at run time in vector mode.
{code}
explain select n_name from supplier_orc s join ( select n_name, n_nationkey
from nation_orc n join region_orc r on n.n_regionkey = r.r_regionkey and
r.r_name = 'XYZ') n1 on s.s_nationkey = n1.n_nationkey;
{code}
Here n_name is a string and all other fields are int.
The stack trace:
{code}
java.lang.RuntimeException: Hive Runtime Error while closing operators
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:260)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector cannot be cast to
org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
at
org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapperBatch.evaluateBatch(VectorHashKeyWrapperBatch.java:116)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.processOp(VectorMapJoinOperator.java:280)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at
org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.processOp(VectorSelectOperator.java:133)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.flushOutput(VectorMapJoinOperator.java:246)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.closeOp(VectorMapJoinOperator.java:253)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:574)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:585)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:234)
... 8 more
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6349) Column name map is broken
[
https://issues.apache.org/jira/browse/HIVE-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HIVE-6349:
---
Status: Patch Available (was: Open)
Column name map is broken
--
Key: HIVE-6349
URL: https://issues.apache.org/jira/browse/HIVE-6349
Project: Hive
Issue Type: Sub-task
Reporter: Jitendra Nath Pandey
Assignee: Jitendra Nath Pandey
Attachments: HIVE-6349.1.patch
Following query results in exception at run time in vector mode.
{code}
explain select n_name from supplier_orc s join ( select n_name, n_nationkey
from nation_orc n join region_orc r on n.n_regionkey = r.r_regionkey and
r.r_name = 'XYZ') n1 on s.s_nationkey = n1.n_nationkey;
{code}
Here n_name is a string and all other fields are int.
The stack trace:
{code}
java.lang.RuntimeException: Hive Runtime Error while closing operators
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:260)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector cannot be cast to
org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
at
org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapperBatch.evaluateBatch(VectorHashKeyWrapperBatch.java:116)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.processOp(VectorMapJoinOperator.java:280)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at
org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.processOp(VectorSelectOperator.java:133)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.flushOutput(VectorMapJoinOperator.java:246)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.closeOp(VectorMapJoinOperator.java:253)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:574)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:585)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:234)
... 8 more
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6716) ORC struct throws NPE for tables with inner structs having null values
[
https://issues.apache.org/jira/browse/HIVE-6716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944602#comment-13944602
]
Hive QA commented on HIVE-6716:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12635937/HIVE-6716.1.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 5442 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hive.service.cli.thrift.TestThriftBinaryCLIService.testExecuteStatementAsync
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1930/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1930/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12635937
ORC struct throws NPE for tables with inner structs having null values
---
Key: HIVE-6716
URL: https://issues.apache.org/jira/browse/HIVE-6716
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0, 0.14.0
Reporter: Prasanth J
Assignee: Prasanth J
Labels: orcfile
Attachments: HIVE-6716.1.patch
ORCStruct should return null when object passed to
getStructFieldsDataAsList(Object obj) is null.
{code}
public ListObject getStructFieldsDataAsList(Object object) {
OrcStruct struct = (OrcStruct) object;
ListObject result = new ArrayListObject(struct.fields.length);
{code}
In the above code struct.fields will throw NPE if struct is NULL.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6711) ORC maps uses getMapSize() from MapOI which is unreliable
[ https://issues.apache.org/jira/browse/HIVE-6711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944609#comment-13944609 ] Prasanth J commented on HIVE-6711: -- Test failure is not related. ORC maps uses getMapSize() from MapOI which is unreliable - Key: HIVE-6711 URL: https://issues.apache.org/jira/browse/HIVE-6711 Project: Hive Issue Type: Bug Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Prasanth J Assignee: Prasanth J Labels: orcfile Fix For: 0.13.0, 0.14.0 Attachments: HIVE-6711.1.patch HIVE-6707 had issues with map size. getMapSize() of LazyMap and LazyBinaryMap does not deserialize the keys and count the number of unique keys. Since getMapSize() may return non-distinct count of keys, the length of maps stored using ORC's map tree writer will not be in sync with actual map size. As a result of this RLE reader will try to read beyond the disk range expecting more map entries and will throw exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6707) Lazy maps are broken (LazyMap and LazyBinaryMap)
[
https://issues.apache.org/jira/browse/HIVE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944607#comment-13944607
]
Prasanth J commented on HIVE-6707:
--
Test failure is not related.
Lazy maps are broken (LazyMap and LazyBinaryMap)
Key: HIVE-6707
URL: https://issues.apache.org/jira/browse/HIVE-6707
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.5.0, 0.6.0, 0.7.0, 0.8.0, 0.9.0, 0.10.0, 0.11.0,
0.12.0, 0.13.0
Reporter: Prasanth J
Assignee: Prasanth J
Priority: Critical
Labels: serde
Fix For: 0.13.0, 0.14.0
Attachments: HIVE-6707.1.patch
LazyPrimitive and LazyBinaryPrimitive overrides hashcode method in HIVE-949.
But it failed to override equals() method. As a result, LazyMap and
LazyBinaryMap will end up having multiple values for the same key. Both
LazyMap and LazyBinaryMap uses LinkedHashMap, so the expected behaviour is to
have a single value per unique key.
In the following code from LazyMap (LazyBinaryMap also has same code segment)
{code}
LazyPrimitive?, ? lazyKey = uncheckedGetKey(i);
if (lazyKey == null) {
continue;
}
Object key = lazyKey.getObject();
if (key != null !cachedMap.containsKey(key)) {
{code}
lazyKey.hashcode() returns the writable object's hashcode. The containsKeys()
method of hash map
(http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/HashMap.java#366)
checks if the hashcode are same, if so then it uses equals() method to
verify if the key already exists. Since LazyPrimitive does not override
equals() method it falls back to use Object equals(). Object equals() will
return true only if both object are exactly the same (this == obj).
So in the above code segment, even if the key already exists, the new value
will be inserted with hash collision resulting in more number of map entries.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6716) ORC struct throws NPE for tables with inner structs having null values
[
https://issues.apache.org/jira/browse/HIVE-6716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944610#comment-13944610
]
Prasanth J commented on HIVE-6716:
--
Test failures are not related.
ORC struct throws NPE for tables with inner structs having null values
---
Key: HIVE-6716
URL: https://issues.apache.org/jira/browse/HIVE-6716
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0, 0.14.0
Reporter: Prasanth J
Assignee: Prasanth J
Labels: orcfile
Attachments: HIVE-6716.1.patch
ORCStruct should return null when object passed to
getStructFieldsDataAsList(Object obj) is null.
{code}
public ListObject getStructFieldsDataAsList(Object object) {
OrcStruct struct = (OrcStruct) object;
ListObject result = new ArrayListObject(struct.fields.length);
{code}
In the above code struct.fields will throw NPE if struct is NULL.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6331) HIVE-5279 deprecated UDAF class without explanation/documentation/alternative
[
https://issues.apache.org/jira/browse/HIVE-6331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944641#comment-13944641
]
Hive QA commented on HIVE-6331:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12635984/HIVE-6331.3.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 5442 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1932/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1932/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12635984
HIVE-5279 deprecated UDAF class without explanation/documentation/alternative
-
Key: HIVE-6331
URL: https://issues.apache.org/jira/browse/HIVE-6331
Project: Hive
Issue Type: Bug
Reporter: Lars Francke
Assignee: Lars Francke
Priority: Minor
Attachments: HIVE-5279.1.patch, HIVE-6331.2.patch, HIVE-6331.3.patch
HIVE-5279 added a @Deprecated annotation to the {{UDAF}} class. The comment
in that class says {quote}UDAF classes are REQUIRED to inherit from this
class.{quote}
One of these two needs to be updated. Either remove the annotation or
document why it was deprecated and what to use instead.
Unfortunately [~navis] did not leave any documentation about his intentions.
I'm happy to provide a patch once I know the intentions.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6707) Lazy maps are broken (LazyMap and LazyBinaryMap)
[
https://issues.apache.org/jira/browse/HIVE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-6707:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks Prasanth!
Lazy maps are broken (LazyMap and LazyBinaryMap)
Key: HIVE-6707
URL: https://issues.apache.org/jira/browse/HIVE-6707
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.5.0, 0.6.0, 0.7.0, 0.8.0, 0.9.0, 0.10.0, 0.11.0,
0.12.0, 0.13.0
Reporter: Prasanth J
Assignee: Prasanth J
Priority: Critical
Labels: serde
Fix For: 0.13.0, 0.14.0
Attachments: HIVE-6707.1.patch
LazyPrimitive and LazyBinaryPrimitive overrides hashcode method in HIVE-949.
But it failed to override equals() method. As a result, LazyMap and
LazyBinaryMap will end up having multiple values for the same key. Both
LazyMap and LazyBinaryMap uses LinkedHashMap, so the expected behaviour is to
have a single value per unique key.
In the following code from LazyMap (LazyBinaryMap also has same code segment)
{code}
LazyPrimitive?, ? lazyKey = uncheckedGetKey(i);
if (lazyKey == null) {
continue;
}
Object key = lazyKey.getObject();
if (key != null !cachedMap.containsKey(key)) {
{code}
lazyKey.hashcode() returns the writable object's hashcode. The containsKeys()
method of hash map
(http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/HashMap.java#366)
checks if the hashcode are same, if so then it uses equals() method to
verify if the key already exists. Since LazyPrimitive does not override
equals() method it falls back to use Object equals(). Object equals() will
return true only if both object are exactly the same (this == obj).
So in the above code segment, even if the key already exists, the new value
will be inserted with hash collision resulting in more number of map entries.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6711) ORC maps uses getMapSize() from MapOI which is unreliable
[ https://issues.apache.org/jira/browse/HIVE-6711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-6711: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk. Thanks Prasanth! ORC maps uses getMapSize() from MapOI which is unreliable - Key: HIVE-6711 URL: https://issues.apache.org/jira/browse/HIVE-6711 Project: Hive Issue Type: Bug Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Prasanth J Assignee: Prasanth J Labels: orcfile Fix For: 0.13.0, 0.14.0 Attachments: HIVE-6711.1.patch HIVE-6707 had issues with map size. getMapSize() of LazyMap and LazyBinaryMap does not deserialize the keys and count the number of unique keys. Since getMapSize() may return non-distinct count of keys, the length of maps stored using ORC's map tree writer will not be in sync with actual map size. As a result of this RLE reader will try to read beyond the disk range expecting more map entries and will throw exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5652) Improve JavaDoc of UDF class
[
https://issues.apache.org/jira/browse/HIVE-5652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944683#comment-13944683
]
Hive QA commented on HIVE-5652:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12636192/HIVE-5652.4.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5442 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1933/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1933/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12636192
Improve JavaDoc of UDF class
Key: HIVE-5652
URL: https://issues.apache.org/jira/browse/HIVE-5652
Project: Hive
Issue Type: Improvement
Components: Documentation
Reporter: Lars Francke
Assignee: Lars Francke
Priority: Trivial
Attachments: HIVE-5652.1.patch, HIVE-5652.2.patch, HIVE-5652.3.patch,
HIVE-5652.4.patch
I think the JavaDoc for the UDF class can be improved. I'll attach a patch
shortly.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6714) Fix getMapSize() of LazyMap
[ https://issues.apache.org/jira/browse/HIVE-6714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-6714: - Status: Patch Available (was: Open) Fix getMapSize() of LazyMap --- Key: HIVE-6714 URL: https://issues.apache.org/jira/browse/HIVE-6714 Project: Hive Issue Type: Sub-task Components: Serializers/Deserializers Affects Versions: 0.13.0, 0.14.0 Reporter: Prasanth J Assignee: Prasanth J Fix For: 0.13.0, 0.14.0 Attachments: HIVE-6714.1.patch, HIVE-6714.2.patch HIVE-6707 fixed an issue with lazy maps. Lazy maps scans for keys using key separators and maintains a counter for the number of keys i.e; map size. However keys may include duplicates in which returning the number of keys for map size will be wrong as after deserialization the actual map will not have duplicate keys. This is a subtask of HIVE-6707 to fix the getMapSize() api. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6708) ConstantVectorExpression should create copies of data objects rather than referencing them
[
https://issues.apache.org/jira/browse/HIVE-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944707#comment-13944707
]
Hive QA commented on HIVE-6708:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12636090/HIVE-6708.2.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 5444 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_expressions
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
org.apache.hadoop.hive.ql.exec.vector.TestVectorizationContext.testIfConditionalExprs
org.apache.hive.service.cli.thrift.TestThriftBinaryCLIService.testExecuteStatementAsync
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1936/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1936/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12636090
ConstantVectorExpression should create copies of data objects rather than
referencing them
--
Key: HIVE-6708
URL: https://issues.apache.org/jira/browse/HIVE-6708
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Attachments: HIVE-6708-1.patch, HIVE-6708.2.patch
1. ConstantVectorExpression vector should be updated for bytecolumnvectors
and decimalColumnVectors. The current code changes the reference to the
vector which might be shared across multiple columns
2. VectorizationContext.foldConstantsForUnaryExpression(ExprNodeDesc
exprDesc) has a minor bug as to when to constant fold the expression.
The following code should replace the corresponding piece of code in the
trunk.
..
GenericUDF gudf = ((ExprNodeGenericFuncDesc) exprDesc).getGenericUDF();
if (gudf instanceof GenericUDFOPNegative || gudf instanceof
GenericUDFOPPositive
|| castExpressionUdfs.contains(gudf.getClass())
...
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6543) TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync is failing sometimes
[ https://issues.apache.org/jira/browse/HIVE-6543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944712#comment-13944712 ] Vaibhav Gumashta commented on HIVE-6543: [~thejas] It seems this got overriden by HIVE-6580. Should we reopen this or create a new jira? TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync is failing sometimes - Key: HIVE-6543 URL: https://issues.apache.org/jira/browse/HIVE-6543 Project: Hive Issue Type: Task Components: Tests Reporter: Navis Assignee: Navis Priority: Trivial Fix For: 0.14.0 Attachments: HIVE-6543.1.patch.txt NO PRECOMMIT TESTS The test uses CREATE TABLE NON_EXISTING_TAB (ID STRING) location 'hdfs://localhost:1/a/b/c' query for intended fail but it seemed not fail so quickly in testbed. Just making the query worse (replacing hdfs to invalid, etc.) would be enough. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6707) Lazy maps are broken (LazyMap and LazyBinaryMap)
[
https://issues.apache.org/jira/browse/HIVE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944725#comment-13944725
]
Harish Butani commented on HIVE-6707:
-
+1 for 0.13
Lazy maps are broken (LazyMap and LazyBinaryMap)
Key: HIVE-6707
URL: https://issues.apache.org/jira/browse/HIVE-6707
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.5.0, 0.6.0, 0.7.0, 0.8.0, 0.9.0, 0.10.0, 0.11.0,
0.12.0, 0.13.0
Reporter: Prasanth J
Assignee: Prasanth J
Priority: Critical
Labels: serde
Fix For: 0.13.0, 0.14.0
Attachments: HIVE-6707.1.patch
LazyPrimitive and LazyBinaryPrimitive overrides hashcode method in HIVE-949.
But it failed to override equals() method. As a result, LazyMap and
LazyBinaryMap will end up having multiple values for the same key. Both
LazyMap and LazyBinaryMap uses LinkedHashMap, so the expected behaviour is to
have a single value per unique key.
In the following code from LazyMap (LazyBinaryMap also has same code segment)
{code}
LazyPrimitive?, ? lazyKey = uncheckedGetKey(i);
if (lazyKey == null) {
continue;
}
Object key = lazyKey.getObject();
if (key != null !cachedMap.containsKey(key)) {
{code}
lazyKey.hashcode() returns the writable object's hashcode. The containsKeys()
method of hash map
(http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/HashMap.java#366)
checks if the hashcode are same, if so then it uses equals() method to
verify if the key already exists. Since LazyPrimitive does not override
equals() method it falls back to use Object equals(). Object equals() will
return true only if both object are exactly the same (this == obj).
So in the above code segment, even if the key already exists, the new value
will be inserted with hash collision resulting in more number of map entries.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
