[jira] [Updated] (HIVE-6578) Use ORC file footer statistics through StatsProvidingRecordReader interface for analyze command
[ https://issues.apache.org/jira/browse/HIVE-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-6578: - Attachment: HIVE-6578.4.patch.txt reuploading for jenkins Use ORC file footer statistics through StatsProvidingRecordReader interface for analyze command --- Key: HIVE-6578 URL: https://issues.apache.org/jira/browse/HIVE-6578 Project: Hive Issue Type: New Feature Affects Versions: 0.13.0 Reporter: Prasanth J Assignee: Prasanth J Labels: orcfile Attachments: HIVE-6578.1.patch, HIVE-6578.2.patch, HIVE-6578.3.patch, HIVE-6578.4.patch, HIVE-6578.4.patch.txt ORC provides file level statistics which can be used in analyze partialscan and noscan cases to compute basic statistics like number of rows, number of files, total file size and raw data size. On the writer side, a new interface was added earlier (StatsProvidingRecordWriter) that exposed stats when writing a table. Similarly, a new interface StatsProvidingRecordReader can be added which when implemented should provide stats that are gathered by the underlying file format. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6455) Scalable dynamic partitioning and bucketing optimization
[ https://issues.apache.org/jira/browse/HIVE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-6455: - Attachment: HIVE-6455.17.patch.txt Reuploading for jenkins Scalable dynamic partitioning and bucketing optimization Key: HIVE-6455 URL: https://issues.apache.org/jira/browse/HIVE-6455 Project: Hive Issue Type: New Feature Components: Query Processor Affects Versions: 0.13.0 Reporter: Prasanth J Assignee: Prasanth J Labels: optimization Attachments: HIVE-6455.1.patch, HIVE-6455.1.patch, HIVE-6455.10.patch, HIVE-6455.10.patch, HIVE-6455.11.patch, HIVE-6455.12.patch, HIVE-6455.13.patch, HIVE-6455.13.patch, HIVE-6455.14.patch, HIVE-6455.15.patch, HIVE-6455.16.patch, HIVE-6455.17.patch, HIVE-6455.17.patch.txt, HIVE-6455.2.patch, HIVE-6455.3.patch, HIVE-6455.4.patch, HIVE-6455.4.patch, HIVE-6455.5.patch, HIVE-6455.6.patch, HIVE-6455.7.patch, HIVE-6455.8.patch, HIVE-6455.9.patch, HIVE-6455.9.patch The current implementation of dynamic partition works by keeping at least one record writer open per dynamic partition directory. In case of bucketing there can be multispray file writers which further adds up to the number of open record writers. The record writers of column oriented file format (like ORC, RCFile etc.) keeps some sort of in-memory buffers (value buffer or compression buffers) open all the time to buffer up the rows and compress them before flushing it to disk. Since these buffers are maintained per column basis the amount of constant memory that will required at runtime increases as the number of partitions and number of columns per partition increases. This often leads to OutOfMemory (OOM) exception in mappers or reducers depending on the number of open record writers. Users often tune the JVM heapsize (runtime memory) to get over such OOM issues. With this optimization, the dynamic partition columns and bucketing columns (in case of bucketed tables) are sorted before being fed to the reducers. Since the partitioning and bucketing columns are sorted, each reducers can keep only one record writer open at any time thereby reducing the memory pressure on the reducers. This optimization is highly scalable as the number of partition and number of columns per partition increases at the cost of sorting the columns. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6656) Bug in ORC Timestamp reader returns wrong nanoseconds
[ https://issues.apache.org/jira/browse/HIVE-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-6656: - Attachment: HIVE-6656.1.patch.txt reuploading for jenkins Bug in ORC Timestamp reader returns wrong nanoseconds - Key: HIVE-6656 URL: https://issues.apache.org/jira/browse/HIVE-6656 Project: Hive Issue Type: Bug Affects Versions: 0.13.0, 0.14.0 Reporter: Prasanth J Assignee: Prasanth J Labels: orcfile Attachments: HIVE-6656.1.patch, HIVE-6656.1.patch.txt ORC timestamp writer stores the number of trailing zeros in 3 LSB bits. There is a bug in parsing nanosecond logic that returns incorrect value. Input: 1999-01-01 00:00:00.9 Output: 1999-01-01 00:00:00.463129087 The fix for this is parseNanos() should first right shift by 3 and then typecast to int. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6677) HBaseSerDe needs to be refactored
[
https://issues.apache.org/jira/browse/HIVE-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13936456#comment-13936456
]
Hive QA commented on HIVE-6677:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12634948/HIVE-6677.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 5406 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
org.apache.hadoop.hive.hbase.TestHBaseSerDe.testHBaseSerDeCompositeKeyWithSeparator
org.apache.hadoop.hive.hbase.TestHBaseSerDe.testHBaseSerDeCompositeKeyWithoutSeparator
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1847/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1847/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12634948
HBaseSerDe needs to be refactored
-
Key: HIVE-6677
URL: https://issues.apache.org/jira/browse/HIVE-6677
Project: Hive
Issue Type: Improvement
Affects Versions: 0.10.0, 0.11.0, 0.12.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Attachments: HIVE-6677.1.patch, HIVE-6677.patch
The code in HBaseSerde seems very complex and hard to be extend to support
new features such as adding generic compound key (HIVE-6411) and Compound key
filter (HIVE-6290), especially when handling key/field serialization. Hope
this task will clean up the code a bit and make it ready for new extensions.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-5959) SQL std auth - bootstrap SUPERUSER roles
[ https://issues.apache.org/jira/browse/HIVE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13936461#comment-13936461 ] Lefty Leverenz commented on HIVE-5959: -- Added a subsection to Authentication/Authorization in Configuration Properties, and documented *hive.users.in.admin.role*: * [Configuration Properties: SQL Standard Based Authorization |https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-SQLStandardBasedAuthorization] But I didn't find any other config parameters for SQL standard authorization. I'll keep the separate section for now, but if no new parameters are added then it might get merged into Hive Client Security later on. SQL std auth - bootstrap SUPERUSER roles Key: HIVE-5959 URL: https://issues.apache.org/jira/browse/HIVE-5959 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Assignee: Ashutosh Chauhan Fix For: 0.13.0 Attachments: HIVE-5959.2.patch, HIVE-5959.3.patch, HIVE-5959.5.patch, HIVE-5959.6.patch, HIVE-5959.7.patch, HIVE-5959.patch Original Estimate: 72h Time Spent: 24h Remaining Estimate: 0h SUPERUSER and PUBLIC are two roles that are always present, these need to be added automatically on metastore startup. This would be similar to creation of the default database from HMSHandler.init(). A config param in hive-site.xml will be used to specify the list of users who belong to the SUPERUSER role. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-4764) Support Kerberos HTTP authentication for HiveServer2 running in http mode
[ https://issues.apache.org/jira/browse/HIVE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-4764: --- Status: Open (was: Patch Available) Support Kerberos HTTP authentication for HiveServer2 running in http mode - Key: HIVE-4764 URL: https://issues.apache.org/jira/browse/HIVE-4764 Project: Hive Issue Type: Sub-task Components: HiveServer2 Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Vaibhav Gumashta Fix For: 0.13.0 Attachments: HIVE-4764.1.patch, HIVE-4764.2.patch, HIVE-4764.3.patch, HIVE-4764.4.patch, HIVE-4764.5.patch Support Kerberos authentication for HiveServer2 running in http mode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-4764) Support Kerberos HTTP authentication for HiveServer2 running in http mode
[ https://issues.apache.org/jira/browse/HIVE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-4764: --- Attachment: (was: HIVE-4764.6.patch) Support Kerberos HTTP authentication for HiveServer2 running in http mode - Key: HIVE-4764 URL: https://issues.apache.org/jira/browse/HIVE-4764 Project: Hive Issue Type: Sub-task Components: HiveServer2 Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Vaibhav Gumashta Fix For: 0.13.0 Attachments: HIVE-4764.1.patch, HIVE-4764.2.patch, HIVE-4764.3.patch, HIVE-4764.4.patch, HIVE-4764.5.patch Support Kerberos authentication for HiveServer2 running in http mode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-4764) Support Kerberos HTTP authentication for HiveServer2 running in http mode
[ https://issues.apache.org/jira/browse/HIVE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-4764: --- Status: Patch Available (was: Open) Support Kerberos HTTP authentication for HiveServer2 running in http mode - Key: HIVE-4764 URL: https://issues.apache.org/jira/browse/HIVE-4764 Project: Hive Issue Type: Sub-task Components: HiveServer2 Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Vaibhav Gumashta Fix For: 0.13.0 Attachments: HIVE-4764.1.patch, HIVE-4764.2.patch, HIVE-4764.3.patch, HIVE-4764.4.patch, HIVE-4764.5.patch, HIVE-4764.6.patch Support Kerberos authentication for HiveServer2 running in http mode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-4764) Support Kerberos HTTP authentication for HiveServer2 running in http mode
[ https://issues.apache.org/jira/browse/HIVE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-4764: --- Attachment: HIVE-4764.6.patch Support Kerberos HTTP authentication for HiveServer2 running in http mode - Key: HIVE-4764 URL: https://issues.apache.org/jira/browse/HIVE-4764 Project: Hive Issue Type: Sub-task Components: HiveServer2 Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Vaibhav Gumashta Fix For: 0.13.0 Attachments: HIVE-4764.1.patch, HIVE-4764.2.patch, HIVE-4764.3.patch, HIVE-4764.4.patch, HIVE-4764.5.patch, HIVE-4764.6.patch Support Kerberos authentication for HiveServer2 running in http mode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6432) Remove deprecated methods in HCatalog
[ https://issues.apache.org/jira/browse/HIVE-6432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-6432: --- Attachment: 6432-addendum.patch 6432-full.patch Good catch, Yin Szehon, yes, my mvn repo still had an older copy of the hbase storage handler in it when I tested the packaging. I've fixed that now. I'm attaching up a few more files for this patch. 6432-addendum.patch : A patch that applies on top of current trunk (git hash 73ca6021628cfb7ae3eaa13122e28f1980756da1 ) to fix this issue. 6432-full.patch : A patch that applies if the current 6432 commits on trunk are rolled back (I have not named these in a manner that allows the precommit tests to automatically pick them up yet, because I'm not certain which state to apply them to, since Xuefu says he's reverted the commit, but I currently see this patch still committed in. I'll edit/re-upload as necessary to fit what's needed.) Remove deprecated methods in HCatalog - Key: HIVE-6432 URL: https://issues.apache.org/jira/browse/HIVE-6432 Project: Hive Issue Type: Task Components: HCatalog Affects Versions: 0.14.0 Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Fix For: 0.14.0 Attachments: 6432-addendum.patch, 6432-full.patch, HIVE-6432.patch, HIVE-6432.wip.1.patch, HIVE-6432.wip.2.patch, hcat.6432.test.out There are a lot of methods in HCatalog that have been deprecated in HCatalog 0.5, and some that were recently deprecated in Hive 0.11 (joint release with HCatalog). The goal for HCatalog deprecation is that in general, after something has been deprecated, it is expected to stay around for 2 releases, which means hive-0.13 will be the last release to ship with all the methods that were deprecated in hive-0.11 (the org.apache.hcatalog.* files should all be removed afterwards), and it is also good for us to clean out and nuke all other older deprecated methods. We should take this on early in a dev/release cycle to allow us time to resolve all fallout, so I propose that we remove all HCatalog deprecated methods after we branch out 0.13 and 0.14 becomes trunk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6668) When auto join convert is on and noconditionaltask is off, ConditionalResolverCommonJoin fails to resolve map joins.
[
https://issues.apache.org/jira/browse/HIVE-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-6668:
Attachment: HIVE-6668.2.patch.txt
When auto join convert is on and noconditionaltask is off,
ConditionalResolverCommonJoin fails to resolve map joins.
Key: HIVE-6668
URL: https://issues.apache.org/jira/browse/HIVE-6668
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0, 0.14.0
Reporter: Yin Huai
Assignee: Navis
Priority: Blocker
Fix For: 0.13.0
Attachments: HIVE-6668.1.patch.txt, HIVE-6668.2.patch.txt
I tried the following query today ...
{code:sql}
set mapred.job.map.memory.mb=2048;
set mapred.job.reduce.memory.mb=2048;
set mapred.map.child.java.opts=-server -Xmx3072m
-Djava.net.preferIPv4Stack=true;
set mapred.reduce.child.java.opts=-server -Xmx3072m
-Djava.net.preferIPv4Stack=true;
set mapred.reduce.tasks=60;
set hive.stats.autogather=false;
set hive.exec.parallel=false;
set hive.enforce.bucketing=true;
set hive.enforce.sorting=true;
set hive.map.aggr=true;
set hive.optimize.bucketmapjoin=true;
set hive.optimize.bucketmapjoin.sortedmerge=true;
set hive.mapred.reduce.tasks.speculative.execution=false;
set hive.auto.convert.join=true;
set hive.auto.convert.sortmerge.join=true;
set hive.auto.convert.sortmerge.join.noconditionaltask=false;
set hive.auto.convert.join.noconditionaltask=false;
set hive.auto.convert.join.noconditionaltask.size=1;
set hive.optimize.reducededuplication=true;
set hive.optimize.reducededuplication.min.reducer=1;
set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;
set hive.mapjoin.smalltable.filesize=4500;
set hive.optimize.index.filter=false;
set hive.vectorized.execution.enabled=false;
set hive.optimize.correlation=false;
select
i_item_id,
s_state,
avg(ss_quantity) agg1,
avg(ss_list_price) agg2,
avg(ss_coupon_amt) agg3,
avg(ss_sales_price) agg4
FROM store_sales
JOIN date_dim on (store_sales.ss_sold_date_sk = date_dim.d_date_sk)
JOIN item on (store_sales.ss_item_sk = item.i_item_sk)
JOIN customer_demographics on (store_sales.ss_cdemo_sk =
customer_demographics.cd_demo_sk)
JOIN store on (store_sales.ss_store_sk = store.s_store_sk)
where
cd_gender = 'F' and
cd_marital_status = 'U' and
cd_education_status = 'Primary' and
d_year = 2002 and
s_state in ('GA','PA', 'LA', 'SC', 'MI', 'AL')
group by i_item_id, s_state with rollup
order by
i_item_id,
s_state
limit 100;
{code}
The log shows ...
{code}
14/03/14 17:05:02 INFO plan.ConditionalResolverCommonJoin: Failed to resolve
driver alias (threshold : 4500, length mapping : {store=94175,
store_sales=48713909726, item=39798667, customer_demographics=1660831,
date_dim=2275902})
Stage-27 is filtered out by condition resolver.
14/03/14 17:05:02 INFO exec.Task: Stage-27 is filtered out by condition
resolver.
Stage-28 is filtered out by condition resolver.
14/03/14 17:05:02 INFO exec.Task: Stage-28 is filtered out by condition
resolver.
Stage-3 is selected by condition resolver.
{code}
Stage-3 is a reduce join. Actually, the resolver should pick the map join
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6668) When auto join convert is on and noconditionaltask is off, ConditionalResolverCommonJoin fails to resolve map joins.
[
https://issues.apache.org/jira/browse/HIVE-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-6668:
Status: Patch Available (was: Open)
kick test
When auto join convert is on and noconditionaltask is off,
ConditionalResolverCommonJoin fails to resolve map joins.
Key: HIVE-6668
URL: https://issues.apache.org/jira/browse/HIVE-6668
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0, 0.14.0
Reporter: Yin Huai
Assignee: Navis
Priority: Blocker
Fix For: 0.13.0
Attachments: HIVE-6668.1.patch.txt, HIVE-6668.2.patch.txt
I tried the following query today ...
{code:sql}
set mapred.job.map.memory.mb=2048;
set mapred.job.reduce.memory.mb=2048;
set mapred.map.child.java.opts=-server -Xmx3072m
-Djava.net.preferIPv4Stack=true;
set mapred.reduce.child.java.opts=-server -Xmx3072m
-Djava.net.preferIPv4Stack=true;
set mapred.reduce.tasks=60;
set hive.stats.autogather=false;
set hive.exec.parallel=false;
set hive.enforce.bucketing=true;
set hive.enforce.sorting=true;
set hive.map.aggr=true;
set hive.optimize.bucketmapjoin=true;
set hive.optimize.bucketmapjoin.sortedmerge=true;
set hive.mapred.reduce.tasks.speculative.execution=false;
set hive.auto.convert.join=true;
set hive.auto.convert.sortmerge.join=true;
set hive.auto.convert.sortmerge.join.noconditionaltask=false;
set hive.auto.convert.join.noconditionaltask=false;
set hive.auto.convert.join.noconditionaltask.size=1;
set hive.optimize.reducededuplication=true;
set hive.optimize.reducededuplication.min.reducer=1;
set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;
set hive.mapjoin.smalltable.filesize=4500;
set hive.optimize.index.filter=false;
set hive.vectorized.execution.enabled=false;
set hive.optimize.correlation=false;
select
i_item_id,
s_state,
avg(ss_quantity) agg1,
avg(ss_list_price) agg2,
avg(ss_coupon_amt) agg3,
avg(ss_sales_price) agg4
FROM store_sales
JOIN date_dim on (store_sales.ss_sold_date_sk = date_dim.d_date_sk)
JOIN item on (store_sales.ss_item_sk = item.i_item_sk)
JOIN customer_demographics on (store_sales.ss_cdemo_sk =
customer_demographics.cd_demo_sk)
JOIN store on (store_sales.ss_store_sk = store.s_store_sk)
where
cd_gender = 'F' and
cd_marital_status = 'U' and
cd_education_status = 'Primary' and
d_year = 2002 and
s_state in ('GA','PA', 'LA', 'SC', 'MI', 'AL')
group by i_item_id, s_state with rollup
order by
i_item_id,
s_state
limit 100;
{code}
The log shows ...
{code}
14/03/14 17:05:02 INFO plan.ConditionalResolverCommonJoin: Failed to resolve
driver alias (threshold : 4500, length mapping : {store=94175,
store_sales=48713909726, item=39798667, customer_demographics=1660831,
date_dim=2275902})
Stage-27 is filtered out by condition resolver.
14/03/14 17:05:02 INFO exec.Task: Stage-27 is filtered out by condition
resolver.
Stage-28 is filtered out by condition resolver.
14/03/14 17:05:02 INFO exec.Task: Stage-28 is filtered out by condition
resolver.
Stage-3 is selected by condition resolver.
{code}
Stage-3 is a reduce join. Actually, the resolver should pick the map join
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HIVE-6552) Failed to generate new mapJoin operator by exception : Big Table Alias is null
[ https://issues.apache.org/jira/browse/HIVE-6552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis resolved HIVE-6552. - Resolution: Fixed Fix Version/s: 0.13.0 Fixed by HIVE-6403 Failed to generate new mapJoin operator by exception : Big Table Alias is null -- Key: HIVE-6552 URL: https://issues.apache.org/jira/browse/HIVE-6552 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.12.0 Environment: Hive version: getBranch() : bigwheel-m16-0.12.0 getBuildVersion(): 0.12.0.2.0.6.1-101 from 8b1b43ece7c96d3cf38fda84414b23e3b707026e by jenkins source checksum 1c1e5eb051cefce14af4d621654dc423 getDate(): Wed Jan 8 22:20:16 PST 2014 getRevision(): 8b1b43ece7c96d3cf38fda84414b23e3b707026e getSrcChecksum() : 1c1e5eb051cefce14af4d621654dc423 getUrl() : git://c64-s17/grid/0/workspace/BIGTOP-HDP_RPM_REPO-bigwheel-M16/label/centos6-builds/bigtop-0.5/build/hive/rpm/BUILD/hive-0.12.0.2.0.6.1 getUser(): jenkins getVersion() : 0.12.0.2.0.6.1-101 OS: Red Hat Enterprise Linux Server release 6.4 x86_64 JVM: java version 1.6.0_31 Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode) Hadoop: Hadoop 2.2.0.2.0.6.0-101 Subversion g...@github.com:hortonworks/hadoop.git -r b07b2906c36defd389c8b5bd22bebc1bead8115b Compiled by jenkins on 2014-01-09T05:18Z Compiled with protoc 2.5.0 From source with checksum 704f1e463ebc4fb89353011407e965 Reporter: Martin Kudlej Fix For: 0.13.0 I've tried BigTop test for UNIQUEJOIN: CREATE TABLE T1(key STRING, val STRING) STORED AS TEXTFILE; CREATE TABLE T2(key STRING, val STRING) STORED AS TEXTFILE; CREATE TABLE T3(key STRING, val STRING) STORED AS TEXTFILE; LOAD DATA LOCAL INPATH 'seed_data_files/T1.txt' INTO TABLE T1; LOAD DATA LOCAL INPATH 'seed_data_files/T2.txt' INTO TABLE T2; LOAD DATA LOCAL INPATH 'seed_data_files/T3.txt' INTO TABLE T3; FROM UNIQUEJOIN PRESERVE T1 a (a.key), PRESERVE T2 b (b.key), PRESERVE T3 c (c.key) SELECT a.key, b.key, c.key; where T1.txt is: 111 212 313 717 818 828 and T2.txt is: 222 313 414 515 818 818 and T3.txt is: 212 414 616 717 if hive.auto.convert.join=false it works and result is: 1 NULLNULL 2 2 2 3 3 NULL NULL 4 4 NULL 5 NULL NULL NULL6 7 NULL7 8 8 NULL 8 8 NULL 8 8 NULL 8 8 NULL but hive.auto.convert.join=true it failed: FROM UNIQUEJOIN PRESERVE T1 a (a.key), PRESERVE T2 b (b.key), PRESERVE T3 c (c.key) SELECT a.key, b.key, c.key org.apache.hadoop.hive.ql.parse.SemanticException: Big Table Alias is null at org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:225) at org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256) at org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248) at org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191) at org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:480) at org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182) at org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111) at org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194) at org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139) at org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79) at org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:90) at org.apache.hadoop.hive.ql.parse.MapReduceCompiler.compile(MapReduceCompiler.java:300) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8410) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:441) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:342) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1000) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) at
[jira] [Commented] (HIVE-6677) HBaseSerDe needs to be refactored
[
https://issues.apache.org/jira/browse/HIVE-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13937182#comment-13937182
]
Hive QA commented on HIVE-6677:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12634951/HIVE-6677.1.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5406 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1848/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1848/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12634951
HBaseSerDe needs to be refactored
-
Key: HIVE-6677
URL: https://issues.apache.org/jira/browse/HIVE-6677
Project: Hive
Issue Type: Improvement
Affects Versions: 0.10.0, 0.11.0, 0.12.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Attachments: HIVE-6677.1.patch, HIVE-6677.patch
The code in HBaseSerde seems very complex and hard to be extend to support
new features such as adding generic compound key (HIVE-6411) and Compound key
filter (HIVE-6290), especially when handling key/field serialization. Hope
this task will clean up the code a bit and make it ready for new extensions.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-3746) Fix HS2 ResultSet Serialization Performance Regression
[ https://issues.apache.org/jira/browse/HIVE-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13937195#comment-13937195 ] zhuyu commented on HIVE-3746: - how to solve it? main: [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-exec --- [INFO] Compiling 7 source files to /mnt/public/workspace/linux/hive-0.13/ql/target/classes [INFO] [INFO] [INFO] Building Hive Service 0.13.0 [INFO] Downloading: http://www.datanucleus.org/downloads/maven2/org/apache/hive/hive-exec/0.13.0/hive-exec-0.13.0-tests.jar Downloading: http://repo.maven.apache.org/maven2/org/apache/hive/hive-exec/0.13.0/hive-exec-0.13.0-tests.jar [INFO] [INFO] Reactor Summary: [INFO] [INFO] Hive .. SUCCESS [2.648s] [INFO] Hive Ant Utilities SUCCESS [1.184s] [INFO] Hive Shims Common . SUCCESS [1.419s] [INFO] Hive Shims 0.20 ... SUCCESS [0.739s] [INFO] Hive Shims Secure Common .. SUCCESS [1.021s] [INFO] Hive Shims 0.20S .. SUCCESS [0.796s] [INFO] Hive Shims 0.23 ... SUCCESS [2.539s] [INFO] Hive Shims SUCCESS [0.415s] [INFO] Hive Common ... SUCCESS [4.014s] [INFO] Hive Serde SUCCESS [1.722s] [INFO] Hive Metastore SUCCESS [2.513s] [INFO] Hive Query Language ... SUCCESS [7.495s] [INFO] Hive Service .. FAILURE [2.921s] [INFO] Hive JDBC . SKIPPED [INFO] Hive Beeline .. SKIPPED [INFO] Hive CLI .. SKIPPED [INFO] Hive Contrib .. SKIPPED [INFO] Hive HBase Handler SKIPPED [INFO] Hive HCatalog . SKIPPED [INFO] Hive HCatalog Core SKIPPED [INFO] Hive HCatalog Pig Adapter . SKIPPED [INFO] Hive HCatalog Server Extensions ... SKIPPED [INFO] Hive HCatalog Webhcat Java Client . SKIPPED [INFO] Hive HCatalog Webhcat . SKIPPED [INFO] Hive HCatalog HBase Storage Handler ... SKIPPED [INFO] Hive HWI .. SKIPPED [INFO] Hive ODBC . SKIPPED [INFO] Hive Shims Aggregator . SKIPPED [INFO] Hive TestUtils SKIPPED [INFO] Hive Packaging SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 31.330s [INFO] Finished at: Mon Mar 17 00:05:57 HKT 2014 [INFO] Final Memory: 41M/236M [INFO] [ERROR] Failed to execute goal on project hive-service: Could not resolve dependencies for project org.apache.hive:hive-service:jar:0.13.0: Could not find artifact org.apache.hive:hive-exec:jar:tests:0.13.0 in datanucleus (http://www.datanucleus.org/downloads/maven2) - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hive-service Fix HS2 ResultSet Serialization Performance Regression -- Key: HIVE-3746 URL: https://issues.apache.org/jira/browse/HIVE-3746 Project: Hive Issue Type: Sub-task Components: HiveServer2, Server Infrastructure Reporter: Carl Steinbach Assignee: Navis Labels: HiveServer2, jdbc, thrift Fix For: 0.13.0 Attachments: HIVE-3746.1.patch.txt, HIVE-3746.2.patch.txt, HIVE-3746.3.patch.txt, HIVE-3746.4.patch.txt, HIVE-3746.5.patch.txt, HIVE-3746.6.patch.txt,
[jira] [Updated] (HIVE-6657) Add test coverage for Kerberos authentication implementation using Hadoop's miniKdc
[ https://issues.apache.org/jira/browse/HIVE-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-6657: -- Attachment: HIVE-6657.4.patch re-attaching for pre-commit run Add test coverage for Kerberos authentication implementation using Hadoop's miniKdc --- Key: HIVE-6657 URL: https://issues.apache.org/jira/browse/HIVE-6657 Project: Hive Issue Type: Improvement Components: Authentication, Testing Infrastructure, Tests Affects Versions: 0.13.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-6657.2.patch, HIVE-6657.3.patch, HIVE-6657.4.patch, HIVE-6657.4.patch Hadoop 2.3 includes miniKdc module. This provides a KDC that can be used by downstream projects to implement unit tests for Kerberos authentication code. Hive has lot of code related to Kerberos and delegation token for authentication, as well as accessing secure hadoop resources. This pretty much has no coverage in the unit tests. We needs to add unit tests using miniKdc module. Note that Hadoop 2.3 doesn't include a secure mini-cluster. Until that is available, we can at least test authentication for components like HiveServer2, Metastore and WebHCat. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6660) HiveServer2 running in non-http mode closes server socket for an SSL connection after the 1st request
[
https://issues.apache.org/jira/browse/HIVE-6660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasad Mujumdar updated HIVE-6660:
--
Attachment: HIVE-6660.1.patch
re-attaching for pre-commit run
HiveServer2 running in non-http mode closes server socket for an SSL
connection after the 1st request
-
Key: HIVE-6660
URL: https://issues.apache.org/jira/browse/HIVE-6660
Project: Hive
Issue Type: Bug
Components: HiveServer2, JDBC
Affects Versions: 0.13.0
Reporter: Vaibhav Gumashta
Assignee: Prasad Mujumdar
Priority: Blocker
Fix For: 0.13.0
Attachments: HIVE-6660.1.patch, HIVE-6660.1.patch, hive-site.xml
*Beeline connection string:*
{code}
!connect
jdbc:hive2://host:1/;ssl=true;sslTrustStore=/usr/share/doc/hive-0.13.0.2.1.1.0/examples/files/truststore.jks;trustStorePassword=HiveJdbc
vgumashta vgumashta org.apache.hive.jdbc.HiveDriver
{code}
*Error:*
{code}
pool-7-thread-1, handling exception: java.net.SocketTimeoutException: Read
timed out
pool-7-thread-1, called close()
pool-7-thread-1, called closeInternal(true)
pool-7-thread-1, SEND TLSv1 ALERT: warning, description = close_notify
Padded plaintext before ENCRYPTION: len = 32
: 01 00 BE 72 AC 10 3B FA 4E 01 A5 DE 9B 14 16 AF ...r..;.N...
0010: 4E DD 7A 29 AD B4 09 09 09 09 09 09 09 09 09 09 N.z)
pool-7-thread-1, WRITE: TLSv1 Alert, length = 32
[Raw write]: length = 37
: 15 03 01 00 20 6C 37 82 A8 52 40 DA FB 83 2D CD l7..R@...-.
0010: 96 9F F0 B7 22 17 E1 04 C1 D1 93 1B C4 39 5A B0 9Z.
0020: A2 3F 5D 7D 2D .?].-
pool-7-thread-1, called closeSocket(selfInitiated)
pool-7-thread-1, called close()
pool-7-thread-1, called closeInternal(true)
pool-7-thread-1, called close()
pool-7-thread-1, called closeInternal(true)
{code}
*Subsequent queries fail:*
{code}
main, WRITE: TLSv1 Application Data, length = 144
main, handling exception: java.net.SocketException: Broken pipe
%% Invalidated: [Session-1, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
main, SEND TLSv1 ALERT: fatal, description = unexpected_message
Padded plaintext before ENCRYPTION: len = 32
: 02 0A 52 C3 18 B1 C1 38 DB 3F B6 D1 C5 CA 14 9C ..R8.?..
0010: A5 38 4C 01 31 69 09 09 09 09 09 09 09 09 09 09 .8L.1i..
main, WRITE: TLSv1 Alert, length = 32
main, Exception sending alert: java.net.SocketException: Broken pipe
main, called closeSocket()
Error: org.apache.thrift.transport.TTransportException:
java.net.SocketException: Broken pipe (state=08S01,code=0)
java.sql.SQLException: org.apache.thrift.transport.TTransportException:
java.net.SocketException: Broken pipe
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:226)
at org.apache.hive.beeline.Commands.execute(Commands.java:736)
at org.apache.hive.beeline.Commands.sql(Commands.java:657)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:796)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:659)
at
org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:368)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:351)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.thrift.transport.TTransportException:
java.net.SocketException: Broken pipe
at
org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
at
org.apache.thrift.transport.TSaslTransport.flush(TSaslTransport.java:471)
at
org.apache.thrift.transport.TSaslClientTransport.flush(TSaslClientTransport.java:37)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)
at
org.apache.hive.service.cli.thrift.TCLIService$Client.send_ExecuteStatement(TCLIService.java:219)
at
org.apache.hive.service.cli.thrift.TCLIService$Client.ExecuteStatement(TCLIService.java:211)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:220)
... 11 more
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:377)
at
Review Request 19265: HIVE-6677: HBaseSerDe needs to be refactored
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19265/ --- Review request for hive. Bugs: HIVE-6677 https://issues.apache.org/jira/browse/HIVE-6677 Repository: hive-git Description --- Pure refactoring. For the purpose of what is described in the JIRA. Mostly, separating key serialization from column serialization. When possible, static methods are used. Some code cleanup is made. All serde properties, including those dedicated for HBaseSerde, is captured in class HBaseSerDeParameters. Diffs - hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 2cd65cb hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java PRE-CREATION hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 29e5da5 hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java db69ae5 Diff: https://reviews.apache.org/r/19265/diff/ Testing --- No new tests. Test suite passed successfully. Thanks, Xuefu Zhang
[jira] [Commented] (HIVE-6677) HBaseSerDe needs to be refactored
[ https://issues.apache.org/jira/browse/HIVE-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13937227#comment-13937227 ] Xuefu Zhang commented on HIVE-6677: --- The above test failure is unrelated. Patch is ready to be reviewed. RB: https://reviews.apache.org/r/19265/ HBaseSerDe needs to be refactored - Key: HIVE-6677 URL: https://issues.apache.org/jira/browse/HIVE-6677 Project: Hive Issue Type: Improvement Affects Versions: 0.10.0, 0.11.0, 0.12.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Attachments: HIVE-6677.1.patch, HIVE-6677.patch The code in HBaseSerde seems very complex and hard to be extend to support new features such as adding generic compound key (HIVE-6411) and Compound key filter (HIVE-6290), especially when handling key/field serialization. Hope this task will clean up the code a bit and make it ready for new extensions. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 19265: HIVE-6677: HBaseSerDe needs to be refactored
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19265/ --- (Updated March 16, 2014, 5:33 p.m.) Review request for hive. Bugs: HIVE-6677 https://issues.apache.org/jira/browse/HIVE-6677 Repository: hive-git Description --- Pure refactoring. For the purpose of what is described in the JIRA. Mostly, separating key serialization from column serialization. When possible, static methods are used. Some code cleanup is made. All serde properties, including those dedicated for HBaseSerde, is captured in class HBaseSerDeParameters. Diffs (updated) - hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 2cd65cb hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java PRE-CREATION hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 29e5da5 hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java db69ae5 Diff: https://reviews.apache.org/r/19265/diff/ Testing --- No new tests. Test suite passed successfully. Thanks, Xuefu Zhang
[jira] [Updated] (HIVE-6677) HBaseSerDe needs to be refactored
[ https://issues.apache.org/jira/browse/HIVE-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-6677: -- Attachment: HIVE-6677.2.patch Patch #2 removed a few trailing spaces/tabs. I don't think new test run is necessary. So, we don't need wait for that. HBaseSerDe needs to be refactored - Key: HIVE-6677 URL: https://issues.apache.org/jira/browse/HIVE-6677 Project: Hive Issue Type: Improvement Affects Versions: 0.10.0, 0.11.0, 0.12.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Attachments: HIVE-6677.1.patch, HIVE-6677.2.patch, HIVE-6677.patch The code in HBaseSerde seems very complex and hard to be extend to support new features such as adding generic compound key (HIVE-6411) and Compound key filter (HIVE-6290), especially when handling key/field serialization. Hope this task will clean up the code a bit and make it ready for new extensions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6677) HBaseSerDe needs to be refactored
[ https://issues.apache.org/jira/browse/HIVE-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13937231#comment-13937231 ] Xuefu Zhang commented on HIVE-6677: --- [~navis] Could you please review and make sure it fits the need of the HIVE-6411? While it does not match exactly the refactoring work that I suggested in HIVE-6411 RB, it should make the work much easier. HBaseSerDe needs to be refactored - Key: HIVE-6677 URL: https://issues.apache.org/jira/browse/HIVE-6677 Project: Hive Issue Type: Improvement Affects Versions: 0.10.0, 0.11.0, 0.12.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Attachments: HIVE-6677.1.patch, HIVE-6677.2.patch, HIVE-6677.patch The code in HBaseSerde seems very complex and hard to be extend to support new features such as adding generic compound key (HIVE-6411) and Compound key filter (HIVE-6290), especially when handling key/field serialization. Hope this task will clean up the code a bit and make it ready for new extensions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6578) Use ORC file footer statistics through StatsProvidingRecordReader interface for analyze command
[
https://issues.apache.org/jira/browse/HIVE-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13937304#comment-13937304
]
Hive QA commented on HIVE-6578:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12634958/HIVE-6578.4.patch.txt
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5407 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1852/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1852/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12634958
Use ORC file footer statistics through StatsProvidingRecordReader interface
for analyze command
---
Key: HIVE-6578
URL: https://issues.apache.org/jira/browse/HIVE-6578
Project: Hive
Issue Type: New Feature
Affects Versions: 0.13.0
Reporter: Prasanth J
Assignee: Prasanth J
Labels: orcfile
Attachments: HIVE-6578.1.patch, HIVE-6578.2.patch, HIVE-6578.3.patch,
HIVE-6578.4.patch, HIVE-6578.4.patch.txt
ORC provides file level statistics which can be used in analyze partialscan
and noscan cases to compute basic statistics like number of rows, number of
files, total file size and raw data size. On the writer side, a new interface
was added earlier (StatsProvidingRecordWriter) that exposed stats when
writing a table. Similarly, a new interface StatsProvidingRecordReader can be
added which when implemented should provide stats that are gathered by the
underlying file format.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6681) Describe table sometimes shows from deserializer for column comments
[ https://issues.apache.org/jira/browse/HIVE-6681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6681: --- Attachment: HIVE-6681.patch Initial patch to get Hive QA run. Describe table sometimes shows from deserializer for column comments -- Key: HIVE-6681 URL: https://issues.apache.org/jira/browse/HIVE-6681 Project: Hive Issue Type: Bug Components: Metastore, Serializers/Deserializers Affects Versions: 0.11.0, 0.12.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-6681.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6681) Describe table sometimes shows from deserializer for column comments
[ https://issues.apache.org/jira/browse/HIVE-6681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6681: --- Status: Patch Available (was: Open) Describe table sometimes shows from deserializer for column comments -- Key: HIVE-6681 URL: https://issues.apache.org/jira/browse/HIVE-6681 Project: Hive Issue Type: Bug Components: Metastore, Serializers/Deserializers Affects Versions: 0.12.0, 0.11.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-6681.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6620) UDF printf doesn't take either CHAR or VARCHAR as the first argument
[
https://issues.apache.org/jira/browse/HIVE-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-6620:
--
Attachment: HIVE-6620.1.patch
Patch #1 included the new tests that were added but missed when the previous
patch was generated.
UDF printf doesn't take either CHAR or VARCHAR as the first argument
Key: HIVE-6620
URL: https://issues.apache.org/jira/browse/HIVE-6620
Project: Hive
Issue Type: Bug
Components: UDF
Affects Versions: 0.12.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Attachments: HIVE-6620.1.patch, HIVE-6620.patch, HIVE-6620.patch,
HIVE-6620.patch
{code}
hive desc vc;
OK
c char(5) None
vcvarchar(7) None
s string None
hive select printf(c) from vc;
FAILED: SemanticException [Error 10016]: Line 1:14 Argument type mismatch
'c': Argument 1 of function PRINTF must be string, but char(5) was found.
{code}
However, if the argument is string type, the query runs successfully.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6656) Bug in ORC Timestamp reader returns wrong nanoseconds
[
https://issues.apache.org/jira/browse/HIVE-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13937342#comment-13937342
]
Hive QA commented on HIVE-6656:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12634959/HIVE-6656.1.patch.txt
{color:green}SUCCESS:{color} +1 5408 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1853/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1853/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12634959
Bug in ORC Timestamp reader returns wrong nanoseconds
-
Key: HIVE-6656
URL: https://issues.apache.org/jira/browse/HIVE-6656
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0, 0.14.0
Reporter: Prasanth J
Assignee: Prasanth J
Labels: orcfile
Attachments: HIVE-6656.1.patch, HIVE-6656.1.patch.txt
ORC timestamp writer stores the number of trailing zeros in 3 LSB bits. There
is a bug in parsing nanosecond logic that returns incorrect value.
Input:
1999-01-01 00:00:00.9
Output:
1999-01-01 00:00:00.463129087
The fix for this is parseNanos() should first right shift by 3 and then
typecast to int.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6666) Metastore init scripts should always populate the version information at the end
[ https://issues.apache.org/jira/browse/HIVE-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-: -- Attachment: HIVE-.1.patch Metastore init scripts should always populate the version information at the end Key: HIVE- URL: https://issues.apache.org/jira/browse/HIVE- Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.13.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-.1.patch, HIVE-.1.patch The metastore schema create scripts for 0.13 and 0.14 (current trunk) has multiple other operations after setting the schema version. This is problematic as any failure in those later operations would leave metastore in inconsistent state, and yet with valid version information. The schemaTool depends on the schema version details. Recording the schema version should be the last step in schema initialization script. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-4293) Predicates following UDTF operator are removed by PPD
[
https://issues.apache.org/jira/browse/HIVE-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Butani updated HIVE-4293:
Status: Open (was: Patch Available)
Predicates following UDTF operator are removed by PPD
-
Key: HIVE-4293
URL: https://issues.apache.org/jira/browse/HIVE-4293
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Critical
Attachments: D9933.6.patch, HIVE-4293.10.patch,
HIVE-4293.11.patch.txt, HIVE-4293.12.patch, HIVE-4293.7.patch.txt,
HIVE-4293.8.patch.txt, HIVE-4293.9.patch.txt, HIVE-4293.D9933.1.patch,
HIVE-4293.D9933.2.patch, HIVE-4293.D9933.3.patch, HIVE-4293.D9933.4.patch,
HIVE-4293.D9933.5.patch
For example,
{noformat}
explain SELECT value from (
select explode(array(key, value)) as (value) from (
select * FROM src WHERE key 200
) A
) B WHERE value 300
;
{noformat}
Makes plan like this, removing last predicates
{noformat}
TableScan
alias: src
Filter Operator
predicate:
expr: (key 200.0)
type: boolean
Select Operator
expressions:
expr: array(key,value)
type: arraystring
outputColumnNames: _col0
UDTF Operator
function name: explode
Select Operator
expressions:
expr: col
type: string
outputColumnNames: _col0
File Output Operator
compressed: false
GlobalTableId: 0
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-4293) Predicates following UDTF operator are removed by PPD
[
https://issues.apache.org/jira/browse/HIVE-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Butani updated HIVE-4293:
Status: Patch Available (was: Open)
resubmit patch
Predicates following UDTF operator are removed by PPD
-
Key: HIVE-4293
URL: https://issues.apache.org/jira/browse/HIVE-4293
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Critical
Attachments: D9933.6.patch, HIVE-4293.10.patch,
HIVE-4293.11.patch.txt, HIVE-4293.12.patch, HIVE-4293.7.patch.txt,
HIVE-4293.8.patch.txt, HIVE-4293.9.patch.txt, HIVE-4293.D9933.1.patch,
HIVE-4293.D9933.2.patch, HIVE-4293.D9933.3.patch, HIVE-4293.D9933.4.patch,
HIVE-4293.D9933.5.patch
For example,
{noformat}
explain SELECT value from (
select explode(array(key, value)) as (value) from (
select * FROM src WHERE key 200
) A
) B WHERE value 300
;
{noformat}
Makes plan like this, removing last predicates
{noformat}
TableScan
alias: src
Filter Operator
predicate:
expr: (key 200.0)
type: boolean
Select Operator
expressions:
expr: array(key,value)
type: arraystring
outputColumnNames: _col0
UDTF Operator
function name: explode
Select Operator
expressions:
expr: col
type: string
outputColumnNames: _col0
File Output Operator
compressed: false
GlobalTableId: 0
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: Review Request 19165: HIVE-6643: Add a check for cross products in plans and output a warning
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19165/ --- (Updated March 17, 2014, 12:42 a.m.) Review request for hive and Gunther Hagleitner. Changes --- changes suggested by gunther Bugs: HIVE-6643 https://issues.apache.org/jira/browse/HIVE-6643 Repository: hive-git Description --- Now that we support old style join syntax, it is easy to write queries that generate a plan with a cross product. For e.g. say you have A join B join C join D on A.x = B.x and A.y = D.y and C.z = D.z So the JoinTree is: A — B __ D — C Since we don't reorder join graphs, we will end up with a cross product between (A join B) and C Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 56d68f5 conf/hive-default.xml.template 906ea8c itests/qtest/pom.xml f8b81a2 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CrossProductCheck.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java d593d08 ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java a5e6cbf ql/src/test/queries/clientpositive/cross_product_check_1.q PRE-CREATION ql/src/test/queries/clientpositive/cross_product_check_2.q PRE-CREATION ql/src/test/results/clientpositive/cross_product_check_1.q.out PRE-CREATION ql/src/test/results/clientpositive/cross_product_check_2.q.out PRE-CREATION ql/src/test/results/clientpositive/tez/cross_product_check_1.q.out PRE-CREATION ql/src/test/results/clientpositive/tez/cross_product_check_2.q.out PRE-CREATION Diff: https://reviews.apache.org/r/19165/diff/ Testing --- added new tests Thanks, Harish Butani
[jira] [Updated] (HIVE-6643) Add a check for cross products in plans and output a warning
[ https://issues.apache.org/jira/browse/HIVE-6643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Butani updated HIVE-6643: Attachment: HIVE-6643.2.patch Add a check for cross products in plans and output a warning Key: HIVE-6643 URL: https://issues.apache.org/jira/browse/HIVE-6643 Project: Hive Issue Type: Bug Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-6643.1.patch, HIVE-6643.2.patch Now that we support old style join syntax, it is easy to write queries that generate a plan with a cross product. For e.g. say you have A join B join C join D on A.x = B.x and A.y = D.y and C.z = D.z So the JoinTree is: A — B |__ D — C Since we don't reorder join graphs, we will end up with a cross product between (A join B) and C -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6643) Add a check for cross products in plans and output a warning
[ https://issues.apache.org/jira/browse/HIVE-6643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Butani updated HIVE-6643: Status: Open (was: Patch Available) Add a check for cross products in plans and output a warning Key: HIVE-6643 URL: https://issues.apache.org/jira/browse/HIVE-6643 Project: Hive Issue Type: Bug Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-6643.1.patch, HIVE-6643.2.patch Now that we support old style join syntax, it is easy to write queries that generate a plan with a cross product. For e.g. say you have A join B join C join D on A.x = B.x and A.y = D.y and C.z = D.z So the JoinTree is: A — B |__ D — C Since we don't reorder join graphs, we will end up with a cross product between (A join B) and C -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6643) Add a check for cross products in plans and output a warning
[ https://issues.apache.org/jira/browse/HIVE-6643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Butani updated HIVE-6643: Status: Patch Available (was: Open) Add a check for cross products in plans and output a warning Key: HIVE-6643 URL: https://issues.apache.org/jira/browse/HIVE-6643 Project: Hive Issue Type: Bug Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-6643.1.patch, HIVE-6643.2.patch Now that we support old style join syntax, it is easy to write queries that generate a plan with a cross product. For e.g. say you have A join B join C join D on A.x = B.x and A.y = D.y and C.z = D.z So the JoinTree is: A — B |__ D — C Since we don't reorder join graphs, we will end up with a cross product between (A join B) and C -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-4764) Support Kerberos HTTP authentication for HiveServer2 running in http mode
[
https://issues.apache.org/jira/browse/HIVE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13937390#comment-13937390
]
Hive QA commented on HIVE-4764:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12634966/HIVE-4764.6.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5397 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1854/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1854/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12634966
Support Kerberos HTTP authentication for HiveServer2 running in http mode
-
Key: HIVE-4764
URL: https://issues.apache.org/jira/browse/HIVE-4764
Project: Hive
Issue Type: Sub-task
Components: HiveServer2
Affects Versions: 0.13.0
Reporter: Thejas M Nair
Assignee: Vaibhav Gumashta
Fix For: 0.13.0
Attachments: HIVE-4764.1.patch, HIVE-4764.2.patch, HIVE-4764.3.patch,
HIVE-4764.4.patch, HIVE-4764.5.patch, HIVE-4764.6.patch
Support Kerberos authentication for HiveServer2 running in http mode.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6681) Describe table sometimes shows from deserializer for column comments
[ https://issues.apache.org/jira/browse/HIVE-6681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6681: --- Status: Open (was: Patch Available) Describe table sometimes shows from deserializer for column comments -- Key: HIVE-6681 URL: https://issues.apache.org/jira/browse/HIVE-6681 Project: Hive Issue Type: Bug Components: Metastore, Serializers/Deserializers Affects Versions: 0.12.0, 0.11.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-6681.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6681) Describe table sometimes shows from deserializer for column comments
[ https://issues.apache.org/jira/browse/HIVE-6681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6681: --- Attachment: HIVE-6681.2.patch Describe table sometimes shows from deserializer for column comments -- Key: HIVE-6681 URL: https://issues.apache.org/jira/browse/HIVE-6681 Project: Hive Issue Type: Bug Components: Metastore, Serializers/Deserializers Affects Versions: 0.11.0, 0.12.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-6681.2.patch, HIVE-6681.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6681) Describe table sometimes shows from deserializer for column comments
[ https://issues.apache.org/jira/browse/HIVE-6681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6681: --- Status: Patch Available (was: Open) Describe table sometimes shows from deserializer for column comments -- Key: HIVE-6681 URL: https://issues.apache.org/jira/browse/HIVE-6681 Project: Hive Issue Type: Bug Components: Metastore, Serializers/Deserializers Affects Versions: 0.12.0, 0.11.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-6681.2.patch, HIVE-6681.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6668) When auto join convert is on and noconditionaltask is off, ConditionalResolverCommonJoin fails to resolve map joins.
[
https://issues.apache.org/jira/browse/HIVE-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13937420#comment-13937420
]
Hive QA commented on HIVE-6668:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12634971/HIVE-6668.2.patch.txt
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 5406 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_hook
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1855/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1855/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12634971
When auto join convert is on and noconditionaltask is off,
ConditionalResolverCommonJoin fails to resolve map joins.
Key: HIVE-6668
URL: https://issues.apache.org/jira/browse/HIVE-6668
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0, 0.14.0
Reporter: Yin Huai
Assignee: Navis
Priority: Blocker
Fix For: 0.13.0
Attachments: HIVE-6668.1.patch.txt, HIVE-6668.2.patch.txt
I tried the following query today ...
{code:sql}
set mapred.job.map.memory.mb=2048;
set mapred.job.reduce.memory.mb=2048;
set mapred.map.child.java.opts=-server -Xmx3072m
-Djava.net.preferIPv4Stack=true;
set mapred.reduce.child.java.opts=-server -Xmx3072m
-Djava.net.preferIPv4Stack=true;
set mapred.reduce.tasks=60;
set hive.stats.autogather=false;
set hive.exec.parallel=false;
set hive.enforce.bucketing=true;
set hive.enforce.sorting=true;
set hive.map.aggr=true;
set hive.optimize.bucketmapjoin=true;
set hive.optimize.bucketmapjoin.sortedmerge=true;
set hive.mapred.reduce.tasks.speculative.execution=false;
set hive.auto.convert.join=true;
set hive.auto.convert.sortmerge.join=true;
set hive.auto.convert.sortmerge.join.noconditionaltask=false;
set hive.auto.convert.join.noconditionaltask=false;
set hive.auto.convert.join.noconditionaltask.size=1;
set hive.optimize.reducededuplication=true;
set hive.optimize.reducededuplication.min.reducer=1;
set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;
set hive.mapjoin.smalltable.filesize=4500;
set hive.optimize.index.filter=false;
set hive.vectorized.execution.enabled=false;
set hive.optimize.correlation=false;
select
i_item_id,
s_state,
avg(ss_quantity) agg1,
avg(ss_list_price) agg2,
avg(ss_coupon_amt) agg3,
avg(ss_sales_price) agg4
FROM store_sales
JOIN date_dim on (store_sales.ss_sold_date_sk = date_dim.d_date_sk)
JOIN item on (store_sales.ss_item_sk = item.i_item_sk)
JOIN customer_demographics on (store_sales.ss_cdemo_sk =
customer_demographics.cd_demo_sk)
JOIN store on (store_sales.ss_store_sk = store.s_store_sk)
where
cd_gender = 'F' and
cd_marital_status = 'U' and
cd_education_status = 'Primary' and
d_year = 2002 and
s_state in ('GA','PA', 'LA', 'SC', 'MI', 'AL')
group by i_item_id, s_state with rollup
order by
i_item_id,
s_state
limit 100;
{code}
The log shows ...
{code}
14/03/14 17:05:02 INFO plan.ConditionalResolverCommonJoin: Failed to resolve
driver alias (threshold : 4500, length mapping : {store=94175,
store_sales=48713909726, item=39798667, customer_demographics=1660831,
date_dim=2275902})
Stage-27 is filtered out by condition resolver.
14/03/14 17:05:02 INFO exec.Task: Stage-27 is filtered out by condition
resolver.
Stage-28 is filtered out by condition resolver.
14/03/14 17:05:02 INFO exec.Task: Stage-28 is filtered out by condition
resolver.
Stage-3 is selected by condition resolver.
{code}
Stage-3 is a reduce join. Actually, the resolver should pick the map join
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6660) HiveServer2 running in non-http mode closes server socket for an SSL connection after the 1st request
[
https://issues.apache.org/jira/browse/HIVE-6660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13937446#comment-13937446
]
Hive QA commented on HIVE-6660:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12634989/HIVE-6660.1.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 5406 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1856/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1856/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12634989
HiveServer2 running in non-http mode closes server socket for an SSL
connection after the 1st request
-
Key: HIVE-6660
URL: https://issues.apache.org/jira/browse/HIVE-6660
Project: Hive
Issue Type: Bug
Components: HiveServer2, JDBC
Affects Versions: 0.13.0
Reporter: Vaibhav Gumashta
Assignee: Prasad Mujumdar
Priority: Blocker
Fix For: 0.13.0
Attachments: HIVE-6660.1.patch, HIVE-6660.1.patch, hive-site.xml
*Beeline connection string:*
{code}
!connect
jdbc:hive2://host:1/;ssl=true;sslTrustStore=/usr/share/doc/hive-0.13.0.2.1.1.0/examples/files/truststore.jks;trustStorePassword=HiveJdbc
vgumashta vgumashta org.apache.hive.jdbc.HiveDriver
{code}
*Error:*
{code}
pool-7-thread-1, handling exception: java.net.SocketTimeoutException: Read
timed out
pool-7-thread-1, called close()
pool-7-thread-1, called closeInternal(true)
pool-7-thread-1, SEND TLSv1 ALERT: warning, description = close_notify
Padded plaintext before ENCRYPTION: len = 32
: 01 00 BE 72 AC 10 3B FA 4E 01 A5 DE 9B 14 16 AF ...r..;.N...
0010: 4E DD 7A 29 AD B4 09 09 09 09 09 09 09 09 09 09 N.z)
pool-7-thread-1, WRITE: TLSv1 Alert, length = 32
[Raw write]: length = 37
: 15 03 01 00 20 6C 37 82 A8 52 40 DA FB 83 2D CD l7..R@...-.
0010: 96 9F F0 B7 22 17 E1 04 C1 D1 93 1B C4 39 5A B0 9Z.
0020: A2 3F 5D 7D 2D .?].-
pool-7-thread-1, called closeSocket(selfInitiated)
pool-7-thread-1, called close()
pool-7-thread-1, called closeInternal(true)
pool-7-thread-1, called close()
pool-7-thread-1, called closeInternal(true)
{code}
*Subsequent queries fail:*
{code}
main, WRITE: TLSv1 Application Data, length = 144
main, handling exception: java.net.SocketException: Broken pipe
%% Invalidated: [Session-1, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
main, SEND TLSv1 ALERT: fatal, description = unexpected_message
Padded plaintext before ENCRYPTION: len = 32
: 02 0A 52 C3 18 B1 C1 38 DB 3F B6 D1 C5 CA 14 9C ..R8.?..
0010: A5 38 4C 01 31 69 09 09 09 09 09 09 09 09 09 09 .8L.1i..
main, WRITE: TLSv1 Alert, length = 32
main, Exception sending alert: java.net.SocketException: Broken pipe
main, called closeSocket()
Error: org.apache.thrift.transport.TTransportException:
java.net.SocketException: Broken pipe (state=08S01,code=0)
java.sql.SQLException: org.apache.thrift.transport.TTransportException:
java.net.SocketException: Broken pipe
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:226)
at org.apache.hive.beeline.Commands.execute(Commands.java:736)
at org.apache.hive.beeline.Commands.sql(Commands.java:657)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:796)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:659)
at
org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:368)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:351)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.thrift.transport.TTransportException:
java.net.SocketException: Broken pipe
at
org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
at
[jira] [Commented] (HIVE-6331) HIVE-5279 deprecated UDAF class without explanation/documentation/alternative
[
https://issues.apache.org/jira/browse/HIVE-6331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13937468#comment-13937468
]
Lefty Leverenz commented on HIVE-6331:
--
Trivial review comments:
1. support should be supports since UDAF is singular:
{code}
+ * Optional for a UDAF class (by implementing these two methods, the user
declares
+ * that the UDAF support partial aggregations):
{code}
2. Second li should be indented like the first one:
{code}
+ * ol
+ * liImplement a single method called {@code evaluatePartial} that returns
the PARTIAL aggregation result.
+ * {@code evaluatePartial} should never return {@code null} or an Exception
will be thrown./li
+ * liImplement a single method called {@code aggregatePartial} that takes a
PARTIAL
{code}
HIVE-5279 deprecated UDAF class without explanation/documentation/alternative
-
Key: HIVE-6331
URL: https://issues.apache.org/jira/browse/HIVE-6331
Project: Hive
Issue Type: Bug
Reporter: Lars Francke
Assignee: Lars Francke
Priority: Minor
Attachments: HIVE-5279.1.patch
HIVE-5279 added a @Deprecated annotation to the {{UDAF}} class. The comment
in that class says {quote}UDAF classes are REQUIRED to inherit from this
class.{quote}
One of these two needs to be updated. Either remove the annotation or
document why it was deprecated and what to use instead.
Unfortunately [~navis] did not leave any documentation about his intentions.
I'm happy to provide a patch once I know the intentions.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6468) HS2 out of memory error when curl sends a get request
[
https://issues.apache.org/jira/browse/HIVE-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13937470#comment-13937470
]
Lefty Leverenz commented on HIVE-6468:
--
I added this sentence to the second paragraph of the wiki's Beeline section:
{quote}
In remote mode HiveServer2 only accepts valid Thrift calls; you cannot call it
via http or telnet (HIVE-6468).
{quote}
Please review and correct if necessary.
* [Beeline -- New Command Line Shell
|https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-Beeline–NewCommandLineShell]
HS2 out of memory error when curl sends a get request
-
Key: HIVE-6468
URL: https://issues.apache.org/jira/browse/HIVE-6468
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Environment: Centos 6.3, hive 12, hadoop-2.2
Reporter: Abin Shahab
Assignee: Navis
Attachments: HIVE-6468.1.patch.txt
We see an out of memory error when we run simple beeline calls.
(The hive.server2.transport.mode is binary)
curl localhost:1
Exception in thread pool-2-thread-8 java.lang.OutOfMemoryError: Java heap
space
at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:181)
at
org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
at
org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:189)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-5963) Update build instructions in Getting Started wikidoc
[
https://issues.apache.org/jira/browse/HIVE-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13937474#comment-13937474
]
Lefty Leverenz commented on HIVE-5963:
--
Another reminder.
Update build instructions in Getting Started wikidoc
Key: HIVE-5963
URL: https://issues.apache.org/jira/browse/HIVE-5963
Project: Hive
Issue Type: Bug
Components: Documentation
Affects Versions: 0.13.0
Reporter: Lefty Leverenz
Labels: documentation
In the Getting Started wikidoc, [Building Hive from
Source|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-BuildingHivefromSource]
still uses {{ant}} instead of {{mvn}}:
{quote}
$ ant clean package
{quote}
and:
{quote}
$ ant clean package -Dhadoop.version=0.23.3 -Dhadoop-0.23.version=0.23.3
-Dhadoop.mr.rev=23
$ ant clean package -Dhadoop.version=2.0.0-alpha
-Dhadoop-0.23.version=2.0.0-alpha -Dhadoop.mr.rev=23
{quote}
Perhaps these instructions should be retained for earlier releases, but Maven
instructions need to be added for 0.13.0.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
