[jira] [Updated] (HIVE-16280) Table level REPL LOAD doesn't return a valid dump path.
[ https://issues.apache.org/jira/browse/HIVE-16280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16280: Priority: Minor (was: Major) > Table level REPL LOAD doesn't return a valid dump path. > --- > > Key: HIVE-16280 > URL: https://issues.apache.org/jira/browse/HIVE-16280 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 2.2.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Minor > > REPL load from a table level dump always fails with exception > "SemanticException org.apache.hadoop.hive.ql.parse.SemanticException: > org.apache.hadoop.hive.ql.parse.SemanticException: Invalid path". > Either the dump path obtained from DUMP command was incorrect or dump didn't > work well. > Steps: > 1. REPL dump default.t1; > ---+ > dump_dir last_repl_id > ---+ > /tmp/dump/1487588522621 58 > ---+ > 2. REPL load repl.t2 from '/tmp/dump/1487588522621'; > Error: Error while compiling statement: FAILED: SemanticException > org.apache.hadoop.hive.ql.parse.SemanticException: > org.apache.hadoop.hive.ql.parse.SemanticException: Invalid path > (state=42000,code=4) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16218) Support view replication when REPL LOAD on a renamed database.
[ https://issues.apache.org/jira/browse/HIVE-16218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16218: Priority: Minor (was: Major) > Support view replication when REPL LOAD on a renamed database. > -- > > Key: HIVE-16218 > URL: https://issues.apache.org/jira/browse/HIVE-16218 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Minor > > If REPL LOAD is applied on a renamed database, then need to update the alias > query string (part of view original and expanded text in Table object) to > point to destination database name else the view will still run the query on > source database. > While creating the CreateViewDesc part of ImportDesc instantiation should > take care of this. > This ticket is a follow up of parent JIRA HIVE-15556. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16186) REPL DUMP shows last event ID of the database even if we use LIMIT option.
[ https://issues.apache.org/jira/browse/HIVE-16186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946627#comment-15946627 ] Sankar Hariappan commented on HIVE-16186: - Thanks [~vgumashta] for the commit! > REPL DUMP shows last event ID of the database even if we use LIMIT option. > -- > > Key: HIVE-16186 > URL: https://issues.apache.org/jira/browse/HIVE-16186 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR > Fix For: 2.2.0, 2.3.0 > > Attachments: HIVE-16186.01.patch, HIVE-16186.02.patch, > HIVE-16186.03.patch > > > Looks like LIMIT option doesn't work well with REPL DUMP. > 0: jdbc:hive2://localhost:10001/default> REPL DUMP default FROM 170 LIMIT 1; > +--+---+ > | dump_dir | last_repl_id | > +--+---+ > | /tmp/dump/1489395053411 | 195 | > +--+---+ -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16318) LLAP cache: address some issues in 2.2/2.3
[
https://issues.apache.org/jira/browse/HIVE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946625#comment-15946625
]
Hive QA commented on HIVE-16318:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860976/HIVE-16318.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10518 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=141)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=172)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4432/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4432/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4432/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860976 - PreCommit-HIVE-Build
> LLAP cache: address some issues in 2.2/2.3
> --
>
> Key: HIVE-16318
> URL: https://issues.apache.org/jira/browse/HIVE-16318
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16318.01.patch, HIVE-16318.02.patch,
> HIVE-16318.03.patch, HIVE-16318.04.patch, HIVE-16318.patch
>
>
> We've run into HIVE-16233 and HIVE-15665 and given that 2.2 and 2.3 releases
> are approaching we are going to add workarounds for them, and then commit the
> above patches and revert the workarounds as soon as we can.
> Unfortunately this will result in cache wasting some memory on some datasets,
> but the alternatives, when they are encountered (usually only on large
> datasets), are worse.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15434) Add UDF to allow interrogation of uniontype values
[
https://issues.apache.org/jira/browse/HIVE-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946578#comment-15946578
]
Ashutosh Chauhan commented on HIVE-15434:
-
+1
> Add UDF to allow interrogation of uniontype values
> --
>
> Key: HIVE-15434
> URL: https://issues.apache.org/jira/browse/HIVE-15434
> Project: Hive
> Issue Type: New Feature
> Components: UDF
>Affects Versions: 2.1.1
>Reporter: David Maughan
>Assignee: David Maughan
> Attachments: HIVE-15434.01.patch, HIVE-15434.02.patch
>
>
> h2. Overview
> As stated in the documention:
> {quote}
> UNIONTYPE support is incomplete The UNIONTYPE datatype was introduced in Hive
> 0.7.0 (HIVE-537), but full support for this type in Hive remains incomplete.
> Queries that reference UNIONTYPE fields in JOIN (HIVE-2508), WHERE, and GROUP
> BY clauses will fail, and Hive does not define syntax to extract the tag or
> value fields of a UNIONTYPE. This means that UNIONTYPEs are effectively
> look-at-only.
> {quote}
> It is essential to have a usable uniontype. Until full support is added to
> Hive users should at least have the ability to inspect and extract values for
> further comparison or transformation.
> h2. Proposal
> I propose to add a GenericUDF that has 2 modes of operation. Consider the
> following schema and data that contains a union:
> Schema:
> {code}
> struct>
> {code}
> Query:
> {code}
> hive> select field1 from thing;
> {0:0}
> {1:"one"}
> {code}
> h4. Explode to Struct
> This method will recursively convert all unions within the type to structs
> with fields named {{tag_n}}, {{n}} being the tag number. Only the {{tag_*}}
> field that matches the tag of the union will be populated with the value. In
> the case above the schema of field1 will be converted to:
> {code}
> struct
> {code}
> {code}
> hive> select extract_union(field1) from thing;
> {"tag_0":0,"tag_1":null}
> {"tag_0":null,"tag_1":one}
> {code}
> {code}
> hive> select extract_union(field1).tag_0 from thing;
> 0
> null
> {code}
> h4. Extract the specified tag
> This method will simply extract the value of the specified tag. If the tag
> number matches then the value is returned, if it does not, then null is
> returned.
> {code}
> hive> select extract_union(field1, 0) from thing;
> 0
> null
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-14455) upgrade httpclient, httpcore to match updated hadoop dependency
[ https://issues.apache.org/jira/browse/HIVE-14455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946572#comment-15946572 ] Ashutosh Chauhan commented on HIVE-14455: - +1 > upgrade httpclient, httpcore to match updated hadoop dependency > --- > > Key: HIVE-14455 > URL: https://issues.apache.org/jira/browse/HIVE-14455 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-14455.1.patch, HIVE-14455.1.patch > > > Hive was having a newer version of httpclient and httpcore since 1.2.0 > (HIVE-9709), when compared to Hadoop 2.x versions, to be able to make use of > newer apis in httpclient 4.4. > There was security issue in the older version of httpclient and httpcore > that hadoop was using, and as a result moved to httpclient 4.5.2 and > httpcore 4.4.4 (HADOOP-12767). > As hadoop was using the older version of these libraries and they often end > up earlier in the classpath, we have had bunch of difficulties in different > environments with class/method not found errors. > Now, that hadoops dependencies in versions with security fix are newer and > have the API that hive needs, we can be on the same version. For older > versions of hadoop this version update doesn't matter as the difference is > already there. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16239) remove useless hiveserver
[
https://issues.apache.org/jira/browse/HIVE-16239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946568#comment-15946568
]
Hive QA commented on HIVE-16239:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860968/HIVE-16239.2-branch-2.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 10477 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_null (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
(batchId=118)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_fast_stats
(batchId=89)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.createTable (batchId=847)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJarWithoutAddDriverClazz[0]
(batchId=287)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[0] (batchId=287)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[1] (batchId=287)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd
(batchId=488)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4431/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4431/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4431/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860968 - PreCommit-HIVE-Build
> remove useless hiveserver
> -
>
> Key: HIVE-16239
> URL: https://issues.apache.org/jira/browse/HIVE-16239
> Project: Hive
> Issue Type: Bug
> Components: CLI
>Affects Versions: 2.0.1, 2.1.1
>Reporter: Fei Hui
>Assignee: Fei Hui
> Attachments: HIVE-16239.1-branch-2.0.patch,
> HIVE-16239.1-branch-2.1.patch, HIVE-16239.2-branch-2.0.patch,
> HIVE-16239.2-branch-2.1.patch
>
>
> {quote}
> [hadoop@header hive]$ hive --service hiveserver
> Starting Hive Thrift Server
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/opt/apps/apache-hive-2.0.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/apps/spark-1.6.2-bin-hadoop2.7/lib/spark-assembly-1.6.2-hadoop2.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/apps/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Exception in thread "main" java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.service.HiveServer
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {quote}
> hiveserver does not exist, we should remove hiveserver from cli on branch-2.0
> After removing it, we get useful message
> {quote}
> Service hiveserver not found
> Available Services: beeline cli hbaseimport hbaseschematool help
> hiveburninclient hiveserver2 hplsql hwi jar lineage llap metastore metatool
> orcfiledump rcfilecat schemaTool version
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16302) Add junit dependency to hive-shims-common to compile with Hadoop 2.8+
[
https://issues.apache.org/jira/browse/HIVE-16302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946561#comment-15946561
]
Ashutosh Chauhan commented on HIVE-16302:
-
+1
> Add junit dependency to hive-shims-common to compile with Hadoop 2.8+
> -
>
> Key: HIVE-16302
> URL: https://issues.apache.org/jira/browse/HIVE-16302
> Project: Hive
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
> Attachments: HIVE-16302.01.patch
>
>
> Compile error when setting hadoop.version to 2.9.0-SNAPSHOT:
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:testCompile
> (default-testCompile) on project hive-shims-common: Compilation failure:
> Compilation failure:
> [ERROR]
> /Users/ajisaka/git/hive/shims/common/src/main/test/org/apache/hadoop/hive/io/TestHdfsUtils.java:[34,17]
> package org.junit does not exist
> [ERROR]
> /Users/ajisaka/git/hive/shims/common/src/main/test/org/apache/hadoop/hive/io/TestHdfsUtils.java
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16302) Add junit dependency to hive-shims-common to compile with Hadoop 2.8+
[
https://issues.apache.org/jira/browse/HIVE-16302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946562#comment-15946562
]
Ashutosh Chauhan commented on HIVE-16302:
-
+1
> Add junit dependency to hive-shims-common to compile with Hadoop 2.8+
> -
>
> Key: HIVE-16302
> URL: https://issues.apache.org/jira/browse/HIVE-16302
> Project: Hive
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
> Attachments: HIVE-16302.01.patch
>
>
> Compile error when setting hadoop.version to 2.9.0-SNAPSHOT:
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:testCompile
> (default-testCompile) on project hive-shims-common: Compilation failure:
> Compilation failure:
> [ERROR]
> /Users/ajisaka/git/hive/shims/common/src/main/test/org/apache/hadoop/hive/io/TestHdfsUtils.java:[34,17]
> package org.junit does not exist
> [ERROR]
> /Users/ajisaka/git/hive/shims/common/src/main/test/org/apache/hadoop/hive/io/TestHdfsUtils.java
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16325) Tez session refresh based on a time interval fails
[ https://issues.apache.org/jira/browse/HIVE-16325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-16325: -- Attachment: HIVE-16325.02.patch Updated patch. The SessionState only becomes available while starting the SessionPool. > Tez session refresh based on a time interval fails > -- > > Key: HIVE-16325 > URL: https://issues.apache.org/jira/browse/HIVE-16325 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Critical > Attachments: HIVE-16325.01.patch, HIVE-16325.02.patch > > > TezSessionSTate setup requires access to SessionState (which is a little > absurd). > The expire / renew threads are not setup with any SessionState. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HIVE-16317) CASE .. NULL in JOIN condition can trigger SemanticException
[
https://issues.apache.org/jira/browse/HIVE-16317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946554#comment-15946554
]
Remus Rusanu edited comment on HIVE-16317 at 3/29/17 5:37 AM:
--
The original query31 has a more complex where clause that gets pulled into the
JOIN by CBO and this cause CBO to fail:
{noformat}
explain with ss as (
select ca_county,d_qoy, d_year,sum(ss_ext_sales_price) as store_sales
from store_sales,date_dim,customer_address
where ss_sold_date_sk = d_date_sk and ss_addr_sk=ca_address_sk
group by ca_county,d_qoy, d_year),
ws as (
select ca_county,d_qoy, d_year,sum(ws_ext_sales_price) as web_sales
from web_sales,date_dim,customer_address
where ws_sold_date_sk = d_date_sk and ws_bill_addr_sk=ca_address_sk
group by ca_county,d_qoy, d_year)
select ss1.ca_county ,ss1.d_year ,ws2.web_sales/ws1.web_sales
web_q1_q2_increase ,ss2.store_sales/ss1.store_sales store_q1_q2_increase
,ws3.web_sales/ws2.web_sales web_q2_q3_increase
,ss3.store_sales/ss2.store_sales store_q2_q3_increase
from ss ss1 ,ss ss2 ,ss ss3 ,ws ws1 ,ws ws2 ,ws ws3
where ss1.d_qoy = 1 and
ss1.d_year = 1998 and
ss1.ca_county = ss2.ca_county and
ss2.d_qoy = 2 and
ss2.d_year = 1998 and
ss2.ca_county = ss3.ca_county and
ss3.d_qoy = 3 and
ss3.d_year = 1998 and
ss1.ca_county = ws1.ca_county and
ws1.d_qoy = 1 and
ws1.d_year = 1998 and
ws1.ca_county = ws2.ca_county and
ws2.d_qoy = 2 and
ws2.d_year = 1998 and
ws1.ca_county = ws3.ca_county and
ws3.d_qoy = 3 and
ws3.d_year =1998 and
case when ws1.web_sales > 0 then ws2.web_sales/ws1.web_sales else null end >
case when ss1.store_sales > 0 then ss2.store_sales/ss1.store_sales else
null end and
case when ws2.web_sales > 0 then ws3.web_sales/ws2.web_sales else null end >
case when ss2.store_sales > 0 then ss3.store_sales/ss2.store_sales else null
end
order by web_q1_q2_increase;
{noformat}
was (Author: rusanu):
The original query31 has a more complex where clause that gets pulled into the
JOIN:
{noformat}
explain with ss as (
select ca_county,d_qoy, d_year,sum(ss_ext_sales_price) as store_sales
from store_sales,date_dim,customer_address
where ss_sold_date_sk = d_date_sk and ss_addr_sk=ca_address_sk
group by ca_county,d_qoy, d_year),
ws as (
select ca_county,d_qoy, d_year,sum(ws_ext_sales_price) as web_sales
from web_sales,date_dim,customer_address
where ws_sold_date_sk = d_date_sk and ws_bill_addr_sk=ca_address_sk
group by ca_county,d_qoy, d_year)
select ss1.ca_county ,ss1.d_year ,ws2.web_sales/ws1.web_sales
web_q1_q2_increase ,ss2.store_sales/ss1.store_sales store_q1_q2_increase
,ws3.web_sales/ws2.web_sales web_q2_q3_increase
,ss3.store_sales/ss2.store_sales store_q2_q3_increase
from ss ss1 ,ss ss2 ,ss ss3 ,ws ws1 ,ws ws2 ,ws ws3
where ss1.d_qoy = 1 and
ss1.d_year = 1998 and
ss1.ca_county = ss2.ca_county and
ss2.d_qoy = 2 and
ss2.d_year = 1998 and
ss2.ca_county = ss3.ca_county and
ss3.d_qoy = 3 and
ss3.d_year = 1998 and
ss1.ca_county = ws1.ca_county and
ws1.d_qoy = 1 and
ws1.d_year = 1998 and
ws1.ca_county = ws2.ca_county and
ws2.d_qoy = 2 and
ws2.d_year = 1998 and
ws1.ca_county = ws3.ca_county and
ws3.d_qoy = 3 and
ws3.d_year =1998 and
case when ws1.web_sales > 0 then ws2.web_sales/ws1.web_sales else null end >
case when ss1.store_sales > 0 then ss2.store_sales/ss1.store_sales else
null end and
case when ws2.web_sales > 0 then ws3.web_sales/ws2.web_sales else null end >
case when ss2.store_sales > 0 then ss3.store_sales/ss2.store_sales else null
end
order by web_q1_q2_increase;
{noformat}
> CASE .. NULL in JOIN condition can trigger SemanticException
>
>
> Key: HIVE-16317
> URL: https://issues.apache.org/jira/browse/HIVE-16317
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
>
> {noformat}
> hive> explain select a.key from src a join src b on case when a.key=b.key
> then 1 else null end;
> FAILED: SemanticException tok_null encountered with 0 children
> {noformat}
> {noformat}
> "2489e686-bf69-4f37-b733-72491351644b main"
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondPopulateAlias(SemanticAnalyzer.java:2629)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondition(SemanticAnalyzer.java:2904)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondition(SemanticAnalyzer.java:2808)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genJoinTree(SemanticAnalyzer.java:8916)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10506)
> at
> org.apache
[jira] [Commented] (HIVE-16317) CASE .. NULL in JOIN condition can trigger SemanticException
[
https://issues.apache.org/jira/browse/HIVE-16317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946554#comment-15946554
]
Remus Rusanu commented on HIVE-16317:
-
The original query31 has a more complex where clause that gets pulled into the
JOIN:
{noformat}
explain with ss as (
select ca_county,d_qoy, d_year,sum(ss_ext_sales_price) as store_sales
from store_sales,date_dim,customer_address
where ss_sold_date_sk = d_date_sk and ss_addr_sk=ca_address_sk
group by ca_county,d_qoy, d_year),
ws as (
select ca_county,d_qoy, d_year,sum(ws_ext_sales_price) as web_sales
from web_sales,date_dim,customer_address
where ws_sold_date_sk = d_date_sk and ws_bill_addr_sk=ca_address_sk
group by ca_county,d_qoy, d_year)
select ss1.ca_county ,ss1.d_year ,ws2.web_sales/ws1.web_sales
web_q1_q2_increase ,ss2.store_sales/ss1.store_sales store_q1_q2_increase
,ws3.web_sales/ws2.web_sales web_q2_q3_increase
,ss3.store_sales/ss2.store_sales store_q2_q3_increase
from ss ss1 ,ss ss2 ,ss ss3 ,ws ws1 ,ws ws2 ,ws ws3
where ss1.d_qoy = 1 and
ss1.d_year = 1998 and
ss1.ca_county = ss2.ca_county and
ss2.d_qoy = 2 and
ss2.d_year = 1998 and
ss2.ca_county = ss3.ca_county and
ss3.d_qoy = 3 and
ss3.d_year = 1998 and
ss1.ca_county = ws1.ca_county and
ws1.d_qoy = 1 and
ws1.d_year = 1998 and
ws1.ca_county = ws2.ca_county and
ws2.d_qoy = 2 and
ws2.d_year = 1998 and
ws1.ca_county = ws3.ca_county and
ws3.d_qoy = 3 and
ws3.d_year =1998 and
case when ws1.web_sales > 0 then ws2.web_sales/ws1.web_sales else null end >
case when ss1.store_sales > 0 then ss2.store_sales/ss1.store_sales else
null end and
case when ws2.web_sales > 0 then ws3.web_sales/ws2.web_sales else null end >
case when ss2.store_sales > 0 then ss3.store_sales/ss2.store_sales else null
end
order by web_q1_q2_increase;
{noformat}
> CASE .. NULL in JOIN condition can trigger SemanticException
>
>
> Key: HIVE-16317
> URL: https://issues.apache.org/jira/browse/HIVE-16317
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
>
> {noformat}
> hive> explain select a.key from src a join src b on case when a.key=b.key
> then 1 else null end;
> FAILED: SemanticException tok_null encountered with 0 children
> {noformat}
> {noformat}
> "2489e686-bf69-4f37-b733-72491351644b main"
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondPopulateAlias(SemanticAnalyzer.java:2629)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondition(SemanticAnalyzer.java:2904)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondition(SemanticAnalyzer.java:2808)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genJoinTree(SemanticAnalyzer.java:8916)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10506)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10419)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:408)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:165)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:511)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1316)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1456)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1236)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1226)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
[jira] [Commented] (HIVE-16288) Add blobstore tests for ORC and RCFILE file formats
[ https://issues.apache.org/jira/browse/HIVE-16288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946552#comment-15946552 ] Ashutosh Chauhan commented on HIVE-16288: - +1 > Add blobstore tests for ORC and RCFILE file formats > --- > > Key: HIVE-16288 > URL: https://issues.apache.org/jira/browse/HIVE-16288 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Attachments: HIVE-16288.patch > > > This patch adds four tests each for ORC and RCFILE when running against > blobstore filesystems: > * Test for bucketed tables > * Test for nonpartitioned tables > * Test for partitioned tables > * Test for partitioned tables with nonstandard partition locations -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16315) Describe table doesn't show num of partitions
[ https://issues.apache.org/jira/browse/HIVE-16315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946538#comment-15946538 ] Ashutosh Chauhan commented on HIVE-16315: - seems like qfile needs updating. Also, can you create a RB with updated golden files. > Describe table doesn't show num of partitions > - > > Key: HIVE-16315 > URL: https://issues.apache.org/jira/browse/HIVE-16315 > Project: Hive > Issue Type: Bug >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-16315.1.patch > > > This doesn't comply with our wiki: > https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-Examples -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16308) PreExecutePrinter and PostExecutePrinter should log to INFO level instead of ERROR
[ https://issues.apache.org/jira/browse/HIVE-16308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946533#comment-15946533 ] Ashutosh Chauhan commented on HIVE-16308: - cc: [~prasanth_j] > PreExecutePrinter and PostExecutePrinter should log to INFO level instead of > ERROR > -- > > Key: HIVE-16308 > URL: https://issues.apache.org/jira/browse/HIVE-16308 > Project: Hive > Issue Type: Bug > Components: Test >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-16308.1.patch > > > Many of the pre and post hook printers log info at the ERROR level, which is > confusing since they aren't errors. They should log to the INFO level. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16309) Hive Test Commands failure should be printed in hive.log in addition to stderr
[
https://issues.apache.org/jira/browse/HIVE-16309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946529#comment-15946529
]
Ashutosh Chauhan commented on HIVE-16309:
-
+1
> Hive Test Commands failure should be printed in hive.log in addition to stderr
> --
>
> Key: HIVE-16309
> URL: https://issues.apache.org/jira/browse/HIVE-16309
> Project: Hive
> Issue Type: Bug
> Components: Test
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16309.1.patch
>
>
> When the {{QTestUtil.executeTestCommand}} runs a test only command, and the
> command fails, the exception info is printed to stdout rather than the log
> file. Printing out to the logs file will better help debuggability of these
> failures.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[ https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946521#comment-15946521 ] Gabriel Zimmerman commented on HIVE-4605: - I am seeing the same issue and maybe this sheds some light. Using Hive version 1.0.0 and below (haven't tried later versions) on EMR, this error happens consistently. The table uses its own storage handler but when I use "create external table" to create the table, somehow Hive thinks the table is native, which means that it runs commit on the FS paths causing the error (see stack trace above). Removing external fixed the issue. Maybe overwrite in the above case had a similar result? > Hive job fails while closing reducer output - Unable to rename > -- > > Key: HIVE-4605 > URL: https://issues.apache.org/jira/browse/HIVE-4605 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1 > Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010 > x86_64 x86_64 x86_64 GNU/Linux > Hadoop 1.1.2 >Reporter: Link Qian >Assignee: Brock Noland > Attachments: HIVE-4605.patch > > > 1, create a table with ORC storage model > create table iparea_analysis_orc (network int, ip string, ) > stored as ORC; > 2, insert table iparea_analysis_orc select network, ip, , the script > success, but failed after add *OVERWRITE* keyword. the main error log list > as here. > ava.lang.RuntimeException: Hive Runtime Error while closing operators: Unable > to rename output from: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 > to: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 > at > org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename > output from: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 > to: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at > org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309) > ... 7 more -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15269) Dynamic Min-Max/BloomFilter runtime-filtering for Tez
[
https://issues.apache.org/jira/browse/HIVE-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946520#comment-15946520
]
Carter Shanklin commented on HIVE-15269:
There are a few settings here that should be documented
hive.tez.dynamic.semijoin.reduction
hive.tez.dynamic.semijoin.reduction.threshold
hive.tez.bigtable.minsize.semijoin.reduction
Should this ticket get opened back up for a TODOC label?
> Dynamic Min-Max/BloomFilter runtime-filtering for Tez
> -
>
> Key: HIVE-15269
> URL: https://issues.apache.org/jira/browse/HIVE-15269
> Project: Hive
> Issue Type: New Feature
> Components: Tez
>Reporter: Jason Dere
>Assignee: Deepak Jaiswal
> Fix For: 2.2.0
>
> Attachments: HIVE-15269.10.patch, HIVE-15269.11.patch,
> HIVE-15269.12.patch, HIVE-15269.13.patch, HIVE-15269.14.patch,
> HIVE-15269.15.patch, HIVE-15269.16.patch, HIVE-15269.17.patch,
> HIVE-15269.18.patch, HIVE-15269.19.patch, HIVE-15269.1.patch,
> HIVE-15269.2.patch, HIVE-15269.3.patch, HIVE-15269.4.patch,
> HIVE-15269.5.patch, HIVE-15269.6.patch, HIVE-15269.7.patch,
> HIVE-15269.8.patch, HIVE-15269.9.patch
>
>
> If a dimension table and fact table are joined:
> {noformat}
> select *
> from store join store_sales on (store.id = store_sales.store_id)
> where store.s_store_name = 'My Store'
> {noformat}
> One optimization that can be done is to get the min/max store id values that
> come out of the scan/filter of the store table, and send this min/max value
> (via Tez edge) to the task which is scanning the store_sales table.
> We can add a BETWEEN(min, max) predicate to the store_sales TableScan, where
> this predicate can be pushed down to the storage handler (for example for ORC
> formats). Pushing a min/max predicate to the ORC reader would allow us to
> avoid having to entire whole row groups during the table scan.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16209) Vectorization: Add support for complex types to VectorExtractRow and VectorAssignRow
[
https://issues.apache.org/jira/browse/HIVE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946507#comment-15946507
]
Hive QA commented on HIVE-16209:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860966/HIVE-16209.3.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10518 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_casts]
(batchId=76)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_part]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_part_all_primitive]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_table]
(batchId=141)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_casts]
(batchId=155)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=172)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4430/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4430/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4430/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860966 - PreCommit-HIVE-Build
> Vectorization: Add support for complex types to VectorExtractRow and
> VectorAssignRow
>
>
> Key: HIVE-16209
> URL: https://issues.apache.org/jira/browse/HIVE-16209
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Matt McCline
>Assignee: Teddy Choi
>Priority: Critical
> Attachments: HIVE-16209.1.patch, HIVE-16209.3.patch
>
>
> Supports complex types in non-native VectorReduceSink, row mode Text
> Vectorization, and some cases of Vectorized Schema Evolution.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16317) CASE .. NULL in JOIN condition can trigger SemanticException
[
https://issues.apache.org/jira/browse/HIVE-16317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946494#comment-15946494
]
Vineet Garg commented on HIVE-16317:
{{SemanticAnalyzer.parseJoinCondPopulateAlias}} doesn't handle {{TOK_NULL}} and
eventually results into an exception. If we add a case for {{TOK_NULL}} we
eventually run into runtime error
{noformat}
Vertex failed, vertexName=Reducer 2, vertexId=vertex_1490758435339_0001_1_02,
diagnostics=[Task failed, taskId=task_1490758435339_0001_1_02_00,
diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
failure ) :
attempt_1490758435339_0001_1_02_00_0:java.lang.RuntimeException:
java.lang.RuntimeException: Hive Runtime Error while closing operators:
java.lang.Integer cannot be cast to java.lang.Boolean
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Hive Runtime Error while closing
operators: java.lang.Integer cannot be cast to java.lang.Boolean
{noformat}
If I try simplified query {code:SQL} select a.key from src a join src b on
2{code} I see the following error with MR
{noformat}
java.lang.Exception: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {"key":"238","value":"val_238"}
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
~[hadoop-mapreduce-client-common-2.7.2.jar:?]
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
[hadoop-mapreduce-client-common-2.7.2.jar:?]
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {"key":"238","value":"val_238"}
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:169)
~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
~[hadoop-mapreduce-client-core-2.7.2.jar:?]
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
~[hadoop-mapreduce-client-core-2.7.2.jar:?]
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
~[hadoop-mapreduce-client-core-2.7.2.jar:?]
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
~[hadoop-mapreduce-client-common-2.7.2.jar:?]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_101]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[?:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[?:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_101]
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row {"key":"238","value":"val_238"}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:562)
~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:160)
~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
~[hadoop-mapreduce-client-core-2.7.2.jar:?]
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
~[hadoop-mapreduce-client-core-2.7.2.jar:?]
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
~[hadoop-mapreduce-client-core-2.7.2.jar:?]
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.
[jira] [Updated] (HIVE-14165) Remove Hive file listing during split computation
[ https://issues.apache.org/jira/browse/HIVE-14165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-14165: Status: Open (was: Patch Available) > Remove Hive file listing during split computation > - > > Key: HIVE-14165 > URL: https://issues.apache.org/jira/browse/HIVE-14165 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.1.0 >Reporter: Abdullah Yousufi >Assignee: Sahil Takiar > Attachments: HIVE-14165.02.patch, HIVE-14165.03.patch, > HIVE-14165.04.patch, HIVE-14165.05.patch, HIVE-14165.06.patch, > HIVE-14165.07.patch, HIVE-14165.patch > > > The Hive side listing in FetchOperator.java is unnecessary, since Hadoop's > FileInputFormat.java will list the files during split computation anyway to > determine their size. One way to remove this is to catch the > InvalidInputFormat exception thrown by FileInputFormat#getSplits() on the > Hive side instead of doing the file listing beforehand. > For S3 select queries on partitioned tables, this results in a 2x speedup. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15396) Basic Stats are not collected when for managed tables with LOCATION specified
[
https://issues.apache.org/jira/browse/HIVE-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-15396:
Attachment: HIVE-15396.3.patch
> Basic Stats are not collected when for managed tables with LOCATION specified
> -
>
> Key: HIVE-15396
> URL: https://issues.apache.org/jira/browse/HIVE-15396
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-15396.1.patch, HIVE-15396.2.patch,
> HIVE-15396.3.patch
>
>
> Basic stats are not collected when a managed table is created with a
> specified {{LOCATION}} clause.
> {code}
> 0: jdbc:hive2://localhost:1> create table hdfs_1 (col int);
> 0: jdbc:hive2://localhost:1> describe formatted hdfs_1;
> +---++-+
> | col_name| data_type
>| comment |
> +---++-+
> | # col_name| data_type
>| comment |
> | | NULL
>| NULL|
> | col | int
>| |
> | | NULL
>| NULL|
> | # Detailed Table Information | NULL
>| NULL|
> | Database: | default
>| NULL|
> | Owner:| anonymous
>| NULL|
> | CreateTime: | Wed Mar 22 18:09:19 PDT 2017
>| NULL|
> | LastAccessTime: | UNKNOWN
>| NULL|
> | Retention:| 0
>| NULL|
> | Location: | file:/warehouse/hdfs_1 | NULL
> |
> | Table Type: | MANAGED_TABLE
>| NULL|
> | Table Parameters: | NULL
>| NULL|
> | | COLUMN_STATS_ACCURATE
>| {\"BASIC_STATS\":\"true\"} |
> | | numFiles
>| 0 |
> | | numRows
>| 0 |
> | | rawDataSize
>| 0 |
> | | totalSize
>| 0 |
> | | transient_lastDdlTime
>| 1490231359 |
> | | NULL
>| NULL|
> | # Storage Information | NULL
>| NULL|
> | SerDe Library:|
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | NULL
>|
> | InputFormat: | org.apache.hadoop.mapred.TextInputFormat
>| NULL|
> | OutputFormat: |
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | NULL
>|
> | Compressed: | No
>| NULL|
> | Num Buckets: | -1
>| NULL|
> | Bucket Columns: | []
>| NULL|
> | Sort Columns: | []
>| NULL|
> | Storage Desc Params: | NULL
>| NULL|
> | | serialization.format
>| 1 |
> +
[jira] [Commented] (HIVE-16323) HS2 JDOPersistenceManagerFactory.pmCache leaks after HIVE-14204
[
https://issues.apache.org/jira/browse/HIVE-16323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946466#comment-15946466
]
Hive QA commented on HIVE-16323:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860955/PM_leak.png
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4429/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4429/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4429/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-03-29 03:25:34.444
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4429/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-03-29 03:25:34.446
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 5814c11 HIVE-16319. LLAP: Better handling of an empty wait
queue, should try scheduling checks. (Siddharth Seth, reviewed by Prasanth
Jayachandran)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 5814c11 HIVE-16319. LLAP: Better handling of an empty wait
queue, should try scheduling checks. (Siddharth Seth, reviewed by Prasanth
Jayachandran)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-03-29 03:25:35.297
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
fatal: unrecognized input
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860955 - PreCommit-HIVE-Build
> HS2 JDOPersistenceManagerFactory.pmCache leaks after HIVE-14204
> ---
>
> Key: HIVE-16323
> URL: https://issues.apache.org/jira/browse/HIVE-16323
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-16323.1.patch, PM_leak.png
>
>
> Hive.loadDynamicPartitions creates threads with new embedded rawstore, but
> never close them, thus we leak PersistenceManager one per such thread.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16323) HS2 JDOPersistenceManagerFactory.pmCache leaks after HIVE-14204
[
https://issues.apache.org/jira/browse/HIVE-16323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946465#comment-15946465
]
Hive QA commented on HIVE-16323:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860955/PM_leak.png
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4428/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4428/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4428/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-03-29 03:24:57.534
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4428/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-03-29 03:24:57.537
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 5814c11 HIVE-16319. LLAP: Better handling of an empty wait
queue, should try scheduling checks. (Siddharth Seth, reviewed by Prasanth
Jayachandran)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 5814c11 HIVE-16319. LLAP: Better handling of an empty wait
queue, should try scheduling checks. (Siddharth Seth, reviewed by Prasanth
Jayachandran)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-03-29 03:24:58.562
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
fatal: unrecognized input
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860955 - PreCommit-HIVE-Build
> HS2 JDOPersistenceManagerFactory.pmCache leaks after HIVE-14204
> ---
>
> Key: HIVE-16323
> URL: https://issues.apache.org/jira/browse/HIVE-16323
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-16323.1.patch, PM_leak.png
>
>
> Hive.loadDynamicPartitions creates threads with new embedded rawstore, but
> never close them, thus we leak PersistenceManager one per such thread.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16318) LLAP cache: address some issues in 2.2/2.3
[
https://issues.apache.org/jira/browse/HIVE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946464#comment-15946464
]
Hive QA commented on HIVE-16318:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860976/HIVE-16318.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10518 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=141)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=172)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4427/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4427/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4427/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860976 - PreCommit-HIVE-Build
> LLAP cache: address some issues in 2.2/2.3
> --
>
> Key: HIVE-16318
> URL: https://issues.apache.org/jira/browse/HIVE-16318
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16318.01.patch, HIVE-16318.02.patch,
> HIVE-16318.03.patch, HIVE-16318.04.patch, HIVE-16318.patch
>
>
> We've run into HIVE-16233 and HIVE-15665 and given that 2.2 and 2.3 releases
> are approaching we are going to add workarounds for them, and then commit the
> above patches and revert the workarounds as soon as we can.
> Unfortunately this will result in cache wasting some memory on some datasets,
> but the alternatives, when they are encountered (usually only on large
> datasets), are worse.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16228) Support subqueries in complex expression in SELECT clause
[
https://issues.apache.org/jira/browse/HIVE-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946432#comment-15946432
]
Hive QA commented on HIVE-16228:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860943/HIVE-16091.2.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10518 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=172)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4425/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4425/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4425/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860943 - PreCommit-HIVE-Build
> Support subqueries in complex expression in SELECT clause
> -
>
> Key: HIVE-16228
> URL: https://issues.apache.org/jira/browse/HIVE-16228
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-16091.2.patch, HIVE-16228.1.patch
>
>
> HIVE-16091 added support for subqueries in SELECT clause but restricted
> subqueries to top level expressions (more detail is at [LINK |
> https://cwiki.apache.org/confluence/display/Hive/Subqueries+in+SELECT])
> This restriction will be relaxed to allow subqueries in all kind of
> expression except UDAF (including UDAs and UDTFs).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16325) Tez session refresh based on a time interval fails
[ https://issues.apache.org/jira/browse/HIVE-16325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946403#comment-15946403 ] Siddharth Seth commented on HIVE-16325: --- Don't think the pool works when doAs is set to true? > Tez session refresh based on a time interval fails > -- > > Key: HIVE-16325 > URL: https://issues.apache.org/jira/browse/HIVE-16325 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Critical > Attachments: HIVE-16325.01.patch > > > TezSessionSTate setup requires access to SessionState (which is a little > absurd). > The expire / renew threads are not setup with any SessionState. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HIVE-16325) Tez session refresh based on a time interval fails
[ https://issues.apache.org/jira/browse/HIVE-16325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946403#comment-15946403 ] Siddharth Seth edited comment on HIVE-16325 at 3/29/17 2:01 AM: Don't think the pool works when doAs is set to true? This is also equivalent to how all the initial sessions are created, so I think it is safe. was (Author: sseth): Don't think the pool works when doAs is set to true? > Tez session refresh based on a time interval fails > -- > > Key: HIVE-16325 > URL: https://issues.apache.org/jira/browse/HIVE-16325 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Critical > Attachments: HIVE-16325.01.patch > > > TezSessionSTate setup requires access to SessionState (which is a little > absurd). > The expire / renew threads are not setup with any SessionState. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16325) Tez session refresh based on a time interval fails
[ https://issues.apache.org/jira/browse/HIVE-16325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-16325: -- Status: Patch Available (was: Open) > Tez session refresh based on a time interval fails > -- > > Key: HIVE-16325 > URL: https://issues.apache.org/jira/browse/HIVE-16325 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Critical > Attachments: HIVE-16325.01.patch > > > TezSessionSTate setup requires access to SessionState (which is a little > absurd). > The expire / renew threads are not setup with any SessionState. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16325) Tez session refresh based on a time interval fails
[ https://issues.apache.org/jira/browse/HIVE-16325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946399#comment-15946399 ] Sergey Shelukhin commented on HIVE-16325: - Hmm.. For the the acls/user case, is it valid to use single session for all the restarts? If yes +1 > Tez session refresh based on a time interval fails > -- > > Key: HIVE-16325 > URL: https://issues.apache.org/jira/browse/HIVE-16325 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Critical > Attachments: HIVE-16325.01.patch > > > TezSessionSTate setup requires access to SessionState (which is a little > absurd). > The expire / renew threads are not setup with any SessionState. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Work started] (HIVE-16325) Tez session refresh based on a time interval fails
[ https://issues.apache.org/jira/browse/HIVE-16325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-16325 started by Siddharth Seth. - > Tez session refresh based on a time interval fails > -- > > Key: HIVE-16325 > URL: https://issues.apache.org/jira/browse/HIVE-16325 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Critical > Attachments: HIVE-16325.01.patch > > > TezSessionSTate setup requires access to SessionState (which is a little > absurd). > The expire / renew threads are not setup with any SessionState. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Work stopped] (HIVE-16325) Tez session refresh based on a time interval fails
[ https://issues.apache.org/jira/browse/HIVE-16325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-16325 stopped by Siddharth Seth. - > Tez session refresh based on a time interval fails > -- > > Key: HIVE-16325 > URL: https://issues.apache.org/jira/browse/HIVE-16325 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Critical > Attachments: HIVE-16325.01.patch > > > TezSessionSTate setup requires access to SessionState (which is a little > absurd). > The expire / renew threads are not setup with any SessionState. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16325) Tez session refresh based on a time interval fails
[ https://issues.apache.org/jira/browse/HIVE-16325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-16325: -- Attachment: HIVE-16325.01.patch [~sershe] - can you please take a look. > Tez session refresh based on a time interval fails > -- > > Key: HIVE-16325 > URL: https://issues.apache.org/jira/browse/HIVE-16325 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Critical > Attachments: HIVE-16325.01.patch > > > TezSessionSTate setup requires access to SessionState (which is a little > absurd). > The expire / renew threads are not setup with any SessionState. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16325) Tez session refresh based on a time interval fails
[ https://issues.apache.org/jira/browse/HIVE-16325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth reassigned HIVE-16325: - > Tez session refresh based on a time interval fails > -- > > Key: HIVE-16325 > URL: https://issues.apache.org/jira/browse/HIVE-16325 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Critical > > TezSessionSTate setup requires access to SessionState (which is a little > absurd). > The expire / renew threads are not setup with any SessionState. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16318) LLAP cache: address some issues in 2.2/2.3
[
https://issues.apache.org/jira/browse/HIVE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946372#comment-15946372
]
Hive QA commented on HIVE-16318:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860972/HIVE-16318.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10518 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=231)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=141)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=172)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4424/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4424/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4424/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860972 - PreCommit-HIVE-Build
> LLAP cache: address some issues in 2.2/2.3
> --
>
> Key: HIVE-16318
> URL: https://issues.apache.org/jira/browse/HIVE-16318
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16318.01.patch, HIVE-16318.02.patch,
> HIVE-16318.03.patch, HIVE-16318.04.patch, HIVE-16318.patch
>
>
> We've run into HIVE-16233 and HIVE-15665 and given that 2.2 and 2.3 releases
> are approaching we are going to add workarounds for them, and then commit the
> above patches and revert the workarounds as soon as we can.
> Unfortunately this will result in cache wasting some memory on some datasets,
> but the alternatives, when they are encountered (usually only on large
> datasets), are worse.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16318) LLAP cache: address some issues in 2.2/2.3
[ https://issues.apache.org/jira/browse/HIVE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946359#comment-15946359 ] Siddharth Seth commented on HIVE-16318: --- +1. Thanks > LLAP cache: address some issues in 2.2/2.3 > -- > > Key: HIVE-16318 > URL: https://issues.apache.org/jira/browse/HIVE-16318 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16318.01.patch, HIVE-16318.02.patch, > HIVE-16318.03.patch, HIVE-16318.04.patch, HIVE-16318.patch > > > We've run into HIVE-16233 and HIVE-15665 and given that 2.2 and 2.3 releases > are approaching we are going to add workarounds for them, and then commit the > above patches and revert the workarounds as soon as we can. > Unfortunately this will result in cache wasting some memory on some datasets, > but the alternatives, when they are encountered (usually only on large > datasets), are worse. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16324) Truncate table should not work when EXTERNAL property of table is true
[
https://issues.apache.org/jira/browse/HIVE-16324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-16324:
---
Description:
Currently if you create an external table using the command {{CREATE EXTERNAL
TABLE table_name}} the {{TRUNCATE table table_name}} command fails as expected
because only managed tables should be allowed to be truncated.
But if you set the external property of a previously managed table using
{{ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='true')}}, truncate table
command does not object and deletes all the data from the external table.
Eg: This works but it should not ..
{noformat}
0: jdbc:hive2://localhost:1/default> create table test_ext2 (col1 string);
No rows affected (0.424 seconds)
0: jdbc:hive2://localhost:1/default> alter table test_ext2 set
tblproperties ('EXTERNAL'='true');
No rows affected (0.149 seconds)
0: jdbc:hive2://localhost:1/default> insert into table test_ext2 values
("test");
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
future versions. Consider using a different execution engine (i.e. spark, tez)
or using Hive 1.X releases.
No rows affected (3.447 seconds)
0: jdbc:hive2://localhost:1/default>
0: jdbc:hive2://localhost:1/default>
0: jdbc:hive2://localhost:1/default> select * from test_ext2;
+-+
| test_ext2.col1 |
+-+
| test|
+-+
1 row selected (0.147 seconds)
0: jdbc:hive2://localhost:1/default> truncate table test_ext2;
No rows affected (0.138 seconds)
0: jdbc:hive2://localhost:1/default> select * from test_ext2;
+-+
| test_ext2.col1 |
+-+
+-+
No rows selected (0.134 seconds)
0: jdbc:hive2://localhost:1/default>
{noformat}
was:
Currently if you create an external table using the command {{CREATE EXTERNAL
TABLE table_name}} the {{TRUNCATE table table_name}} command fails as expected
because only managed tables should be allowed to be truncated.
But if you set the external property of a previously managed table using
{{ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='true')}}, truncate table
command does not object and deletes all the data from the external table.
Eg: This works but it should now ..
{noformat}
0: jdbc:hive2://localhost:1/default> create table test_ext2 (col1 string);
No rows affected (0.424 seconds)
0: jdbc:hive2://localhost:1/default> alter table test_ext2 set
tblproperties ('EXTERNAL'='true');
No rows affected (0.149 seconds)
0: jdbc:hive2://localhost:1/default> insert into table test_ext2 values
("test");
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
future versions. Consider using a different execution engine (i.e. spark, tez)
or using Hive 1.X releases.
No rows affected (3.447 seconds)
0: jdbc:hive2://localhost:1/default>
0: jdbc:hive2://localhost:1/default>
0: jdbc:hive2://localhost:1/default> select * from test_ext2;
+-+
| test_ext2.col1 |
+-+
| test|
+-+
1 row selected (0.147 seconds)
0: jdbc:hive2://localhost:1/default> truncate table test_ext2;
No rows affected (0.138 seconds)
0: jdbc:hive2://localhost:1/default> select * from test_ext2;
+-+
| test_ext2.col1 |
+-+
+-+
No rows selected (0.134 seconds)
0: jdbc:hive2://localhost:1/default>
{noformat}
> Truncate table should not work when EXTERNAL property of table is true
> --
>
> Key: HIVE-16324
> URL: https://issues.apache.org/jira/browse/HIVE-16324
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>
> Currently if you create an external table using the command {{CREATE EXTERNAL
> TABLE table_name}} the {{TRUNCATE table table_name}} command fails as
> expected because only managed tables should be allowed to be truncated.
> But if you set the external property of a previously managed table using
> {{ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='true')}}, truncate
> table command does not object and deletes all the data from the external
> table.
> Eg: This works but it should not ..
> {noformat}
> 0: jdbc:hive2://localhost:1/default> create table test_ext2 (col1 string);
> No rows affected (0.424 seconds)
> 0: jdbc:hive2://localhost:1/default> alter table test_ext2 set
> tblproperties ('EXTERNAL'='true');
> No rows affected (0.149 seconds)
> 0: jdbc:hive2://localhost:1/default> insert into table test_ext2 values
> ("test");
> WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
> future versions. Consider using a different execution engine (i.e. spark,
> tez) or using
[jira] [Updated] (HIVE-16324) Truncate table should not work when EXTERNAL property of table is true
[
https://issues.apache.org/jira/browse/HIVE-16324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-16324:
---
Description:
Currently if you create an external table using the command {{CREATE EXTERNAL
TABLE table_name}} the {{TRUNCATE table table_name}} command fails as expected
because only managed tables should be allowed to be truncated.
But if you set the external property of a previously managed table using
{{ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='true')}}, truncate table
command does not object and deletes all the data from the external table.
Eg: This works but it should now ..
{noformat}
0: jdbc:hive2://localhost:1/default> create table test_ext2 (col1 string);
No rows affected (0.424 seconds)
0: jdbc:hive2://localhost:1/default> alter table test_ext2 set
tblproperties ('EXTERNAL'='true');
No rows affected (0.149 seconds)
0: jdbc:hive2://localhost:1/default> insert into table test_ext2 values
("test");
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
future versions. Consider using a different execution engine (i.e. spark, tez)
or using Hive 1.X releases.
No rows affected (3.447 seconds)
0: jdbc:hive2://localhost:1/default>
0: jdbc:hive2://localhost:1/default>
0: jdbc:hive2://localhost:1/default> select * from test_ext2;
+-+
| test_ext2.col1 |
+-+
| test|
+-+
1 row selected (0.147 seconds)
0: jdbc:hive2://localhost:1/default> truncate table test_ext2;
No rows affected (0.138 seconds)
0: jdbc:hive2://localhost:1/default> select * from test_ext2;
+-+
| test_ext2.col1 |
+-+
+-+
No rows selected (0.134 seconds)
0: jdbc:hive2://localhost:1/default>
{noformat}
was:
Currently if you create an external table using the command {{CREATE EXTERNAL
TABLE table_name}} the {{TRUNCATE table table_name}} command fails as expected
because only managed tables should be allowed to be truncated.
But if you set the external property of a previously managed table using
{{ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='true')}}, truncate table
command does not object and deletes all the data from the external table.
> Truncate table should not work when EXTERNAL property of table is true
> --
>
> Key: HIVE-16324
> URL: https://issues.apache.org/jira/browse/HIVE-16324
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>
> Currently if you create an external table using the command {{CREATE EXTERNAL
> TABLE table_name}} the {{TRUNCATE table table_name}} command fails as
> expected because only managed tables should be allowed to be truncated.
> But if you set the external property of a previously managed table using
> {{ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='true')}}, truncate
> table command does not object and deletes all the data from the external
> table.
> Eg: This works but it should now ..
> {noformat}
> 0: jdbc:hive2://localhost:1/default> create table test_ext2 (col1 string);
> No rows affected (0.424 seconds)
> 0: jdbc:hive2://localhost:1/default> alter table test_ext2 set
> tblproperties ('EXTERNAL'='true');
> No rows affected (0.149 seconds)
> 0: jdbc:hive2://localhost:1/default> insert into table test_ext2 values
> ("test");
> WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
> future versions. Consider using a different execution engine (i.e. spark,
> tez) or using Hive 1.X releases.
> No rows affected (3.447 seconds)
> 0: jdbc:hive2://localhost:1/default>
> 0: jdbc:hive2://localhost:1/default>
> 0: jdbc:hive2://localhost:1/default> select * from test_ext2;
> +-+
> | test_ext2.col1 |
> +-+
> | test|
> +-+
> 1 row selected (0.147 seconds)
> 0: jdbc:hive2://localhost:1/default> truncate table test_ext2;
> No rows affected (0.138 seconds)
> 0: jdbc:hive2://localhost:1/default> select * from test_ext2;
> +-+
> | test_ext2.col1 |
> +-+
> +-+
> No rows selected (0.134 seconds)
> 0: jdbc:hive2://localhost:1/default>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16318) LLAP cache: address some issues in 2.2/2.3
[ https://issues.apache.org/jira/browse/HIVE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-16318: Attachment: HIVE-16318.04.patch Taking out the extra headroom > LLAP cache: address some issues in 2.2/2.3 > -- > > Key: HIVE-16318 > URL: https://issues.apache.org/jira/browse/HIVE-16318 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16318.01.patch, HIVE-16318.02.patch, > HIVE-16318.03.patch, HIVE-16318.04.patch, HIVE-16318.patch > > > We've run into HIVE-16233 and HIVE-15665 and given that 2.2 and 2.3 releases > are approaching we are going to add workarounds for them, and then commit the > above patches and revert the workarounds as soon as we can. > Unfortunately this will result in cache wasting some memory on some datasets, > but the alternatives, when they are encountered (usually only on large > datasets), are worse. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15880) Allow insert overwrite and truncate table query to use auto.purge table property
[
https://issues.apache.org/jira/browse/HIVE-15880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946351#comment-15946351
]
Vihang Karajgaonkar commented on HIVE-15880:
Hi [~ctang.ma] Thanks for the review. I changed the implementation of
{{HiveMetaStoreFsImpl.deleteDir}} so that it reuses FileUtils.moveToTrash
method like you suggested.
Also investigated encrypted tables, I realized that in case of encrypted table
the {{INSERT OVERWRITE TABLE encrypted_table ...}} command was anyways ignoring
the error thrown when old data was moved to Trash. It logs a warning and then
deletes the data when moving to Trash to fails. With my patch there would be no
behavioral change with respect to encrypted tables. With the patch, if the
table is encrypted and auto.purge is true, then it doesn't attempt to move the
data to trash in the first place, so the warning would not be logged as well.
Similarly, for truncate table. Both the commands use FileUtils.moveToTrash in
the backend to delete the data which catches the exception when table is
encrypted and simply logs the warning. So with the patch both there would not
be any change with respect to encrypted tables for both INSERT OVERWRITE and
TRUNCATE commands since they were already ignoring the Trash.
This would still be more efficient when the tables are located on S3 since the
code does not attempt to move them to trash as well.
Also, found another unrelated bug during this investigation regarding the
inconsistency with respect to "EXTERNAL" property and EXTERNAL table. Created
HIVE-16324 for that.
Updating the patch which includes additional q file test and modifies the
existing truncate table tests to work around HIVE-16324
> Allow insert overwrite and truncate table query to use auto.purge table
> property
>
>
> Key: HIVE-15880
> URL: https://issues.apache.org/jira/browse/HIVE-15880
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-15880.01.patch, HIVE-15880.02.patch,
> HIVE-15880.03.patch, HIVE-15880.04.patch, HIVE-15880.05.patch
>
>
> It seems inconsistent that auto.purge property is not considered when we do a
> INSERT OVERWRITE while it is when we do a DROP TABLE
> Drop table doesn't move table data to Trash when auto.purge is set to true
> {noformat}
> > create table temp(col1 string, col2 string);
> No rows affected (0.064 seconds)
> > alter table temp set tblproperties('auto.purge'='true');
> No rows affected (0.083 seconds)
> > insert into temp values ('test', 'test'), ('test2', 'test2');
> No rows affected (25.473 seconds)
> # hdfs dfs -ls /user/hive/warehouse/temp
> Found 1 items
> -rwxrwxrwt 3 hive hive 22 2017-02-09 13:03
> /user/hive/warehouse/temp/00_0
> #
> > drop table temp;
> No rows affected (0.242 seconds)
> # hdfs dfs -ls /user/hive/warehouse/temp
> ls: `/user/hive/warehouse/temp': No such file or directory
> #
> # sudo -u hive hdfs dfs -ls /user/hive/.Trash/Current/user/hive/warehouse
> #
> {noformat}
> INSERT OVERWRITE query moves the table data to Trash even when auto.purge is
> set to true
> {noformat}
> > create table temp(col1 string, col2 string);
> > alter table temp set tblproperties('auto.purge'='true');
> > insert into temp values ('test', 'test'), ('test2', 'test2');
> # hdfs dfs -ls /user/hive/warehouse/temp
> Found 1 items
> -rwxrwxrwt 3 hive hive 22 2017-02-09 13:07
> /user/hive/warehouse/temp/00_0
> #
> > insert overwrite table temp select * from dummy;
> # hdfs dfs -ls /user/hive/warehouse/temp
> Found 1 items
> -rwxrwxrwt 3 hive hive 26 2017-02-09 13:08
> /user/hive/warehouse/temp/00_0
> # sudo -u hive hdfs dfs -ls /user/hive/.Trash/Current/user/hive/warehouse
> Found 1 items
> drwx-- - hive hive 0 2017-02-09 13:08
> /user/hive/.Trash/Current/user/hive/warehouse/temp
> #
> {noformat}
> While move operations are not very costly on HDFS it could be significant
> overhead on slow FileSystems like S3. This could improve the performance of
> {{INSERT OVERWRITE TABLE}} queries especially when there are large number of
> partitions on tables located on S3 should the user wish to set auto.purge
> property to true
> Similarly {{TRUNCATE TABLE}} query on a table with {{auto.purge}} property
> set true should not move the data to Trash
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16318) LLAP cache: address some issues in 2.2/2.3
[ https://issues.apache.org/jira/browse/HIVE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946350#comment-15946350 ] Siddharth Seth commented on HIVE-16318: --- Nit: "metadataFraction < 0" -> "metadataFraction <= 0" ? > LLAP cache: address some issues in 2.2/2.3 > -- > > Key: HIVE-16318 > URL: https://issues.apache.org/jira/browse/HIVE-16318 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16318.01.patch, HIVE-16318.02.patch, > HIVE-16318.03.patch, HIVE-16318.patch > > > We've run into HIVE-16233 and HIVE-15665 and given that 2.2 and 2.3 releases > are approaching we are going to add workarounds for them, and then commit the > above patches and revert the workarounds as soon as we can. > Unfortunately this will result in cache wasting some memory on some datasets, > but the alternatives, when they are encountered (usually only on large > datasets), are worse. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16324) Truncate table should not work when EXTERNAL property of table is true
[
https://issues.apache.org/jira/browse/HIVE-16324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar reassigned HIVE-16324:
--
> Truncate table should not work when EXTERNAL property of table is true
> --
>
> Key: HIVE-16324
> URL: https://issues.apache.org/jira/browse/HIVE-16324
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>
> Currently if you create an external table using the command {{CREATE EXTERNAL
> TABLE table_name}} the {{TRUNCATE table table_name}} command fails as
> expected because only managed tables should be allowed to be truncated.
> But if you set the external property of a previously managed table using
> {{ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='true')}}, truncate
> table command does not object and deletes all the data from the external
> table.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15880) Allow insert overwrite and truncate table query to use auto.purge table property
[
https://issues.apache.org/jira/browse/HIVE-15880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-15880:
---
Attachment: HIVE-15880.05.patch
> Allow insert overwrite and truncate table query to use auto.purge table
> property
>
>
> Key: HIVE-15880
> URL: https://issues.apache.org/jira/browse/HIVE-15880
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-15880.01.patch, HIVE-15880.02.patch,
> HIVE-15880.03.patch, HIVE-15880.04.patch, HIVE-15880.05.patch
>
>
> It seems inconsistent that auto.purge property is not considered when we do a
> INSERT OVERWRITE while it is when we do a DROP TABLE
> Drop table doesn't move table data to Trash when auto.purge is set to true
> {noformat}
> > create table temp(col1 string, col2 string);
> No rows affected (0.064 seconds)
> > alter table temp set tblproperties('auto.purge'='true');
> No rows affected (0.083 seconds)
> > insert into temp values ('test', 'test'), ('test2', 'test2');
> No rows affected (25.473 seconds)
> # hdfs dfs -ls /user/hive/warehouse/temp
> Found 1 items
> -rwxrwxrwt 3 hive hive 22 2017-02-09 13:03
> /user/hive/warehouse/temp/00_0
> #
> > drop table temp;
> No rows affected (0.242 seconds)
> # hdfs dfs -ls /user/hive/warehouse/temp
> ls: `/user/hive/warehouse/temp': No such file or directory
> #
> # sudo -u hive hdfs dfs -ls /user/hive/.Trash/Current/user/hive/warehouse
> #
> {noformat}
> INSERT OVERWRITE query moves the table data to Trash even when auto.purge is
> set to true
> {noformat}
> > create table temp(col1 string, col2 string);
> > alter table temp set tblproperties('auto.purge'='true');
> > insert into temp values ('test', 'test'), ('test2', 'test2');
> # hdfs dfs -ls /user/hive/warehouse/temp
> Found 1 items
> -rwxrwxrwt 3 hive hive 22 2017-02-09 13:07
> /user/hive/warehouse/temp/00_0
> #
> > insert overwrite table temp select * from dummy;
> # hdfs dfs -ls /user/hive/warehouse/temp
> Found 1 items
> -rwxrwxrwt 3 hive hive 26 2017-02-09 13:08
> /user/hive/warehouse/temp/00_0
> # sudo -u hive hdfs dfs -ls /user/hive/.Trash/Current/user/hive/warehouse
> Found 1 items
> drwx-- - hive hive 0 2017-02-09 13:08
> /user/hive/.Trash/Current/user/hive/warehouse/temp
> #
> {noformat}
> While move operations are not very costly on HDFS it could be significant
> overhead on slow FileSystems like S3. This could improve the performance of
> {{INSERT OVERWRITE TABLE}} queries especially when there are large number of
> partitions on tables located on S3 should the user wish to set auto.purge
> property to true
> Similarly {{TRUNCATE TABLE}} query on a table with {{auto.purge}} property
> set true should not move the data to Trash
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16318) LLAP cache: address some issues in 2.2/2.3
[ https://issues.apache.org/jira/browse/HIVE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946342#comment-15946342 ] Siddharth Seth commented on HIVE-16318: --- Mostly looks good. +1. Strongly prefer it if the fraction of on heap cache was not affecting per executor memory. LLAP_DAEMON_XMX_HEADROOM gives control over this value. Think overall configuration and sizing will be a little easier without having to think about 1 more parameter. e.g. ContainerSize=132G. CacheSize=20G. Heap=100G. memPerExecutor=4G. numExecutors=25. If the on-heap cache needs to be factored in (10%): change Xmx to a higher value - 110. Executor size remains unchanged. ContainerSize and Heap increase by the amount that the on heap cache will use. OTOH, with an automatic reduction. Setting the heap to 110G, and a factor of 10%, 11G will be used for the cache. The executor size calculations become a little more complicated (4G executors still required). Some of the calculations for executors - io.sort.mb, noconditionaltasksize, unordered buffers are absolute values - based on the initially specified container size (or per executor memory). Would be better to keep those as is. (There's explicit validations for some of these values to be within executor memory, which could cause errors) Not automatically doing this means we're not accounting for cache usage, and another parameter would need to be changed. There is a knob available though. Setting it automatically reduces memory, but doesn't really fix the other parameters which have already been computed based on the available memory. 10G vs 11G when the value is set to 10% will not make that much of a difference. Without increasing the size though, executor memory can be reduced by quite a bit. TL;DR - Simpler configuration via containerSize, Xmx OR heap reservation, instead of automatically subtracting from heap. Also not sure how accurate this measure is - considering Java objects. > LLAP cache: address some issues in 2.2/2.3 > -- > > Key: HIVE-16318 > URL: https://issues.apache.org/jira/browse/HIVE-16318 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16318.01.patch, HIVE-16318.02.patch, > HIVE-16318.03.patch, HIVE-16318.patch > > > We've run into HIVE-16233 and HIVE-15665 and given that 2.2 and 2.3 releases > are approaching we are going to add workarounds for them, and then commit the > above patches and revert the workarounds as soon as we can. > Unfortunately this will result in cache wasting some memory on some datasets, > but the alternatives, when they are encountered (usually only on large > datasets), are worse. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15173) Allow dec as an alias for decimal
[ https://issues.apache.org/jira/browse/HIVE-15173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-15173: Attachment: HIVE-15173.2.patch > Allow dec as an alias for decimal > - > > Key: HIVE-15173 > URL: https://issues.apache.org/jira/browse/HIVE-15173 > Project: Hive > Issue Type: Sub-task > Components: Parser >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-15173.1.patch, HIVE-15173.1.patch, > HIVE-15173.2.patch, HIVE-15173.patch > > > Standard allows dec as an alias for decimal -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15173) Allow dec as an alias for decimal
[ https://issues.apache.org/jira/browse/HIVE-15173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-15173: Status: Patch Available (was: Open) > Allow dec as an alias for decimal > - > > Key: HIVE-15173 > URL: https://issues.apache.org/jira/browse/HIVE-15173 > Project: Hive > Issue Type: Sub-task > Components: Parser >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-15173.1.patch, HIVE-15173.1.patch, > HIVE-15173.2.patch, HIVE-15173.patch > > > Standard allows dec as an alias for decimal -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15173) Allow dec as an alias for decimal
[ https://issues.apache.org/jira/browse/HIVE-15173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-15173: Status: Open (was: Patch Available) > Allow dec as an alias for decimal > - > > Key: HIVE-15173 > URL: https://issues.apache.org/jira/browse/HIVE-15173 > Project: Hive > Issue Type: Sub-task > Components: Parser >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-15173.1.patch, HIVE-15173.1.patch, HIVE-15173.patch > > > Standard allows dec as an alias for decimal -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16318) LLAP cache: address some issues in 2.2/2.3
[ https://issues.apache.org/jira/browse/HIVE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946305#comment-15946305 ] Sergey Shelukhin commented on HIVE-16318: - cacheMetrics.setCacheCapacityTotal(totalMemorySize); call should also add metaMem > LLAP cache: address some issues in 2.2/2.3 > -- > > Key: HIVE-16318 > URL: https://issues.apache.org/jira/browse/HIVE-16318 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16318.01.patch, HIVE-16318.02.patch, > HIVE-16318.03.patch, HIVE-16318.patch > > > We've run into HIVE-16233 and HIVE-15665 and given that 2.2 and 2.3 releases > are approaching we are going to add workarounds for them, and then commit the > above patches and revert the workarounds as soon as we can. > Unfortunately this will result in cache wasting some memory on some datasets, > but the alternatives, when they are encountered (usually only on large > datasets), are worse. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16318) LLAP cache: address some issues in 2.2/2.3
[ https://issues.apache.org/jira/browse/HIVE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-16318: Attachment: HIVE-16318.03.patch > LLAP cache: address some issues in 2.2/2.3 > -- > > Key: HIVE-16318 > URL: https://issues.apache.org/jira/browse/HIVE-16318 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16318.01.patch, HIVE-16318.02.patch, > HIVE-16318.03.patch, HIVE-16318.patch > > > We've run into HIVE-16233 and HIVE-15665 and given that 2.2 and 2.3 releases > are approaching we are going to add workarounds for them, and then commit the > above patches and revert the workarounds as soon as we can. > Unfortunately this will result in cache wasting some memory on some datasets, > but the alternatives, when they are encountered (usually only on large > datasets), are worse. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16318) LLAP cache: address some issues in 2.2/2.3
[ https://issues.apache.org/jira/browse/HIVE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-16318: Attachment: (was: HIVE-16318.03.patch) > LLAP cache: address some issues in 2.2/2.3 > -- > > Key: HIVE-16318 > URL: https://issues.apache.org/jira/browse/HIVE-16318 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16318.01.patch, HIVE-16318.02.patch, > HIVE-16318.03.patch, HIVE-16318.patch > > > We've run into HIVE-16233 and HIVE-15665 and given that 2.2 and 2.3 releases > are approaching we are going to add workarounds for them, and then commit the > above patches and revert the workarounds as soon as we can. > Unfortunately this will result in cache wasting some memory on some datasets, > but the alternatives, when they are encountered (usually only on large > datasets), are worse. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16228) Support subqueries in complex expression in SELECT clause
[
https://issues.apache.org/jira/browse/HIVE-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946299#comment-15946299
]
Hive QA commented on HIVE-16228:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860943/HIVE-16091.2.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10518 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=172)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4423/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4423/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4423/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860943 - PreCommit-HIVE-Build
> Support subqueries in complex expression in SELECT clause
> -
>
> Key: HIVE-16228
> URL: https://issues.apache.org/jira/browse/HIVE-16228
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-16091.2.patch, HIVE-16228.1.patch
>
>
> HIVE-16091 added support for subqueries in SELECT clause but restricted
> subqueries to top level expressions (more detail is at [LINK |
> https://cwiki.apache.org/confluence/display/Hive/Subqueries+in+SELECT])
> This restriction will be relaxed to allow subqueries in all kind of
> expression except UDAF (including UDAs and UDTFs).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16318) LLAP cache: address some issues in 2.2/2.3
[ https://issues.apache.org/jira/browse/HIVE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-16318: Attachment: HIVE-16318.03.patch The patch that takes the fraction off xmx and actually reserves it too cc [~prasanth_j] [~sseth] > LLAP cache: address some issues in 2.2/2.3 > -- > > Key: HIVE-16318 > URL: https://issues.apache.org/jira/browse/HIVE-16318 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16318.01.patch, HIVE-16318.02.patch, > HIVE-16318.03.patch, HIVE-16318.patch > > > We've run into HIVE-16233 and HIVE-15665 and given that 2.2 and 2.3 releases > are approaching we are going to add workarounds for them, and then commit the > above patches and revert the workarounds as soon as we can. > Unfortunately this will result in cache wasting some memory on some datasets, > but the alternatives, when they are encountered (usually only on large > datasets), are worse. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16239) remove useless hiveserver
[
https://issues.apache.org/jira/browse/HIVE-16239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fei Hui updated HIVE-16239:
---
Attachment: HIVE-16239.2-branch-2.1.patch
HIVE-16239.2-branch-2.0.patch
[~Ferd] Thanks. Update patches base on latest code
> remove useless hiveserver
> -
>
> Key: HIVE-16239
> URL: https://issues.apache.org/jira/browse/HIVE-16239
> Project: Hive
> Issue Type: Bug
> Components: CLI
>Affects Versions: 2.0.1, 2.1.1
>Reporter: Fei Hui
>Assignee: Fei Hui
> Attachments: HIVE-16239.1-branch-2.0.patch,
> HIVE-16239.1-branch-2.1.patch, HIVE-16239.2-branch-2.0.patch,
> HIVE-16239.2-branch-2.1.patch
>
>
> {quote}
> [hadoop@header hive]$ hive --service hiveserver
> Starting Hive Thrift Server
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/opt/apps/apache-hive-2.0.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/apps/spark-1.6.2-bin-hadoop2.7/lib/spark-assembly-1.6.2-hadoop2.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/apps/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Exception in thread "main" java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.service.HiveServer
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {quote}
> hiveserver does not exist, we should remove hiveserver from cli on branch-2.0
> After removing it, we get useful message
> {quote}
> Service hiveserver not found
> Available Services: beeline cli hbaseimport hbaseschematool help
> hiveburninclient hiveserver2 hplsql hwi jar lineage llap metastore metatool
> orcfiledump rcfilecat schemaTool version
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16209) Vectorization: Add support for complex types to VectorExtractRow and VectorAssignRow
[ https://issues.apache.org/jira/browse/HIVE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-16209: -- Attachment: HIVE-16209.3.patch This third patch fixed schema evolution test failures, fixed a timestamp precision issue, and removed irrelevant changes. > Vectorization: Add support for complex types to VectorExtractRow and > VectorAssignRow > > > Key: HIVE-16209 > URL: https://issues.apache.org/jira/browse/HIVE-16209 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Matt McCline >Assignee: Teddy Choi >Priority: Critical > Attachments: HIVE-16209.1.patch, HIVE-16209.3.patch > > > Supports complex types in non-native VectorReduceSink, row mode Text > Vectorization, and some cases of Vectorized Schema Evolution. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16319) LLAP: Better handling of an empty wait queue, should try scheduling checks
[ https://issues.apache.org/jira/browse/HIVE-16319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-16319: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) > LLAP: Better handling of an empty wait queue, should try scheduling checks > -- > > Key: HIVE-16319 > URL: https://issues.apache.org/jira/browse/HIVE-16319 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 3.0.0 > > Attachments: HIVE-16319.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16318) LLAP cache: address some issues in 2.2/2.3
[ https://issues.apache.org/jira/browse/HIVE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946258#comment-15946258 ] Prasanth Jayachandran commented on HIVE-16318: -- Changes other than where fraction is chosen (heapsize vs cachesize) looks good to me. +1 > LLAP cache: address some issues in 2.2/2.3 > -- > > Key: HIVE-16318 > URL: https://issues.apache.org/jira/browse/HIVE-16318 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16318.01.patch, HIVE-16318.02.patch, > HIVE-16318.patch > > > We've run into HIVE-16233 and HIVE-15665 and given that 2.2 and 2.3 releases > are approaching we are going to add workarounds for them, and then commit the > above patches and revert the workarounds as soon as we can. > Unfortunately this will result in cache wasting some memory on some datasets, > but the alternatives, when they are encountered (usually only on large > datasets), are worse. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946247#comment-15946247 ] Kalyan commented on HIVE-15691: --- Hi [~ekoifman], if possible please update the document with below text. StrictRegexWriter Class StrictRegexWriter implements the RecordWriter interface. It accepts input records, regex that in text format and writes them to Hive. It converts the text record using proper regex directly into an Object using RegexSerDe, which is then passed on to the underlying AcidOutputFormat's record updater for the appropriate bucket. See Javadoc. I am not getting write permission to update the docs. > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan >Priority: Critical > Fix For: 1.3.0, 1.2.2, 2.3.0, 3.0.0 > > Attachments: HIVE-15691.1-branch-1.2.patch, > HIVE-15691.1-branch-1.patch, HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.3.patch, HIVE-15691.4.patch, HIVE-15691.5.patch, > HIVE-15691-branch-1.2.patch, HIVE-15691-branch-1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16318) LLAP cache: address some issues in 2.2/2.3
[ https://issues.apache.org/jira/browse/HIVE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946249#comment-15946249 ] Siddharth Seth commented on HIVE-16318: --- bq. Note that with this temporary approach, the memory is completely taken out of data cache... Actually, given that we don't alloc on-heap, we could set metadata cache to a fraction of data cache, w/o reducing the latter. Thoughts? Makes sense to me. Also, the metadata cache being a fraction of heapsize may make sizing the entire YARN container a little easier. I don't think we should change the executor memory though. > LLAP cache: address some issues in 2.2/2.3 > -- > > Key: HIVE-16318 > URL: https://issues.apache.org/jira/browse/HIVE-16318 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16318.01.patch, HIVE-16318.02.patch, > HIVE-16318.patch > > > We've run into HIVE-16233 and HIVE-15665 and given that 2.2 and 2.3 releases > are approaching we are going to add workarounds for them, and then commit the > above patches and revert the workarounds as soon as we can. > Unfortunately this will result in cache wasting some memory on some datasets, > but the alternatives, when they are encountered (usually only on large > datasets), are worse. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15396) Basic Stats are not collected when for managed tables with LOCATION specified
[
https://issues.apache.org/jira/browse/HIVE-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946219#comment-15946219
]
Hive QA commented on HIVE-15396:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860930/HIVE-15396.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 817 failed/errored test(s), 6297 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_custom_key2]
(batchId=222)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_custom_key]
(batchId=222)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_joins]
(batchId=222)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=222)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=222)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=222)
org.apache.hadoop.hive.cli.TestBeeLineDriver.org.apache.hadoop.hive.cli.TestBeeLineDriver
(batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ctas_hdfs_to_blobstore]
(batchId=234)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_blobstore]
(batchId=234)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_local]
(batchId=234)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_warehouse]
(batchId=234)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_local_to_blobstore]
(batchId=234)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_blobstore]
(batchId=234)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_blobstore_nonpart]
(batchId=234)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_local]
(batchId=234)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_warehouse]
(batchId=234)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_warehouse_nonpart]
(batchId=234)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_local_to_blobstore]
(batchId=234)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_dynamic_partitions]
(batchId=234)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_table]
(batchId=234)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_directory]
(batchId=234)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions]
(batchId=234)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[write_final_output_blobstore]
(batchId=234)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=27)
org.apac
[jira] [Commented] (HIVE-16319) LLAP: Better handling of an empty wait queue, should try scheduling checks
[ https://issues.apache.org/jira/browse/HIVE-16319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946185#comment-15946185 ] Siddharth Seth commented on HIVE-16319: --- Test failures are not related. Committing. > LLAP: Better handling of an empty wait queue, should try scheduling checks > -- > > Key: HIVE-16319 > URL: https://issues.apache.org/jira/browse/HIVE-16319 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-16319.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-11529) Loop optimization for SIMD in IfExprLong*Long*
[
https://issues.apache.org/jira/browse/HIVE-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946152#comment-15946152
]
Carter Shanklin commented on HIVE-11529:
[~teddy.choi] did this ever make it in?
> Loop optimization for SIMD in IfExprLong*Long*
> --
>
> Key: HIVE-11529
> URL: https://issues.apache.org/jira/browse/HIVE-11529
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Minor
> Attachments: HIVE-11529.2.patch, HIVE-11529.3.patch,
> HIVE-11529.4.patch, HIVE-11529.patch
>
>
> HIVE-10238 showed that ?: operator can be optimized with bitwise operators
> when it only handles LongColumnVectors. There are more similar cases as
> following.
> IfExprLongColumnLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> vector2[i] : arg3Scalar);{code}
> IfExprLongScalarLongColumn: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : vector3[i]);{code}
> IfExprLongScalarLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : arg3Scalar);{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16319) LLAP: Better handling of an empty wait queue, should try scheduling checks
[
https://issues.apache.org/jira/browse/HIVE-16319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946146#comment-15946146
]
Hive QA commented on HIVE-16319:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860931/HIVE-16319.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10518 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=141)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=172)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4421/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4421/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4421/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860931 - PreCommit-HIVE-Build
> LLAP: Better handling of an empty wait queue, should try scheduling checks
> --
>
> Key: HIVE-16319
> URL: https://issues.apache.org/jira/browse/HIVE-16319
> Project: Hive
> Issue Type: Bug
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-16319.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16323) HS2 JDOPersistenceManagerFactory.pmCache leaks after HIVE-14204
[ https://issues.apache.org/jira/browse/HIVE-16323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-16323: -- Attachment: PM_leak.png > HS2 JDOPersistenceManagerFactory.pmCache leaks after HIVE-14204 > --- > > Key: HIVE-16323 > URL: https://issues.apache.org/jira/browse/HIVE-16323 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-16323.1.patch, PM_leak.png > > > Hive.loadDynamicPartitions creates threads with new embedded rawstore, but > never close them, thus we leak PersistenceManager one per such thread. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16323) HS2 JDOPersistenceManagerFactory.pmCache leaks after HIVE-14204
[ https://issues.apache.org/jira/browse/HIVE-16323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-16323: -- Status: Patch Available (was: Open) > HS2 JDOPersistenceManagerFactory.pmCache leaks after HIVE-14204 > --- > > Key: HIVE-16323 > URL: https://issues.apache.org/jira/browse/HIVE-16323 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-16323.1.patch > > > Hive.loadDynamicPartitions creates threads with new embedded rawstore, but > never close them, thus we leak PersistenceManager one per such thread. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16323) HS2 JDOPersistenceManagerFactory.pmCache leaks after HIVE-14204
[ https://issues.apache.org/jira/browse/HIVE-16323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-16323: -- Attachment: HIVE-16323.1.patch > HS2 JDOPersistenceManagerFactory.pmCache leaks after HIVE-14204 > --- > > Key: HIVE-16323 > URL: https://issues.apache.org/jira/browse/HIVE-16323 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-16323.1.patch > > > Hive.loadDynamicPartitions creates threads with new embedded rawstore, but > never close them, thus we leak PersistenceManager one per such thread. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16323) HS2 JDOPersistenceManagerFactory.pmCache leaks after HIVE-14204
[ https://issues.apache.org/jira/browse/HIVE-16323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai reassigned HIVE-16323: - > HS2 JDOPersistenceManagerFactory.pmCache leaks after HIVE-14204 > --- > > Key: HIVE-16323 > URL: https://issues.apache.org/jira/browse/HIVE-16323 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Daniel Dai >Assignee: Daniel Dai > > Hive.loadDynamicPartitions creates threads with new embedded rawstore, but > never close them, thus we leak PersistenceManager one per such thread. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16305) Additional Datanucleus ClassLoaderResolverImpl leaks causing HS2 OOM
[ https://issues.apache.org/jira/browse/HIVE-16305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-16305: -- Target Version/s: 2.3.0, 3.0.0 (was: 3.0.0) Further committed to branch-2. > Additional Datanucleus ClassLoaderResolverImpl leaks causing HS2 OOM > > > Key: HIVE-16305 > URL: https://issues.apache.org/jira/browse/HIVE-16305 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 2.3.0, 3.0.0 > > Attachments: ClassLoaderResolverImpl_1.png, > ClassLoaderResolverImpl_2.png, ClassLoaderResolverImpl_3.png, > HIVE-16305.1.patch > > > This is a followup for HIVE-16160. We see additional ClassLoaderResolverImpl > leaks even with the patch. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16305) Additional Datanucleus ClassLoaderResolverImpl leaks causing HS2 OOM
[ https://issues.apache.org/jira/browse/HIVE-16305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-16305: -- Target Version/s: 3.0.0 (was: 2.3.0, 3.0.0) > Additional Datanucleus ClassLoaderResolverImpl leaks causing HS2 OOM > > > Key: HIVE-16305 > URL: https://issues.apache.org/jira/browse/HIVE-16305 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 2.3.0, 3.0.0 > > Attachments: ClassLoaderResolverImpl_1.png, > ClassLoaderResolverImpl_2.png, ClassLoaderResolverImpl_3.png, > HIVE-16305.1.patch > > > This is a followup for HIVE-16160. We see additional ClassLoaderResolverImpl > leaks even with the patch. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16305) Additional Datanucleus ClassLoaderResolverImpl leaks causing HS2 OOM
[ https://issues.apache.org/jira/browse/HIVE-16305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-16305: -- Fix Version/s: 2.3.0 > Additional Datanucleus ClassLoaderResolverImpl leaks causing HS2 OOM > > > Key: HIVE-16305 > URL: https://issues.apache.org/jira/browse/HIVE-16305 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 2.3.0, 3.0.0 > > Attachments: ClassLoaderResolverImpl_1.png, > ClassLoaderResolverImpl_2.png, ClassLoaderResolverImpl_3.png, > HIVE-16305.1.patch > > > This is a followup for HIVE-16160. We see additional ClassLoaderResolverImpl > leaks even with the patch. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16305) Additional Datanucleus ClassLoaderResolverImpl leaks causing HS2 OOM
[ https://issues.apache.org/jira/browse/HIVE-16305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-16305: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Patch committed to master. > Additional Datanucleus ClassLoaderResolverImpl leaks causing HS2 OOM > > > Key: HIVE-16305 > URL: https://issues.apache.org/jira/browse/HIVE-16305 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 3.0.0 > > Attachments: ClassLoaderResolverImpl_1.png, > ClassLoaderResolverImpl_2.png, ClassLoaderResolverImpl_3.png, > HIVE-16305.1.patch > > > This is a followup for HIVE-16160. We see additional ClassLoaderResolverImpl > leaks even with the patch. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16318) LLAP cache: address some issues in 2.2/2.3
[ https://issues.apache.org/jira/browse/HIVE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-16318: Attachment: HIVE-16318.02.patch Updated based on CR feedback. I think for realistic cases, 15% is enough; the only reason we saw it because dataset was huge so basically metadata took all the cache. Note that with this temporary approach, the memory is completely taken out of data cache... Actually, given that we don't alloc on-heap, we could set metadata cache to a fraction of data cache, w/o reducing the latter. Thoughts? > LLAP cache: address some issues in 2.2/2.3 > -- > > Key: HIVE-16318 > URL: https://issues.apache.org/jira/browse/HIVE-16318 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16318.01.patch, HIVE-16318.02.patch, > HIVE-16318.patch > > > We've run into HIVE-16233 and HIVE-15665 and given that 2.2 and 2.3 releases > are approaching we are going to add workarounds for them, and then commit the > above patches and revert the workarounds as soon as we can. > Unfortunately this will result in cache wasting some memory on some datasets, > but the alternatives, when they are encountered (usually only on large > datasets), are worse. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16318) LLAP cache: address some issues in 2.2/2.3
[
https://issues.apache.org/jira/browse/HIVE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946063#comment-15946063
]
Hive QA commented on HIVE-16318:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860926/HIVE-16318.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10518 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=141)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[load_dyn_part5]
(batchId=115)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=172)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4420/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4420/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4420/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860926 - PreCommit-HIVE-Build
> LLAP cache: address some issues in 2.2/2.3
> --
>
> Key: HIVE-16318
> URL: https://issues.apache.org/jira/browse/HIVE-16318
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16318.01.patch, HIVE-16318.patch
>
>
> We've run into HIVE-16233 and HIVE-15665 and given that 2.2 and 2.3 releases
> are approaching we are going to add workarounds for them, and then commit the
> above patches and revert the workarounds as soon as we can.
> Unfortunately this will result in cache wasting some memory on some datasets,
> but the alternatives, when they are encountered (usually only on large
> datasets), are worse.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16318) LLAP cache: address some issues in 2.2/2.3
[ https://issues.apache.org/jira/browse/HIVE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946056#comment-15946056 ] Prasanth Jayachandran commented on HIVE-16318: -- 90-10 split for data-metadata is probably small. This may lead to some more regression as queries will start reading file footers now. May be make the split configurable and deprecate it so that it can be removed in 3.0 when metadata moves offheap? > LLAP cache: address some issues in 2.2/2.3 > -- > > Key: HIVE-16318 > URL: https://issues.apache.org/jira/browse/HIVE-16318 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16318.01.patch, HIVE-16318.patch > > > We've run into HIVE-16233 and HIVE-15665 and given that 2.2 and 2.3 releases > are approaching we are going to add workarounds for them, and then commit the > above patches and revert the workarounds as soon as we can. > Unfortunately this will result in cache wasting some memory on some datasets, > but the alternatives, when they are encountered (usually only on large > datasets), are worse. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16322) Create a TestNegativeEncryptedCliDriver for negative tests
[ https://issues.apache.org/jira/browse/HIVE-16322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar reassigned HIVE-16322: -- > Create a TestNegativeEncryptedCliDriver for negative tests > -- > > Key: HIVE-16322 > URL: https://issues.apache.org/jira/browse/HIVE-16322 > Project: Hive > Issue Type: Test > Components: Hive >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar > > Currently there is no negative test CLI driver for running encrypted tests. > The current mechanism to test negative tests for encrypted q-file tests is to > set hive.cli.ignore.error=true and let the error occur and compared in q.out > file. we should create a negative CLI driver just like TestNegativeCliDriver > for encrypted use-cases. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16321) Possible deadlock in metastore with Acid workload
[
https://issues.apache.org/jira/browse/HIVE-16321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946045#comment-15946045
]
Eugene Koifman commented on HIVE-16321:
---
HIVE-13842 could be a workaround for this
> Possible deadlock in metastore with Acid workload
> -
>
> Key: HIVE-16321
> URL: https://issues.apache.org/jira/browse/HIVE-16321
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>
> TxnStore.MutexAPI is a mechanism how different Metastore instances can
> coordinate their operations. It uses a JDBCConnection to achieve it.
> In some cases this may lead to deadlock. TxnHandler uses a connection pool
> of fixed size. Suppose you X simultaneous calls to TxnHandlerlock(), where
> X is >= size of the pool. This take all connections form the pool, so when
> {noformat}
> handle = getMutexAPI().acquireLock(MUTEX_KEY.CheckLock.name());
> {noformat}
> is executed in _TxnHandler.checkLock(Connection dbConn, long extLockId)_ the
> pool is empty and the system is deadlocked.
> MutexAPI can't use the same connection as the operation it's protecting.
> (TxnHandler.checkLock(Connection dbConn, long extLockId) is an example).
> We could make MutexAPI use a separate connection pool (size > 'primary' conn
> pool).
> Or we could make TxnHandler.lock(LockRequest rqst) return immediately after
> enqueueing the lock with the expectation that the caller will always follow
> up with a call to checkLock(CheckLockRequest rqst).
> cc [~f1sherox]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16318) LLAP cache: address some issues in 2.2/2.3
[ https://issues.apache.org/jira/browse/HIVE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-16318: Attachment: HIVE-16318.01.patch A small update to metrics. > LLAP cache: address some issues in 2.2/2.3 > -- > > Key: HIVE-16318 > URL: https://issues.apache.org/jira/browse/HIVE-16318 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16318.01.patch, HIVE-16318.patch > > > We've run into HIVE-16233 and HIVE-15665 and given that 2.2 and 2.3 releases > are approaching we are going to add workarounds for them, and then commit the > above patches and revert the workarounds as soon as we can. > Unfortunately this will result in cache wasting some memory on some datasets, > but the alternatives, when they are encountered (usually only on large > datasets), are worse. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16228) Support subqueries in complex expression in SELECT clause
[ https://issues.apache.org/jira/browse/HIVE-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946029#comment-15946029 ] Vineet Garg commented on HIVE-16228: RB link is created and attached to the JIRA. > Support subqueries in complex expression in SELECT clause > - > > Key: HIVE-16228 > URL: https://issues.apache.org/jira/browse/HIVE-16228 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-16091.2.patch, HIVE-16228.1.patch > > > HIVE-16091 added support for subqueries in SELECT clause but restricted > subqueries to top level expressions (more detail is at [LINK | > https://cwiki.apache.org/confluence/display/Hive/Subqueries+in+SELECT]) > This restriction will be relaxed to allow subqueries in all kind of > expression except UDAF (including UDAs and UDTFs). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16228) Support subqueries in complex expression in SELECT clause
[ https://issues.apache.org/jira/browse/HIVE-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-16228: --- Status: Patch Available (was: Open) > Support subqueries in complex expression in SELECT clause > - > > Key: HIVE-16228 > URL: https://issues.apache.org/jira/browse/HIVE-16228 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-16091.2.patch, HIVE-16228.1.patch > > > HIVE-16091 added support for subqueries in SELECT clause but restricted > subqueries to top level expressions (more detail is at [LINK | > https://cwiki.apache.org/confluence/display/Hive/Subqueries+in+SELECT]) > This restriction will be relaxed to allow subqueries in all kind of > expression except UDAF (including UDAs and UDTFs). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16228) Support subqueries in complex expression in SELECT clause
[ https://issues.apache.org/jira/browse/HIVE-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-16228: --- Attachment: HIVE-16091.2.patch > Support subqueries in complex expression in SELECT clause > - > > Key: HIVE-16228 > URL: https://issues.apache.org/jira/browse/HIVE-16228 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-16091.2.patch, HIVE-16228.1.patch > > > HIVE-16091 added support for subqueries in SELECT clause but restricted > subqueries to top level expressions (more detail is at [LINK | > https://cwiki.apache.org/confluence/display/Hive/Subqueries+in+SELECT]) > This restriction will be relaxed to allow subqueries in all kind of > expression except UDAF (including UDAs and UDTFs). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16228) Support subqueries in complex expression in SELECT clause
[ https://issues.apache.org/jira/browse/HIVE-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-16228: --- Attachment: (was: HIVE-16091.2.patch) > Support subqueries in complex expression in SELECT clause > - > > Key: HIVE-16228 > URL: https://issues.apache.org/jira/browse/HIVE-16228 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-16091.2.patch, HIVE-16228.1.patch > > > HIVE-16091 added support for subqueries in SELECT clause but restricted > subqueries to top level expressions (more detail is at [LINK | > https://cwiki.apache.org/confluence/display/Hive/Subqueries+in+SELECT]) > This restriction will be relaxed to allow subqueries in all kind of > expression except UDAF (including UDAs and UDTFs). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16228) Support subqueries in complex expression in SELECT clause
[ https://issues.apache.org/jira/browse/HIVE-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-16228: --- Status: Open (was: Patch Available) > Support subqueries in complex expression in SELECT clause > - > > Key: HIVE-16228 > URL: https://issues.apache.org/jira/browse/HIVE-16228 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-16091.2.patch, HIVE-16228.1.patch > > > HIVE-16091 added support for subqueries in SELECT clause but restricted > subqueries to top level expressions (more detail is at [LINK | > https://cwiki.apache.org/confluence/display/Hive/Subqueries+in+SELECT]) > This restriction will be relaxed to allow subqueries in all kind of > expression except UDAF (including UDAs and UDTFs). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16309) Hive Test Commands failure should be printed in hive.log in addition to stderr
[
https://issues.apache.org/jira/browse/HIVE-16309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945992#comment-15945992
]
Sahil Takiar commented on HIVE-16309:
-
[~aihuaxu], [~vihangk1] could you review? This is an improvement to the test
infra for running HDFS encryption tests. When commands such as {{CRYPTO
CREATE_KEY}} fail, the whole stack trace is now printed out.
Test failures are all flaky tests:
HIVE-16320 - TestBeeLineDriver.testCliDriver[escape_comments]
HIVE-16256 - TestCliDriver.testCliDriver[comments]
HIVE-16312 - TestHCatClient.testTransportFailure
> Hive Test Commands failure should be printed in hive.log in addition to stderr
> --
>
> Key: HIVE-16309
> URL: https://issues.apache.org/jira/browse/HIVE-16309
> Project: Hive
> Issue Type: Bug
> Components: Test
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16309.1.patch
>
>
> When the {{QTestUtil.executeTestCommand}} runs a test only command, and the
> command fails, the exception info is printed to stdout rather than the log
> file. Printing out to the logs file will better help debuggability of these
> failures.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16312) Flaky test: TestHCatClient.testTransportFailure
[
https://issues.apache.org/jira/browse/HIVE-16312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945982#comment-15945982
]
Sahil Takiar commented on HIVE-16312:
-
More detailed stack-trace:
{code}
java.lang.AssertionError: The expected exception was never thrown.
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure(TestHCatClient.java:462)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
{code}
> Flaky test: TestHCatClient.testTransportFailure
> ---
>
> Key: HIVE-16312
> URL: https://issues.apache.org/jira/browse/HIVE-16312
> Project: Hive
> Issue Type: Sub-task
>Reporter: Barna Zsombor Klara
>
> The test has been failing consistently for 10+ builds.
> Error message:
> {code}
> Error Message
> The expected exception was never thrown.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16309) Hive Test Commands failure should be printed in hive.log in addition to stderr
[
https://issues.apache.org/jira/browse/HIVE-16309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945983#comment-15945983
]
Hive QA commented on HIVE-16309:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860924/HIVE-16309.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10517 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=172)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4419/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4419/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4419/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860924 - PreCommit-HIVE-Build
> Hive Test Commands failure should be printed in hive.log in addition to stderr
> --
>
> Key: HIVE-16309
> URL: https://issues.apache.org/jira/browse/HIVE-16309
> Project: Hive
> Issue Type: Bug
> Components: Test
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16309.1.patch
>
>
> When the {{QTestUtil.executeTestCommand}} runs a test only command, and the
> command fails, the exception info is printed to stdout rather than the log
> file. Printing out to the logs file will better help debuggability of these
> failures.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-13370) Add test for HIVE-11470
[ https://issues.apache.org/jira/browse/HIVE-13370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945966#comment-15945966 ] Pengcheng Xiong commented on HIVE-13370: [~daijy], is it ready to be committed? Thanks. > Add test for HIVE-11470 > --- > > Key: HIVE-13370 > URL: https://issues.apache.org/jira/browse/HIVE-13370 > Project: Hive > Issue Type: Bug >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan >Priority: Blocker > Attachments: HIVE-13370.patch > > > HIVE-11470 added capability to handle NULL dynamic partitioning keys > properly. However, it did not add a test for the case, we should have one so > we don't have future regressions of the same. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15829) LLAP text cache: disable memory tracking on the writer
[ https://issues.apache.org/jira/browse/HIVE-15829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-15829: Target Version/s: 2.2.0, 2.1.2, 2.3.0 (was: 2.1.2, 2.3.0) > LLAP text cache: disable memory tracking on the writer > -- > > Key: HIVE-15829 > URL: https://issues.apache.org/jira/browse/HIVE-15829 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Fix For: 3.0.0, 2.3.0 > > Attachments: HIVE-15829.patch > > > See ORC-141 and HIVE-15672 for context -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16228) Support subqueries in complex expression in SELECT clause
[ https://issues.apache.org/jira/browse/HIVE-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945973#comment-15945973 ] Ashutosh Chauhan commented on HIVE-16228: - Can you create a RB for this? > Support subqueries in complex expression in SELECT clause > - > > Key: HIVE-16228 > URL: https://issues.apache.org/jira/browse/HIVE-16228 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-16091.2.patch, HIVE-16228.1.patch > > > HIVE-16091 added support for subqueries in SELECT clause but restricted > subqueries to top level expressions (more detail is at [LINK | > https://cwiki.apache.org/confluence/display/Hive/Subqueries+in+SELECT]) > This restriction will be relaxed to allow subqueries in all kind of > expression except UDAF (including UDAs and UDTFs). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15829) LLAP text cache: disable memory tracking on the writer
[ https://issues.apache.org/jira/browse/HIVE-15829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-15829: Resolution: Fixed Fix Version/s: 2.3.0 3.0.0 Status: Resolved (was: Patch Available) [~owen.omalley] if ORC project dependency and text cache are both part of 2.2, this needs to go in 2.2 too > LLAP text cache: disable memory tracking on the writer > -- > > Key: HIVE-15829 > URL: https://issues.apache.org/jira/browse/HIVE-15829 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Fix For: 3.0.0, 2.3.0 > > Attachments: HIVE-15829.patch > > > See ORC-141 and HIVE-15672 for context -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16311) Improve the performance for FastHiveDecimalImpl.fastDivide
[ https://issues.apache.org/jira/browse/HIVE-16311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945962#comment-15945962 ] Pengcheng Xiong commented on HIVE-16311: Hello, I am deferring this to Hive 3.0 as we are going to cut the first RC and it is not marked as blocker. Please feel free to commit to the branch if this can be resolved before the release. > Improve the performance for FastHiveDecimalImpl.fastDivide > -- > > Key: HIVE-16311 > URL: https://issues.apache.org/jira/browse/HIVE-16311 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: Colin Ma >Assignee: Colin Ma > Fix For: 3.0.0 > > Attachments: HIVE-16311.001.patch > > > FastHiveDecimalImpl.fastDivide is poor performance when evaluate the > expression as 12345.67/123.45 > There are 2 points can be improved: > 1. Don't always use HiveDecimal.MAX_SCALE as scale when do the > BigDecimal.divide. > 2. Get the precision for BigInteger in a fast way if possible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-14077) add implicit decimal arithmetic q test, fix issues if found
[ https://issues.apache.org/jira/browse/HIVE-14077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14077: Resolution: Fixed Fix Version/s: 2.3.0 3.0.0 Status: Resolved (was: Patch Available) Just committed actually ;) Thanks for the review and discussion. > add implicit decimal arithmetic q test, fix issues if found > > > Key: HIVE-14077 > URL: https://issues.apache.org/jira/browse/HIVE-14077 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Fix For: 3.0.0, 2.3.0 > > Attachments: HIVE-14077.patch > > > See comments in that JIRA -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16311) Improve the performance for FastHiveDecimalImpl.fastDivide
[ https://issues.apache.org/jira/browse/HIVE-16311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16311: --- Target Version/s: 3.0.0 (was: 2.2.0) > Improve the performance for FastHiveDecimalImpl.fastDivide > -- > > Key: HIVE-16311 > URL: https://issues.apache.org/jira/browse/HIVE-16311 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: Colin Ma >Assignee: Colin Ma > Fix For: 3.0.0 > > Attachments: HIVE-16311.001.patch > > > FastHiveDecimalImpl.fastDivide is poor performance when evaluate the > expression as 12345.67/123.45 > There are 2 points can be improved: > 1. Don't always use HiveDecimal.MAX_SCALE as scale when do the > BigDecimal.divide. > 2. Get the precision for BigInteger in a fast way if possible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16311) Improve the performance for FastHiveDecimalImpl.fastDivide
[ https://issues.apache.org/jira/browse/HIVE-16311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16311: --- Fix Version/s: (was: 2.2.0) 3.0.0 > Improve the performance for FastHiveDecimalImpl.fastDivide > -- > > Key: HIVE-16311 > URL: https://issues.apache.org/jira/browse/HIVE-16311 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: Colin Ma >Assignee: Colin Ma > Fix For: 3.0.0 > > Attachments: HIVE-16311.001.patch > > > FastHiveDecimalImpl.fastDivide is poor performance when evaluate the > expression as 12345.67/123.45 > There are 2 points can be improved: > 1. Don't always use HiveDecimal.MAX_SCALE as scale when do the > BigDecimal.divide. > 2. Get the precision for BigInteger in a fast way if possible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-14077) add implicit decimal arithmetic q test, fix issues if found
[ https://issues.apache.org/jira/browse/HIVE-14077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945961#comment-15945961 ] Pengcheng Xiong commented on HIVE-14077: [~sershe] ready to commit? > add implicit decimal arithmetic q test, fix issues if found > > > Key: HIVE-14077 > URL: https://issues.apache.org/jira/browse/HIVE-14077 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Attachments: HIVE-14077.patch > > > See comments in that JIRA -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16304) LLAP: Throw SERVICE_UNAVAILABLE only if there are pending tasks
[ https://issues.apache.org/jira/browse/HIVE-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945955#comment-15945955 ] Prasanth Jayachandran commented on HIVE-16304: -- Can this check be moved to where the timeout monitor is scheduled startTimeoutMonitor(). Better to have all the checks together in one place. > LLAP: Throw SERVICE_UNAVAILABLE only if there are pending tasks > --- > > Key: HIVE-16304 > URL: https://issues.apache.org/jira/browse/HIVE-16304 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0, 3.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-16304.1.patch > > > Ignore service timeouts if there are no pending requests in LLAP task > scheduler. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16319) LLAP: Better handling of an empty wait queue, should try scheduling checks
[ https://issues.apache.org/jira/browse/HIVE-16319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945950#comment-15945950 ] Prasanth Jayachandran commented on HIVE-16319: -- lgtm, +1. Pending tests > LLAP: Better handling of an empty wait queue, should try scheduling checks > -- > > Key: HIVE-16319 > URL: https://issues.apache.org/jira/browse/HIVE-16319 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-16319.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16301) Prepare branch-2 for 2.3 development.
[ https://issues.apache.org/jira/browse/HIVE-16301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945947#comment-15945947 ] Pengcheng Xiong commented on HIVE-16301: LGTM, i think we just need more changes. Could u resubmit the patch with the changes for a ptest run? After that, I think we will be good. > Prepare branch-2 for 2.3 development. > - > > Key: HIVE-16301 > URL: https://issues.apache.org/jira/browse/HIVE-16301 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.3.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-16301.patch > > > branch-2 is now being used for 2.3.0 development. The build files will need > to reflect this change. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16228) Support subqueries in complex expression in SELECT clause
[ https://issues.apache.org/jira/browse/HIVE-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945940#comment-15945940 ] Vineet Garg commented on HIVE-16228: Thanks for the tests [~ashutoshc]. These tests found bunch of issues in the code. > Support subqueries in complex expression in SELECT clause > - > > Key: HIVE-16228 > URL: https://issues.apache.org/jira/browse/HIVE-16228 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-16091.2.patch, HIVE-16228.1.patch > > > HIVE-16091 added support for subqueries in SELECT clause but restricted > subqueries to top level expressions (more detail is at [LINK | > https://cwiki.apache.org/confluence/display/Hive/Subqueries+in+SELECT]) > This restriction will be relaxed to allow subqueries in all kind of > expression except UDAF (including UDAs and UDTFs). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16228) Support subqueries in complex expression in SELECT clause
[ https://issues.apache.org/jira/browse/HIVE-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-16228: --- Status: Patch Available (was: Open) > Support subqueries in complex expression in SELECT clause > - > > Key: HIVE-16228 > URL: https://issues.apache.org/jira/browse/HIVE-16228 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-16091.2.patch, HIVE-16228.1.patch > > > HIVE-16091 added support for subqueries in SELECT clause but restricted > subqueries to top level expressions (more detail is at [LINK | > https://cwiki.apache.org/confluence/display/Hive/Subqueries+in+SELECT]) > This restriction will be relaxed to allow subqueries in all kind of > expression except UDAF (including UDAs and UDTFs). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16228) Support subqueries in complex expression in SELECT clause
[ https://issues.apache.org/jira/browse/HIVE-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-16228: --- Attachment: HIVE-16091.2.patch > Support subqueries in complex expression in SELECT clause > - > > Key: HIVE-16228 > URL: https://issues.apache.org/jira/browse/HIVE-16228 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-16091.2.patch, HIVE-16228.1.patch > > > HIVE-16091 added support for subqueries in SELECT clause but restricted > subqueries to top level expressions (more detail is at [LINK | > https://cwiki.apache.org/confluence/display/Hive/Subqueries+in+SELECT]) > This restriction will be relaxed to allow subqueries in all kind of > expression except UDAF (including UDAs and UDTFs). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16228) Support subqueries in complex expression in SELECT clause
[ https://issues.apache.org/jira/browse/HIVE-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-16228: --- Status: Open (was: Patch Available) > Support subqueries in complex expression in SELECT clause > - > > Key: HIVE-16228 > URL: https://issues.apache.org/jira/browse/HIVE-16228 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-16228.1.patch > > > HIVE-16091 added support for subqueries in SELECT clause but restricted > subqueries to top level expressions (more detail is at [LINK | > https://cwiki.apache.org/confluence/display/Hive/Subqueries+in+SELECT]) > This restriction will be relaxed to allow subqueries in all kind of > expression except UDAF (including UDAs and UDTFs). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
