date:20181026

[jira] [Commented] (HIVE-20786) Maven Build Failed with group id is too big

2018-10-26 Thread Vihang Karajgaonkar (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665929#comment-16665929
 ] 

Vihang Karajgaonkar commented on HIVE-20786:


Hi [~szehon] can you update the packaging/pom.xml as well? Rest looks good to 
me. Thanks!

> Maven Build Failed with group id is too big 
> 
>
> Key: HIVE-20786
> URL: https://issues.apache.org/jira/browse/HIVE-20786
> Project: Hive
>  Issue Type: Bug
>  Components: Standalone Metastore
> Environment:  
> OS: MacOS 10.13.6
> Java:
> {code}
> java version "1.8.0_192"
> Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
> {code}
> Maven:
> {code}
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 
> 2018-06-18T02:33:14+08:00)
> Maven home: /usr/local/Cellar/maven/3.5.4/libexec
> Java version: 1.8.0_192, vendor: Oracle Corporation, runtime: 
> /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre
> Default locale: en_CN, platform encoding: UTF-8
> OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"
> {code}
>  
>  
>Reporter: PENG Zhengshuai
>Assignee: Szehon Ho
>Priority: Major
>  Labels: maven
> Attachments: HIVE-20786.patch, hive_build_error.log
>
>
> When executing
> {code}
> mvn clean install -DskipTests
> {code}
> Build Failed:
> {code}
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Storage API 2.7.0-SNAPSHOT  SUCCESS [  5.299 
> s]
> [INFO] Hive 4.0.0-SNAPSHOT  SUCCESS [  0.750 
> s]
> [INFO] Hive Classifications ... SUCCESS [  1.057 
> s]
> [INFO] Hive Shims Common .. SUCCESS [  3.882 
> s]
> [INFO] Hive Shims 0.23  SUCCESS [  5.020 
> s]
> [INFO] Hive Shims Scheduler ... SUCCESS [  2.587 
> s]
> [INFO] Hive Shims . SUCCESS [  2.038 
> s]
> [INFO] Hive Common  SUCCESS [  6.921 
> s]
> [INFO] Hive Service RPC ... SUCCESS [  3.503 
> s]
> [INFO] Hive Serde . SUCCESS [  6.322 
> s]
> [INFO] Hive Standalone Metastore .. FAILURE [  0.557 
> s]
> [INFO] Hive Standalone Metastore Common Code .. SKIPPED
> [INFO] Hive Metastore . SKIPPED
> [INFO] Hive Vector-Code-Gen Utilities . SKIPPED
> [INFO] Hive Llap Common ... SKIPPED
> [INFO] Hive Llap Client ... SKIPPED
> [INFO] Hive Llap Tez .. SKIPPED
> [INFO] Hive Spark Remote Client ... SKIPPED
> [INFO] Hive Metastore Server .. SKIPPED
> [INFO] Hive Query Language  SKIPPED
> [INFO] Hive Llap Server ... SKIPPED
> [INFO] Hive Service ... SKIPPED
> [INFO] Hive Accumulo Handler .. SKIPPED
> [INFO] Hive JDBC .. SKIPPED
> [INFO] Hive Beeline ... SKIPPED
> [INFO] Hive CLI ... SKIPPED
> [INFO] Hive Contrib ... SKIPPED
> [INFO] Hive Druid Handler . SKIPPED
> [INFO] Hive HBase Handler . SKIPPED
> [INFO] Hive JDBC Handler .. SKIPPED
> [INFO] Hive HCatalog .. SKIPPED
> [INFO] Hive HCatalog Core . SKIPPED
> [INFO] Hive HCatalog Pig Adapter .. SKIPPED
> [INFO] Hive HCatalog Server Extensions  SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
> [INFO] Hive HCatalog Webhcat .. SKIPPED
> [INFO] Hive HCatalog Streaming  SKIPPED
> [INFO] Hive HPL/SQL ... SKIPPED
> [INFO] Hive Streaming . SKIPPED
> [INFO] Hive Llap External Client .. SKIPPED
> [INFO] Hive Shims Aggregator .. SKIPPED
> [INFO] Hive Kryo Registrator .. SKIPPED
> [INFO] Hive TestUtils . SKIPPED
> [INFO] Hive Kafka Storage Handler . SKIPPED
> [INFO] Hive Packaging

[jira] [Updated] (HIVE-20825) Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in length) causing the "Not a valid ORC file" error

2018-10-26 Thread Tom Zeng (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom Zeng updated HIVE-20825:

Attachment: hive-merge-invalid-orc-repro.log
hive-merge-invalid-orc-repro.hql

> Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in 
> length) causing the "Not a valid ORC file" error
> --
>
> Key: HIVE-20825
> URL: https://issues.apache.org/jira/browse/HIVE-20825
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, ORC, Transactions
>Affects Versions: 2.2.0, 2.3.1, 2.3.2
> Environment: Hive 2.3.x on Amazon EMR 5.8.0 to 5.18.0
>Reporter: Tom Zeng
>Priority: Major
> Attachments: hive-merge-invalid-orc-repro.hql, 
> hive-merge-invalid-orc-repro.log
>
>
> When using Hive ACID Merge (supported with the ORC format) to update/insert 
> data, bucket files with 0 byte or 3 bytes (file content is three character: 
> ORC) are generated during MERGE INTO operations which finish with no errors. 
> Subsequent queries on the base table will get "Not a valid ORC file" error.
>  
> The following script can be used to reproduce the issue:
> set hive.auto.convert.join=false;
> set hive.enforce.bucketing=true;
> set hive.exec.dynamic.partition.mode = nonstrict;
> set hive.support.concurrency=true;
> set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> drop table if exists mergedelta_txt_1;
> drop table if exists mergedelta_txt_2;
> CREATE TABLE mergedelta_txt_1 (
> id_str varchar(12), time_key int, value bigint)
> PARTITIONED BY (date_key int)
> ROW FORMAT DELIMITED
> STORED AS TEXTFILE;
> CREATE TABLE mergedelta_txt_2 (
> id_str varchar(12), time_key int, value bigint)
> PARTITIONED BY (date_key int)
> ROW FORMAT DELIMITED
> STORED AS TEXTFILE;
> INSERT INTO TABLE mergedelta_txt_1
> partition(date_key=20170103)
> VALUES
>  ("AB94LIENR0",46700,12345676836978),
>  ("AB94LIENR1",46825,12345676836978),
>  ("AB94LIENS0",46709,12345676836978),
>  ("AB94LIENS1",46834,12345676836978),
>  ("AB94LIENT0",46709,12345676836978),
>  ("AB94LIENT1",46834,12345676836978),
>  ("AB94LIENU0",46718,12345676836978),
>  ("AB94LIENU1",46844,12345676836978),
>  ("AB94LIENV0",46719,12345676836978),
>  ("AB94LIENV1",46844,12345676836978),
>  ("AB94LIENW0",46728,12345676836978),
>  ("AB94LIENW1",46854,12345676836978),
>  ("AB94LIENX0",46728,12345676836978),
>  ("AB94LIENX1",46854,12345676836978),
>  ("AB94LIENY0",46737,12345676836978),
>  ("AB94LIENY1",46863,12345676836978),
>  ("AB94LIENZ0",46738,12345676836978),
>  ("AB94LIENZ1",46863,12345676836978),
>  ("AB94LIERA0",47176,12345676836982),
>  ("AB94LIERA1",47302,12345676836982);
> INSERT INTO TABLE mergedelta_txt_2
> partition(date_key=20170103)
> VALUES 
>  ("AB94LIENT1",46834,12345676836978),
>  ("AB94LIENU0",46718,12345676836978),
>  ("AB94LIENU1",46844,12345676836978),
>  ("AB94LIENV0",46719,12345676836978),
>  ("AB94LIENV1",46844,12345676836978),
>  ("AB94LIENW0",46728,12345676836978),
>  ("AB94LIENW1",46854,12345676836978),
>  ("AB94LIENX0",46728,12345676836978),
>  ("AB94LIENX1",46854,12345676836978),
>  ("AB94LIENY0",46737,12345676836978),
>  ("AB94LIENY1",46863,12345676836978),
>  ("AB94LIENZ0",46738,12345676836978),
>  ("AB94LIENZ1",46863,12345676836978),
>  ("AB94LIERA0",47176,12345676836982),
>  ("AB94LIERA1",47302,12345676836982),
>  ("AB94LIERA2",47418,12345676836982),
>  ("AB94LIERB0",47176,12345676836982),
>  ("AB94LIERB1",47302,12345676836982),
>  ("AB94LIERB2",47418,12345676836982),
>  ("AB94LIERC0",47185,12345676836982);
> DROP TABLE IF EXISTS mergebase_1;
> CREATE TABLE mergebase_1 (
> id_str varchar(12) , time_key int , value bigint)
> PARTITIONED BY (date_key int)
> CLUSTERED BY (id_str,time_key) INTO 32 BUCKETS
> STORED AS ORC
> TBLPROPERTIES (
>  'orc.compress'='SNAPPY',
>  'pk_columns'='id_str,date_key,time_key',
>  'NO_AUTO_COMPACTION'='true',
>  'transactional'='true');
> MERGE INTO mergebase_1 AS base
> USING (SELECT * 
>  FROM (
>  SELECT id_str ,time_key ,value, date_key, rank() OVER (PARTITION BY 
> id_str,date_key,time_key ORDER BY id_str,date_key,time_key) AS rk 
>  FROM mergedelta_txt_1
>  DISTRIBUTE BY date_key
>  ) rankedtbl 
>  WHERE rankedtbl.rk=1
> ) AS delta
> ON delta.id_str=base.id_str AND delta.date_key=base.date_key AND 
> delta.time_key=base.time_key
> WHEN MATCHED THEN UPDATE SET value=delta.value
> WHEN NOT MATCHED THEN INSERT VALUES ( delta.id_str , delta.time_key , 
> delta.value, delta.date_key);
> MERGE INTO mergebase_1 AS base
> USING (SELECT * 
>  FROM (
>  SELECT id_str ,time_key ,value, date_key, rank() OVER (PARTITION BY 
> id_str,date_key,time_key ORDER BY id_str,date_key,time_key) AS rk 
>  FROM mergedelta_txt_2
>  DISTRIBUTE BY date_key
>  ) rankedtbl 
>  WHERE

[jira] [Commented] (HIVE-20486) Kafka: Use Row SerDe + vectorization

2018-10-26 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665906#comment-16665906
 ] 

Hive QA commented on HIVE-20486:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12945812/HIVE-20486.3.patch

{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15507 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[kafka_storage_handler]
 (batchId=194)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/14648/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14648/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14648/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12945812 - PreCommit-HIVE-Build

> Kafka: Use Row SerDe + vectorization
> 
>
> Key: HIVE-20486
> URL: https://issues.apache.org/jira/browse/HIVE-20486
> Project: Hive
>  Issue Type: Improvement
>  Components: kafka integration
>Reporter: Gopal V
>Assignee: slim bouguerra
>Priority: Major
>  Labels: kafka, vectorization
> Fix For: 4.0.0
>
> Attachments: HIVE-20486.3.patch, HIVE-20486.3.patch, HIVE-20486.patch
>
>
> KafkaHandler returns unvectorized rows which causes the operators downstream 
> to be slower and sub-optimal.
> Hive has a vectorization shim which allows Kafka streams without complex 
> projections to be wrapped into a vectorized reader via 
> {{hive.vectorized.use.row.serde.deserialize}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20486) Kafka: Use Row SerDe + vectorization

2018-10-26 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665893#comment-16665893
 ] 

Hive QA commented on HIVE-20486:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
56s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
42s{color} | {color:blue} itests/util in master has 48 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
36s{color} | {color:blue} ql in master has 2317 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m  
8s{color} | {color:red} kafka-handler: The patch generated 1 new + 1 unchanged 
- 0 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 55s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14648/dev-support/hive-personality.sh
 |
| git revision | master / 1002e89 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14648/yetus/diff-checkstyle-kafka-handler.txt
 |
| modules | C: itests/util kafka-handler ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14648/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Kafka: Use Row SerDe + vectorization
> 
>
> Key: HIVE-20486
> URL: https://issues.apache.org/jira/browse/HIVE-20486
> Project: Hive
>  Issue Type: Improvement
>  Components: kafka integration
>Reporter: Gopal V
>Assignee: slim bouguerra
>Priority: Major
>  Labels: kafka, vectorization
> Fix For: 4.0.0
>
> Attachments: HIVE-20486.3.patch, HIVE-20486.3.patch, HIVE-20486.patch
>
>
> KafkaHandler returns unvectorized rows which causes the operators downstream 
> to be slower and sub-optimal.
> Hive has a vectorization shim which allows Kafka streams without complex 
> projections to be wrapped into a vectorized reader via 
> {{hive.vectorized.use.row.serde.deserialize}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (HIVE-20816) FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)

2018-10-26 Thread Rajesh Balamohan (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan reopened HIVE-20816:
-

> FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)
> --
>
> Key: HIVE-20816
> URL: https://issues.apache.org/jira/browse/HIVE-20816
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 2.3.2
>Reporter: Rajesh Balamohan
>Priority: Minor
>  Labels: fastdecimal
> Attachments: HIVE-20816.1.patch
>
>
> {noformat}
> with t1 as (
> ...
> ...
> )
> select id, max(abs(c1))) from t1 group by id;
> {noformat}
> throws the following exception
> {noformat}
> g.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
>  at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1126)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
> 
> ...
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1084)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1123)
> ... 18 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.RuntimeException: Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1047)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1067)
> ... 19 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20816) FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)

2018-10-26 Thread Rajesh Balamohan (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665819#comment-16665819
 ] 

Rajesh Balamohan commented on HIVE-20816:
-

reopening this bug as this is happening in 
\{{FastHiveDecimalImpl:doDecimalToBinaryConversion}}.

> FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)
> --
>
> Key: HIVE-20816
> URL: https://issues.apache.org/jira/browse/HIVE-20816
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 2.3.2
>Reporter: Rajesh Balamohan
>Priority: Minor
>  Labels: fastdecimal
> Attachments: HIVE-20816.1.patch
>
>
> {noformat}
> with t1 as (
> ...
> ...
> )
> select id, max(abs(c1))) from t1 group by id;
> {noformat}
> throws the following exception
> {noformat}
> g.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
>  at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1126)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
> 
> ...
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1084)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1123)
> ... 18 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.RuntimeException: Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1047)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1067)
> ... 19 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20815) JdbcRecordReader.next shall not eat exception

2018-10-26 Thread Thejas M Nair (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665794#comment-16665794
 ] 

Thejas M Nair commented on HIVE-20815:
--

[~daijy]
Can you also add a UT for this ?
I assume a query with wrong connection params would work ?


> JdbcRecordReader.next shall not eat exception
> -
>
> Key: HIVE-20815
> URL: https://issues.apache.org/jira/browse/HIVE-20815
> Project: Hive
>  Issue Type: Bug
>  Components: StorageHandler
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20815.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-20815) JdbcRecordReader.next shall not eat exception

2018-10-26 Thread Thejas M Nair (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665794#comment-16665794
 ] 

Thejas M Nair edited comment on HIVE-20815 at 10/27/18 12:29 AM:
-

[~daijy]
Can you also add a UT for this ?
I assume a table definition with wrong connection params would work ?



was (Author: thejas):
[~daijy]
Can you also add a UT for this ?
I assume a query with wrong connection params would work ?


> JdbcRecordReader.next shall not eat exception
> -
>
> Key: HIVE-20815
> URL: https://issues.apache.org/jira/browse/HIVE-20815
> Project: Hive
>  Issue Type: Bug
>  Components: StorageHandler
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20815.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-18202) Automatically migrate hbase.table.name to hbase.mapreduce.hfileoutputformat.table.name for hbase-based table

2018-10-26 Thread Aihua Xu (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-18202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu reassigned HIVE-18202:
---

Assignee: Aihua Xu

> Automatically migrate hbase.table.name to 
> hbase.mapreduce.hfileoutputformat.table.name for hbase-based table
> 
>
> Key: HIVE-18202
> URL: https://issues.apache.org/jira/browse/HIVE-18202
> Project: Hive
>  Issue Type: Sub-task
>  Components: HBase Handler
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18202.1.patch, HIVE-18202.patch.addendum
>
>
> The property name for Hbase table mapping is changed from hbase.table.name to 
> hbase.mapreduce.hfileoutputformat.table.name in HBase 2.
> We can include such upgrade for existing hbase-based tables in DB upgrade 
> script to automatically change such values.
> For the new tables, the query will be like:
> create table hbase_table(key int, val string) stored by 
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties 
> ('hbase.columns.mapping' = ':key,cf:val') tblproperties 
> ('hbase.mapreduce.hfileoutputformat.table.name' = 
> 'positive_hbase_handler_bulk')



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-18366) Update HBaseSerDe to use hbase.mapreduce.hfileoutputformat.table.name instead of hbase.table.name as the table name property

2018-10-26 Thread Aihua Xu (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-18366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu reassigned HIVE-18366:
---

Assignee: Aihua Xu

> Update HBaseSerDe to use hbase.mapreduce.hfileoutputformat.table.name instead 
> of hbase.table.name as the table name property
> 
>
> Key: HIVE-18366
> URL: https://issues.apache.org/jira/browse/HIVE-18366
> Project: Hive
>  Issue Type: Sub-task
>  Components: HBase Handler
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18366.1.patch, HIVE-18366.2.patch
>
>
> HBase 2.0 changes the table name property to 
> hbase.mapreduce.hfileoutputformat.table.name. HiveHFileOutputFormat is using 
> the new property name while HiveHBaseTableOutputFormat is not. If we create 
> the table as follows, HiveHBaseTableOutputFormat is used which still uses the 
> old property hbase.table.name.
> {noformat}
> create table hbase_table2(key int, val string) stored by 
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties 
> ('hbase.columns.mapping' = ':key,cf:val') tblproperties 
> ('hbase.mapreduce.hfileoutputformat.table.name' = 
> 'positive_hbase_handler_bulk')
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20805) Hive does not copy source data when importing as non-hive user

2018-10-26 Thread Thejas M Nair (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-20805:
-
Attachment: HIVE-20805.1.patch

> Hive does not copy source data when importing as non-hive user 
> ---
>
> Key: HIVE-20805
> URL: https://issues.apache.org/jira/browse/HIVE-20805
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, HiveServer2
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: Thejas M Nair
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20805.1.patch
>
>
> while loading data to a managed table from user given path, Hive uses move 
> operation to copy data from user location to table location. In case move can 
> not be used due to permission issue or mismatched encryption zone etc, hive 
> uses copy and then deletes the files from source location to keep to behavior 
> same. But in case the user does not have write access to the source location, 
> delete will fail with file permission exception and load operation will fail. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-20805) Hive does not copy source data when importing as non-hive user

2018-10-26 Thread Thejas M Nair (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair reassigned HIVE-20805:


Assignee: mahesh kumar behera  (was: Thejas M Nair)

> Hive does not copy source data when importing as non-hive user 
> ---
>
> Key: HIVE-20805
> URL: https://issues.apache.org/jira/browse/HIVE-20805
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, HiveServer2
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20805.1.patch
>
>
> while loading data to a managed table from user given path, Hive uses move 
> operation to copy data from user location to table location. In case move can 
> not be used due to permission issue or mismatched encryption zone etc, hive 
> uses copy and then deletes the files from source location to keep to behavior 
> same. But in case the user does not have write access to the source location, 
> delete will fail with file permission exception and load operation will fail. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-20805) Hive does not copy source data when importing as non-hive user

2018-10-26 Thread Thejas M Nair (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair reassigned HIVE-20805:


Assignee: Thejas M Nair  (was: mahesh kumar behera)

> Hive does not copy source data when importing as non-hive user 
> ---
>
> Key: HIVE-20805
> URL: https://issues.apache.org/jira/browse/HIVE-20805
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, HiveServer2
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: Thejas M Nair
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20805.1.patch
>
>
> while loading data to a managed table from user given path, Hive uses move 
> operation to copy data from user location to table location. In case move can 
> not be used due to permission issue or mismatched encryption zone etc, hive 
> uses copy and then deletes the files from source location to keep to behavior 
> same. But in case the user does not have write access to the source location, 
> delete will fail with file permission exception and load operation will fail. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20805) Hive does not copy source data when importing as non-hive user

2018-10-26 Thread Thejas M Nair (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665786#comment-16665786
 ] 

Thejas M Nair commented on HIVE-20805:
--

+1 pending tests


> Hive does not copy source data when importing as non-hive user 
> ---
>
> Key: HIVE-20805
> URL: https://issues.apache.org/jira/browse/HIVE-20805
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, HiveServer2
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20805.1.patch
>
>
> while loading data to a managed table from user given path, Hive uses move 
> operation to copy data from user location to table location. In case move can 
> not be used due to permission issue or mismatched encryption zone etc, hive 
> uses copy and then deletes the files from source location to keep to behavior 
> same. But in case the user does not have write access to the source location, 
> delete will fail with file permission exception and load operation will fail. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-20823) Make Compactor run in a transaction

2018-10-26 Thread Eugene Koifman (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman reassigned HIVE-20823:
-


> Make Compactor run in a transaction
> ---
>
> Key: HIVE-20823
> URL: https://issues.apache.org/jira/browse/HIVE-20823
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> Have compactor open a transaction and run the job in that transaction.
> # make compactor produced base/delta include this txn id in the folder name, 
> e.g. base_7_c17 where 17 is the txnid.
> # add {{CQ_TXN_ID bigint}} to COMPACTION_QUEUE and COMPLETED_COMPACTIONS to 
> record this txn id
> # make sure {{AcidUtils.getAcidState()}} pays attention to this transaction 
> on read and ignores this dir if this txn id is not committed in the current 
> snapshot
> ## this means not only validWriteIdList but ValidTxnIdList should be passed 
> along in config (if it isn't yet)
> # once this is done, {{CompactorMR.createCompactorMarker()}} can be 
> eliminated and {{AcidUtils.isValidBase}} modified accordingly
> # modify Cleaner so that it doesn't clean old files until new file is visible 
> to all readers
> # 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler

2018-10-26 Thread Bharathkrishna Guruvayoor Murali (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharathkrishna Guruvayoor Murali updated HIVE-20512:

Attachment: HIVE-20512.4.patch

> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch, HIVE-20512.2.patch, 
> HIVE-20512.3.patch, HIVE-20512.4.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but 
> we should improve the methodology for how frequently we log this info. 
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if (currentThreshold >= 100) {
>   return currentThreshold + 100;
> }
> return 10 * currentThreshold;
>   }
> {code}
> The issue is that after a while, the increase by 10x factor means that you 
> have to process a huge # of records before this gets triggered.
> A better approach would be to log this info at a given interval. This would 
> help in debugging tasks that are seemingly hung.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-20822) Improvements to push computation to JDBC from Calcite

2018-10-26 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez reassigned HIVE-20822:
--


> Improvements to push computation to JDBC from Calcite
> -
>
> Key: HIVE-20822
> URL: https://issues.apache.org/jira/browse/HIVE-20822
> Project: Hive
>  Issue Type: Improvement
>  Components: StorageHandler
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20793) add RP namespacing to workload management

2018-10-26 Thread Prasanth Jayachandran (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665756#comment-16665756
 ] 

Prasanth Jayachandran commented on HIVE-20793:
--

+1, pending tests

> add RP namespacing to workload management
> -
>
> Key: HIVE-20793
> URL: https://issues.apache.org/jira/browse/HIVE-20793
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20793.01.nogen.patch, HIVE-20793.01.patch, 
> HIVE-20793.02.nogen.patch, HIVE-20793.02.patch, HIVE-20793.nogen.patch, 
> HIVE-20793.patch
>
>
> The idea is to be able to use the same warehouse for multiple clusters in the 
> cloud use cases. This scenario is not currently supported by WM.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20793) add RP namespacing to workload management

2018-10-26 Thread Sergey Shelukhin (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665753#comment-16665753
 ] 

Sergey Shelukhin commented on HIVE-20793:
-

Addressed the feedback, fixed a bug. Also fixed some old issue that I noticed 
while fixing the bug - having the child pools take the entire capacity of the 
parent (i.e. 1.0) should be allowed. Fixed that too.

> add RP namespacing to workload management
> -
>
> Key: HIVE-20793
> URL: https://issues.apache.org/jira/browse/HIVE-20793
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20793.01.nogen.patch, HIVE-20793.01.patch, 
> HIVE-20793.02.nogen.patch, HIVE-20793.02.patch, HIVE-20793.nogen.patch, 
> HIVE-20793.patch
>
>
> The idea is to be able to use the same warehouse for multiple clusters in the 
> cloud use cases. This scenario is not currently supported by WM.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20793) add RP namespacing to workload management

2018-10-26 Thread Sergey Shelukhin (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-20793:

Attachment: HIVE-20793.02.patch

> add RP namespacing to workload management
> -
>
> Key: HIVE-20793
> URL: https://issues.apache.org/jira/browse/HIVE-20793
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20793.01.nogen.patch, HIVE-20793.01.patch, 
> HIVE-20793.02.nogen.patch, HIVE-20793.02.patch, HIVE-20793.nogen.patch, 
> HIVE-20793.patch
>
>
> The idea is to be able to use the same warehouse for multiple clusters in the 
> cloud use cases. This scenario is not currently supported by WM.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20793) add RP namespacing to workload management

2018-10-26 Thread Sergey Shelukhin (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-20793:

Attachment: HIVE-20793.02.nogen.patch

> add RP namespacing to workload management
> -
>
> Key: HIVE-20793
> URL: https://issues.apache.org/jira/browse/HIVE-20793
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20793.01.nogen.patch, HIVE-20793.01.patch, 
> HIVE-20793.02.nogen.patch, HIVE-20793.nogen.patch, HIVE-20793.patch
>
>
> The idea is to be able to use the same warehouse for multiple clusters in the 
> cloud use cases. This scenario is not currently supported by WM.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20792) Inserting timestamp with zones truncates the data

2018-10-26 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665749#comment-16665749
 ] 

Hive QA commented on HIVE-20792:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12945619/HIVE-20792.3.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 15508 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/14647/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14647/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14647/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12945619 - PreCommit-HIVE-Build

> Inserting timestamp with zones truncates the data
> -
>
> Key: HIVE-20792
> URL: https://issues.apache.org/jira/browse/HIVE-20792
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Affects Versions: 3.1.0
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20792.1.patch, HIVE-20792.2.patch, 
> HIVE-20792.3.patch
>
>
> For example with the table:
> {code}
> CREATE TABLE myTable
> (
> a TIMESTAMP
> )
> STORED AS ORC
> tblproperties("transactional"="true");
> {code}
> The following inserts store the wrong data:
> {code}
> INSERT INTO myTable VALUES("2018-10-19 10:35:00 UTC"); -> 2018-10-19 
> 00:00:00.0
> INSERT INTO myTable VALUES("2018-10-19 10:35:00 ZZZ"); -> 2018-10-19 
> 00:00:00.0
> {code}
> The second one should fail since ZZZ is not a time zone.
> Similarly if the column is of type DATE,
> {code}
> INSERT INTO myTableDate VALUES("2018-10-19 "); -> 2018-10-19
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20792) Inserting timestamp with zones truncates the data

2018-10-26 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665733#comment-16665733
 ] 

Hive QA commented on HIVE-20792:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 2s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
31s{color} | {color:blue} common in master has 65 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
36s{color} | {color:blue} serde in master has 198 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
47s{color} | {color:blue} ql in master has 2317 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
15s{color} | {color:red} serde: The patch generated 1 new + 191 unchanged - 1 
fixed = 192 total (was 192) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 47s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14647/dev-support/hive-personality.sh
 |
| git revision | master / 1002e89 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14647/yetus/diff-checkstyle-serde.txt
 |
| modules | C: common serde ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14647/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Inserting timestamp with zones truncates the data
> -
>
> Key: HIVE-20792
> URL: https://issues.apache.org/jira/browse/HIVE-20792
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Affects Versions: 3.1.0
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20792.1.patch, HIVE-20792.2.patch, 
> HIVE-20792.3.patch
>
>
> For example with the table:
> {code}
> CREATE TABLE myTable
> (
> a TIMESTAMP
> )
> STORED AS ORC
> tblproperties("transactional"="true");
> {code}
> The following inserts store the wrong data:
> {code}
> INSERT INTO myTable VALUES("2018-10-19 10:35:00 UTC"); -> 2018-10-19 
> 00:00:00.0
> INSERT INTO myTable VALUES("2018-10-19 10:35:00 ZZZ"); -> 2018-10-19 
> 00:00:00.0
> {code}
> The second one should

[jira] [Commented] (HIVE-20816) FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)

2018-10-26 Thread Rajesh Balamohan (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665719#comment-16665719
 ] 

Rajesh Balamohan commented on HIVE-20816:
-

[~gopalv] - This issue happened with the fix in HIVE-19085

> FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)
> --
>
> Key: HIVE-20816
> URL: https://issues.apache.org/jira/browse/HIVE-20816
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 2.3.2
>Reporter: Rajesh Balamohan
>Priority: Minor
>  Labels: fastdecimal
> Attachments: HIVE-20816.1.patch
>
>
> {noformat}
> with t1 as (
> ...
> ...
> )
> select id, max(abs(c1))) from t1 group by id;
> {noformat}
> throws the following exception
> {noformat}
> g.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
>  at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1126)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
> 
> ...
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1084)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1123)
> ... 18 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.RuntimeException: Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1047)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1067)
> ... 19 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20821) Rewrite SUM0 into SUM + COALESCE combination

2018-10-26 Thread Gopal V (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665686#comment-16665686
 ] 

Gopal V commented on HIVE-20821:


LGTM  +1  - tests pending

> Rewrite SUM0 into SUM + COALESCE combination
> 
>
> Key: HIVE-20821
> URL: https://issues.apache.org/jira/browse/HIVE-20821
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20821.patch
>
>
> Since SUM0 is not vectorized, but SUM + COALESCE are.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20821) Rewrite SUM0 into SUM + COALESCE combination

2018-10-26 Thread Jesus Camacho Rodriguez (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665684#comment-16665684
 ] 

Jesus Camacho Rodriguez commented on HIVE-20821:


Cc [~gopalv] [~ndembla]

> Rewrite SUM0 into SUM + COALESCE combination
> 
>
> Key: HIVE-20821
> URL: https://issues.apache.org/jira/browse/HIVE-20821
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20821.patch
>
>
> Since SUM0 is not vectorized, but SUM + COALESCE are.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20821) Rewrite SUM0 into SUM + COALESCE combination

2018-10-26 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-20821:
---
Attachment: HIVE-20821.patch

> Rewrite SUM0 into SUM + COALESCE combination
> 
>
> Key: HIVE-20821
> URL: https://issues.apache.org/jira/browse/HIVE-20821
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20821.patch
>
>
> Since SUM0 is not vectorized, but SUM + COALESCE are.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (HIVE-20821) Rewrite SUM0 into SUM + COALESCE combination

2018-10-26 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-20821 started by Jesus Camacho Rodriguez.
--
> Rewrite SUM0 into SUM + COALESCE combination
> 
>
> Key: HIVE-20821
> URL: https://issues.apache.org/jira/browse/HIVE-20821
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20821.patch
>
>
> Since SUM0 is not vectorized, but SUM + COALESCE are.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20821) Rewrite SUM0 into SUM + COALESCE combination

2018-10-26 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-20821:
---
Status: Patch Available  (was: In Progress)

> Rewrite SUM0 into SUM + COALESCE combination
> 
>
> Key: HIVE-20821
> URL: https://issues.apache.org/jira/browse/HIVE-20821
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20821.patch
>
>
> Since SUM0 is not vectorized, but SUM + COALESCE are.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-20821) Rewrite SUM0 into SUM + COALESCE combination

2018-10-26 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez reassigned HIVE-20821:
--


> Rewrite SUM0 into SUM + COALESCE combination
> 
>
> Key: HIVE-20821
> URL: https://issues.apache.org/jira/browse/HIVE-20821
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Since SUM0 is not vectorized, but SUM + COALESCE are.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20605) merge master-tez092 branch into master

2018-10-26 Thread Sergey Shelukhin (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665676#comment-16665676
 ] 

Sergey Shelukhin commented on HIVE-20605:
-

merged master into branch to make sure we stay on top of conflicts... I will 
just keep doing this in this JIRA

> merge master-tez092 branch into master
> --
>
> Key: HIVE-20605
> URL: https://issues.apache.org/jira/browse/HIVE-20605
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>
> I got tired of waiting for Tez 0.92 release (it's been pending for half a 
> year) so I created a branch to prevent various patches from conflicting with 
> each other.
> This jira is to merge them into master after Tez 0.92 is finally released.
> The jiras here: 
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20HIVE%20AND%20fixVersion%20%3D%20master-tez092
>  should then be updated with the corresponding Hive release version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20820) MV partition on clause position

2018-10-26 Thread Vineet Garg (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665638#comment-16665638
 ] 

Vineet Garg commented on HIVE-20820:


+1 pending tests

> MV partition on clause position
> ---
>
> Key: HIVE-20820
> URL: https://issues.apache.org/jira/browse/HIVE-20820
> Project: Hive
>  Issue Type: Bug
>  Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-20820.patch
>
>
> It should obey the following syntax as per 
> https://cwiki.apache.org/confluence/display/Hive/Materialized+views:
> {code}
> CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
>   [DISABLE REWRITE]
>   [COMMENT materialized_view_comment]
>   [PARTITIONED ON (col_name, ...)]
>   [
> [ROW FORMAT row_format]
> [STORED AS file_format]
>   | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
>   ]
>   [LOCATION hdfs_path]
>   [TBLPROPERTIES (property_name=property_value, ...)]
> AS
> ;
> {code}
> Currently it is positioned just before TBLPROPERTIES.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-10459) Add materialized views to Hive

2018-10-26 Thread Jesus Camacho Rodriguez (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-10459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665627#comment-16665627
 ] 

Jesus Camacho Rodriguez edited comment on HIVE-10459 at 10/26/18 8:06 PM:
--

[~thai.bui], would you mind to send your question to dev mailing list and we 
can continue discussion over there? Thanks

I am going to close this issue as initial work for implementing materialized 
views support has been mostly completed. Documentation can be found in: 
https://cwiki.apache.org/confluence/display/Hive/Materialized+views


was (Author: jcamachorodriguez):
[~thai.bui], would you mind to send your question to mailing list and we can 
continue discussion over there? Thanks

I am going to close this issue as initial work for implementing materialized 
views support has been mostly completed. Documentation can be found in: 
https://cwiki.apache.org/confluence/display/Hive/Materialized+views

> Add materialized views to Hive
> --
>
> Key: HIVE-10459
> URL: https://issues.apache.org/jira/browse/HIVE-10459
> Project: Hive
>  Issue Type: New Feature
>  Components: Views
>Reporter: Alan Gates
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Fix For: 3.2.0
>
>
> Materialized views are useful as ways to store either alternate versions of 
> data (e.g. same data, different sort order) or derivatives of data sets (e.g. 
> commonly used aggregates).  It is useful to store these as materialized views 
> rather than as tables because it can give the optimizer the ability to 
> understand how data sets are related.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20820) MV partition on clause position

2018-10-26 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-20820:
---
Attachment: HIVE-20820.patch

> MV partition on clause position
> ---
>
> Key: HIVE-20820
> URL: https://issues.apache.org/jira/browse/HIVE-20820
> Project: Hive
>  Issue Type: Bug
>  Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-20820.patch
>
>
> It should obey the following syntax as per 
> https://cwiki.apache.org/confluence/display/Hive/Materialized+views:
> {code}
> CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
>   [DISABLE REWRITE]
>   [COMMENT materialized_view_comment]
>   [PARTITIONED ON (col_name, ...)]
>   [
> [ROW FORMAT row_format]
> [STORED AS file_format]
>   | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
>   ]
>   [LOCATION hdfs_path]
>   [TBLPROPERTIES (property_name=property_value, ...)]
> AS
> ;
> {code}
> Currently it is positioned just before TBLPROPERTIES.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-20820) MV partition on clause position

2018-10-26 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez reassigned HIVE-20820:
--


> MV partition on clause position
> ---
>
> Key: HIVE-20820
> URL: https://issues.apache.org/jira/browse/HIVE-20820
> Project: Hive
>  Issue Type: Bug
>  Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
>
> It should obey the following syntax as per 
> https://cwiki.apache.org/confluence/display/Hive/Materialized+views:
> {code}
> CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
>   [DISABLE REWRITE]
>   [COMMENT materialized_view_comment]
>   [PARTITIONED ON (col_name, ...)]
>   [
> [ROW FORMAT row_format]
> [STORED AS file_format]
>   | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
>   ]
>   [LOCATION hdfs_path]
>   [TBLPROPERTIES (property_name=property_value, ...)]
> AS
> ;
> {code}
> Currently it is positioned just before TBLPROPERTIES.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (HIVE-20820) MV partition on clause position

2018-10-26 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-20820 started by Jesus Camacho Rodriguez.
--
> MV partition on clause position
> ---
>
> Key: HIVE-20820
> URL: https://issues.apache.org/jira/browse/HIVE-20820
> Project: Hive
>  Issue Type: Bug
>  Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
>
> It should obey the following syntax as per 
> https://cwiki.apache.org/confluence/display/Hive/Materialized+views:
> {code}
> CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
>   [DISABLE REWRITE]
>   [COMMENT materialized_view_comment]
>   [PARTITIONED ON (col_name, ...)]
>   [
> [ROW FORMAT row_format]
> [STORED AS file_format]
>   | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
>   ]
>   [LOCATION hdfs_path]
>   [TBLPROPERTIES (property_name=property_value, ...)]
> AS
> ;
> {code}
> Currently it is positioned just before TBLPROPERTIES.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20820) MV partition on clause position

2018-10-26 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-20820:
---
Status: Patch Available  (was: In Progress)

> MV partition on clause position
> ---
>
> Key: HIVE-20820
> URL: https://issues.apache.org/jira/browse/HIVE-20820
> Project: Hive
>  Issue Type: Bug
>  Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
>
> It should obey the following syntax as per 
> https://cwiki.apache.org/confluence/display/Hive/Materialized+views:
> {code}
> CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
>   [DISABLE REWRITE]
>   [COMMENT materialized_view_comment]
>   [PARTITIONED ON (col_name, ...)]
>   [
> [ROW FORMAT row_format]
> [STORED AS file_format]
>   | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
>   ]
>   [LOCATION hdfs_path]
>   [TBLPROPERTIES (property_name=property_value, ...)]
> AS
> ;
> {code}
> Currently it is positioned just before TBLPROPERTIES.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-10459) Add materialized views to Hive

2018-10-26 Thread Jesus Camacho Rodriguez (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-10459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665627#comment-16665627
 ] 

Jesus Camacho Rodriguez commented on HIVE-10459:


[~thai.bui], would you mind to send your question to mailing list and we can 
continue discussion over there? Thanks

I am going to close this issue as initial work for implementing materialized 
views support has been mostly completed. Documentation can be found in: 
https://cwiki.apache.org/confluence/display/Hive/Materialized+views

> Add materialized views to Hive
> --
>
> Key: HIVE-10459
> URL: https://issues.apache.org/jira/browse/HIVE-10459
> Project: Hive
>  Issue Type: New Feature
>  Components: Views
>Reporter: Alan Gates
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Fix For: 3.2.0
>
>
> Materialized views are useful as ways to store either alternate versions of 
> data (e.g. same data, different sort order) or derivatives of data sets (e.g. 
> commonly used aggregates).  It is useful to store these as materialized views 
> rather than as tables because it can give the optimizer the ability to 
> understand how data sets are related.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HIVE-10459) Add materialized views to Hive

2018-10-26 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-10459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez resolved HIVE-10459.

   Resolution: Fixed
Fix Version/s: 3.2.0
   4.0.0

> Add materialized views to Hive
> --
>
> Key: HIVE-10459
> URL: https://issues.apache.org/jira/browse/HIVE-10459
> Project: Hive
>  Issue Type: New Feature
>  Components: Views
>Reporter: Alan Gates
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Fix For: 4.0.0, 3.2.0
>
>
> Materialized views are useful as ways to store either alternate versions of 
> data (e.g. same data, different sort order) or derivatives of data sets (e.g. 
> commonly used aggregates).  It is useful to store these as materialized views 
> rather than as tables because it can give the optimizer the ability to 
> understand how data sets are related.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-10459) Add materialized views to Hive

2018-10-26 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-10459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-10459:
---
Fix Version/s: (was: 4.0.0)

> Add materialized views to Hive
> --
>
> Key: HIVE-10459
> URL: https://issues.apache.org/jira/browse/HIVE-10459
> Project: Hive
>  Issue Type: New Feature
>  Components: Views
>Reporter: Alan Gates
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Fix For: 3.2.0
>
>
> Materialized views are useful as ways to store either alternate versions of 
> data (e.g. same data, different sort order) or derivatives of data sets (e.g. 
> commonly used aggregates).  It is useful to store these as materialized views 
> rather than as tables because it can give the optimizer the ability to 
> understand how data sets are related.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-20819) Leaking Metastore connections when HADOOP_USER_NAME environmental variable is set

2018-10-26 Thread Roohi Syeda (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roohi Syeda reassigned HIVE-20819:
--


> Leaking Metastore connections when HADOOP_USER_NAME environmental variable is 
> set
> -
>
> Key: HIVE-20819
> URL: https://issues.apache.org/jira/browse/HIVE-20819
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Roohi Syeda
>Assignee: Roohi Syeda
>Priority: Minor
>
> Leaking Metastore connections when HADOOP_USER_NAME environmental variable is 
> set.
> The connections created are in ESTABLISHED state and never closed



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20793) add RP namespacing to workload management

2018-10-26 Thread Prasanth Jayachandran (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665579#comment-16665579
 ] 

Prasanth Jayachandran commented on HIVE-20793:
--

some minor comments. mostly looks good. 

> add RP namespacing to workload management
> -
>
> Key: HIVE-20793
> URL: https://issues.apache.org/jira/browse/HIVE-20793
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20793.01.nogen.patch, HIVE-20793.01.patch, 
> HIVE-20793.nogen.patch, HIVE-20793.patch
>
>
> The idea is to be able to use the same warehouse for multiple clusters in the 
> cloud use cases. This scenario is not currently supported by WM.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-10459) Add materialized views to Hive

2018-10-26 Thread Thai Bui (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-10459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665550#comment-16665550
 ] 

Thai Bui commented on HIVE-10459:
-

[~jcamachorodriguez] [~alangates] Would it be possible to support materialized 
views on external tables? For examples, there are cases where the tables are 
mostly immutable, or dropped & recreated as external tables with date 
table_name___mm_dd postfix, with a stable view on top of that such tables 
as table_name -> table_name___mm_dd, materialized views could work with 
cache invalidations and query result caching.

I'm currently hacking Hive to support such use cases since my company would 
mostly rely on immutable datasets on AWS S3 (by removing the transactional 
hardcoded constraints). However, I think Hive should support this feature as a 
first class citizen since the industry is moving towards more workloads in the 
cloud.

The feature in https://issues.apache.org/jira/browse/HIVE-19154 could be used 
to invalidate cache & push notifications externally into Hive via the cloud 
blob storage's change events as well. Although I am not familiar with the 
internal of Hive enough to understand how everything could work together.

> Add materialized views to Hive
> --
>
> Key: HIVE-10459
> URL: https://issues.apache.org/jira/browse/HIVE-10459
> Project: Hive
>  Issue Type: New Feature
>  Components: Views
>Reporter: Alan Gates
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Materialized views are useful as ways to store either alternate versions of 
> data (e.g. same data, different sort order) or derivatives of data sets (e.g. 
> commonly used aggregates).  It is useful to store these as materialized views 
> rather than as tables because it can give the optimizer the ability to 
> understand how data sets are related.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.

2018-10-26 Thread Sankar Hariappan (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665492#comment-16665492
 ] 

Sankar Hariappan edited comment on HIVE-20682 at 10/26/18 6:43 PM:
---

[~pvary], Thanks for your thoughts!
 * H1 is sessionHive object and will be preserved in session object through out 
the life time of session. HMS connection of sessionHive will be closed only 
when we close the session.
 * H2 is local to given thread and will be closed when we set sessionHive again 
in next query. ThreadLocalHive.set() method closes the previous thread local 
Hive object H2 before overwriting it.
 * Also, if an async thread reallocates Hive object (let's say H3), then it 
will be closed by Hive.closeCurrent call when the thread exits.
 * So, I don't think there would be any HMS connection leak here.
 * parentHive is always sessionHive object where allowClose flag is false. So, 
the "assert (!parentHive.allowClose());" will never fail.

But I agree, there is a chance that each and every query re-creates Hive object 
if the HMS relevant configs are changed in between queries. I missed this part 
where sessionConf would be changed when user sets any Hive configurations via 
cli. 

So, I think, the earlier thread reference count solution would solve this 
issue. [~maheshk114], please comment if you think otherwise.

Btw, there is one another issue where Hive.get(sessionConf) directly stores the 
reference to sessionConf in Hive object which makes isCompatible() method to 
return true always.
{code:java}
private static Hive getInternal(HiveConf c, boolean needsRefresh, boolean 
isFastCheck,
boolean doRegisterAllFns) throws HiveException {
  Hive db = hiveDB.get();
  if (db == null || !db.isCurrentUserOwner() || needsRefresh
  || (c != null && !isCompatible(db, c, isFastCheck))) {
if (db != null) {
  LOG.debug("Creating new db. db = " + db + ", needsRefresh = " + 
needsRefresh +
  ", db.isCurrentUserOwner = " + db.isCurrentUserOwner());
  closeCurrent();
}
db = create(c, doRegisterAllFns);
  }
  if (c != null) {
db.conf = c;
  }
  return db;
}{code}
So, as of today, Thread local Hive object never gets recreated even if we 
change any HMS configs. Only REPL commands re-create Hive object as it uses 
different HiveConf object not sessionConf to set HMS configs.
  
 This issue can be resolved by always cloning HiveConf object instead of 
storing the reference to input HiveConf.

But, it will be problem for Hive.getWithFastCheck(conf). I think, this method 
is added as an optimisation to avoid checking individual HMS configs. It only 
checks if the input conf object is same as the one stored in Hive object and if 
not, then re-create Hive object.

So, this needs to be fixed carefully. 

Please share your thoughts on this.

cc [~daijy], [~thejas], [~maheshk114]

 


was (Author: sankarh):
[~pvary], Thanks for your thoughts!
 * H1 is sessionHive object and will be preserved in session object through out 
the life time of session. HMS connection of sessionHive will be closed only 
when we close the session.
 * H2 is local to given thread and will be closed when we set sessionHive again 
in next query. ThreadLocalHive.set() method closes the previous thread local 
Hive object H2 before overwriting it.
 * Also, if an async thread reallocates Hive object (let's say H3), then it 
will be closed by Hive.closeCurrent call when the thread exits.
 * So, I don't think there would be any HMS connection leak here.
 * parentHive is always sessionHive object where allowClose flag is false. So, 
the "assert (!parentHive.allowClose());" will never fail.

But I agree, there is a chance that each and every query re-creates Hive object 
if the HMS relevant configs are changed in between queries. I missed this part 
where sessionConf would be changed when user sets any Hive configurations via 
cli. 

So, I think, the earlier thread reference count solution would solve this 
issue. [~maheshk114], please comment if you think otherwise.

Btw, there is one another issue where Hive.get(sessionConf) directly stores the 
reference to sessionConf in Hive object which makes isCompatible() method to 
return true always.
{code:java}
private static Hive getInternal(HiveConf c, boolean needsRefresh, boolean 
isFastCheck,
boolean doRegisterAllFns) throws HiveException {
  Hive db = hiveDB.get();
  if (db == null || !db.isCurrentUserOwner() || needsRefresh
  || (c != null && !isCompatible(db, c, isFastCheck))) {
if (db != null) {
  LOG.debug("Creating new db. db = " + db + ", needsRefresh = " + 
needsRefresh +
  ", db.isCurrentUserOwner = " + db.isCurrentUserOwner());
  closeCurrent();
}
db = create(c, doRegisterAllFns);
  }
  if (c != null) {
db.conf = c;
  }
  return db;
}{code}
So, as of today, Thread local Hive object never gets recreated even if we

[jira] [Updated] (HIVE-20486) Kafka: Use Row SerDe + vectorization

2018-10-26 Thread slim bouguerra (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

slim bouguerra updated HIVE-20486:
--
Attachment: HIVE-20486.3.patch

> Kafka: Use Row SerDe + vectorization
> 
>
> Key: HIVE-20486
> URL: https://issues.apache.org/jira/browse/HIVE-20486
> Project: Hive
>  Issue Type: Improvement
>  Components: kafka integration
>Reporter: Gopal V
>Assignee: slim bouguerra
>Priority: Major
>  Labels: kafka, vectorization
> Fix For: 4.0.0
>
> Attachments: HIVE-20486.3.patch, HIVE-20486.3.patch, HIVE-20486.patch
>
>
> KafkaHandler returns unvectorized rows which causes the operators downstream 
> to be slower and sub-optimal.
> Hive has a vectorization shim which allows Kafka streams without complex 
> projections to be wrapped into a vectorized reader via 
> {{hive.vectorized.use.row.serde.deserialize}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20748) Disable materialized view rewriting when plan pattern is not allowed

2018-10-26 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-20748:
---
Attachment: HIVE-20748.02.patch

> Disable materialized view rewriting when plan pattern is not allowed
> 
>
> Key: HIVE-20748
> URL: https://issues.apache.org/jira/browse/HIVE-20748
> Project: Hive
>  Issue Type: Bug
>  Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20748.01.patch, HIVE-20748.01.patch, 
> HIVE-20748.02.patch, HIVE-20748.02.patch, HIVE-20748.patch
>
>
> For instance, currently rewriting algorithm does not support some operators. 
> Or we cannot have non-deterministic function in the MV definition. In those 
> cases, we should fail either when we try to create the MV with rewriting 
> enabled, or when when we enable the rewriting for a MV already created.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.

2018-10-26 Thread Sankar Hariappan (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665492#comment-16665492
 ] 

Sankar Hariappan edited comment on HIVE-20682 at 10/26/18 6:16 PM:
---

[~pvary], Thanks for your thoughts!
 * H1 is sessionHive object and will be preserved in session object through out 
the life time of session. HMS connection of sessionHive will be closed only 
when we close the session.
 * H2 is local to given thread and will be closed when we set sessionHive again 
in next query. ThreadLocalHive.set() method closes the previous thread local 
Hive object H2 before overwriting it.
 * Also, if an async thread reallocates Hive object (let's say H3), then it 
will be closed by Hive.closeCurrent call when the thread exits.
 * So, I don't think there would be any HMS connection leak here.
 * parentHive is always sessionHive object where allowClose flag is false. So, 
the "assert (!parentHive.allowClose());" will never fail.

But I agree, there is a chance that each and every query re-creates Hive object 
if the HMS relevant configs are changed in between queries. I missed this part 
where sessionConf would be changed when user sets any Hive configurations via 
cli. 

So, I think, the earlier thread reference count solution would solve this 
issue. [~maheshk114], please comment if you think otherwise.

Btw, there is one another issue where Hive.get(sessionConf) directly stores the 
reference to sessionConf in Hive object which makes isCompatible() method to 
return true always.
{code:java}
private static Hive getInternal(HiveConf c, boolean needsRefresh, boolean 
isFastCheck,
boolean doRegisterAllFns) throws HiveException {
  Hive db = hiveDB.get();
  if (db == null || !db.isCurrentUserOwner() || needsRefresh
  || (c != null && !isCompatible(db, c, isFastCheck))) {
if (db != null) {
  LOG.debug("Creating new db. db = " + db + ", needsRefresh = " + 
needsRefresh +
  ", db.isCurrentUserOwner = " + db.isCurrentUserOwner());
  closeCurrent();
}
db = create(c, doRegisterAllFns);
  }
  if (c != null) {
db.conf = c;
  }
  return db;
}{code}
So, as of today, Thread local Hive object never gets recreated even if we 
change any HMS configs. Only REPL commands re-create Hive object as it uses 
different HiveConf object not sessionConf to set HMS configs.
 
This issue can be resolved by always cloning HiveConf object instead of storing 
the input HiveConf reference.

But, it will be problem for Hive.getWithFastCheck(conf). I think, this method 
is added as an optimisation to avoid checking individual HMS configs. It only 
checks if the input conf object is same as the one stored in Hive object and if 
not, then re-create Hive object.

So, this needs to be fixed carefully. 

Please share your thoughts on this.

cc [~daijy], [~thejas], [~maheshk114]

 


was (Author: sankarh):
[~pvary], Thanks for your thoughts!
 * H1 is sessionHive object and will be preserved in session object through out 
the life time of session. HMS connection of sessionHive will be closed only 
when we close the session.
 * H2 is local to given thread and will be closed when we set sessionHive again 
in next query. ThreadLocalHive.set() method closes the previous thread local 
Hive object H2 before overwriting it.
 * Also, if an async thread reallocates Hive object (let's say H3), then it 
will be closed by Hive.closeCurrent call when the thread exits.
 * So, I don't think there would be any HMS connection leak here.
 * parentHive is always sessionHive object where allowClose flag is false. So, 
the "assert (!parentHive.allowClose());" will never fail.

But I agree, there is a chance that each and every query re-creates Hive object 
if the HMS relevant configs are changed in between queries. I missed this part 
where sessionConf would be changed when user sets any Hive configurations via 
cli. 

So, I think, the earlier thread reference count solution would solve this 
issue. [~maheshk114], please comment if you think otherwise.

Btw, there is one another issue where Hive.get(sessionConf) directly stores the 
reference to sessionConf in Hive object which makes isCompatible() method to 
return true always.
{code:java}
private static Hive getInternal(HiveConf c, boolean needsRefresh, boolean 
isFastCheck,
boolean doRegisterAllFns) throws HiveException {
  Hive db = hiveDB.get();
  if (db == null || !db.isCurrentUserOwner() || needsRefresh
  || (c != null && !isCompatible(db, c, isFastCheck))) {
if (db != null) {
  LOG.debug("Creating new db. db = " + db + ", needsRefresh = " + 
needsRefresh +
  ", db.isCurrentUserOwner = " + db.isCurrentUserOwner());
  closeCurrent();
}
db = create(c, doRegisterAllFns);
  }
  if (c != null) {
db.conf = c;
  }
  return db;
}{code}
So, as of today, Thread local Hive object never gets recreated even if we 
change

[jira] [Commented] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.

2018-10-26 Thread Sankar Hariappan (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665492#comment-16665492
 ] 

Sankar Hariappan commented on HIVE-20682:
-

[~pvary], Thanks for your thoughts!
 * H1 is sessionHive object and will be preserved in session object through out 
the life time of session. HMS connection of sessionHive will be closed only 
when we close the session.
 * H2 is local to given thread and will be closed when we set sessionHive again 
in next query. ThreadLocalHive.set() method closes the previous thread local 
Hive object H2 before overwriting it.
 * Also, if an async thread reallocates Hive object (let's say H3), then it 
will be closed by Hive.closeCurrent call when the thread exits.
 * So, I don't think there would be any HMS connection leak here.
 * parentHive is always sessionHive object where allowClose flag is false. So, 
the "assert (!parentHive.allowClose());" will never fail.

But I agree, there is a chance that each and every query re-creates Hive object 
if the HMS relevant configs are changed in between queries. I missed this part 
where sessionConf would be changed when user sets any Hive configurations via 
cli. 

So, I think, the earlier thread reference count solution would solve this 
issue. [~maheshk114], please comment if you think otherwise.

Btw, there is one another issue where Hive.get(sessionConf) directly stores the 
reference to sessionConf in Hive object which makes isCompatible() method to 
return true always.
{code:java}
private static Hive getInternal(HiveConf c, boolean needsRefresh, boolean 
isFastCheck,
boolean doRegisterAllFns) throws HiveException {
  Hive db = hiveDB.get();
  if (db == null || !db.isCurrentUserOwner() || needsRefresh
  || (c != null && !isCompatible(db, c, isFastCheck))) {
if (db != null) {
  LOG.debug("Creating new db. db = " + db + ", needsRefresh = " + 
needsRefresh +
  ", db.isCurrentUserOwner = " + db.isCurrentUserOwner());
  closeCurrent();
}
db = create(c, doRegisterAllFns);
  }
  if (c != null) {
db.conf = c;
  }
  return db;
}{code}
So, as of today, Thread local Hive object never gets recreated even if we 
change any HMS configs. Only REPL commands re-creates Hive object as it uses 
different HiveConf object not sessionConf to set HMS configs.
if (db == null || !db.isCurrentUserOwner() || needsRefresh
|| (c != null && !isCompatible(db, c, isFastCheck))) {
This issue can be resolved by always cloning HiveConf object instead of storing 
the input HiveConf reference.

But, it will be problem for Hive.getWithFastCheck(conf). I think, this method 
is added as an optimisation to avoid checking individual HMS configs. It only 
checks if the input conf object is same as the one stored in Hive object and if 
not, then re-create Hive object.

So, this needs to be fixed carefully. 

Please share your thoughts on this.

cc [~daijy], [~thejas], [~maheshk114]

 

> Async query execution can potentially fail if shared sessionHive is closed by 
> master thread.
> 
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-20682.01.patch, HIVE-20682.02.patch, 
> HIVE-20682.03.patch, HIVE-20682.04.patch
>
>
> *Problem description:*
> The master thread initializes the *sessionHive* object in *HiveSessionImpl* 
> class when we open a new session for a client connection and by default all 
> queries from this connection shares the same sessionHive object. 
> If the master thread executes a *synchronous* query, it closes the 
> sessionHive object (referred via thread local hiveDb) if  
> {{Hive.isCompatible}} returns false and sets new Hive object in thread local 
> HiveDb but doesn't change the sessionHive object in the session. Whereas, 
> *asynchronous* query execution via async threads never closes the sessionHive 
> object and it just creates a new one if needed and sets it as their thread 
> local hiveDb.
> So, the problem can happen in the case where an *asynchronous* query is being 
> executed by async threads refers to sessionHive object and the master thread 
> receives a *synchronous* query that closes the same sessionHive object. 
> Also, each query execution overwrites the thread local hiveDb object to 
> sessionHive object which potentially leaks a metastore connection if the 
> previous synchronous query execution re-created the Hive object.
> *Possible Fix:*
> The *sessionHive* object could be shared my multiple threads and so it 
> shouldn't be

[jira] [Commented] (HIVE-20486) Kafka: Use Row SerDe + vectorization

2018-10-26 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665483#comment-16665483
 ] 

Hive QA commented on HIVE-20486:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12945632/HIVE-20486.3.patch

{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15500 tests 
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=195)

[druidmini_dynamic_partition.q,druidmini_test_ts.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test_insert.q]
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=196)

[druidmini_masking.q,druidmini_test1.q,druidkafkamini_basic.q,druidmini_joins.q,druid_timestamptz.q]
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[kafka_storage_handler]
 (batchId=194)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/14646/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14646/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14646/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12945632 - PreCommit-HIVE-Build

> Kafka: Use Row SerDe + vectorization
> 
>
> Key: HIVE-20486
> URL: https://issues.apache.org/jira/browse/HIVE-20486
> Project: Hive
>  Issue Type: Improvement
>  Components: kafka integration
>Reporter: Gopal V
>Assignee: slim bouguerra
>Priority: Major
>  Labels: kafka, vectorization
> Fix For: 4.0.0
>
> Attachments: HIVE-20486.3.patch, HIVE-20486.patch
>
>
> KafkaHandler returns unvectorized rows which causes the operators downstream 
> to be slower and sub-optimal.
> Hive has a vectorization shim which allows Kafka streams without complex 
> projections to be wrapped into a vectorized reader via 
> {{hive.vectorized.use.row.serde.deserialize}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20486) Kafka: Use Row SerDe + vectorization

2018-10-26 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665441#comment-16665441
 ] 

Hive QA commented on HIVE-20486:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
42s{color} | {color:blue} itests/util in master has 48 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
45s{color} | {color:blue} ql in master has 2317 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m  
8s{color} | {color:red} kafka-handler: The patch generated 1 new + 1 unchanged 
- 0 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 13s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14646/dev-support/hive-personality.sh
 |
| git revision | master / 1002e89 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14646/yetus/diff-checkstyle-kafka-handler.txt
 |
| modules | C: itests/util kafka-handler ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14646/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Kafka: Use Row SerDe + vectorization
> 
>
> Key: HIVE-20486
> URL: https://issues.apache.org/jira/browse/HIVE-20486
> Project: Hive
>  Issue Type: Improvement
>  Components: kafka integration
>Reporter: Gopal V
>Assignee: slim bouguerra
>Priority: Major
>  Labels: kafka, vectorization
> Fix For: 4.0.0
>
> Attachments: HIVE-20486.3.patch, HIVE-20486.patch
>
>
> KafkaHandler returns unvectorized rows which causes the operators downstream 
> to be slower and sub-optimal.
> Hive has a vectorization shim which allows Kafka streams without complex 
> projections to be wrapped into a vectorized reader via 
> {{hive.vectorized.use.row.serde.deserialize}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20816) FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)

2018-10-26 Thread Gopal V (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665418#comment-16665418
 ] 

Gopal V commented on HIVE-20816:


[~rajesh.balamohan]: that fix does cover this issue directly - right?

> FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)
> --
>
> Key: HIVE-20816
> URL: https://issues.apache.org/jira/browse/HIVE-20816
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 2.3.2
>Reporter: Rajesh Balamohan
>Priority: Minor
>  Labels: fastdecimal
> Attachments: HIVE-20816.1.patch
>
>
> {noformat}
> with t1 as (
> ...
> ...
> )
> select id, max(abs(c1))) from t1 group by id;
> {noformat}
> throws the following exception
> {noformat}
> g.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
>  at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1126)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
> 
> ...
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1084)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1123)
> ... 18 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.RuntimeException: Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1047)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1067)
> ... 19 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HIVE-20816) FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)

2018-10-26 Thread Gopal V (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V resolved HIVE-20816.

Resolution: Duplicate

> FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)
> --
>
> Key: HIVE-20816
> URL: https://issues.apache.org/jira/browse/HIVE-20816
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 2.3.2
>Reporter: Rajesh Balamohan
>Priority: Minor
>  Labels: fastdecimal
> Attachments: HIVE-20816.1.patch
>
>
> {noformat}
> with t1 as (
> ...
> ...
> )
> select id, max(abs(c1))) from t1 group by id;
> {noformat}
> throws the following exception
> {noformat}
> g.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
>  at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1126)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
> 
> ...
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1084)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1123)
> ... 18 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.RuntimeException: Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1047)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1067)
> ... 19 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20816) FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)

2018-10-26 Thread Gopal V (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-20816:
---
Status: Open  (was: Patch Available)

> FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)
> --
>
> Key: HIVE-20816
> URL: https://issues.apache.org/jira/browse/HIVE-20816
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 2.3.2
>Reporter: Rajesh Balamohan
>Priority: Minor
>  Labels: fastdecimal
> Attachments: HIVE-20816.1.patch
>
>
> {noformat}
> with t1 as (
> ...
> ...
> )
> select id, max(abs(c1))) from t1 group by id;
> {noformat}
> throws the following exception
> {noformat}
> g.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
>  at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1126)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
> 
> ...
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1084)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1123)
> ... 18 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.RuntimeException: Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1047)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1067)
> ... 19 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-12971) Hive Support for Kudu

2018-10-26 Thread Tomas Hudik (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-12971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665411#comment-16665411
 ] 

Tomas Hudik commented on HIVE-12971:


I do hope there will be some positive change after Cloudera Hortonworks deal :)

> Hive Support for Kudu
> -
>
> Key: HIVE-12971
> URL: https://issues.apache.org/jira/browse/HIVE-12971
> Project: Hive
>  Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: Lenni Kuff
>Assignee: Janaki Lahorani
>Priority: Major
>
> JIRA for tracking work related to Hive/Kudu integration.
> It would be useful to allow Kudu data to be accessible via Hive. This would 
> involve creating a Kudu SerDe/StorageHandler and implementing support for 
> QUERY and DML commands like SELECT, INSERT, UPDATE, and DELETE. Kudu 
> Input/OutputFormats classes already exist. The work can be staged to support 
> this functionality incrementally.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-20638) Upgrade version of Jetty to 9.3.25.v20180904

2018-10-26 Thread Thejas M Nair (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665392#comment-16665392
 ] 

Thejas M Nair edited comment on HIVE-20638 at 10/26/18 4:39 PM:


pushed to master.
Thanks for the contribution [~abstractdog]!



was (Author: thejas):
pushed to master.


> Upgrade version of Jetty to 9.3.25.v20180904
> 
>
> Key: HIVE-20638
> URL: https://issues.apache.org/jira/browse/HIVE-20638
> Project: Hive
>  Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20638.01.patch, HIVE-20638.02.patch, 
> HIVE-20638.03.patch, HIVE-20638.03.patch, HIVE-20638.04.patch, 
> HIVE-20638.05.patch
>
>
> Current version is 9.3.20.v20170531



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20638) Upgrade version of Jetty to 9.3.25.v20180904

2018-10-26 Thread Thejas M Nair (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665392#comment-16665392
 ] 

Thejas M Nair commented on HIVE-20638:
--

pushed to master.


> Upgrade version of Jetty to 9.3.25.v20180904
> 
>
> Key: HIVE-20638
> URL: https://issues.apache.org/jira/browse/HIVE-20638
> Project: Hive
>  Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20638.01.patch, HIVE-20638.02.patch, 
> HIVE-20638.03.patch, HIVE-20638.03.patch, HIVE-20638.04.patch, 
> HIVE-20638.05.patch
>
>
> Current version is 9.3.20.v20170531



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20638) Upgrade version of Jetty to 9.3.25.v20180904

2018-10-26 Thread Thejas M Nair (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-20638:
-
   Resolution: Fixed
Fix Version/s: 4.0.0
   Status: Resolved  (was: Patch Available)

> Upgrade version of Jetty to 9.3.25.v20180904
> 
>
> Key: HIVE-20638
> URL: https://issues.apache.org/jira/browse/HIVE-20638
> Project: Hive
>  Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20638.01.patch, HIVE-20638.02.patch, 
> HIVE-20638.03.patch, HIVE-20638.03.patch, HIVE-20638.04.patch, 
> HIVE-20638.05.patch
>
>
> Current version is 9.3.20.v20170531



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20812) Update jetty dependency to 9.3.25.v20180904

2018-10-26 Thread Thejas M Nair (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-20812:
-
Resolution: Duplicate
Status: Resolved  (was: Patch Available)

> Update jetty dependency to 9.3.25.v20180904
> ---
>
> Key: HIVE-20812
> URL: https://issues.apache.org/jira/browse/HIVE-20812
> Project: Hive
>  Issue Type: Task
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
>Priority: Critical
> Attachments: HIVE-20812.1.patch
>
>
> The jetty version 9.3.20.v20170531 being used currently in master has several 
> CVE associated with it.
> Version 9.3.25.v20180904 has those issues resolved.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20638) Upgrade version of Jetty to 9.3.25.v20180904

2018-10-26 Thread Thejas M Nair (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665390#comment-16665390
 ] 

Thejas M Nair commented on HIVE-20638:
--

+1
committing in a bit


> Upgrade version of Jetty to 9.3.25.v20180904
> 
>
> Key: HIVE-20638
> URL: https://issues.apache.org/jira/browse/HIVE-20638
> Project: Hive
>  Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-20638.01.patch, HIVE-20638.02.patch, 
> HIVE-20638.03.patch, HIVE-20638.03.patch, HIVE-20638.04.patch, 
> HIVE-20638.05.patch
>
>
> Current version is 9.3.20.v20170531



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20782) Cleaning some unused code

2018-10-26 Thread slim bouguerra (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

slim bouguerra updated HIVE-20782:
--
Attachment: HIVE-20782.3.patch

> Cleaning some unused code
> -
>
> Key: HIVE-20782
> URL: https://issues.apache.org/jira/browse/HIVE-20782
> Project: Hive
>  Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: Teddy Choi
>Priority: Major
> Attachments: HIVE-20782.2.patch, HIVE-20782.2.patch, 
> HIVE-20782.3.patch, HIVE-20782.patch
>
>
> Am making my way into the vectorize code and trying understand the APIs. Ran 
> into this unused one, i guess it is not used anymore.
> [~ashutoshc] maybe can explain as you are the main contributor to this file 
> {code} 
> a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedSerde.java{code}
>  ?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20817) Reading Timestamp datatype via HiveServer2 gives errors

2018-10-26 Thread mahesh kumar behera (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-20817:
---
Description: 
CREATE TABLE JdbcBasicRead ( empno int, desg string,empname string,doj 
timestamp,Salary float,mgrid smallint, deptno tinyint ) ROW FORMAT DELIMITED 
FIELDS TERMINATED BY ',';

LOAD DATA LOCAL INPATH '/tmp/art_jdbc/hive/input/input_7columns.txt' OVERWRITE 
INTO TABLE JdbcBasicRead;

Sample Data.
—
7369,M,SMITH,1980-12-17 17:07:29.234234,5000.00,7902,20
7499,X,ALLEN,1981-02-20 17:07:29.234234,1250.00,7698,30
7521,X,WARD,1981-02-22 17:07:29.234234,01600.57,7698,40
7566,M,JONES,1981-04-02 17:07:29.234234,02975.65,7839,10
7654,X,MARTIN,1981-09-28 17:07:29.234234,01250.00,7698,20
7698,M,BLAKE,1981-05-01 17:07:29.234234,2850.98,7839,30
7782,M,CLARK,1981-06-09 17:07:29.234234,02450.00,7839,20
—

Select statement: SELECT empno, desg, empname, doj, salary, mgrid, deptno FROM 
JdbcBasicWrite

{code}
2018-09-25T07:11:03,222 WARN [HiveServer2-Handler-Pool: Thread-83]: 
thrift.ThriftCLIService (:()) - Error fetching results:
org.apache.hive.service.cli.HiveSQLException: java.lang.ClassCastException: 
org.apache.hadoop.hive.common.type.Timestamp cannot be cast to 
java.sql.Timestamp
at 
org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:469)
 ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at 
org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:328)
 ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at 
org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:910)
 ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source) ~[?:?]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_112]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
at 
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
 ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at 
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
 ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at 
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
 ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112]
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
 ~[hadoop-common-3.1.1.3.0.1.0-187.jar:?]
at 
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
 ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at com.sun.proxy.$Proxy46.fetchResults(Unknown Source) ~[?:?]
at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:564) 
~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:786)
 ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
 ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
 ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) 
~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) 
~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at 
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
 ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
 ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
~[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
Caused by: java.lang.ClassCastException: 
org.apache.hadoop.hive.common.type.Timestamp cannot be cast to 
java.sql.Timestamp
at org.apache.hive.service.cli.ColumnValue.toTColumnValue(ColumnValue.java:203) 
~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at org.apache.hive.service.cli.RowBasedSet.addRow(RowBasedSet.java:60) 
~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at org.apache.hive.service.cli.RowBasedSet.addRow(RowBasedSet.java:32) 
~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at 
org.apache.hive.service.cli.operation.SQLOperation.prepareFromRow(SQLOperation.java:517)

[jira] [Updated] (HIVE-20782) Cleaning some unused code

2018-10-26 Thread slim bouguerra (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

slim bouguerra updated HIVE-20782:
--
Attachment: (was: HIVE-20768.2.patch)

> Cleaning some unused code
> -
>
> Key: HIVE-20782
> URL: https://issues.apache.org/jira/browse/HIVE-20782
> Project: Hive
>  Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: Teddy Choi
>Priority: Major
> Attachments: HIVE-20782.2.patch, HIVE-20782.2.patch, 
> HIVE-20782.3.patch, HIVE-20782.patch
>
>
> Am making my way into the vectorize code and trying understand the APIs. Ran 
> into this unused one, i guess it is not used anymore.
> [~ashutoshc] maybe can explain as you are the main contributor to this file 
> {code} 
> a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedSerde.java{code}
>  ?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-20817) Reading Timestamp datatype via HiveServer2 gives errors

2018-10-26 Thread mahesh kumar behera (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera reassigned HIVE-20817:
--


> Reading Timestamp datatype via HiveServer2 gives errors
> ---
>
> Key: HIVE-20817
> URL: https://issues.apache.org/jira/browse/HIVE-20817
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Fix For: 4.0.0
>
>
> CREATE TABLE JdbcBasicRead ( empno int, desg string,empname string,doj 
> timestamp,Salary float,mgrid smallint, deptno tinyint ) ROW FORMAT DELIMITED 
> FIELDS TERMINATED BY ',';
> LOAD DATA LOCAL INPATH '/tmp/art_jdbc/hive/input/input_7columns.txt' 
> OVERWRITE INTO TABLE JdbcBasicRead;
> Sample Data.
> —
> 7369,M,SMITH,1980-12-17 17:07:29.234234,5000.00,7902,20
> 7499,X,ALLEN,1981-02-20 17:07:29.234234,1250.00,7698,30
> 7521,X,WARD,1981-02-22 17:07:29.234234,01600.57,7698,40
> 7566,M,JONES,1981-04-02 17:07:29.234234,02975.65,7839,10
> 7654,X,MARTIN,1981-09-28 17:07:29.234234,01250.00,7698,20
> 7698,M,BLAKE,1981-05-01 17:07:29.234234,2850.98,7839,30
> 7782,M,CLARK,1981-06-09 17:07:29.234234,02450.00,7839,20
> —
> Select statement: SELECT empno, desg, empname, doj, salary, mgrid, deptno 
> FROM JdbcBasicWrite
> {code}
> 2018-09-25T07:11:03,222 WARN [HiveServer2-Handler-Pool: Thread-83]: 
> thrift.ThriftCLIService (:()) - Error fetching results:
> org.apache.hive.service.cli.HiveSQLException: java.lang.ClassCastException: 
> org.apache.hadoop.hive.common.type.Timestamp cannot be cast to 
> java.sql.Timestamp
> at 
> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:469)
>  ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
> at 
> org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:328)
>  ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
> at 
> org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:910)
>  ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
> at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source) ~[?:?]
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_112]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
> at 
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
>  ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
> at 
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
>  ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
> at 
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
>  ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112]
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
>  ~[hadoop-common-3.1.1.3.0.1.0-187.jar:?]
> at 
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
>  ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
> at com.sun.proxy.$Proxy46.fetchResults(Unknown Source) ~[?:?]
> at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:564) 
> ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
> at 
> org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:786)
>  ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
> at 
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
>  ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
> at 
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
>  ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) 
> ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) 
> ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
> at 
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
>  ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
> at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>  ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  ~[?:1.8.0_112]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  ~[?:1.8.0_112]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]

[jira] [Updated] (HIVE-20486) Kafka: Use Row SerDe + vectorization

2018-10-26 Thread slim bouguerra (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

slim bouguerra updated HIVE-20486:
--
Labels: kafka vectorization  (was: vectorization)

> Kafka: Use Row SerDe + vectorization
> 
>
> Key: HIVE-20486
> URL: https://issues.apache.org/jira/browse/HIVE-20486
> Project: Hive
>  Issue Type: Improvement
>  Components: kafka integration
>Reporter: Gopal V
>Assignee: slim bouguerra
>Priority: Major
>  Labels: kafka, vectorization
> Fix For: 4.0.0
>
> Attachments: HIVE-20486.3.patch, HIVE-20486.patch
>
>
> KafkaHandler returns unvectorized rows which causes the operators downstream 
> to be slower and sub-optimal.
> Hive has a vectorization shim which allows Kafka streams without complex 
> projections to be wrapped into a vectorized reader via 
> {{hive.vectorized.use.row.serde.deserialize}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20735) Address some of the review comments plus Kerberos support

2018-10-26 Thread slim bouguerra (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

slim bouguerra updated HIVE-20735:
--
Labels: kafka kerberos  (was: )

> Address some of the review comments plus Kerberos support
> -
>
> Key: HIVE-20735
> URL: https://issues.apache.org/jira/browse/HIVE-20735
> Project: Hive
>  Issue Type: Sub-task
>  Components: kafka integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
>  Labels: kafka, kerberos
> Fix For: 4.0.0
>
> Attachments: HIVE-20735.patch
>
>
> As part of the review comments we agreed to:
> # remove start and end offsets columns
> # remove the best effort mode
> # make the 2pc as default protocol for EOS
> Also this patch will include an additional enhancement to add kerberos 
> support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20486) Kafka: Use Row SerDe + vectorization

2018-10-26 Thread slim bouguerra (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

slim bouguerra updated HIVE-20486:
--
Labels: vectorization  (was: )

> Kafka: Use Row SerDe + vectorization
> 
>
> Key: HIVE-20486
> URL: https://issues.apache.org/jira/browse/HIVE-20486
> Project: Hive
>  Issue Type: Improvement
>  Components: kafka integration
>Reporter: Gopal V
>Assignee: slim bouguerra
>Priority: Major
>  Labels: kafka, vectorization
> Fix For: 4.0.0
>
> Attachments: HIVE-20486.3.patch, HIVE-20486.patch
>
>
> KafkaHandler returns unvectorized rows which causes the operators downstream 
> to be slower and sub-optimal.
> Hive has a vectorization shim which allows Kafka streams without complex 
> projections to be wrapped into a vectorized reader via 
> {{hive.vectorized.use.row.serde.deserialize}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20816) FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)

2018-10-26 Thread Rajesh Balamohan (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-20816:

Issue Type: Bug  (was: Improvement)

> FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)
> --
>
> Key: HIVE-20816
> URL: https://issues.apache.org/jira/browse/HIVE-20816
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 2.3.2
>Reporter: Rajesh Balamohan
>Priority: Major
>  Labels: fastdecimal
> Attachments: HIVE-20816.1.patch
>
>
> {noformat}
> with t1 as (
> ...
> ...
> )
> select id, max(abs(c1))) from t1 group by id;
> {noformat}
> throws the following exception
> {noformat}
> g.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
>  at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1126)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
> 
> ...
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1084)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1123)
> ... 18 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.RuntimeException: Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1047)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1067)
> ... 19 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20816) FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)

2018-10-26 Thread Rajesh Balamohan (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-20816:

Priority: Minor  (was: Major)

> FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)
> --
>
> Key: HIVE-20816
> URL: https://issues.apache.org/jira/browse/HIVE-20816
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 2.3.2
>Reporter: Rajesh Balamohan
>Priority: Minor
>  Labels: fastdecimal
> Attachments: HIVE-20816.1.patch
>
>
> {noformat}
> with t1 as (
> ...
> ...
> )
> select id, max(abs(c1))) from t1 group by id;
> {noformat}
> throws the following exception
> {noformat}
> g.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
>  at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1126)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
> 
> ...
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1084)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1123)
> ... 18 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.RuntimeException: Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1047)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1067)
> ... 19 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20816) FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)

2018-10-26 Thread Rajesh Balamohan (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-20816:

Status: Patch Available  (was: Open)

> FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)
> --
>
> Key: HIVE-20816
> URL: https://issues.apache.org/jira/browse/HIVE-20816
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Affects Versions: 2.3.2
>Reporter: Rajesh Balamohan
>Priority: Major
>  Labels: fastdecimal
> Attachments: HIVE-20816.1.patch
>
>
> {noformat}
> with t1 as (
> ...
> ...
> )
> select id, max(abs(c1))) from t1 group by id;
> {noformat}
> throws the following exception
> {noformat}
> g.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
>  at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1126)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
> 
> ...
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1084)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1123)
> ... 18 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.RuntimeException: Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1047)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1067)
> ... 19 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20816) FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)

2018-10-26 Thread Rajesh Balamohan (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-20816:

Attachment: HIVE-20816.1.patch

> FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)
> --
>
> Key: HIVE-20816
> URL: https://issues.apache.org/jira/browse/HIVE-20816
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Affects Versions: 2.3.2
>Reporter: Rajesh Balamohan
>Priority: Major
>  Labels: fastdecimal
> Attachments: HIVE-20816.1.patch
>
>
> {noformat}
> with t1 as (
> ...
> ...
> )
> select id, max(abs(c1))) from t1 group by id;
> {noformat}
> throws the following exception
> {noformat}
> g.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
>  at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1126)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
> 
> ...
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1084)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1123)
> ... 18 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.RuntimeException: Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1047)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1067)
> ... 19 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-20816) FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)

2018-10-26 Thread Rajesh Balamohan (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665272#comment-16665272
 ] 

Rajesh Balamohan edited comment on HIVE-20816 at 10/26/18 3:02 PM:
---

Related ticket: https://issues.apache.org/jira/browse/HIVE-19085

https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/storage-api/src/java/org/apache/hadoop/hive/common/type/FastHiveDecimalImpl.java#L2537

If all words are 0 && byteIndex is 0 as well, we should return 0 instead of 
processing further. 



was (Author: rajesh.balamohan):
Related ticket: https://issues.apache.org/jira/browse/HIVE-19085

https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/storage-api/src/java/org/apache/hadoop/hive/common/type/FastHiveDecimalImpl.java#L2537

If all words are 0 && byteIndex is 0 as well, we should return 0 instead of 
proceeding further. 


> FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)
> --
>
> Key: HIVE-20816
> URL: https://issues.apache.org/jira/browse/HIVE-20816
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Affects Versions: 2.3.2
>Reporter: Rajesh Balamohan
>Priority: Major
>  Labels: fastdecimal
>
> {noformat}
> with t1 as (
> ...
> ...
> )
> select id, max(abs(c1))) from t1 group by id;
> {noformat}
> throws the following exception
> {noformat}
> g.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
>  at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1126)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
> 
> ...
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1084)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1123)
> ... 18 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.RuntimeException: Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1047)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1067)
> ... 19 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20816) FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)

2018-10-26 Thread Rajesh Balamohan (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665272#comment-16665272
 ] 

Rajesh Balamohan commented on HIVE-20816:
-

Related ticket: https://issues.apache.org/jira/browse/HIVE-19085

https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/storage-api/src/java/org/apache/hadoop/hive/common/type/FastHiveDecimalImpl.java#L2537

If all words are 0 && byteIndex is 0 as well, we should return 0 instead of 
proceeding further. 


> FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)
> --
>
> Key: HIVE-20816
> URL: https://issues.apache.org/jira/browse/HIVE-20816
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Affects Versions: 2.3.2
>Reporter: Rajesh Balamohan
>Priority: Major
>  Labels: fastdecimal
>
> {noformat}
> with t1 as (
> ...
> ...
> )
> select id, max(abs(c1))) from t1 group by id;
> {noformat}
> throws the following exception
> {noformat}
> g.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
>  at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1126)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
> 
> ...
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1084)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1123)
> ... 18 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.RuntimeException: Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1047)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1067)
> ... 19 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20816) FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)

2018-10-26 Thread Rajesh Balamohan (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-20816:

Component/s: Hive

> FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)
> --
>
> Key: HIVE-20816
> URL: https://issues.apache.org/jira/browse/HIVE-20816
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Affects Versions: 2.3.2
>Reporter: Rajesh Balamohan
>Priority: Major
>  Labels: fastdecimal
>
> {noformat}
> with t1 as (
> ...
> ...
> )
> select id, max(abs(c1))) from t1 group by id;
> {noformat}
> throws the following exception
> {noformat}
> g.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
>  at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1126)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
> 
> ...
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1084)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1123)
> ... 18 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.RuntimeException: Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1047)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1067)
> ... 19 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20816) FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)

2018-10-26 Thread Rajesh Balamohan (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-20816:

Labels: fastdecimal  (was: )

> FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)
> --
>
> Key: HIVE-20816
> URL: https://issues.apache.org/jira/browse/HIVE-20816
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Affects Versions: 2.3.2
>Reporter: Rajesh Balamohan
>Priority: Major
>  Labels: fastdecimal
>
> {noformat}
> with t1 as (
> ...
> ...
> )
> select id, max(abs(c1))) from t1 group by id;
> {noformat}
> throws the following exception
> {noformat}
> g.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
>  at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1126)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
> 
> ...
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1084)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1123)
> ... 18 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.RuntimeException: Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1047)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1067)
> ... 19 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-20608) Incorrect handling of sql command args in hive service leading to misleading error messages

2018-10-26 Thread Soumabrata Chakraborty (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665241#comment-16665241
 ] 

Soumabrata Chakraborty edited comment on HIVE-20608 at 10/26/18 2:22 PM:
-

Hi [~daijy]

Yes - the code you pointed out from ExecuteStatementOperation.java does trim 
the statement and at this moment is the only point from which 
HiveCommandOperation's runInternal() method may be reached.

However,  I believe HiveCommandOperation.java and runInternal() should be self 
sufficient and should not depend on operation in another class to trim the 
statement for it.

In fact, if you look at the current code for HiveCommandOperation.java then it 
does trim the statement on line 109 (self sufficiency) but fails to use this 
trimmed version in next line (i.e. line 110).  This is exactly what my change 
fixes. 

References: 
 # 
[https://github.com/apache/hive/blob/0d7015468611c485c01bb80cdbe4c89e4e68b680/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L109]
 # 
[https://github.com/apache/hive/pull/442/files#diff-459cb15e0bcb0c7f00727e0dcf63a3b0]
 

I agree that functionally it is not a bug at the moment.  That said, I think 
this change shall make the code more robust.

I leave it up to your discretion to decide on the course of action.


was (Author: soumabrata):
Hi [~daijy]

Yes - the code you pointed out from ExecuteStatementOperation.java does trim 
the statement and at this moment is the only point from which 
HiveCommandOperation's runInternal() method may be reached.

However,  I believe HiveCommandOperation.java and runInternal() should be self 
sufficient and should not depend on operation in another class to trim the 
statement for it.

In fact, if you look at the current code for HiveCommandOperation.java then it 
does trim the statement on line 109 (self sufficiency) but fails to use this 
trimmed version in next line (i.e. line 110).  This is exactly what my change 
fixes.  (Reference 
https://github.com/apache/hive/blob/0d7015468611c485c01bb80cdbe4c89e4e68b680/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L109)

I agree that functionally it is not a bug at the moment.  That said, I think 
this change shall make the code more robust.

I leave it up to your discretion to decide on the course of action.

> Incorrect handling of sql command args in hive service leading to misleading 
> error messages
> ---
>
> Key: HIVE-20608
> URL: https://issues.apache.org/jira/browse/HIVE-20608
> Project: Hive
>  Issue Type: Bug
>Reporter: Soumabrata Chakraborty
>Assignee: Soumabrata Chakraborty
>Priority: Major
> Attachments: HIVE-20608.2.patch, HIVE-20608.patch
>
>
> *Steps to reproduce:*
> (1) Connect to HiveServer2 using JDBC driver (not via Beeline)
> (2) Execute a set command with a space before set – e.g. " set 
> hive.exec.dynamic.partiton=true"
> (3) The error that is returned says: 
> {code:java}
> Caused by: org.apache.hive.service.cli.HiveSQLException: Error while 
> processing statement: Cannot modify set hive.exec.dynamic.partition at 
> runtime. It is not in list of params that are allowed to be modified at 
> runtime
> {code}
>  (4) However on removing the space before the set command - it works fine.
>  
> *Analysis:*
> Looks like an issue with 
> [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java]
>  
> In the runInternal() method, the untrimmed sql statement is split by space 
> (\\s) but the substring operation to get the command args is done on the 
> trimmed sql statement.  This causes the issue. 
> {code:java}
> String command = getStatement().trim(); 
> String[] tokens = statement.split("\\s"); 
> String commandArgs = command.substring(tokens[0].length()).trim();
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20608) Incorrect handling of sql command args in hive service leading to misleading error messages

2018-10-26 Thread Soumabrata Chakraborty (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665241#comment-16665241
 ] 

Soumabrata Chakraborty commented on HIVE-20608:
---

Hi [~daijy]

Yes - the code you pointed out from ExecuteStatementOperation.java does trim 
the statement and at this moment is the only point from which 
HiveCommandOperation's runInternal() method may be reached.

However,  I believe HiveCommandOperation.java and runInternal() should be self 
sufficient and should not depend on operation in another class to trim the 
statement for it.

In fact, if you look at the current code for HiveCommandOperation.java then it 
does trim the statement on line 109 (self sufficiency) but fails to use this 
trimmed version in next line (i.e. line 110).  This is exactly what my change 
fixes.  (Reference 
https://github.com/apache/hive/blob/0d7015468611c485c01bb80cdbe4c89e4e68b680/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L109)

I agree that functionally it is not a bug at the moment.  That said, I think 
this change shall make the code more robust.

I leave it up to your discretion to decide on the course of action.

> Incorrect handling of sql command args in hive service leading to misleading 
> error messages
> ---
>
> Key: HIVE-20608
> URL: https://issues.apache.org/jira/browse/HIVE-20608
> Project: Hive
>  Issue Type: Bug
>Reporter: Soumabrata Chakraborty
>Assignee: Soumabrata Chakraborty
>Priority: Major
> Attachments: HIVE-20608.2.patch, HIVE-20608.patch
>
>
> *Steps to reproduce:*
> (1) Connect to HiveServer2 using JDBC driver (not via Beeline)
> (2) Execute a set command with a space before set – e.g. " set 
> hive.exec.dynamic.partiton=true"
> (3) The error that is returned says: 
> {code:java}
> Caused by: org.apache.hive.service.cli.HiveSQLException: Error while 
> processing statement: Cannot modify set hive.exec.dynamic.partition at 
> runtime. It is not in list of params that are allowed to be modified at 
> runtime
> {code}
>  (4) However on removing the space before the set command - it works fine.
>  
> *Analysis:*
> Looks like an issue with 
> [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java]
>  
> In the runInternal() method, the untrimmed sql statement is split by space 
> (\\s) but the substring operation to get the command args is done on the 
> trimmed sql statement.  This causes the issue. 
> {code:java}
> String command = getStatement().trim(); 
> String[] tokens = statement.split("\\s"); 
> String commandArgs = command.substring(tokens[0].length()).trim();
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-18045) can VectorizedOrcAcidRowBatchReader be used all the time

2018-10-26 Thread hezhang (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-18045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hezhang reassigned HIVE-18045:
--

Assignee: hezhang  (was: Eugene Koifman)

> can VectorizedOrcAcidRowBatchReader be used all the time
> 
>
> Key: HIVE-18045
> URL: https://issues.apache.org/jira/browse/HIVE-18045
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: hezhang
>Priority: Blocker
>
> Can we use VectorizedOrcAcidRowBatchReader for non-vectorized queries?
> It would just need a wrapper on top of it to turn VRBs into rows.
> This would mean there is just 1 acid reader to maintain - not 2.
> Would this be an issue for sorted reader/SMB support?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-18045) can VectorizedOrcAcidRowBatchReader be used all the time

2018-10-26 Thread hezhang (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-18045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hezhang reassigned HIVE-18045:
--

Assignee: (was: hezhang)

> can VectorizedOrcAcidRowBatchReader be used all the time
> 
>
> Key: HIVE-18045
> URL: https://issues.apache.org/jira/browse/HIVE-18045
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Reporter: Eugene Koifman
>Priority: Blocker
>
> Can we use VectorizedOrcAcidRowBatchReader for non-vectorized queries?
> It would just need a wrapper on top of it to turn VRBs into rows.
> This would mean there is just 1 acid reader to maintain - not 2.
> Would this be an issue for sorted reader/SMB support?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-16455) ADD JAR command leaks JAR Files

2018-10-26 Thread hezhang (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665161#comment-16665161
 ] 

hezhang edited comment on HIVE-16455 at 10/26/18 1:29 PM:
--

[~wanghaihua] hi 

I have encountered the same problem in hive 2.1.1。 hive-2.1.1 still leaking jar 
file。
 
What version do you use?


was (Author: heiheizhang):
[~wanghaihua],[~aihuaxu] hi 

I have encountered the same problem in hive 2.1.1。 hive-2.1.1 still leaking jar 
file。 how shoud I resolve this issue?

> ADD JAR command leaks JAR Files
> ---
>
> Key: HIVE-16455
> URL: https://issues.apache.org/jira/browse/HIVE-16455
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-16455.1.patch
>
>
> HiveServer2 is leaking file handles when using ADD JAR statement and the JAR 
> file added is not used in the query itself.
> {noformat}
> beeline> !connect jdbc:hive2://localhost:1 admin
> 0: jdbc:hive2://localhost:1> create table test_leak (a int);
> 0: jdbc:hive2://localhost:1> insert into test_leak Values (1);
> -- Exit beeline terminal; Find PID of HiveServer2
> [root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
> 0
> [root@host-10-17-80-111 ~]# beeline -u jdbc:hive2://localhost:1/default 
> -n admin
> And run the command "ADD JAR hdfs:///tmp/hive-contrib.jar; select * from 
> test_leak"
> [root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
> 1
> java29588 hive  391u   REG  252,3125987  2099944 
> /tmp/57d98f5b-1e53-44e2-876b-6b4323ac24db_resources/hive-contrib.jar (deleted)
> java29588 hive  392u   REG  252,3125987  2099946 
> /tmp/eb3184ad-7f15-4a77-a10d-87717ae634d1_resources/hive-contrib.jar (deleted)
> java29588 hive  393r   REG  252,3125987  2099825 
> /tmp/e29dccfc-5708-4254-addb-7a8988fc0500_resources/hive-contrib.jar (deleted)
> java29588 hive  394r   REG  252,3125987  2099833 
> /tmp/5153dd4a-a606-4f53-b02c-d606e7e56985_resources/hive-contrib.jar (deleted)
> java29588 hive  395r   REG  252,3125987  2099827 
> /tmp/ff3cdb05-917f-43c0-830a-b293bf397a23_resources/hive-contrib.jar (deleted)
> java29588 hive  396r   REG  252,3125987  2099822 
> /tmp/60531b66-5985-421e-8eb5-eeac31fdf964_resources/hive-contrib.jar (deleted)
> java29588 hive  397r   REG  252,3125987  2099831 
> /tmp/78878921-455c-438c-9735-447566ed8381_resources/hive-contrib.jar (deleted)
> java29588 hive  399r   REG  252,3125987  2099835 
> /tmp/0e5d7990-30cc-4248-9058-587f7f1ff211_resources/hive-contrib.jar (deleted)
> {noformat}
> You can see the the session directory (and therefore anything in it) is set 
> to delete only on exit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-16455) ADD JAR command leaks JAR Files

2018-10-26 Thread hezhang (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665161#comment-16665161
 ] 

hezhang commented on HIVE-16455:


[~wanghaihua],[~aihuaxu] hi 

I have encountered the same problem in hive 2.1.1。 hive-2.1.1 still leaking jar 
file。 how shoud I resolve this issue?

> ADD JAR command leaks JAR Files
> ---
>
> Key: HIVE-16455
> URL: https://issues.apache.org/jira/browse/HIVE-16455
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-16455.1.patch
>
>
> HiveServer2 is leaking file handles when using ADD JAR statement and the JAR 
> file added is not used in the query itself.
> {noformat}
> beeline> !connect jdbc:hive2://localhost:1 admin
> 0: jdbc:hive2://localhost:1> create table test_leak (a int);
> 0: jdbc:hive2://localhost:1> insert into test_leak Values (1);
> -- Exit beeline terminal; Find PID of HiveServer2
> [root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
> 0
> [root@host-10-17-80-111 ~]# beeline -u jdbc:hive2://localhost:1/default 
> -n admin
> And run the command "ADD JAR hdfs:///tmp/hive-contrib.jar; select * from 
> test_leak"
> [root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
> 1
> java29588 hive  391u   REG  252,3125987  2099944 
> /tmp/57d98f5b-1e53-44e2-876b-6b4323ac24db_resources/hive-contrib.jar (deleted)
> java29588 hive  392u   REG  252,3125987  2099946 
> /tmp/eb3184ad-7f15-4a77-a10d-87717ae634d1_resources/hive-contrib.jar (deleted)
> java29588 hive  393r   REG  252,3125987  2099825 
> /tmp/e29dccfc-5708-4254-addb-7a8988fc0500_resources/hive-contrib.jar (deleted)
> java29588 hive  394r   REG  252,3125987  2099833 
> /tmp/5153dd4a-a606-4f53-b02c-d606e7e56985_resources/hive-contrib.jar (deleted)
> java29588 hive  395r   REG  252,3125987  2099827 
> /tmp/ff3cdb05-917f-43c0-830a-b293bf397a23_resources/hive-contrib.jar (deleted)
> java29588 hive  396r   REG  252,3125987  2099822 
> /tmp/60531b66-5985-421e-8eb5-eeac31fdf964_resources/hive-contrib.jar (deleted)
> java29588 hive  397r   REG  252,3125987  2099831 
> /tmp/78878921-455c-438c-9735-447566ed8381_resources/hive-contrib.jar (deleted)
> java29588 hive  399r   REG  252,3125987  2099835 
> /tmp/0e5d7990-30cc-4248-9058-587f7f1ff211_resources/hive-contrib.jar (deleted)
> {noformat}
> You can see the the session directory (and therefore anything in it) is set 
> to delete only on exit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Issue Comment Deleted] (HIVE-16455) ADD JAR command leaks JAR Files

2018-10-26 Thread hezhang (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hezhang updated HIVE-16455:
---
Comment: was deleted

(was: [~wanghaihua] hi )

> ADD JAR command leaks JAR Files
> ---
>
> Key: HIVE-16455
> URL: https://issues.apache.org/jira/browse/HIVE-16455
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-16455.1.patch
>
>
> HiveServer2 is leaking file handles when using ADD JAR statement and the JAR 
> file added is not used in the query itself.
> {noformat}
> beeline> !connect jdbc:hive2://localhost:1 admin
> 0: jdbc:hive2://localhost:1> create table test_leak (a int);
> 0: jdbc:hive2://localhost:1> insert into test_leak Values (1);
> -- Exit beeline terminal; Find PID of HiveServer2
> [root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
> 0
> [root@host-10-17-80-111 ~]# beeline -u jdbc:hive2://localhost:1/default 
> -n admin
> And run the command "ADD JAR hdfs:///tmp/hive-contrib.jar; select * from 
> test_leak"
> [root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
> 1
> java29588 hive  391u   REG  252,3125987  2099944 
> /tmp/57d98f5b-1e53-44e2-876b-6b4323ac24db_resources/hive-contrib.jar (deleted)
> java29588 hive  392u   REG  252,3125987  2099946 
> /tmp/eb3184ad-7f15-4a77-a10d-87717ae634d1_resources/hive-contrib.jar (deleted)
> java29588 hive  393r   REG  252,3125987  2099825 
> /tmp/e29dccfc-5708-4254-addb-7a8988fc0500_resources/hive-contrib.jar (deleted)
> java29588 hive  394r   REG  252,3125987  2099833 
> /tmp/5153dd4a-a606-4f53-b02c-d606e7e56985_resources/hive-contrib.jar (deleted)
> java29588 hive  395r   REG  252,3125987  2099827 
> /tmp/ff3cdb05-917f-43c0-830a-b293bf397a23_resources/hive-contrib.jar (deleted)
> java29588 hive  396r   REG  252,3125987  2099822 
> /tmp/60531b66-5985-421e-8eb5-eeac31fdf964_resources/hive-contrib.jar (deleted)
> java29588 hive  397r   REG  252,3125987  2099831 
> /tmp/78878921-455c-438c-9735-447566ed8381_resources/hive-contrib.jar (deleted)
> java29588 hive  399r   REG  252,3125987  2099835 
> /tmp/0e5d7990-30cc-4248-9058-587f7f1ff211_resources/hive-contrib.jar (deleted)
> {noformat}
> You can see the the session directory (and therefore anything in it) is set 
> to delete only on exit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-16455) ADD JAR command leaks JAR Files

2018-10-26 Thread hezhang (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665157#comment-16665157
 ] 

hezhang commented on HIVE-16455:


[~wanghaihua] hi 

> ADD JAR command leaks JAR Files
> ---
>
> Key: HIVE-16455
> URL: https://issues.apache.org/jira/browse/HIVE-16455
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-16455.1.patch
>
>
> HiveServer2 is leaking file handles when using ADD JAR statement and the JAR 
> file added is not used in the query itself.
> {noformat}
> beeline> !connect jdbc:hive2://localhost:1 admin
> 0: jdbc:hive2://localhost:1> create table test_leak (a int);
> 0: jdbc:hive2://localhost:1> insert into test_leak Values (1);
> -- Exit beeline terminal; Find PID of HiveServer2
> [root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
> 0
> [root@host-10-17-80-111 ~]# beeline -u jdbc:hive2://localhost:1/default 
> -n admin
> And run the command "ADD JAR hdfs:///tmp/hive-contrib.jar; select * from 
> test_leak"
> [root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
> 1
> java29588 hive  391u   REG  252,3125987  2099944 
> /tmp/57d98f5b-1e53-44e2-876b-6b4323ac24db_resources/hive-contrib.jar (deleted)
> java29588 hive  392u   REG  252,3125987  2099946 
> /tmp/eb3184ad-7f15-4a77-a10d-87717ae634d1_resources/hive-contrib.jar (deleted)
> java29588 hive  393r   REG  252,3125987  2099825 
> /tmp/e29dccfc-5708-4254-addb-7a8988fc0500_resources/hive-contrib.jar (deleted)
> java29588 hive  394r   REG  252,3125987  2099833 
> /tmp/5153dd4a-a606-4f53-b02c-d606e7e56985_resources/hive-contrib.jar (deleted)
> java29588 hive  395r   REG  252,3125987  2099827 
> /tmp/ff3cdb05-917f-43c0-830a-b293bf397a23_resources/hive-contrib.jar (deleted)
> java29588 hive  396r   REG  252,3125987  2099822 
> /tmp/60531b66-5985-421e-8eb5-eeac31fdf964_resources/hive-contrib.jar (deleted)
> java29588 hive  397r   REG  252,3125987  2099831 
> /tmp/78878921-455c-438c-9735-447566ed8381_resources/hive-contrib.jar (deleted)
> java29588 hive  399r   REG  252,3125987  2099835 
> /tmp/0e5d7990-30cc-4248-9058-587f7f1ff211_resources/hive-contrib.jar (deleted)
> {noformat}
> You can see the the session directory (and therefore anything in it) is set 
> to delete only on exit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-16455) ADD JAR command leaks JAR Files

2018-10-26 Thread hezhang (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665159#comment-16665159
 ] 

hezhang commented on HIVE-16455:


[~wanghaihua] hi 

> ADD JAR command leaks JAR Files
> ---
>
> Key: HIVE-16455
> URL: https://issues.apache.org/jira/browse/HIVE-16455
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-16455.1.patch
>
>
> HiveServer2 is leaking file handles when using ADD JAR statement and the JAR 
> file added is not used in the query itself.
> {noformat}
> beeline> !connect jdbc:hive2://localhost:1 admin
> 0: jdbc:hive2://localhost:1> create table test_leak (a int);
> 0: jdbc:hive2://localhost:1> insert into test_leak Values (1);
> -- Exit beeline terminal; Find PID of HiveServer2
> [root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
> 0
> [root@host-10-17-80-111 ~]# beeline -u jdbc:hive2://localhost:1/default 
> -n admin
> And run the command "ADD JAR hdfs:///tmp/hive-contrib.jar; select * from 
> test_leak"
> [root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
> 1
> java29588 hive  391u   REG  252,3125987  2099944 
> /tmp/57d98f5b-1e53-44e2-876b-6b4323ac24db_resources/hive-contrib.jar (deleted)
> java29588 hive  392u   REG  252,3125987  2099946 
> /tmp/eb3184ad-7f15-4a77-a10d-87717ae634d1_resources/hive-contrib.jar (deleted)
> java29588 hive  393r   REG  252,3125987  2099825 
> /tmp/e29dccfc-5708-4254-addb-7a8988fc0500_resources/hive-contrib.jar (deleted)
> java29588 hive  394r   REG  252,3125987  2099833 
> /tmp/5153dd4a-a606-4f53-b02c-d606e7e56985_resources/hive-contrib.jar (deleted)
> java29588 hive  395r   REG  252,3125987  2099827 
> /tmp/ff3cdb05-917f-43c0-830a-b293bf397a23_resources/hive-contrib.jar (deleted)
> java29588 hive  396r   REG  252,3125987  2099822 
> /tmp/60531b66-5985-421e-8eb5-eeac31fdf964_resources/hive-contrib.jar (deleted)
> java29588 hive  397r   REG  252,3125987  2099831 
> /tmp/78878921-455c-438c-9735-447566ed8381_resources/hive-contrib.jar (deleted)
> java29588 hive  399r   REG  252,3125987  2099835 
> /tmp/0e5d7990-30cc-4248-9058-587f7f1ff211_resources/hive-contrib.jar (deleted)
> {noformat}
> You can see the the session directory (and therefore anything in it) is set 
> to delete only on exit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Issue Comment Deleted] (HIVE-16455) ADD JAR command leaks JAR Files

2018-10-26 Thread hezhang (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hezhang updated HIVE-16455:
---
Comment: was deleted

(was: [~wanghaihua] hi )

> ADD JAR command leaks JAR Files
> ---
>
> Key: HIVE-16455
> URL: https://issues.apache.org/jira/browse/HIVE-16455
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-16455.1.patch
>
>
> HiveServer2 is leaking file handles when using ADD JAR statement and the JAR 
> file added is not used in the query itself.
> {noformat}
> beeline> !connect jdbc:hive2://localhost:1 admin
> 0: jdbc:hive2://localhost:1> create table test_leak (a int);
> 0: jdbc:hive2://localhost:1> insert into test_leak Values (1);
> -- Exit beeline terminal; Find PID of HiveServer2
> [root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
> 0
> [root@host-10-17-80-111 ~]# beeline -u jdbc:hive2://localhost:1/default 
> -n admin
> And run the command "ADD JAR hdfs:///tmp/hive-contrib.jar; select * from 
> test_leak"
> [root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
> 1
> java29588 hive  391u   REG  252,3125987  2099944 
> /tmp/57d98f5b-1e53-44e2-876b-6b4323ac24db_resources/hive-contrib.jar (deleted)
> java29588 hive  392u   REG  252,3125987  2099946 
> /tmp/eb3184ad-7f15-4a77-a10d-87717ae634d1_resources/hive-contrib.jar (deleted)
> java29588 hive  393r   REG  252,3125987  2099825 
> /tmp/e29dccfc-5708-4254-addb-7a8988fc0500_resources/hive-contrib.jar (deleted)
> java29588 hive  394r   REG  252,3125987  2099833 
> /tmp/5153dd4a-a606-4f53-b02c-d606e7e56985_resources/hive-contrib.jar (deleted)
> java29588 hive  395r   REG  252,3125987  2099827 
> /tmp/ff3cdb05-917f-43c0-830a-b293bf397a23_resources/hive-contrib.jar (deleted)
> java29588 hive  396r   REG  252,3125987  2099822 
> /tmp/60531b66-5985-421e-8eb5-eeac31fdf964_resources/hive-contrib.jar (deleted)
> java29588 hive  397r   REG  252,3125987  2099831 
> /tmp/78878921-455c-438c-9735-447566ed8381_resources/hive-contrib.jar (deleted)
> java29588 hive  399r   REG  252,3125987  2099835 
> /tmp/0e5d7990-30cc-4248-9058-587f7f1ff211_resources/hive-contrib.jar (deleted)
> {noformat}
> You can see the the session directory (and therefore anything in it) is set 
> to delete only on exit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20796) jdbc URL can contain sensitive information that should not be logged

2018-10-26 Thread Peter Vary (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665128#comment-16665128
 ] 

Peter Vary commented on HIVE-20796:
---

HIVE-20796.05.patch is even better +1 pending tests :D

> jdbc URL can contain sensitive information that should not be logged
> 
>
> Key: HIVE-20796
> URL: https://issues.apache.org/jira/browse/HIVE-20796
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Affects Versions: 4.0.0
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-20796.01.patch, HIVE-20796.02.patch, 
> HIVE-20796.03.patch, HIVE-20796.04.patch, HIVE-20796.05.patch
>
>
> It is possible to put passwords in the jdbc connection url and some jdbc 
> drivers will supposedly use that. (derby, mysql). This information is 
> considered sensitive, and should be masked out, while logging the connection 
> url.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20661) Dynamic partitions loading calls add partition for every partition 1-by-1

2018-10-26 Thread Peter Vary (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-20661:
--
   Resolution: Fixed
Fix Version/s: 4.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master.

Thanks for the patch [~lpinter]!

> Dynamic partitions loading calls add partition for every partition 1-by-1
> -
>
> Key: HIVE-20661
> URL: https://issues.apache.org/jira/browse/HIVE-20661
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Affects Versions: 4.0.0
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20661.01.patch, HIVE-20661.02.patch, 
> HIVE-20661.03.patch, HIVE-20661.04.patch, HIVE-20661.05.patch, 
> HIVE-20661.06.patch, HIVE-20661.07.patch, HIVE-20661.08.patch, 
> HIVE-20661.09.patch, HIVE-20661.10.patch, HIVE-20661.11.patch, 
> HIVE-20661.12.patch
>
>
> Hive.loadDynamicPartitions creates partitions using a threadpool, but the 
> update of the Metastore  via the MetastoreClient is done 1-by-1. This adds 
> unnecessary extra calls. The partitions should be created in one batch. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.

2018-10-26 Thread Peter Vary (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665119#comment-16665119
 ] 

Peter Vary commented on HIVE-20682:
---

[~sankarh] [~maheshk114]: What will happen when the following sequence of 
events happen:
 * Session started - Hive object H1 is created with allowClose=false
 * Async compilation of a long compiling query Q1 is started (BackGroundWork is 
started) - other thread got the H1 as a threadLocal Hive object too.
 * Set some metastore client configuration (set 
hive.metastore.client.socket.timeout=7000)
 * New query Q2 is issued - Hive.get(HiveConf) will detect that the 2 
configurations are not compatible
 ** Which triggers a new Hive creation  H2 
 *** The allowClose for H2 will be true
 *** This will be set as a ThreadLocal for the given thread - but not to the 
session level (I think this is a problem since from now on every thread needs 
to create it's own Hive object - but this is a different problem :))
 ** Also H1 is not closed since allowClose=false
 ** If I did not made any mistakes until this stage then "assert 
(!parentHive.allowClose());" will fail too, so the query will not run.
 * Async compilation of Q1 is finished
 ** closeCurrent() is called - since allowClose is false the HMS connection is 
not closed

Basically we will close the HMS connections only for the session when the 
session is closed, but will create multiple connections if the following is 
true:
{code:java}
if (db == null || !db.isCurrentUserOwner() || needsRefresh
|| (c != null && !isCompatible(db, c, isFastCheck))) {{code}
What do you think?

Is there any mistake in my reasoning?

Thanks,

Peter

> Async query execution can potentially fail if shared sessionHive is closed by 
> master thread.
> 
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-20682.01.patch, HIVE-20682.02.patch, 
> HIVE-20682.03.patch, HIVE-20682.04.patch
>
>
> *Problem description:*
> The master thread initializes the *sessionHive* object in *HiveSessionImpl* 
> class when we open a new session for a client connection and by default all 
> queries from this connection shares the same sessionHive object. 
> If the master thread executes a *synchronous* query, it closes the 
> sessionHive object (referred via thread local hiveDb) if  
> {{Hive.isCompatible}} returns false and sets new Hive object in thread local 
> HiveDb but doesn't change the sessionHive object in the session. Whereas, 
> *asynchronous* query execution via async threads never closes the sessionHive 
> object and it just creates a new one if needed and sets it as their thread 
> local hiveDb.
> So, the problem can happen in the case where an *asynchronous* query is being 
> executed by async threads refers to sessionHive object and the master thread 
> receives a *synchronous* query that closes the same sessionHive object. 
> Also, each query execution overwrites the thread local hiveDb object to 
> sessionHive object which potentially leaks a metastore connection if the 
> previous synchronous query execution re-created the Hive object.
> *Possible Fix:*
> The *sessionHive* object could be shared my multiple threads and so it 
> shouldn't be allowed to be closed by any query execution threads when they 
> re-create the Hive object due to changes in Hive configurations. But the Hive 
> objects created by query execution threads should be closed when the thread 
> exits.
> So, it is proposed to have an *isAllowClose* flag (default: *true*) in Hive 
> object which should be set to *false* for *sessionHive* and would be 
> forcefully closed when the session is closed or released.
> cc [~pvary]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20638) Upgrade version of Jetty to 9.3.25.v20180904

2018-10-26 Thread Laszlo Bodor (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665114#comment-16665114
 ] 

Laszlo Bodor commented on HIVE-20638:
-

tests passed

cc: [~nishantbangarwa] , [~kgyrtkirk]

> Upgrade version of Jetty to 9.3.25.v20180904
> 
>
> Key: HIVE-20638
> URL: https://issues.apache.org/jira/browse/HIVE-20638
> Project: Hive
>  Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-20638.01.patch, HIVE-20638.02.patch, 
> HIVE-20638.03.patch, HIVE-20638.03.patch, HIVE-20638.04.patch, 
> HIVE-20638.05.patch
>
>
> Current version is 9.3.20.v20170531



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.

2018-10-26 Thread Sankar Hariappan (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665081#comment-16665081
 ] 

Sankar Hariappan commented on HIVE-20682:
-

Thanks [~maheshk114] for the review!

[~daijy], can you please review and +1 it?

> Async query execution can potentially fail if shared sessionHive is closed by 
> master thread.
> 
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-20682.01.patch, HIVE-20682.02.patch, 
> HIVE-20682.03.patch, HIVE-20682.04.patch
>
>
> *Problem description:*
> The master thread initializes the *sessionHive* object in *HiveSessionImpl* 
> class when we open a new session for a client connection and by default all 
> queries from this connection shares the same sessionHive object. 
> If the master thread executes a *synchronous* query, it closes the 
> sessionHive object (referred via thread local hiveDb) if  
> {{Hive.isCompatible}} returns false and sets new Hive object in thread local 
> HiveDb but doesn't change the sessionHive object in the session. Whereas, 
> *asynchronous* query execution via async threads never closes the sessionHive 
> object and it just creates a new one if needed and sets it as their thread 
> local hiveDb.
> So, the problem can happen in the case where an *asynchronous* query is being 
> executed by async threads refers to sessionHive object and the master thread 
> receives a *synchronous* query that closes the same sessionHive object. 
> Also, each query execution overwrites the thread local hiveDb object to 
> sessionHive object which potentially leaks a metastore connection if the 
> previous synchronous query execution re-created the Hive object.
> *Possible Fix:*
> The *sessionHive* object could be shared my multiple threads and so it 
> shouldn't be allowed to be closed by any query execution threads when they 
> re-create the Hive object due to changes in Hive configurations. But the Hive 
> objects created by query execution threads should be closed when the thread 
> exits.
> So, it is proposed to have an *isAllowClose* flag (default: *true*) in Hive 
> object which should be set to *false* for *sessionHive* and would be 
> forcefully closed when the session is closed or released.
> cc [~pvary]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20782) Cleaning some unused code

2018-10-26 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665068#comment-16665068
 ] 

Hive QA commented on HIVE-20782:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12945602/HIVE-20768.2.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/14645/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14645/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14645/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-10-26 11:39:22.312
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-14645/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-10-26 11:39:22.315
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
   a99be34..50a96d7  master -> origin/master
+ git reset --hard HEAD
HEAD is now at a99be34 HIVE-20746: HiveProtoHookLogger does not close file at 
end of day. (Harish JP, reviewd by Anishek Agarwal)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 50a96d7 HIVE-20806: Add ASF license for files added in 
HIVE-20679 (Anishek Agarwal, reviewed by Sankar Hariappan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-10-26 11:39:23.324
+ rm -rf ../yetus_PreCommit-HIVE-Build-14645
+ mkdir ../yetus_PreCommit-HIVE-Build-14645
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-14645
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-14645/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: cannot apply binary patch to 'data/files/test.jceks' without full index 
line
error: data/files/test.jceks: patch does not apply
error: patch failed: 
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java:517
Falling back to three-way merge...
Applied patch to 
'ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java' cleanly.
Falling back to three-way merge...
Applied patch to 
'ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTumbledWindow.java'
 cleanly.
Falling back to three-way merge...
Applied patch to 'ql/src/test/queries/clientpositive/tumbled_window_tests.q' 
cleanly.
error: patch failed: 
ql/src/test/results/clientpositive/llap/vector_partitioned_date_time.q.out:233
Falling back to three-way merge...
Applied patch to 
'ql/src/test/results/clientpositive/llap/vector_partitioned_date_time.q.out' 
with conflicts.
error: patch failed: 
ql/src/test/results/clientpositive/perf/spark/query24.q.out:147
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/spark/query24.q.out' 
with conflicts.
error: patch failed: 
ql/src/test/results/clientpositive/perf/spark/query54.q.out:1
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/spark/query54.q.out' 
with conflicts.
error: patch failed: 
ql/src/test/results/clientpositive/perf/tez/query17.q.out:103
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query17.q.out' 
with conflicts.
error: patch failed: 
ql/src/test/results/clientpositive/perf/tez/query24.q.out:114
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query24.q.out' 
with conflicts.
error: patch failed:

[jira] [Commented] (HIVE-18767) Some alterPartitions invocations throw 'NumberFormatException: null'

2018-10-26 Thread Mass Dosage (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665069#comment-16665069
 ] 

Mass Dosage commented on HIVE-18767:


Ah sorry, yes, I got confused (I've been away on holiday!). You're right, so 
we're done here and this issue will be closed when released etc.

> Some alterPartitions invocations throw 'NumberFormatException: null'
> 
>
> Key: HIVE-18767
> URL: https://issues.apache.org/jira/browse/HIVE-18767
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.3.3, 3.1.0, 4.0.0, 3.2.0
>Reporter: Yuming Wang
>Assignee: Mass Dosage
>Priority: Major
> Fix For: 2.4.0, 4.0.0, 3.2.0, 3.1.1, 2.3.4
>
> Attachments: HIVE-18767-branch-2.3.patch, HIVE-18767-branch-2.patch, 
> HIVE-18767-branch-3.1.patch, HIVE-18767-branch-3.patch, HIVE-18767.1.patch, 
> HIVE-18767.2-branch-2.3.patch, HIVE-18767.2-branch-2.patch, 
> HIVE-18767.2-branch-3.1.patch, HIVE-18767.2.branch-3.patch, 
> HIVE-18767.2.patch, HIVE-18767.3-branch-3.1.patch, 
> HIVE-18767.3.branch-3.patch, HIVE-18767.3.patch, 
> HIVE-18767.4-branch-3.1.patch, HIVE-18767.4.branch-3.1.patch, 
> HIVE-18767.4.patch, HIVE-18767.5.patch, HIVE-18767.6.patch
>
>
> Error messages:
> {noformat}
> [info] Cause: java.lang.NumberFormatException: null
> [info] at java.lang.Long.parseLong(Long.java:552)
> [info] at java.lang.Long.parseLong(Long.java:631)
> [info] at 
> org.apache.hadoop.hive.metastore.MetaStoreUtils.isFastStatsSame(MetaStoreUtils.java:315)
> [info] at 
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:605)
> [info] at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:3837)
> [info] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [info] at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [info] at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [info] at java.lang.reflect.Method.invoke(Method.java:498)
> [info] at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> [info] at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> [info] at 
> com.sun.proxy.$Proxy23.alter_partitions_with_environment_context(Unknown 
> Source)
> [info] at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_partitions(HiveMetaStoreClient.java:1527)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18767) Some alterPartitions invocations throw 'NumberFormatException: null'

2018-10-26 Thread Peter Vary (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665066#comment-16665066
 ] 

Peter Vary commented on HIVE-18767:
---

[~massdos...@gmail.com]: It is in 
[https://github.com/apache/hive/commits/branch-3]

branch-3.0 is if we want to release 3.0.1 - which I do not think will ever 
happen. So we should not torture ourselves with that.

What do you think?

> Some alterPartitions invocations throw 'NumberFormatException: null'
> 
>
> Key: HIVE-18767
> URL: https://issues.apache.org/jira/browse/HIVE-18767
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.3.3, 3.1.0, 4.0.0, 3.2.0
>Reporter: Yuming Wang
>Assignee: Mass Dosage
>Priority: Major
> Fix For: 2.4.0, 4.0.0, 3.2.0, 3.1.1, 2.3.4
>
> Attachments: HIVE-18767-branch-2.3.patch, HIVE-18767-branch-2.patch, 
> HIVE-18767-branch-3.1.patch, HIVE-18767-branch-3.patch, HIVE-18767.1.patch, 
> HIVE-18767.2-branch-2.3.patch, HIVE-18767.2-branch-2.patch, 
> HIVE-18767.2-branch-3.1.patch, HIVE-18767.2.branch-3.patch, 
> HIVE-18767.2.patch, HIVE-18767.3-branch-3.1.patch, 
> HIVE-18767.3.branch-3.patch, HIVE-18767.3.patch, 
> HIVE-18767.4-branch-3.1.patch, HIVE-18767.4.branch-3.1.patch, 
> HIVE-18767.4.patch, HIVE-18767.5.patch, HIVE-18767.6.patch
>
>
> Error messages:
> {noformat}
> [info] Cause: java.lang.NumberFormatException: null
> [info] at java.lang.Long.parseLong(Long.java:552)
> [info] at java.lang.Long.parseLong(Long.java:631)
> [info] at 
> org.apache.hadoop.hive.metastore.MetaStoreUtils.isFastStatsSame(MetaStoreUtils.java:315)
> [info] at 
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:605)
> [info] at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:3837)
> [info] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [info] at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [info] at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [info] at java.lang.reflect.Method.invoke(Method.java:498)
> [info] at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> [info] at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> [info] at 
> com.sun.proxy.$Proxy23.alter_partitions_with_environment_context(Unknown 
> Source)
> [info] at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_partitions(HiveMetaStoreClient.java:1527)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18767) Some alterPartitions invocations throw 'NumberFormatException: null'

2018-10-26 Thread Peter Vary (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-18767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-18767:
--
Fix Version/s: 2.3.4
   3.1.1

> Some alterPartitions invocations throw 'NumberFormatException: null'
> 
>
> Key: HIVE-18767
> URL: https://issues.apache.org/jira/browse/HIVE-18767
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.3.3, 3.1.0, 4.0.0, 3.2.0
>Reporter: Yuming Wang
>Assignee: Mass Dosage
>Priority: Major
> Fix For: 2.4.0, 4.0.0, 3.2.0, 3.1.1, 2.3.4
>
> Attachments: HIVE-18767-branch-2.3.patch, HIVE-18767-branch-2.patch, 
> HIVE-18767-branch-3.1.patch, HIVE-18767-branch-3.patch, HIVE-18767.1.patch, 
> HIVE-18767.2-branch-2.3.patch, HIVE-18767.2-branch-2.patch, 
> HIVE-18767.2-branch-3.1.patch, HIVE-18767.2.branch-3.patch, 
> HIVE-18767.2.patch, HIVE-18767.3-branch-3.1.patch, 
> HIVE-18767.3.branch-3.patch, HIVE-18767.3.patch, 
> HIVE-18767.4-branch-3.1.patch, HIVE-18767.4.branch-3.1.patch, 
> HIVE-18767.4.patch, HIVE-18767.5.patch, HIVE-18767.6.patch
>
>
> Error messages:
> {noformat}
> [info] Cause: java.lang.NumberFormatException: null
> [info] at java.lang.Long.parseLong(Long.java:552)
> [info] at java.lang.Long.parseLong(Long.java:631)
> [info] at 
> org.apache.hadoop.hive.metastore.MetaStoreUtils.isFastStatsSame(MetaStoreUtils.java:315)
> [info] at 
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:605)
> [info] at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:3837)
> [info] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [info] at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [info] at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [info] at java.lang.reflect.Method.invoke(Method.java:498)
> [info] at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> [info] at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> [info] at 
> com.sun.proxy.$Proxy23.alter_partitions_with_environment_context(Unknown 
> Source)
> [info] at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_partitions(HiveMetaStoreClient.java:1527)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18767) Some alterPartitions invocations throw 'NumberFormatException: null'

2018-10-26 Thread Peter Vary (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-18767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-18767:
--
  Resolution: Fixed
Target Version/s: 3.1.0, 2.3.3, 4.0.0, 3.2.0  (was: 2.3.3, 3.1.0, 4.0.0, 
3.2.0)
  Status: Resolved  (was: Patch Available)

> Some alterPartitions invocations throw 'NumberFormatException: null'
> 
>
> Key: HIVE-18767
> URL: https://issues.apache.org/jira/browse/HIVE-18767
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.3.3, 3.1.0, 4.0.0, 3.2.0
>Reporter: Yuming Wang
>Assignee: Mass Dosage
>Priority: Major
> Fix For: 2.4.0, 4.0.0, 3.2.0, 3.1.1, 2.3.4
>
> Attachments: HIVE-18767-branch-2.3.patch, HIVE-18767-branch-2.patch, 
> HIVE-18767-branch-3.1.patch, HIVE-18767-branch-3.patch, HIVE-18767.1.patch, 
> HIVE-18767.2-branch-2.3.patch, HIVE-18767.2-branch-2.patch, 
> HIVE-18767.2-branch-3.1.patch, HIVE-18767.2.branch-3.patch, 
> HIVE-18767.2.patch, HIVE-18767.3-branch-3.1.patch, 
> HIVE-18767.3.branch-3.patch, HIVE-18767.3.patch, 
> HIVE-18767.4-branch-3.1.patch, HIVE-18767.4.branch-3.1.patch, 
> HIVE-18767.4.patch, HIVE-18767.5.patch, HIVE-18767.6.patch
>
>
> Error messages:
> {noformat}
> [info] Cause: java.lang.NumberFormatException: null
> [info] at java.lang.Long.parseLong(Long.java:552)
> [info] at java.lang.Long.parseLong(Long.java:631)
> [info] at 
> org.apache.hadoop.hive.metastore.MetaStoreUtils.isFastStatsSame(MetaStoreUtils.java:315)
> [info] at 
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:605)
> [info] at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:3837)
> [info] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [info] at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [info] at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [info] at java.lang.reflect.Method.invoke(Method.java:498)
> [info] at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> [info] at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> [info] at 
> com.sun.proxy.$Proxy23.alter_partitions_with_environment_context(Unknown 
> Source)
> [info] at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_partitions(HiveMetaStoreClient.java:1527)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18767) Some alterPartitions invocations throw 'NumberFormatException: null'

2018-10-26 Thread Mass Dosage (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665056#comment-16665056
 ] 

Mass Dosage commented on HIVE-18767:


So [~pvary] should it show up in 
https://github.com/apache/hive/tree/branch-3.0? I don't see it there?

> Some alterPartitions invocations throw 'NumberFormatException: null'
> 
>
> Key: HIVE-18767
> URL: https://issues.apache.org/jira/browse/HIVE-18767
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.3.3, 3.1.0, 4.0.0, 3.2.0
>Reporter: Yuming Wang
>Assignee: Mass Dosage
>Priority: Major
> Fix For: 2.4.0, 4.0.0, 3.2.0
>
> Attachments: HIVE-18767-branch-2.3.patch, HIVE-18767-branch-2.patch, 
> HIVE-18767-branch-3.1.patch, HIVE-18767-branch-3.patch, HIVE-18767.1.patch, 
> HIVE-18767.2-branch-2.3.patch, HIVE-18767.2-branch-2.patch, 
> HIVE-18767.2-branch-3.1.patch, HIVE-18767.2.branch-3.patch, 
> HIVE-18767.2.patch, HIVE-18767.3-branch-3.1.patch, 
> HIVE-18767.3.branch-3.patch, HIVE-18767.3.patch, 
> HIVE-18767.4-branch-3.1.patch, HIVE-18767.4.branch-3.1.patch, 
> HIVE-18767.4.patch, HIVE-18767.5.patch, HIVE-18767.6.patch
>
>
> Error messages:
> {noformat}
> [info] Cause: java.lang.NumberFormatException: null
> [info] at java.lang.Long.parseLong(Long.java:552)
> [info] at java.lang.Long.parseLong(Long.java:631)
> [info] at 
> org.apache.hadoop.hive.metastore.MetaStoreUtils.isFastStatsSame(MetaStoreUtils.java:315)
> [info] at 
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:605)
> [info] at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:3837)
> [info] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [info] at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [info] at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [info] at java.lang.reflect.Method.invoke(Method.java:498)
> [info] at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> [info] at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> [info] at 
> com.sun.proxy.$Proxy23.alter_partitions_with_environment_context(Unknown 
> Source)
> [info] at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_partitions(HiveMetaStoreClient.java:1527)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20796) jdbc URL can contain sensitive information that should not be logged

2018-10-26 Thread Laszlo Pinter (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Pinter updated HIVE-20796:
-
Attachment: HIVE-20796.05.patch

> jdbc URL can contain sensitive information that should not be logged
> 
>
> Key: HIVE-20796
> URL: https://issues.apache.org/jira/browse/HIVE-20796
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Affects Versions: 4.0.0
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-20796.01.patch, HIVE-20796.02.patch, 
> HIVE-20796.03.patch, HIVE-20796.04.patch, HIVE-20796.05.patch
>
>
> It is possible to put passwords in the jdbc connection url and some jdbc 
> drivers will supposedly use that. (derby, mysql). This information is 
> considered sensitive, and should be masked out, while logging the connection 
> url.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.

2018-10-26 Thread mahesh kumar behera (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664992#comment-16664992
 ] 

mahesh kumar behera commented on HIVE-20682:


patch 4 looks fine to me .

> Async query execution can potentially fail if shared sessionHive is closed by 
> master thread.
> 
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-20682.01.patch, HIVE-20682.02.patch, 
> HIVE-20682.03.patch, HIVE-20682.04.patch
>
>
> *Problem description:*
> The master thread initializes the *sessionHive* object in *HiveSessionImpl* 
> class when we open a new session for a client connection and by default all 
> queries from this connection shares the same sessionHive object. 
> If the master thread executes a *synchronous* query, it closes the 
> sessionHive object (referred via thread local hiveDb) if  
> {{Hive.isCompatible}} returns false and sets new Hive object in thread local 
> HiveDb but doesn't change the sessionHive object in the session. Whereas, 
> *asynchronous* query execution via async threads never closes the sessionHive 
> object and it just creates a new one if needed and sets it as their thread 
> local hiveDb.
> So, the problem can happen in the case where an *asynchronous* query is being 
> executed by async threads refers to sessionHive object and the master thread 
> receives a *synchronous* query that closes the same sessionHive object. 
> Also, each query execution overwrites the thread local hiveDb object to 
> sessionHive object which potentially leaks a metastore connection if the 
> previous synchronous query execution re-created the Hive object.
> *Possible Fix:*
> The *sessionHive* object could be shared my multiple threads and so it 
> shouldn't be allowed to be closed by any query execution threads when they 
> re-create the Hive object due to changes in Hive configurations. But the Hive 
> objects created by query execution threads should be closed when the thread 
> exits.
> So, it is proposed to have an *isAllowClose* flag (default: *true*) in Hive 
> object which should be set to *false* for *sessionHive* and would be 
> forcefully closed when the session is closed or released.
> cc [~pvary]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.

2018-10-26 Thread Sankar Hariappan (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-20682:

Status: Patch Available  (was: Open)

04.patch fixed comments from Mahesh!

[~maheshk114], please take a look

> Async query execution can potentially fail if shared sessionHive is closed by 
> master thread.
> 
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-20682.01.patch, HIVE-20682.02.patch, 
> HIVE-20682.03.patch, HIVE-20682.04.patch
>
>
> *Problem description:*
> The master thread initializes the *sessionHive* object in *HiveSessionImpl* 
> class when we open a new session for a client connection and by default all 
> queries from this connection shares the same sessionHive object. 
> If the master thread executes a *synchronous* query, it closes the 
> sessionHive object (referred via thread local hiveDb) if  
> {{Hive.isCompatible}} returns false and sets new Hive object in thread local 
> HiveDb but doesn't change the sessionHive object in the session. Whereas, 
> *asynchronous* query execution via async threads never closes the sessionHive 
> object and it just creates a new one if needed and sets it as their thread 
> local hiveDb.
> So, the problem can happen in the case where an *asynchronous* query is being 
> executed by async threads refers to sessionHive object and the master thread 
> receives a *synchronous* query that closes the same sessionHive object. 
> Also, each query execution overwrites the thread local hiveDb object to 
> sessionHive object which potentially leaks a metastore connection if the 
> previous synchronous query execution re-created the Hive object.
> *Possible Fix:*
> The *sessionHive* object could be shared my multiple threads and so it 
> shouldn't be allowed to be closed by any query execution threads when they 
> re-create the Hive object due to changes in Hive configurations. But the Hive 
> objects created by query execution threads should be closed when the thread 
> exits.
> So, it is proposed to have an *isAllowClose* flag (default: *true*) in Hive 
> object which should be set to *false* for *sessionHive* and would be 
> forcefully closed when the session is closed or released.
> cc [~pvary]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.

2018-10-26 Thread Sankar Hariappan (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-20682:

Attachment: HIVE-20682.04.patch

> Async query execution can potentially fail if shared sessionHive is closed by 
> master thread.
> 
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-20682.01.patch, HIVE-20682.02.patch, 
> HIVE-20682.03.patch, HIVE-20682.04.patch
>
>
> *Problem description:*
> The master thread initializes the *sessionHive* object in *HiveSessionImpl* 
> class when we open a new session for a client connection and by default all 
> queries from this connection shares the same sessionHive object. 
> If the master thread executes a *synchronous* query, it closes the 
> sessionHive object (referred via thread local hiveDb) if  
> {{Hive.isCompatible}} returns false and sets new Hive object in thread local 
> HiveDb but doesn't change the sessionHive object in the session. Whereas, 
> *asynchronous* query execution via async threads never closes the sessionHive 
> object and it just creates a new one if needed and sets it as their thread 
> local hiveDb.
> So, the problem can happen in the case where an *asynchronous* query is being 
> executed by async threads refers to sessionHive object and the master thread 
> receives a *synchronous* query that closes the same sessionHive object. 
> Also, each query execution overwrites the thread local hiveDb object to 
> sessionHive object which potentially leaks a metastore connection if the 
> previous synchronous query execution re-created the Hive object.
> *Possible Fix:*
> The *sessionHive* object could be shared my multiple threads and so it 
> shouldn't be allowed to be closed by any query execution threads when they 
> re-create the Hive object due to changes in Hive configurations. But the Hive 
> objects created by query execution threads should be closed when the thread 
> exits.
> So, it is proposed to have an *isAllowClose* flag (default: *true*) in Hive 
> object which should be set to *false* for *sessionHive* and would be 
> forcefully closed when the session is closed or released.
> cc [~pvary]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.

2018-10-26 Thread Sankar Hariappan (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-20682:

Status: Open  (was: Patch Available)

> Async query execution can potentially fail if shared sessionHive is closed by 
> master thread.
> 
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-20682.01.patch, HIVE-20682.02.patch, 
> HIVE-20682.03.patch
>
>
> *Problem description:*
> The master thread initializes the *sessionHive* object in *HiveSessionImpl* 
> class when we open a new session for a client connection and by default all 
> queries from this connection shares the same sessionHive object. 
> If the master thread executes a *synchronous* query, it closes the 
> sessionHive object (referred via thread local hiveDb) if  
> {{Hive.isCompatible}} returns false and sets new Hive object in thread local 
> HiveDb but doesn't change the sessionHive object in the session. Whereas, 
> *asynchronous* query execution via async threads never closes the sessionHive 
> object and it just creates a new one if needed and sets it as their thread 
> local hiveDb.
> So, the problem can happen in the case where an *asynchronous* query is being 
> executed by async threads refers to sessionHive object and the master thread 
> receives a *synchronous* query that closes the same sessionHive object. 
> Also, each query execution overwrites the thread local hiveDb object to 
> sessionHive object which potentially leaks a metastore connection if the 
> previous synchronous query execution re-created the Hive object.
> *Possible Fix:*
> The *sessionHive* object could be shared my multiple threads and so it 
> shouldn't be allowed to be closed by any query execution threads when they 
> re-create the Hive object due to changes in Hive configurations. But the Hive 
> objects created by query execution threads should be closed when the thread 
> exits.
> So, it is proposed to have an *isAllowClose* flag (default: *true*) in Hive 
> object which should be set to *false* for *sessionHive* and would be 
> forcefully closed when the session is closed or released.
> cc [~pvary]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

1 2 >

1 - 100 of 103 matches

Mail list logo