[jira] [Created] (HIVE-11798) The Beeline report should not display the header when --showHeader is set to false.

[Chetan Bhat (JIRA)]

Chetan Bhat created HIVE-11798:
--

 Summary: The Beeline report should not display the header when 
--showHeader is set to false.
 Key: HIVE-11798
 URL: https://issues.apache.org/jira/browse/HIVE-11798
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Affects Versions: 0.13.0
 Environment: SUSE Linux 11
Reporter: Chetan Bhat
Priority: Minor


In Beeline tool User sets the --showheader option as false.
In command line interface user inputs the command bin/beeline -u 
jdbc:hive2://10.19.92.183:1 [^] --showHeader=false

Actual Result : The Beeline report displays the column name.

Expected Result : The Beeline report should not display the header when 
--showHeader is set to false.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 38292: HIVE-11768 java.io.DeleteOnExitHook leaks memory on long running Hive Server2 Instances

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38292/
---

Review request for hive.


Repository: hive-git


Description
---

More than 490,000 paths was added to java.io.DeleteOnExitHook on one of our 
long running HiveServer2 instances,taken up more than 100MB on heap.
  Most of the paths contains a suffix of ".pipeout".


Diffs
-

  common/src/java/org/apache/hive/common/util/ShutdownHookManager.java fd2f20a 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 7ed8e5f 
  
service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java
 bcc66cf 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
cc9df76 

Diff: https://reviews.apache.org/r/38292/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 38290: HIVE-11774 Show macro definition for desc function

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38290/
---

(Updated 9 11, 2015, 5:53 오전)


Review request for hive.


Summary (updated)
-

HIVE-11774 Show macro definition for desc function 


Bugs: HIVE-11774
https://issues.apache.org/jira/browse/HIVE-11774


Repository: hive-git


Description
---

Currently, desc function shows nothing for macro. It would be helpful if it 
shows the definition of it.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 734742c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java 
b09b706 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java f1fe30d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 7671d29 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java a5d59ae 
  ql/src/java/org/apache/hadoop/hive/ql/parse/MacroSemanticAnalyzer.java 
e3ba201 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateMacroDesc.java 443614c 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMacro.java 
3f505f2 
  ql/src/test/org/apache/hadoop/hive/ql/plan/TestCreateMacroDesc.java 00a24b9 
  ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFMacro.java 
8bbaa6b 
  ql/src/test/queries/clientpositive/macro.q 47b05ff 
  ql/src/test/results/clientpositive/macro.q.out 76ea250 

Diff: https://reviews.apache.org/r/38290/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 38290: Show macro definition for desc function

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38290/
---

Review request for hive.


Bugs: HIVE-11774
https://issues.apache.org/jira/browse/HIVE-11774


Repository: hive-git


Description
---

Currently, desc function shows nothing for macro. It would be helpful if it 
shows the definition of it.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 734742c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java 
b09b706 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java f1fe30d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 7671d29 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java a5d59ae 
  ql/src/java/org/apache/hadoop/hive/ql/parse/MacroSemanticAnalyzer.java 
e3ba201 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateMacroDesc.java 443614c 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMacro.java 
3f505f2 
  ql/src/test/org/apache/hadoop/hive/ql/plan/TestCreateMacroDesc.java 00a24b9 
  ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFMacro.java 
8bbaa6b 
  ql/src/test/queries/clientpositive/macro.q 47b05ff 
  ql/src/test/results/clientpositive/macro.q.out 76ea250 

Diff: https://reviews.apache.org/r/38290/diff/


Testing
---


Thanks,

Navis Ryu

[jira] [Created] (HIVE-11797) Alter table change columnname doesn't work on avro serde hive table

[Joy SN (JIRA)]

Joy SN created HIVE-11797:
-

 Summary: Alter table change columnname doesn't work on avro serde 
hive table
 Key: HIVE-11797
 URL: https://issues.apache.org/jira/browse/HIVE-11797
 Project: Hive
  Issue Type: Bug
Reporter: Joy SN


We create a table using Avro serde 
Hive table name hive_t1.
Then we try to change the column name.
The commands ends successfully but the name of the column is not modified.

create table if not exists hive_t1
partitioned by (p1 int)
row format SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS
INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES ('avro.schema.literal'='{
  "namespace": "testing.hive.avro.serde",
  "name": "avro_table",
  "type": "record",
  "fields": [
{
  "name":"number1",
  "type":"int",
  "doc":"Order of playing the role"
},
{
  "name":"first_name",
  "type":"string",
  "doc":"first name of actor playing role"
},
{
  "name":"last_name",
  "type":"string",
  "doc":"last name of actor playing role"
},
{
  "name":"extra_field",
  "type":"string",
  "doc:":"an extra field not in the original file",
  "default":"fishfingers and custard"
}
  ]
}');

hive> alter table hive_t1 change column number number1 int;
OK
Time taken: 0.12 seconds
hive> select * from hive_t1 limit 5;
OK
hive_t1.number   hive_t1.first_name   hive_t1.last_name
hive_t1.extra_field  hive_t1.p1
6   Colin   Baker   fishfingers and custard 100
3   Jon Pertwee fishfingers and custard 100
4   Tom Baker   fishfingers and custard 100
5   Peter   Davison fishfingers and custard 100
11  MattSmith   fishfingers and custard 100
Time taken: 0.05 seconds, Fetched: 5 row(s)
hive> describe hive_t1;
OK
col_namedata_type   comment
number  int from deserializer   
first_name  string  from deserializer   
last_name   string  from deserializer   
extra_field string  from deserializer   
p1  int 
 
# Partition Information  
# col_name  data_type   comment 
 
p1  int 
Time taken: 0.051 seconds, Fetched: 10 row(s)

-- Using the below command also the column name is not changed from "number" to 
"number1"

hive> alter table hive_t1 change number number1 int;
OK
Time taken: 0.081 seconds
hive>  describe hive_t1;
OK
col_namedata_type   comment
number  int from deserializer   
first_name  string  from deserializer   
last_name   string  from deserializer   
extra_field string  from deserializer   
p1  int 
 
# Partition Information  
# col_name  data_type   comment 
 
p1  int 
Time taken: 0.054 seconds, Fetched: 10 row(s)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38216: HIVE-11745: Alter table Exchange partition with multiple partition_spec is not working

[Szehon Ho]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38216/#review98547
---


Publishing again as it did not send the email, I added some more comments.

- Szehon Ho


On Sept. 10, 2015, 7:36 p.m., Yongzhi Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38216/
> ---
> 
> (Updated Sept. 10, 2015, 7:36 p.m.)
> 
> 
> Review request for hive, Chao Sun, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-11745
> https://issues.apache.org/jira/browse/HIVE-11745
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Alter table Exchange partition with multiple partition_spec does not work in 
> cluster mode because in rename, the parent folder for destination path does 
> not physically exist. Some files system(hdfs for instance) does not 
> support(or allow) this. Fix by create parent folder first.
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 
> bed621d3eb74f01e54110552f68538afd228018d 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> 1840e76cc567e95e1942d912b8ab0db516d63a3b 
>   ql/src/test/queries/clientpositive/exchgpartition2lel.q PRE-CREATION 
>   ql/src/test/results/clientpositive/exchgpartition2lel.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38216/diff/
> 
> 
> Testing
> ---
> 
> Add minimr unit test.
> 
> 
> Thanks,
> 
> Yongzhi Chen
> 
>

Re: Review Request 38216: HIVE-11745: Alter table Exchange partition with multiple partition_spec is not working

[Szehon Ho]

> On Sept. 10, 2015, 6:02 p.m., Szehon Ho wrote:
> > Please make sure that file permission inheritance works for this feature 
> > (see HadoopShims.getFullFileStatus and HadoopShims.setFullFileStatus).  And 
> > please add a test to FolderPermissionBase after you verified it?

Please make sure to add a unit test in this file, to verify permission 
inheritance works.


> On Sept. 10, 2015, 6:02 p.m., Szehon Ho wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, 
> > line 2552
> > 
> >
> > I think this whole method can be moved to FileUtils for organization.  
> > Also please check if there's any method there already.
> 
> Yongzhi Chen wrote:
> I think it may be better as a private method in the HiveMetaStore class 
> for it will using its private variable wh (hdfs warehouse) .

Actually looking more into the code, this method should not be necessary.  You 
can just call wh.mkdirs directly.  The underlying FileSystem.mkdirs has the 
same semantics as -p, there should be no file system that violates this.  If 
there were, many other partition codes would break..


> On Sept. 10, 2015, 6:02 p.m., Szehon Ho wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, 
> > line 2548
> > 
> >
> > Please make sure we follow the normal code standards (properly indent 
> > the comment, make sure there are spaces after "if", and "catch" does not 
> > have to be a new line.
> 
> Yongzhi Chen wrote:
> Fixed the indent of the comment. If and catch statements.

Thanks, please also put parenthesis around all the if-statements for 
consistency.


- Szehon


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38216/#review98435
---


On Sept. 10, 2015, 7:36 p.m., Yongzhi Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38216/
> ---
> 
> (Updated Sept. 10, 2015, 7:36 p.m.)
> 
> 
> Review request for hive, Chao Sun, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-11745
> https://issues.apache.org/jira/browse/HIVE-11745
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Alter table Exchange partition with multiple partition_spec does not work in 
> cluster mode because in rename, the parent folder for destination path does 
> not physically exist. Some files system(hdfs for instance) does not 
> support(or allow) this. Fix by create parent folder first.
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 
> bed621d3eb74f01e54110552f68538afd228018d 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> 1840e76cc567e95e1942d912b8ab0db516d63a3b 
>   ql/src/test/queries/clientpositive/exchgpartition2lel.q PRE-CREATION 
>   ql/src/test/results/clientpositive/exchgpartition2lel.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38216/diff/
> 
> 
> Testing
> ---
> 
> Add minimr unit test.
> 
> 
> Thanks,
> 
> Yongzhi Chen
> 
>

[jira] [Created] (HIVE-11796) CLI option is not updated when executing the initial files[beeline-cli]

[Ferdinand Xu (JIRA)]

Ferdinand Xu created HIVE-11796:
---

 Summary: CLI option is not updated when executing the initial 
files[beeline-cli]
 Key: HIVE-11796
 URL: https://issues.apache.org/jira/browse/HIVE-11796
 Project: Hive
  Issue Type: Sub-task
Affects Versions: beeline-cli-branch
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
 Fix For: beeline-cli-branch


"Method not supported" is thrown when executing the initial files. This is 
caused by CLI option is not updated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11795) hive-1.2.1 compile spark-1.5.0 output error

[liu wen (JIRA)]

liu wen created HIVE-11795:
--

 Summary: hive-1.2.1 compile spark-1.5.0 output error
 Key: HIVE-11795
 URL: https://issues.apache.org/jira/browse/HIVE-11795
 Project: Hive
  Issue Type: Bug
 Environment: operation system: centos 6.5
softwares: maven 3.3.3; hive-1.2.1; hadoop-2.7.1; jdk_1.7.0_71
All these softwares in directory: /opt/modules/
Reporter: liu wen
Assignee: liu wen


When input command: "mvn clean install -Phadoop-2,dist -DskipTests 
-Dhadoop-23.version=2.7.1 -Dspark.version=1.5.0" to compile hadoo-2.7.1 
spark-1.5.0, system output an error, as follows:
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on 
project spark-client: Compilation failure
[ERROR] 
/opt/modules/apache-hive-1.2.1-src/spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java:[441,11]
 org.apache.hive.spark.client.RemoteDriver.ClientListener is not abstract and 
does not override abstract method 
onBlockUpdated(org.apache.spark.scheduler.SparkListenerBlockUpdated) in 
org.apache.spark.scheduler.SparkListener
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :spark-client



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11794) GBY vectorization appears to process COMPLETE reduce-side GBY incorrectly

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-11794:
---

 Summary: GBY vectorization appears to process COMPLETE reduce-side 
GBY incorrectly
 Key: HIVE-11794
 URL: https://issues.apache.org/jira/browse/HIVE-11794
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Matt McCline


The code in Vectorizer is as such:
{noformat}
boolean isMergePartial = (desc.getMode() != GroupByDesc.Mode.HASH);
{noformat}
then, if it's reduce side:
{noformat}
if (isMergePartial) {
// Reduce Merge-Partial GROUP BY.
// A merge-partial GROUP BY is fed by grouping by keys from 
reduce-shuffle.  It is the
// first (or root) operator for its reduce task.

  } else {
// Reduce Hash GROUP BY or global aggregation.
...
{noformat}

In fact, the comments are missing the COMPLETE mode. Both from the comment:
{noformat}
 COMPLETE: complete 1-phase aggregation: iterate, terminate
...
HASH: For non-distinct the same as PARTIAL1 but use hash-table-based aggregation
...
PARTIAL1: partial aggregation - first phase: iterate, terminatePartial
{noformat}

and from the explain plan like this (the query has multiple stages of 
aggregations over a union; the mapper does a partial hash aggregation for each 
side of the union, which is then followed by mergepartial, and 2nd stage as 
complete):
{noformat}
Reduce Operator Tree:
  Group By Operator
keys: KEY._col0 (type: int), KEY._col1 (type: int), KEY._col2 (type: int), 
KEY._col3 (type: int), KEY._col4 (type: int), KEY._col5 (type: bigint), 
KEY._col6 (type: bigint), KEY._col7 (type: bigint), KEY._col8 (type: bigint), 
KEY._col9 (type: bigint), KEY._col10 (type: bigint), KEY._col11 (type: bigint), 
KEY._col12 (type: bigint)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, 
_col8, _col9, _col10, _col11, _col12
Group By Operator
  aggregations: sum(_col5), sum(_col6), sum(_col7), sum(_col8), sum(_col9), 
sum(_col10), sum(_col11), sum(_col12)
  keys: _col0 (type: int), _col1 (type: int), _col2 (type: int), _col3 
(type: int), _col4 (type: int)
  mode: complete
  outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, 
_col7, _col8, _col9, _col10, _col11, _col12
{noformat}

it seems like COMPLETE is actually the global aggregation, and HASH isn't (or 
may not be).
So, it seems like reduce-side COMPLETE should be handled on the else-path of 
the above if. For map-side, it doesn't check mode at all as far as I can see.
Not sure if additional code changes are necessary after that, it may just work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11793) SHOW LOCKS with DbTxnManager ignores filter options

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-11793:
-

 Summary: SHOW LOCKS with DbTxnManager ignores filter options
 Key: HIVE-11793
 URL: https://issues.apache.org/jira/browse/HIVE-11793
 Project: Hive
  Issue Type: Improvement
  Components: Transactions
Affects Versions: 1.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


https://cwiki.apache.org/confluence/display/Hive/Locking and 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowLocks
 list various options that can be used with SHOW LOCKS, e.g. 

When ACID is enabled, all these options are ignored and a full list is returned.
(also only ext lock id is shown, int lock id is not).

see DDLTask.showLocks() and TxnHandler.showLocks()

requires extending ShowLocksRequest which is a Thrift object



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11792) User explain in tez does not preserve ordering

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-11792:


 Summary: User explain in tez does not preserve ordering
 Key: HIVE-11792
 URL: https://issues.apache.org/jira/browse/HIVE-11792
 Project: Hive
  Issue Type: Bug
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran
 Fix For: 2.0.0


HIVE-11037 introduced user level explain. The output of operator attributes are 
JSONObject which does not preserve ordering. This is causing diffs in test 
output when run under tez vs llap like below

{code}
-partition:{"ts":"2012-01-03+14:46:31","ds":"2012-01-03"}
-
table:{"serde:":"org.apache.hadoop.hive.ql.io.orc.OrcSerde","name:":"default.src_orc_merge_test_part","input
 format:":"org.apache.hadoop.hive.ql.io.orc.OrcInputFormat","output 
format:":"org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat"}
+partition:{"ds":"2012-01-03","ts":"2012-01-03+14:46:31"}
+table:{"name:":"default.src_orc_merge_test_part","input 
format:":"org.apache.hadoop.hive.ql.io.orc.OrcInputFormat","output 
format:":"org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat","serde:":"org.apache.hadoop.hive.ql.io.orc.OrcSerde"}
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 38268: HIVE-10980 Merge of dynamic partitions loads all data to default partition

[Illya Yalovyy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38268/
---

Review request for hive and Gopal V.


Bugs: HIVE-10980
https://issues.apache.org/jira/browse/HIVE-10980


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-10980

Conditions that lead to the issue:
1. Execution engine set to MapReduce
2. Partition columns have different types
3. Both static and dynamic partitions are used in the query
4. Dynamically generated partitions require merge

Result: Final data is loaded to "__HIVE_DEFAULT_PARTITION__".

Steps to reproduce:
set hive.exec.dynamic.partition=true;
set hive.exec.dynamic.partition.mode=strict;
set hive.optimize.sort.dynamic.partition=false;
set hive.merge.mapfiles=true;
set hive.merge.mapredfiles=true;
set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;
set hive.execution.engine=mr;

create external table sdp (
  dataint bigint,
  hour int,
  req string,
  cid string,
  caid string
)
row format delimited
fields terminated by ',';

load data local inpath '../../data/files/dynpartdata1.txt' into table sdp;
load data local inpath '../../data/files/dynpartdata2.txt' into table sdp;
...
load data local inpath '../../data/files/dynpartdataN.txt' into table sdp;

create table tdp (cid string, caid string)
partitioned by (dataint bigint, hour int, req string);

insert overwrite table tdp partition (dataint=20150316, hour=16, req)
select cid, caid, req from sdp where dataint=20150316 and hour=16;

select * from tdp order by caid;
show partitions tdp;

Example of the input file:
20150316,16,reqA,clusterIdA,cacheId1
20150316,16,reqB,clusterIdB,cacheId2 
20150316,16,reqA,clusterIdC,cacheId3  
20150316,16,reqD,clusterIdD,cacheId4
20150316,16,reqA,clusterIdA,cacheId5  

Actual result:
clusterIdA  cacheId12015031616  
__HIVE_DEFAULT_PARTITION__ 
clusterIdA  cacheId12015031616  
__HIVE_DEFAULT_PARTITION__
clusterIdB  cacheId22015031616  
__HIVE_DEFAULT_PARTITION__
clusterIdC  cacheId32015031616  
__HIVE_DEFAULT_PARTITION__
clusterIdD  cacheId42015031616  
__HIVE_DEFAULT_PARTITION__
clusterIdA  cacheId52015031616  
__HIVE_DEFAULT_PARTITION__
clusterIdD  cacheId82015031616  
__HIVE_DEFAULT_PARTITION__
clusterIdB  cacheId92015031616  
__HIVE_DEFAULT_PARTITION__  
  
dataint=20150316/hour=16/req=__HIVE_DEFAULT_PARTITION__


Diffs
-

  data/files/dynpartdata1.txt PRE-CREATION 
  data/files/dynpartdata2.txt PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 4a325fb 
  
ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsUsePartitionColumnsNegative.java
 PRE-CREATION 
  
ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsUsePartitionColumnsPositive.java
 PRE-CREATION 
  ql/src/test/queries/clientpositive/dynpart_merge.q PRE-CREATION 
  ql/src/test/results/clientpositive/dynpart_merge.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/list_bucket_dml_6.q.java1.7.out d223234 
  ql/src/test/results/clientpositive/list_bucket_dml_6.q.java1.8.out f884ace 
  ql/src/test/results/clientpositive/list_bucket_dml_7.q.out 541944d 

Diff: https://reviews.apache.org/r/38268/diff/


Testing
---

1. Added new unit tests
2. Added qtest
3. Updated old qtests


Thanks,

Illya Yalovyy

Re: Review Request 38216: HIVE-11745: Alter table Exchange partition with multiple partition_spec is not working

[Yongzhi Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38216/
---

(Updated Sept. 10, 2015, 7:36 p.m.)


Review request for hive, Chao Sun, Szehon Ho, and Xuefu Zhang.


Bugs: HIVE-11745
https://issues.apache.org/jira/browse/HIVE-11745


Repository: hive-git


Description
---

Alter table Exchange partition with multiple partition_spec does not work in 
cluster mode because in rename, the parent folder for destination path does not 
physically exist. Some files system(hdfs for instance) does not support(or 
allow) this. Fix by create parent folder first.


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 
bed621d3eb74f01e54110552f68538afd228018d 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
1840e76cc567e95e1942d912b8ab0db516d63a3b 
  ql/src/test/queries/clientpositive/exchgpartition2lel.q PRE-CREATION 
  ql/src/test/results/clientpositive/exchgpartition2lel.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/38216/diff/


Testing
---

Add minimr unit test.


Thanks,

Yongzhi Chen

Re: Review Request 37778: HIVE-11634

[Hari Sankar Sivarama Subramaniyan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37778/
---

(Updated Sept. 10, 2015, 6:40 p.m.)


Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and John 
Pullokkaran.


Repository: hive-git


Description
---

Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d2c5885 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 439f616 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/PartitionColumnsSeparator.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/PointLookupOptimizer.java 
d83636d 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/OpProcFactory.java 
7262164 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 6a31689 
  ql/src/test/queries/clientpositive/pcs.q PRE-CREATION 
  ql/src/test/queries/clientpositive/pointlookup.q 1aef2ef 
  ql/src/test/queries/clientpositive/pointlookup2.q 31bebbb 
  ql/src/test/queries/clientpositive/pointlookup3.q 3daa94b 
  ql/src/test/results/clientpositive/pcs.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/pointlookup.q.out 7e19be4 
  ql/src/test/results/clientpositive/pointlookup2.q.out 55edd90 
  ql/src/test/results/clientpositive/pointlookup3.q.out 4cfb97e 

Diff: https://reviews.apache.org/r/37778/diff/


Testing
---

Local testing done. More unit tests coming in the next patch.


Thanks,

Hari Sankar Sivarama Subramaniyan

[jira] [Created] (HIVE-11791) Add test for HIVE-10122

[Illya Yalovyy (JIRA)]

Illya Yalovyy created HIVE-11791:


 Summary: Add test for HIVE-10122
 Key: HIVE-11791
 URL: https://issues.apache.org/jira/browse/HIVE-11791
 Project: Hive
  Issue Type: Test
  Components: Metastore
Affects Versions: 1.1.0
Reporter: Illya Yalovyy
Priority: Minor


Unit tests for PartitionPruner.compactExpr()



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11790) HCatLoader documentation refers to deprecated package.

[Mithun Radhakrishnan (JIRA)]

Mithun Radhakrishnan created HIVE-11790:
---

 Summary: HCatLoader documentation refers to deprecated package.
 Key: HIVE-11790
 URL: https://issues.apache.org/jira/browse/HIVE-11790
 Project: Hive
  Issue Type: Bug
Reporter: Mithun Radhakrishnan
Priority: Trivial


The [HCatLoader documentation 
page|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34013511] 
seems to refer to {{org.apache.hcatalog.pig.HCatLoader}} instead of 
{{org.apache.hive.hcatalog.pig.HCatLoader}}. (Similarly, the {{HCatStorer}} 
documentation might need change.) The old package was deprecated and removed in 
Hive 0.13.

Let's change the documentation to reflect the new package-name.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 37778: HIVE-11634

[Hari Sankar Sivarama Subramaniyan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37778/
---

(Updated Sept. 10, 2015, 5:29 p.m.)


Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and John 
Pullokkaran.


Repository: hive-git


Description
---

Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d2c5885 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 439f616 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/PartitionColumnsSeparator.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/PointLookupOptimizer.java 
d83636d 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/OpProcFactory.java 
7262164 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 6a31689 
  ql/src/test/queries/clientpositive/pcs.q PRE-CREATION 
  ql/src/test/queries/clientpositive/pointlookup.q 1aef2ef 
  ql/src/test/queries/clientpositive/pointlookup2.q 31bebbb 
  ql/src/test/queries/clientpositive/pointlookup3.q 3daa94b 
  ql/src/test/results/clientpositive/pcs.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/pointlookup.q.out 7e19be4 
  ql/src/test/results/clientpositive/pointlookup2.q.out 55edd90 
  ql/src/test/results/clientpositive/pointlookup3.q.out 4cfb97e 

Diff: https://reviews.apache.org/r/37778/diff/


Testing
---

Local testing done. More unit tests coming in the next patch.


Thanks,

Hari Sankar Sivarama Subramaniyan

[jira] [Created] (HIVE-11789) CBO to support multicolumn

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-11789:
--

 Summary: CBO to support multicolumn
 Key: HIVE-11789
 URL: https://issues.apache.org/jira/browse/HIVE-11789
 Project: Hive
  Issue Type: Improvement
  Components: CBO
Affects Versions: 2.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


Support multicolumn in CBO way in/out i.e. translate STRUCT in/to multicolumn.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

HCatalog dynamic partition on a single node

[Huang, Yunkai]

Hi,

I am trying to use the HCatalog dynamic partition feature to write a Hive table 
with partition columns. I was able to write successfully when running in 
Map-Reduce. However, I can't find a way to achieve the same result on a single 
node without running Map-Reduce.
When running in Map-Reduce, I set up the dynamic partition schema with 
HCatOutputFormat.setSchema(...), as shown in the example 
http://quabr.com/14286256/adding-partitions-to-hive-from-a-mapreduce-job.
When running on a single node,  I tried to use HCatWriter (the actual class is 
HCatOutputFormatWriter) obtained through DataTransferFactory.getHCatWriter as 
shown in 
https://cwiki.apache.org/confluence/display/Hive/HCatalog+ReaderWriter. 
However, with that class, I can't find a way to set up the dynamic partition 
schema similar to running in Map-Reduce.
Will this missing feature be addressed in upcoming enhancements? Does anyone 
have a work around currently?






ATTENTION: -

The information contained in this message (including any files transmitted with 
this message) may contain proprietary, trade secret or other confidential 
and/or legally privileged information. Any pricing information contained in 
this message or in any files transmitted with this message is always 
confidential and cannot be shared with any third parties without prior written 
approval from Syncsort. This message is intended to be read only by the 
individual or entity to whom it is addressed or by their designee. If the 
reader of this message is not the intended recipient, you are on notice that 
any use, disclosure, copying or distribution of this message, in any form, is 
strictly prohibited. If you have received this message in error, please 
immediately notify the sender and/or Syncsort and destroy all copies of this 
message in your possession, custody or control.

Patches needing review

[kulkarni.swar...@gmail.com]

Hello all,

I have couple of patches submitted and out for review for some time. If I
can get some help on getting them reviewed and merged, I would highly
appreciate that!

HIVE-11691 (Wiki update for developer debugging. Already one +1 from Lefty)
HIVE-11647 (HBase dependency bump to 1.1.1)
HIVE-11609 (10-100x perf improvement on HBase comp key queries)
HIVE-11590 (Log updates to AvroSerDe. Already one +1)
HIVE-11560 (Fixing a passivity issue introduced by HIVE-8898)
HIVE-10708 (Support to proactively check for avro reader/writer schema
compatibility)

Thanks again for help,
Swarnim

Re: Review Request 37930: HIVE-11609 Fix filter logic for hbase scans via composite keys

[Swarnim Kulkarni]

> On Aug. 31, 2015, 6:02 p.m., Sergey Shelukhin wrote:
> > hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java,
> >  line 244
> > 
> >
> > that seems brittle, can you use actual class objects?

This was already in here. Just moved from a different class.


- Swarnim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37930/#review97100
---


On Aug. 31, 2015, 5:51 a.m., Swarnim Kulkarni wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37930/
> ---
> 
> (Updated Aug. 31, 2015, 5:51 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Gopal V, Navis Ryu, and Sergey 
> Shelukhin.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11609 Fix filter logic for hbase scans via composite keys
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseScanRange.java 
> 8b6432161281bb87e536a164d056b0d1599d7428 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java
>  c0020706477df48d60937c64db2af908665c 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
>  8e72759acd939e8650e4763e55240f49de578770 
>   
> hbase-handler/src/test/org/apache/hadoop/hive/hbase/SampleHBaseKeyFactory3.java
>  712725f208f4b2c9cc166750509c25859518ec3e 
>   hbase-handler/src/test/results/positive/hbase_custom_key3.q.out 
> 25032cc36557245b0ae21c80ba66c9866b9d143d 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 
> 98bce9649b1cddd44be7cd0f9c529985e42874b8 
> 
> Diff: https://reviews.apache.org/r/37930/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Swarnim Kulkarni
> 
>

[jira] [Created] (HIVE-11788) Column stats should be preserved after db/table/partitoin rename

[Chaoyu Tang (JIRA)]

Chaoyu Tang created HIVE-11788:
--

 Summary: Column stats should be preserved after db/table/partitoin 
rename
 Key: HIVE-11788
 URL: https://issues.apache.org/jira/browse/HIVE-11788
 Project: Hive
  Issue Type: Bug
  Components: Metastore, Statistics
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang


Currently we simply delete the column stats after renaming a database, table, 
or partition since there was not an easy way in HMS to update the DB_NAME, 
TABLE_NAME and PARTITION_NAME in TAB_COL_STATS and PART_COL_STATS. With the 
removal of these redundant columns in these tables (HIVE-11786), we will still 
keep column stats in the operation which is not to change a column name or type.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11787) Remove the redundant columns in TAB_COL_STATS and PART_COL_STATS

[Chaoyu Tang (JIRA)]

Chaoyu Tang created HIVE-11787:
--

 Summary: Remove the redundant columns in TAB_COL_STATS and 
PART_COL_STATS
 Key: HIVE-11787
 URL: https://issues.apache.org/jira/browse/HIVE-11787
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang


After HIVE-11786 deprecates the use of redundant columns in TAB_COL_STATS and 
PART_COL_STATS at HMS code level, the column DB_NAME/TABLE_NAME in 
TAB_COL_STATS and DB_NAME/TABLE_NAME/PARTITION_NAME in PART_COL_STATS are 
useless and should be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11786) Deprecate the use of redundant column in colunm stats related tables

[Chaoyu Tang (JIRA)]

Chaoyu Tang created HIVE-11786:
--

 Summary: Deprecate the use of redundant column in colunm stats 
related tables
 Key: HIVE-11786
 URL: https://issues.apache.org/jira/browse/HIVE-11786
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang


The stats tables such as TAB_COL_STATS, PART_COL_STATS have redundant columns 
such as DB_NAME, TABLE_NAME, PARTITION_NAME since these tables already have 
foreign key like TBL_ID, or PART_ID referencing to TBLS or PARTITIONS. 
These redundant columns violate database normalization rules and cause a lot of 
inconvenience (sometimes difficult) in column stats related feature 
implementation. For example, when renaming a table, we have to update 
TABLE_NAME column in these tables as well which is unnecessary.

This JIRA is first to deprecate the use of these columns at HMS code level. A 
followed JIRA is to be opened to focus on DB schema change and upgrade.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11785) Carriage return and new line are processed differently when hive.fetch.task.conversion is set to none

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-11785:
---

 Summary: Carriage return and new line are processed differently 
when hive.fetch.task.conversion is set to none
 Key: HIVE-11785
 URL: https://issues.apache.org/jira/browse/HIVE-11785
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 2.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


Create the table and perform the queries as follows. You will see different 
results when the setting changes. Seems both present incorrect results.

{noformat}
hive> create table repo (lvalue int, charstring string) stored as parquet;
OK
Time taken: 0.34 seconds
hive> load data inpath '/tmp/repo/test.parquet' overwrite into table repo;
Loading data to table default.repo
chgrp: changing ownership of 
'hdfs://nameservice1/user/hive/warehouse/repo/test.parquet': User does not 
belong to hive
Table default.repo stats: [numFiles=1, numRows=0, totalSize=610, rawDataSize=0]
OK
Time taken: 0.732 seconds
hive> set hive.fetch.task.conversion=more;
hive> select * from repo;
OK
1   newline
here
herecarriage return
3   both
here
Time taken: 0.253 seconds, Fetched: 3 row(s)
hive> set hive.fetch.task.conversion=none;
hive> select * from repo;
Query ID = root_20150909113535_e081db8b-ccd9-4c44-aad9-d990ffb8edf3
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1441752031022_0006, Tracking URL = 
http://host-10-17-81-63.coe.cloudera.com:8088/proxy/application_1441752031022_0006/
Kill Command = 
/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/lib/hadoop/bin/hadoop job  
-kill job_1441752031022_0006
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2015-09-09 11:35:54,127 Stage-1 map = 0%,  reduce = 0%
2015-09-09 11:36:04,664 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.98 
sec
MapReduce Total cumulative CPU time: 2 seconds 980 msec
Ended Job = job_1441752031022_0006
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1   Cumulative CPU: 2.98 sec   HDFS Read: 4251 HDFS Write: 
51 SUCCESS
Total MapReduce CPU Time Spent: 2 seconds 980 msec
OK
1   newline
NULLNULL
2   carriage return
NULLNULL
3   both
NULLNULL
Time taken: 25.131 seconds, Fetched: 6 row(s)
hive>
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11784) Extends new cost model to reflect HDFS read/write cost when a new execution phase is created

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-11784:
--

 Summary: Extends new cost model to reflect HDFS read/write cost 
when a new execution phase is created
 Key: HIVE-11784
 URL: https://issues.apache.org/jira/browse/HIVE-11784
 Project: Hive
  Issue Type: Bug
  Components: CBO
Affects Versions: 2.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11783) Extending HPL/SQL parser

[Dmitry Tolpeko (JIRA)]

Dmitry Tolpeko created HIVE-11783:
-

 Summary: Extending HPL/SQL parser
 Key: HIVE-11783
 URL: https://issues.apache.org/jira/browse/HIVE-11783
 Project: Hive
  Issue Type: Improvement
  Components: hpl/sql
Reporter: Dmitry Tolpeko
Assignee: Dmitry Tolpeko


Need to extend procedural SQL parser and synchronize code base by adding 
PART_COUNT, PART_COUNT_BY functions as well as CMP ROW_COUNT, CMP SUM and COPY 
TO HDFS statements.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: HIVE-TRUNK-JAVA8 #111

[hiveqa]

See 


Changes:

[Jimmy Xiang] HIVE-11737: IndexOutOfBounds compiling query with duplicated 
groupby keys (Jimmy, reviewed by Szehon)

[jcamacho] HIVE-11646: CBO: Calcite Operator To Hive Operator (Calcite Return 
Path): fix multiple window spec for PTF operator (Pengcheng Xiong, reviewed by 
Jesus Camacho Rodriguez)

[jcamacho] HIVE-11617: Explain plan for multiple lateral views is very slow 
(Aihua Xu, reviewed by Jesus Camacho Rodriguez)

[Sergio Pena] HIVE-11329: Column prefix in key of hbase column prefix map 
(Wojciech Indyk, reviewed by Sergio Pena)

[Chao Sun] HIVE-11696: Exception when table-level serde is Parquet while 
partition-level serde is JSON (Aihua Xu, reviewed by Chao Sun)

[amareshwari] HIVE-11482 : Adds retrying thrift client for HiveServer2 (Akshay 
Goyal, reviewed by Amareshwari)

[Ashutosh Chauhan] HIVE-11754 : Not reachable code parts in StatsUtils (Navis 
via Ashutosh Chauhan)

--
[...truncated 324 lines...]
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestExecutionPhase.testFailingQFile.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/MockLocalCommandFactory.java
Aptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testIsolatedFailsOnExec.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestExecutionPhase.testPassingUnitTest.approved.txt
Aptest2/src/test/java/org/apache/hive/ptest/execution/TestJIRAService.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestReportingPhase.testExecute.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testShutdownBeforeExec.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testPrepGit.approved.txt
Aptest2/src/test/java/org/apache/hive/ptest/execution/ExtendedAssert.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestPhase.testRsyncFromLocalToRemoteInstancesWithFailureUnknown.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testParallelFailsOnExec.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestPhase.testExecHostsWithFailure.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestCleanupPhase.testExecute.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testIsolatedFailsOnRsyncUnknown.approved.txt
Aptest2/src/test/java/org/apache/hive/ptest/execution/TestPrepPhase.java
Aptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testPrepNone.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testBasic.approved.txt
Aptest2/src/test/java/org/apache/hive/ptest/execution/TestLocalCommand.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testAlternativeTestJVM.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testParallelFailsOnRsync.approved.txt
Aptest2/src/test/java/org/apache/hive/ptest/execution/TestPhase.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestLogDirectoryCleaner.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testPrepSvn.approved.txt
Aptest2/src/test/java/org/apache/hive/ptest/execution/context
A
ptest2/src/test/java/org/apache/hive/ptest/execution/context/TestCloudExecutionContextProvider.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/context/TestCloudComputeService.java
Aptest2/src/test/java/org/apache/hive/ptest/execution/conf
A
ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestTestParser.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestQFileTestBatch.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestTestConfiguration.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestPrepPhase.testExecute.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestPhase.testRsyncFromLocalToRemoteInstancesWithFailureOne.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestExecutionPhase.testFailingUnitTest.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testIsolatedFailsOnRsyncOne.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestPhase.testExecInstancesWithFailure.approved.txt
Aptest2/src/test/java/org/apache/hive/ptest/execution/ssh
A
ptest2/src/test/java/org/apache/hive/ptest/execution/ssh/TestSSHCommandExecutor.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/ssh/TestRSyncCommandExecutor.java
Aptest2/pom.xml
Aptest2/README.md
Checked out revi

[jira] [Created] (HIVE-11782) ORC file statistic collection fails when table contains null value

[Pavel Benes (JIRA)]

Pavel Benes created HIVE-11782:
--

 Summary: ORC file statistic collection fails when table contains 
null value
 Key: HIVE-11782
 URL: https://issues.apache.org/jira/browse/HIVE-11782
 Project: Hive
  Issue Type: Bug
Affects Versions: 0.14.0
Reporter: Pavel Benes
Priority: Critical


Sqoop import of JDBC table fails with following error:

Error: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl$DecimalStatisticsImpl.serialize(ColumnStatisticsImpl.java:585)
at 
org.apache.hadoop.hive.ql.io.orc.WriterImpl$TreeWriter.createRowIndexEntry(WriterImpl.java:769)
at 
org.apache.hadoop.hive.ql.io.orc.WriterImpl$TreeWriter.createRowIndexEntry(WriterImpl.java:775)
at 
org.apache.hadoop.hive.ql.io.orc.WriterImpl.createRowIndexEntry(WriterImpl.java:1978)
at 
org.apache.hadoop.hive.ql.io.orc.WriterImpl.flushStripe(WriterImpl.java:1985)
at 
org.apache.hadoop.hive.ql.io.orc.WriterImpl.checkMemory(WriterImpl.java:322)
at 
org.apache.hadoop.hive.ql.io.orc.MemoryManager.notifyWriters(MemoryManager.java:168)
at 
org.apache.hadoop.hive.ql.io.orc.MemoryManager.addedRow(MemoryManager.java:157)
at org.apache.hadoop.hive.ql.io.orc.WriterImpl.addRow(WriterImpl.java:2276)
at 
org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.write(OrcOutputFormat.java:76)
at 
org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.write(OrcOutputFormat.java:55)
at 
org.apache.hive.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:122)
at 
org.apache.hive.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:54)
at 
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:655)
at 
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at 
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at 
org.apache.sqoop.mapreduce.hcat.SqoopHCatImportMapper.map(SqoopHCatImportMapper.java:52)
at 
org.apache.sqoop.mapreduce.hcat.SqoopHCatImportMapper.map(SqoopHCatImportMapper.java:34)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

The problem is caused by null DECIMAL value in the imported table. The creation 
of ORC table statistics crashes because of this. The issue probably affects 
some other column types as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: hiveserver2 hangs

[Sanjeev Verma]

Thanks sanjeev for your help.

BTW I try to increase the Heap Size of HS2 but seeing the same
exception.from where this exception has originated, it looks like
originated from the thrift client.any idea what operation it is doing with
the given stack.

Local Variable: org.apache.thrift.TByteArrayOutputStream#42
Local Variable: byte[]#5378
at org.apache.thrift.transport.TSaslTransport.write(TSaslTransport.java:446)
at org.apache.thrift.transport.TSaslServerTransport.write(TSasl
ServerTransport.java:41)
at org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryP
rotocol.java:163)
at org.apache.thrift.protocol.TBinaryProtocol.writeString(TBina
ryProtocol.java:186)
Local Variable: byte[]#2
at org.apache.hive.service.cli.thrift.TStringColumn$TStringColu
mnStandardScheme.write(TStringColumn.java:490)
Local Variable: java.util.ArrayList$Itr#1
at org.apache.hive.service.cli.thrift.TStringColumn$TStringColu
mnStandardScheme.write(TStringColumn.java:433)
Local Variable: org.apache.hive.service.cli.thrift.TStringColumn$TStringColu
mnStandardScheme#1
at org.apache.hive.service.cli.thrift.TStringColumn.write(TStri
ngColumn.java:371)
at org.apache.hive.service.cli.thrift.TColumn.standardSchemeWri
teValue(TColumn.java:381)
Local Variable: org.apache.hive.service.cli.thrift.TColumn#504
Local Variable: org.apache.hive.service.cli.thrift.TStringColumn#453
at org.apache.thrift.TUnion$TUnionStandardScheme.write(TUnion.java:244)
at org.apache.thrift.TUnion$TUnionStandardScheme.write(TUnion.java:213)
at org.apache.thrift.TUnion.write(TUnion.java:152)

On Wed, Sep 9, 2015 at 8:19 AM, kulkarni.swar...@gmail.com <
kulkarni.swar...@gmail.com> wrote:

> Sanjeev,
>
> I am going off this exception in the stacktrace that you posted.
>
> "at java.lang.OutOfMemoryError.(OutOfMemoryError.java:48)"
>
> which def. indicates that it's not very happy memory wise. I would def.
> recommend to bump up the memory and see if it helps. If not, we can debug
> further from there.
>
> On Tue, Sep 8, 2015 at 12:17 PM, Sanjeev Verma 
> wrote:
>
>> What this exception implies here? how to identify the problem here.
>> Thanks
>>
>> On Tue, Sep 8, 2015 at 10:44 PM, Sanjeev Verma > > wrote:
>>
>>> We have 8GB HS2 java heap, we have not tried any bumping.
>>>
>>> On Tue, Sep 8, 2015 at 8:14 PM, kulkarni.swar...@gmail.com <
>>> kulkarni.swar...@gmail.com> wrote:
>>>
 How much memory have you currently provided to HS2? Have you tried
 bumping that up?

 On Mon, Sep 7, 2015 at 1:09 AM, Sanjeev Verma <
 sanjeev.verm...@gmail.com> wrote:

> *I am getting the following exception when the HS2 is crashing, any
> idea why it has happening*
>
> "pool-1-thread-121" prio=4 tid=19283 RUNNABLE
> at java.lang.OutOfMemoryError.(OutOfMemoryError.java:48)
> at java.util.Arrays.copyOf(Arrays.java:2271)
> Local Variable: byte[]#1
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
> at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutput
> Stream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
> Local Variable: org.apache.thrift.TByteArrayOutputStream#42
> Local Variable: byte[]#5378
> at org.apache.thrift.transport.TSaslTransport.write(TSaslTransp
> ort.java:446)
> at org.apache.thrift.transport.TSaslServerTransport.write(TSasl
> ServerTransport.java:41)
> at org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryP
> rotocol.java:163)
> at org.apache.thrift.protocol.TBinaryProtocol.writeString(TBina
> ryProtocol.java:186)
> Local Variable: byte[]#2
> at org.apache.hive.service.cli.thrift.TStringColumn$TStringColu
> mnStandardScheme.write(TStringColumn.java:490)
> Local Variable: java.util.ArrayList$Itr#1
> at org.apache.hive.service.cli.thrift.TStringColumn$TStringColu
> mnStandardScheme.write(TStringColumn.java:433)
> Local Variable: org.apache.hive.service.cli.th
> rift.TStringColumn$TStringColumnStandardScheme#1
> at org.apache.hive.service.cli.thrift.TStringColumn.write(TStri
> ngColumn.java:371)
> at org.apache.hive.service.cli.thrift.TColumn.standardSchemeWri
> teValue(TColumn.java:381)
> Local Variable: org.apache.hive.service.cli.thrift.TColumn#504
> Local Variable: org.apache.hive.service.cli.thrift.TStringColumn#453
> at org.apache.thrift.TUnion$TUnionStandardScheme.write(TUnion.j
> ava:244)
> at org.apache.thrift.TUnion$TUnionStandardScheme.write(TUnion.j
> ava:213)
> at org.apache.thrift.TUnion.write(TUnion.java:152)
>
>
>
> On Fri, Aug 21, 2015 at 6:16 AM, kulkarni.swar...@gmail.com <
> kulkarni.swar...@gmail.com> wrote:
>
>> Sanjeev,
>>
>> One possibility is that you are running into[1] which affects hive
>> 0.13. Is it possible for you to apply the patch on [1] and see if it 
>> fixes
>> your problem?
>>
>> [1] https://issues.apache.org/jira/browse/HIVE-1041

[jira] [Created] (HIVE-11781) Remove HiveLimit operator

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-11781:
--

 Summary: Remove HiveLimit operator
 Key: HIVE-11781
 URL: https://issues.apache.org/jira/browse/HIVE-11781
 Project: Hive
  Issue Type: Bug
  Components: CBO
Affects Versions: 2.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


Calcite's Sort contains both: Sort and Limit operator. We should extend that 
one. Further, we should get rid of HiveLimit operator which is never used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11780) Hive support "set role none"

[Dapeng Sun (JIRA)]

Dapeng Sun created HIVE-11780:
-

 Summary: Hive support "set role none"
 Key: HIVE-11780
 URL: https://issues.apache.org/jira/browse/HIVE-11780
 Project: Hive
  Issue Type: Improvement
Reporter: Dapeng Sun
Assignee: Dapeng Sun






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38211: HIVE-11684

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/
---

(Updated Sept. 10, 2015, 8:23 a.m.)


Review request for hive and John Pullokkaran.


Bugs: HIVE-11684
https://issues.apache.org/jira/browse/HIVE-11684


Repository: hive-git


Description
---

Implement limit pushdown through outer join in CBO


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
d2c5885b4808335cdddc826137d95a8a065dabf7 
  itests/src/test/resources/testconfiguration.properties 
bed621d3eb74f01e54110552f68538afd228018d 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
73ae7c4ef90af22847bef9e54863d4651d9c1013 
  ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION 
  ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION 
  ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out 
PRE-CREATION 

Diff: https://reviews.apache.org/r/38211/diff/


Testing
---

Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only 
Tez)


Thanks,

Jesús Camacho Rodríguez

Review Request 38250: HIVE-11779-support hive.cli.pretty.output.num.cols in new CLI

[Ke Jia]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38250/
---

Review request for hive and cheng xu.


Repository: hive-git


Description
---

support hive.cli.pretty.output.num.cols in new CLI


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataPrettyFormatUtils.java
 8f939e6 
  ql/src/test/queries/clientnegative/describe_pretty.q PRE-CREATION 
  ql/src/test/results/clientpositive/describe_pretty.q.out 1c05e0d 

Diff: https://reviews.apache.org/r/38250/diff/


Testing
---

New unit tests add
Existing unit tests pass (will be running them again)
Manual testing


Thanks,

Ke Jia