[jira] [Updated] (HIVE-4518) Counter Strike: Operation Operator

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-4518:
-

Release Note: Added config setting hive.counters.group.name: counter group 
name for counters used during query execution.

> Counter Strike: Operation Operator
> --
>
> Key: HIVE-4518
> URL: https://issues.apache.org/jira/browse/HIVE-4518
> Project: Hive
>  Issue Type: Improvement
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Fix For: 0.13.0
>
> Attachments: HIVE-4518.1.patch, HIVE-4518.10.patch, 
> HIVE-4518.11.patch, HIVE-4518.2.patch, HIVE-4518.3.patch, HIVE-4518.4.patch, 
> HIVE-4518.5.patch, HIVE-4518.6.patch.txt, HIVE-4518.7.patch, 
> HIVE-4518.8.patch, HIVE-4518.9.patch
>
>
> Queries of the form:
> from foo
> insert overwrite table bar partition (p) select ...
> insert overwrite table bar partition (p) select ...
> insert overwrite table bar partition (p) select ...
> Generate a huge amount of counters. The reason is that task.progress is 
> turned on for dynamic partitioning queries.
> The counters not only make queries slower than necessary (up to 50%) you will 
> also eventually run out. That's because we're wrapping them in enum values to 
> comply with hadoop 0.17.
> The real reason we turn task.progress on is that we need CREATED_FILES and 
> FATAL counters to ensure dynamic partitioning queries don't go haywire.
> The counters have counter-intuitive names like C1 through C1000 and don't 
> seem really useful by themselves.
> With hadoop 20+ you don't need to wrap the counters anymore, each operator 
> can simply create and increment counters. That should simplify the code a lot.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6021) Problem in GroupByOperator for handling distinct aggrgations

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13850139#comment-13850139
 ] 

Hive QA commented on HIVE-6021:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12619014/HIVE-6021.2.patch

{color:green}SUCCESS:{color} +1 4790 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/666/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/666/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12619014

> Problem in GroupByOperator for handling distinct aggrgations
> 
>
> Key: HIVE-6021
> URL: https://issues.apache.org/jira/browse/HIVE-6021
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.12.0
>Reporter: Sun Rui
>Assignee: Sun Rui
> Attachments: HIVE-6021.1.patch, HIVE-6021.2.patch
>
>
> Use the following test case with HIVE 0.12:
> {code:sql}
> create table src(key int, value string);
> load data local inpath 'src/data/files/kv1.txt' overwrite into table src;
> set hive.map.aggr=false; 
> select count(key),count(distinct value) from src group by key;
> {code}
> We will get an ArrayIndexOutOfBoundsException from GroupByOperator:
> {code}
> java.lang.RuntimeException: Error in configuring object
>   at 
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>   at 
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>   at 
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>   at 
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:485)
>   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
>   at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:260)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>   ... 5 more
> Caused by: java.lang.RuntimeException: Reduce operator initialization failed
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:159)
>   ... 10 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.initializeOp(GroupByOperator.java:281)
>   at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:377)
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:152)
>   ... 10 more
> {code}
> explain select count(key),count(distinct value) from src group by key;
> {code}
> STAGE PLANS:
>   Stage: Stage-1
> Map Reduce
>   Alias -> Map Operator Tree:
> src 
>   TableScan
> alias: src
> Select Operator
>   expressions:
> expr: key
> type: int
> expr: value
> type: string
>   outputColumnNames: key, value
>   Reduce Output Operator
> key expressions:
>   expr: key
>   type: int
>   expr: value
>   type: string
> sort order: ++
> Map-reduce partition columns:
>   expr: key
>   type: int
> tag: -1
>   Reduce Operator Tree:
> Group By Operator
>   aggregations:
> expr: count(KEY._col0)   // The parameter causes this problem
>^^^
> expr: count(DISTINCT KEY._col1:0._col0)
>   bucketGroup: false
>   keys:
> expr: KEY._col0
> type: int
>   mode: complete
>   outputColumnNames: _col0, _col1, _col2
>   Select Operator
> expressions:
>   expr: _col1
>   type: bigint
>   expr: _col2
>   type: bigint
> outputColumnNames: _col0, _col1
> File Output Operator
>   compressed: false
>   

Re: Review Request 16184: Hive should be able to skip header and footer rows when reading data file for a table (HIVE-5795)

[Thejas Nair]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16184/#review30516
---



ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java


can you add a javadoc comment describing the what the return value is 



ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java


can you add a javadoc comment describing the what the return value is 



ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java


This variable name was used before as well. But can you rename this 
variable to something more descriptive like opNotEOF, while you are making 
these changes? 



ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java


please use braces for the if condition - "if (ret) {.."



ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java


duplicate if block. 



ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java


In your comment in previous reviewboard link, you say that ObjectPair can't 
be used because "I need deep copy of the key and value field through 
ReflectionUtils".
But the use of ReflectionUtils is not happening from within KVPair. I 
haven't understood why ObjectPair can't be used instead.




ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java


can you add this to the comment (unfortunately ReflectionsUtils.copy is 
poorly/incorrectly documented, created HADOOP-10168) 
// copy value from footerBuf to key,value 




ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java


Please don't use same variable name as member variable, it gets very 
confusing! :)
maybe, call it inputEOF ?

pls add comment - 
//read new value into the buffer




ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java


looks like this else statement is not needed. the copy has already been 
done.
Comments applicable to the FetchOperator also



ql/src/test/queries/clientnegative/file_with_header_footer_negative.q


can you also add a test case with a table with empty file ?


- Thejas Nair


On Dec. 11, 2013, 9:19 p.m., Shuaishuai Nie wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16184/
> ---
> 
> (Updated Dec. 11, 2013, 9:19 p.m.)
> 
> 
> Review request for hive, Eric Hanson and Thejas Nair.
> 
> 
> Bugs: hive-5795
> https://issues.apache.org/jira/browse/hive-5795
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Hive should be able to skip header and footer rows when reading data file for 
> a table
> (follow up with review https://reviews.apache.org/r/15663/diff/#index_header)
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fa3e048 
>   conf/hive-default.xml.template c61a0bb 
>   data/files/header_footer_table_1/0001.txt PRE-CREATION 
>   data/files/header_footer_table_1/0002.txt PRE-CREATION 
>   data/files/header_footer_table_1/0003.txt PRE-CREATION 
>   data/files/header_footer_table_2/2012/01/01/0001.txt PRE-CREATION 
>   data/files/header_footer_table_2/2012/01/02/0002.txt PRE-CREATION 
>   data/files/header_footer_table_2/2012/01/03/0003.txt PRE-CREATION 
>   itests/qtest/pom.xml c3cbb89 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java d2b2526 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java 
> dd5cb6b 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 974a5d6 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/TestHiveBinarySearchRecordReader.java
>  85dd975 
>   ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java 
> 0686d9b 
>   ql/src/test/queries/clientnegative/file_with_header_footer_negative.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/file_with_header_footer.q PRE-CREATION 
>   ql/src/test/results/clientnegative/file_with_header_footer_negative.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/file_with_header_footer.q.out 
> PRE-CREATION 
>   serde/if/serde.thrift 2ceb572 
>   
> serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/serdeConstants.java
>  22a6168 
> 
> Diff: https://reviews.apache.org/r/16184/diff/
> 
> 
> Testing
> 

Re: Review Request 15151: Better error reporting by async threads in HiveServer2

[Vaibhav Gumashta]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15151/
---

(Updated Dec. 17, 2013, 5:34 a.m.)


Review request for hive, Carl Steinbach, Prasad Mujumdar, and Thejas Nair.


Changes
---

Rebased on trunk with thrift files.


Bugs: HIVE-5230
https://issues.apache.org/jira/browse/HIVE-5230


Repository: hive-git


Description
---

[HIVE-4617|https://issues.apache.org/jira/browse/HIVE-4617] provides support 
for async execution in HS2. When a background thread gets an error, currently 
the client can only poll for the operation state and also the error with its 
stacktrace is logged. However, it will be useful to provide a richer error 
response like thrift API does with TStatus (which is constructed while building 
a Thrift response object). 


Diffs (updated)
-

  jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java e420b75 
  jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 37975e5 
  service/if/TCLIService.thrift 62a9730 
  service/src/gen/thrift/gen-cpp/TCLIService_types.h 853bb4c 
  service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 7ab1310 
  
service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/ThriftHive.java
 1c44789 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TColumn.java
 497cc01 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetOperationStatusResp.java
 b5c3f01 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTablesReq.java
 1cb5147 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionReq.java
 8ab8297 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionResp.java
 688f790 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java
 8c6c4f0 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRow.java
 0b6772c 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRowSet.java
 db2262d 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TStatus.java
 81c2f16 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TTableSchema.java
 ff5e54d 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TTypeDesc.java
 251f86a 
  service/src/gen/thrift/gen-py/TCLIService/ttypes.py 185ea5b 
  service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb c94acbf 
  service/src/java/org/apache/hive/service/cli/CLIService.java 8c85386 
  service/src/java/org/apache/hive/service/cli/CLIServiceClient.java 14ef54f 
  service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 
9dca874 
  service/src/java/org/apache/hive/service/cli/ICLIService.java f647ce6 
  service/src/java/org/apache/hive/service/cli/OperationState.java 1ec6bd1 
  service/src/java/org/apache/hive/service/cli/OperationStatus.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java 5d1dd5f 
  service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
bcdb67f 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
296f8b3 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
c0e6151 
  service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
e262b72 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
9df110e 
  
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java 
9bb2a0f 
  service/src/test/org/apache/hive/service/cli/CLIServiceTest.java 44d3130 
  service/src/test/org/apache/hive/service/cli/thrift/ThriftCLIServiceTest.java 
ff7166d 

Diff: https://reviews.apache.org/r/15151/diff/


Testing
---


Thanks,

Vaibhav Gumashta

[jira] [Updated] (HIVE-1466) Add NULL DEFINED AS to ROW FORMAT specification

[Prasad Mujumdar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad Mujumdar updated HIVE-1466:
--

  Component/s: SQL
Fix Version/s: 0.13.0
 Release Note: This features enables defining a custom null format for a 
table via 'create table' statement. A custom null format can also be specified 
while exporting data to local filesystem using 'insert overwrite .. local dir' 
statement.

> Add NULL DEFINED AS to ROW FORMAT specification
> ---
>
> Key: HIVE-1466
> URL: https://issues.apache.org/jira/browse/HIVE-1466
> Project: Hive
>  Issue Type: New Feature
>  Components: SQL
>Reporter: Adam Kramer
>Assignee: Prasad Mujumdar
> Fix For: 0.13.0
>
> Attachments: HIVE-1466.1.patch, HIVE-1466.2.patch
>
>
> NULL values are passed to transformers as a literal backslash and a literal 
> N. NULL values are saved when INSERT OVERWRITing LOCAL DIRECTORies as "NULL". 
> This is inconsistent.
> The ROW FORMAT specification of tables should be able to specify the manner 
> in which a null character is represented. ROW FORMAT NULL DEFINED AS '\N' or 
> '\003' or whatever should apply to all instances of table export and saving.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-1466) Add NULL DEFINED AS to ROW FORMAT specification

[Prasad Mujumdar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13850110#comment-13850110
 ] 

Prasad Mujumdar commented on HIVE-1466:
---

[~thejas] Thanks for pointing that out. I just updated the wiki to reflect the 
syntax changes. will add a rel note on the ticket.

I guess it's a good idea to have a doc jira along with the a patch that 
introduces a user facing change (SQL syntax, script/tools etc). The reviewers 
should also verify that before approving the patch.

> Add NULL DEFINED AS to ROW FORMAT specification
> ---
>
> Key: HIVE-1466
> URL: https://issues.apache.org/jira/browse/HIVE-1466
> Project: Hive
>  Issue Type: New Feature
>Reporter: Adam Kramer
>Assignee: Prasad Mujumdar
> Attachments: HIVE-1466.1.patch, HIVE-1466.2.patch
>
>
> NULL values are passed to transformers as a literal backslash and a literal 
> N. NULL values are saved when INSERT OVERWRITing LOCAL DIRECTORies as "NULL". 
> This is inconsistent.
> The ROW FORMAT specification of tables should be able to specify the manner 
> in which a null character is represented. ROW FORMAT NULL DEFINED AS '\N' or 
> '\003' or whatever should apply to all instances of table export and saving.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6040) to_utc_timestamp() not intuitive when cluster timezone is not set to UTC

[Ryan Harris (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryan Harris updated HIVE-6040:
--

Description: 
unix_timestamp() is the root of string to timestamp conversion operations.

Therefore, when local cluster timezone is NOT set to UTC the results produced 
by to_utc_timestamp() may be confusing to the user.

The query below was run on a cluster with the local timezone set to 
'America/Denver'
For reference, 138600 =  2013-12-02 16:00:00 GMT

SELECT
from_unixtime(138600) as test01,
  -- test01 =  2013-12-02 09:00:00 
  -- local timezone applies to from_unixtime()
unix_timestamp(from_unixtime(138600)) as test02, 
  -- test02 = 138600
  -- local timezone applies to both unix_timestamp and from_unixtime()
unix_timestamp('2013-12-02 16:00:00') as test03,
  -- test03 =  1386025200
  -- local timezone applies to from_unixtime()
from_utc_timestamp(138600, 'UTC') as test04,
  -- test04 = 1970-01-16 18:00:00
  -- This demonstrates the bug from HIVE-3454
cast(from_utc_timestamp(cast(138600 as DOUBLE),'UTC') as BIGINT) as test05, 
  -- test05 = 138600
  -- one work-around to HIVE-3454
cast(from_utc_timestamp(138600 * 1.0,'UTC') as BIGINT) as test06, 
  -- test06 = 138600
  -- a second work-around to HIVE-3454
cast(from_utc_timestamp(cast(cast(138600 as DOUBLE) as TIMESTAMP),'UTC') as 
BIGINT) as test07,
  --test07 = 138600
  --direct cast conversion of DOUBLE to TIMESTAMP works
cast(cast('2013-12-02 16:00:00' as TIMESTAMP) as BIGINT) as test08,
  --test08 = 1386025200
  -- same result as test03, cast conversion uses local timezone setting
cast(from_utc_timestamp(cast('2013-12-02 16:00:00' as TIMESTAMP),'UTC') as 
BIGINT) as test09,
  --test09 = 1386025200
  --same as test08 demonstrating consistency even when using 
from_utc_timestamp()
cast(from_utc_timestamp(cast('2013-12-02 16:00:00' as 
TIMESTAMP),'America/Denver') as BIGINT) as test10,
  --test10 = 138600
  --not exactly intuitive, but this is the correct result
cast(from_utc_timestamp(unix_timestamp('2013-12-02 16:00:00'),'America/Denver') 
as BIGINT) as test11,
  --test11= 1360825
  --similar to test10, but producing HIVE-3454 bug
cast(from_utc_timestamp(unix_timestamp('2013-12-02 16:00:00') * 
1.0,'America/Denver') as BIGINT) as test12,
  --test12=138600
  --like test10/test11 with HIVE-3454 work-around.
cast(to_utc_timestamp('2013-12-02 16:00:00','UTC') as BIGINT) as test13,
  --test13=1386025200
  --intuitively this seems to be the correct approach
  --the result is wrong.
cast(cast('2013-12-02 16:00:00' as TIMESTAMP) as BIGINT) as test14,
  --test14=1386025200
  --this result could be confusing to the user
cast( from_utc_timestamp( cast( '2013-12-02 16:00:00' as TIMESTAMP ), 
'\${system:user.timezone}' ) as BIGINT ) as test15,
  --test15=138600
  --like test10, using hiveconf system variable substitution
cast( from_utc_timestamp( unix_timestamp( '2013-12-02 16:00:00') * 1.0, 
'\${system:user.timezone}' ) as BIGINT) as test16,
  --test16=138600
  --like test12, using hiveconf system variable substitution
cast( to_utc_timestamp( from_utc_timestamp( unix_timestamp( '2013-12-02 
16:00:00' ) * 1.0, '\${system:user.timezone}' ), 'UTC') as BIGINT) as test17
  --test17=138600
  --universal work-around, produces results expected by test13
FROM dummy

To work with datetime string conversions using to_utc_timestamp in a way that 
produces consistent results, regardless of cluster timezone configuration, the 
following work-around is required:

to_utc_timestamp(from_utc_timestamp(unix_timestamp('2013-12-02 
16:00:00')*1.0,'\${system:user.timezone}'), '')

One solution could be to add a utc_timestamp() udf to hive and for 
to_utc_timestamp() to use that for implicit datetime string conversions.


  was:
unix_timestamp() is the root of string to timestamp conversion operations.

Therefore, when local cluster timezone is NOT set to UTC the results produced 
by to_utc_timestamp() may be confusing to the user.

The query below was run on a cluster with the local timezone set to 
'America/Denver'
For reference, 138600 =  2013-12-02 16:00:00 GMT

SELECT
from_unixtime(138600) as test01,
  -- test01 =  2013-12-02 09:00:00 
  -- local timezone applies to from_unixtime()
unix_timestamp(from_unixtime(138600)) as test02, 
  -- test02 = 138600
  -- local timezone applies to both unix_timestamp and from_unixtime()
unix_timestamp('2013-12-02 16:00:00') as test03,
  -- test03 =  1386025200
  -- local timezone applies to from_unixtime()
from_utc_timestamp(138600, 'UTC') as test04,
  -- test04 = 1970-01-16 18:00:00
  -- This demonstrates the bug from HIVE-3454
cast(from_utc_timestamp(cast(138600 as DOUBLE),'UTC') as BIGINT) as test05, 
  -- test05 = 138600
  -- one work-around to HIVE-3454
cast(from_utc_timestamp(138600 * 1.0,'UTC') as BIGINT) as test06, 
  -- test06 = 138600
  -- a se

[jira] [Created] (HIVE-6040) to_utc_timestamp() not intuitive when cluster timezone is not set to UTC

[Ryan Harris (JIRA)]

Ryan Harris created HIVE-6040:
-

 Summary: to_utc_timestamp() not intuitive when cluster timezone is 
not set to UTC
 Key: HIVE-6040
 URL: https://issues.apache.org/jira/browse/HIVE-6040
 Project: Hive
  Issue Type: Bug
  Components: Types, UDF
Affects Versions: 0.12.0, 0.11.0, 0.10.0, 0.9.0, 0.8.1, 0.8.0, 0.13.0, 
0.12.1
Reporter: Ryan Harris
Priority: Minor


unix_timestamp() is the root of string to timestamp conversion operations.

Therefore, when local cluster timezone is NOT set to UTC the results produced 
by to_utc_timestamp() may be confusing to the user.

The query below was run on a cluster with the local timezone set to 
'America/Denver'
For reference, 138600 =  2013-12-02 16:00:00 GMT

SELECT
from_unixtime(138600) as test01,
  -- test01 =  2013-12-02 09:00:00 
  -- local timezone applies to from_unixtime()
unix_timestamp(from_unixtime(138600)) as test02, 
  -- test02 = 138600
  -- local timezone applies to both unix_timestamp and from_unixtime()
unix_timestamp('2013-12-02 16:00:00') as test03,
  -- test03 =  1386025200
  -- local timezone applies to from_unixtime()
from_utc_timestamp(138600, 'UTC') as test04,
  -- test04 = 1970-01-16 18:00:00
  -- This demonstrates the bug from HIVE-3454
cast(from_utc_timestamp(cast(138600 as DOUBLE),'UTC') as BIGINT) as test05, 
  -- test05 = 138600
  -- one work-around to HIVE-3454
cast(from_utc_timestamp(138600 * 1.0,'UTC') as BIGINT) as test06, 
  -- test06 = 138600
  -- a second work-around to HIVE-3454
cast(from_utc_timestamp(cast(cast(138600 as DOUBLE) as TIMESTAMP),'UTC') as 
BIGINT) as test07,
  --test07 = 138600
  --direct cast conversion of DOUBLE to TIMESTAMP works
cast(cast('2013-12-02 16:00:00' as TIMESTAMP) as BIGINT) as test08,
  --test08 = 1386025200
  -- same result as test03, cast conversion uses local timezone setting
cast(from_utc_timestamp(cast('2013-12-02 16:00:00' as TIMESTAMP),'UTC') as 
BIGINT) as test09,
  --test09 = 1386025200
  --same as test08 demonstrating consistency even when using 
from_utc_timestamp()
cast(from_utc_timestamp(cast('2013-12-02 16:00:00' as 
TIMESTAMP),'America/Denver') as BIGINT) as test10,
  --test10 = 138600
  --not exactly intuitive, but this is the correct result
cast(from_utc_timestamp(unix_timestamp('2013-12-02 16:00:00'),'America/Denver') 
as BIGINT) as test11,
  --test11= 1360825
  --similar to test10, but producing HIVE-3454 bug
cast(from_utc_timestamp(unix_timestamp('2013-12-02 16:00:00') * 
1.0,'America/Denver') as BIGINT) as test12,
  --test12=138600
  --like test10/test11 with HIVE-3454 work-around.
cast(to_utc_timestamp('2013-12-02 16:00:00','UTC') as BIGINT) as test13,
  --test13=1386025200
  --intuitively this seems to be the correct approach
  --the result is wrong.
cast(cast('2013-12-02 16:00:00' as TIMESTAMP) as BIGINT) as test14,
  --test14=1386025200
  --this result could be confusing to the user
cast(from_utc_timestamp(cast('2013-12-02 16:00:00' as 
TIMESTAMP),'\${system:user.timezone}') as BIGINT) as test15,
  --test15=138600
  --like test10, using hiveconf system variable substitution
cast(from_utc_timestamp(unix_timestamp('2013-12-02 
16:00:00')*1.0,'\${system:user.timezone}') as BIGINT) as test16,
  --test16=138600
  --like test12, using hiveconf system variable substitution
cast(to_utc_timestamp(from_utc_timestamp(unix_timestamp('2013-12-02 
16:00:00')*1.0,'\${system:user.timezone}'), 'UTC') as BIGINT) as test17
  --test17=138600
  --universal work-around, produces results expected by test13
FROM dummy

To work with datetime string conversions using to_utc_timestamp in a way that 
produces consistent results, regardless of cluster timezone configuration, the 
following work-around is required:

to_utc_timestamp(from_utc_timestamp(unix_timestamp('2013-12-02 
16:00:00')*1.0,'\${system:user.timezone}'), '')

One solution could be to add a utc_timestamp() udf to hive and for 
to_utc_timestamp() to use that for implicit datetime string conversions.




--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5829) Rewrite Trim and Pad UDFs based on GenericUDF

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13850090#comment-13850090
 ] 

Hive QA commented on HIVE-5829:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12619002/HIVE-5829.2.patch

{color:green}SUCCESS:{color} +1 4794 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/663/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/663/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12619002

> Rewrite Trim and Pad UDFs based on GenericUDF
> -
>
> Key: HIVE-5829
> URL: https://issues.apache.org/jira/browse/HIVE-5829
> Project: Hive
>  Issue Type: Bug
>Reporter: Mohammad Kamrul Islam
>Assignee: Mohammad Kamrul Islam
> Attachments: HIVE-5829.1.patch, HIVE-5829.2.patch, tmp.HIVE-5829.patch
>
>
> This JIRA includes following UDFs:
> 1. trim()
> 2. ltrim()
> 3. rtrim()
> 4. lpad()
> 5. rpad()



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-1466) Add NULL DEFINED AS to ROW FORMAT specification

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13850063#comment-13850063
 ] 

Thejas M Nair commented on HIVE-1466:
-

Prasad, can you please add a release note to the jira and create a followup 
jira for inclusion in wiki (Lefty or someone else might be able to help with 
incorporation in wiki) or update the wiki page itself directly ?
We should try to ensure that all new features get documented. I think the best 
way to do that is to ensure that documentation is available before feature is 
committed.


> Add NULL DEFINED AS to ROW FORMAT specification
> ---
>
> Key: HIVE-1466
> URL: https://issues.apache.org/jira/browse/HIVE-1466
> Project: Hive
>  Issue Type: New Feature
>Reporter: Adam Kramer
>Assignee: Prasad Mujumdar
> Attachments: HIVE-1466.1.patch, HIVE-1466.2.patch
>
>
> NULL values are passed to transformers as a literal backslash and a literal 
> N. NULL values are saved when INSERT OVERWRITing LOCAL DIRECTORies as "NULL". 
> This is inconsistent.
> The ROW FORMAT specification of tables should be able to specify the manner 
> in which a null character is represented. ROW FORMAT NULL DEFINED AS '\N' or 
> '\003' or whatever should apply to all instances of table export and saving.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Hive-trunk-h0.21 - Build # 2507 - Still Failing

[Apache Jenkins Server]

Changes for Build #2472
[navis] HIVE-4518 : Should be removed files (OptrStatsGroupByHook, etc.)

[navis] HIVE-5839 : BytesRefArrayWritable compareTo violates contract (Xuefu 
Zhang via Navis)

[navis] HIVE-4518 : Missing file (HiveFatalException)

[navis] HIVE-4518 : Counter Strike: Operation Operator (Gunther Hagleitner and 
Jason Dere via Navis)


Changes for Build #2473
[brock] HIVE-4741 - Add Hive config API to modify the restrict list (Prasad 
Mujumdar, Navis via Brock Noland)


Changes for Build #2474
[navis] HIVE-5827 : Incorrect location of logs for failed tests (Vikram Dixit K 
and Szehon Ho via Navis)

[thejas] HIVE-4485 : beeline prints null as empty strings (Thejas Nair reviewed 
by Ashutosh Chauhan)

[brock] HIVE-5704 - A couple of generic UDFs are not in the right 
folder/package (Xuefu Zhang via Brock Noland)

[brock] HIVE-5706 - Move a few numeric UDFs to generic implementations (Xuefu 
Zhang via Brock Noland)

[hashutosh] HIVE-5817 : column name to index mapping in VectorizationContext is 
broken (Remus Rusanu, Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5876 : Split elimination in ORC breaks for partitioned tables 
(Prasanth J via Ashutosh Chauhan)

[hashutosh] HIVE-5886 : [Refactor] Remove unused class JobCloseFeedback 
(Ashutosh Chauhan via Thejas Nair)

[brock] HIVE-5894 - Fix minor PTest2 issues (Brock Noland)


Changes for Build #2475
[brock] HIVE-5755 - Fix hadoop2 execution environment Milestone 1 (Vikram Dixit 
K via Brock Noland)


Changes for Build #2476
[xuefu] HIVE-5893: hive-schema-0.13.0.mysql.sql contains reference to 
nonexistent column (Carl via Xuefu)

[xuefu] HIVE-5684: Serde support for char (Jason via Xuefu)


Changes for Build #2477

Changes for Build #2478

Changes for Build #2479

Changes for Build #2480
[brock] HIVE-5441 - Async query execution doesn't return resultset status 
(Prasad Mujumdar via Thejas M Nair)

[brock] HIVE-5880 - Rename HCatalog HBase Storage Handler artifact id (Brock 
Noland reviewed by Prasad Mujumdar)


Changes for Build #2481

Changes for Build #2482
[ehans] HIVE-5581: Implement vectorized year/month/day... etc. for string 
arguments (Teddy Choi via Eric Hanson)


Changes for Build #2483
[rhbutani] HIVE-5898 Make fetching of column statistics configurable (Prasanth 
Jayachandran via Harish Butani)


Changes for Build #2484
[brock] HIVE-5880 - (Rename HCatalog HBase Storage Handler artifact id) breaks 
packaging (Xuefu Zhang via Brock Noland)


Changes for Build #2485
[xuefu] HIVE-5866: Hive divide operator generates wrong results in certain 
cases (reviewed by Prasad)

[ehans] HIVE-5877: Implement vectorized support for IN as boolean-valued 
expression (Eric Hanson)


Changes for Build #2486
[ehans] HIVE-5895: vectorization handles division by zero differently from 
normal execution (Sergey Shelukhin via Eric Hanson)

[hashutosh] HIVE-5938 : Remove apache.mina dependency for test (Navis via 
Ashutosh Chauhan)

[xuefu] HIVE-5912: Show partition command doesn't support db.table (Yu Zhao via 
Xuefu)

[brock] HIVE-5906 - TestGenericUDFPower should use delta to compare doubles 
(Szehon Ho via Brock Noland)

[brock] HIVE-5855 - Add deprecated methods back to ColumnProjectionUtils (Brock 
Noland reviewed by Navis)

[brock] HIVE-5915 - Shade Kryo dependency (Brock Noland reviewed by Ashutosh 
Chauhan)


Changes for Build #2487
[hashutosh] HIVE-5916 : No need to aggregate statistics collected via counter 
mechanism (Ashutosh Chauhan via Navis)

[xuefu] HIVE-5947: Fix test failure in decimal_udf.q (reviewed by Brock)

[thejas] HIVE-5550 : Import fails for tables created with default text, 
sequence and orc file formats using HCatalog API (Sushanth Sowmyan via Thejas 
Nair)


Changes for Build #2488
[hashutosh] HIVE-5935 : hive.query.string is not provided to FetchTask (Navis 
via Ashutosh Chauhan)

[navis] HIVE-3455 : ANSI CORR(X,Y) is incorrect (Maxim Bolotin via Navis)

[hashutosh] HIVE-5921 : Better heuristics for worst case statistics estimates 
for join, limit and filter operator (Prasanth J via Harish Butani)

[rhbutani] HIVE-5899 NPE during explain extended with char/varchar columns 
(Jason Dere via Harish Butani)


Changes for Build #2489
[xuefu] HIVE-3181: getDatabaseMajor/Minor version does not return values 
(Szehon via Xuefu, reviewed by Navis)

[brock] HIVE-5641 - BeeLineOpts ignores Throwable (Brock Noland reviewed by 
Prasad and Thejas)

[hashutosh] HIVE-5909 : locate and instr throw 
java.nio.BufferUnderflowException when empty string as substring (Navis via 
Ashutosh Chauhan)

[hashutosh] HIVE-5686 : partition column type validation doesn't quite work for 
dates (Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5887 : metastore direct sql doesn't work with oracle (Sergey 
Shelukhin via Ashutosh Chauhan)


Changes for Build #2490

Changes for Build #2491

Changes for Build #2492
[brock] HIVE-5981 - Add hive-unit back to itests pom (Brock Noland reviewed by 
Prasad)


Changes for Build #2493
[xuefu] HIVE-5872: Ma

[jira] [Updated] (HIVE-1466) Add NULL DEFINED AS to ROW FORMAT specification

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-1466:


Issue Type: New Feature  (was: Improvement)

> Add NULL DEFINED AS to ROW FORMAT specification
> ---
>
> Key: HIVE-1466
> URL: https://issues.apache.org/jira/browse/HIVE-1466
> Project: Hive
>  Issue Type: New Feature
>Reporter: Adam Kramer
>Assignee: Prasad Mujumdar
> Attachments: HIVE-1466.1.patch, HIVE-1466.2.patch
>
>
> NULL values are passed to transformers as a literal backslash and a literal 
> N. NULL values are saved when INSERT OVERWRITing LOCAL DIRECTORies as "NULL". 
> This is inconsistent.
> The ROW FORMAT specification of tables should be able to specify the manner 
> in which a null character is represented. ROW FORMAT NULL DEFINED AS '\N' or 
> '\003' or whatever should apply to all instances of table export and saving.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Hive-trunk-hadoop2 - Build # 606 - Still Failing

[Apache Jenkins Server]

Changes for Build #571
[navis] HIVE-4518 : Should be removed files (OptrStatsGroupByHook, etc.)

[navis] HIVE-5839 : BytesRefArrayWritable compareTo violates contract (Xuefu 
Zhang via Navis)

[navis] HIVE-4518 : Missing file (HiveFatalException)

[navis] HIVE-4518 : Counter Strike: Operation Operator (Gunther Hagleitner and 
Jason Dere via Navis)


Changes for Build #572
[brock] HIVE-4741 - Add Hive config API to modify the restrict list (Prasad 
Mujumdar, Navis via Brock Noland)


Changes for Build #573
[navis] HIVE-5827 : Incorrect location of logs for failed tests (Vikram Dixit K 
and Szehon Ho via Navis)

[thejas] HIVE-4485 : beeline prints null as empty strings (Thejas Nair reviewed 
by Ashutosh Chauhan)

[brock] HIVE-5704 - A couple of generic UDFs are not in the right 
folder/package (Xuefu Zhang via Brock Noland)

[brock] HIVE-5706 - Move a few numeric UDFs to generic implementations (Xuefu 
Zhang via Brock Noland)

[hashutosh] HIVE-5817 : column name to index mapping in VectorizationContext is 
broken (Remus Rusanu, Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5876 : Split elimination in ORC breaks for partitioned tables 
(Prasanth J via Ashutosh Chauhan)

[hashutosh] HIVE-5886 : [Refactor] Remove unused class JobCloseFeedback 
(Ashutosh Chauhan via Thejas Nair)

[brock] HIVE-5894 - Fix minor PTest2 issues (Brock Noland)


Changes for Build #574
[brock] HIVE-5755 - Fix hadoop2 execution environment Milestone 1 (Vikram Dixit 
K via Brock Noland)


Changes for Build #575
[xuefu] HIVE-5893: hive-schema-0.13.0.mysql.sql contains reference to 
nonexistent column (Carl via Xuefu)

[xuefu] HIVE-5684: Serde support for char (Jason via Xuefu)


Changes for Build #576

Changes for Build #577

Changes for Build #578

Changes for Build #579
[brock] HIVE-5441 - Async query execution doesn't return resultset status 
(Prasad Mujumdar via Thejas M Nair)

[brock] HIVE-5880 - Rename HCatalog HBase Storage Handler artifact id (Brock 
Noland reviewed by Prasad Mujumdar)


Changes for Build #580
[ehans] HIVE-5581: Implement vectorized year/month/day... etc. for string 
arguments (Teddy Choi via Eric Hanson)


Changes for Build #581
[rhbutani] HIVE-5898 Make fetching of column statistics configurable (Prasanth 
Jayachandran via Harish Butani)


Changes for Build #582
[brock] HIVE-5880 - (Rename HCatalog HBase Storage Handler artifact id) breaks 
packaging (Xuefu Zhang via Brock Noland)


Changes for Build #583
[xuefu] HIVE-5866: Hive divide operator generates wrong results in certain 
cases (reviewed by Prasad)

[ehans] HIVE-5877: Implement vectorized support for IN as boolean-valued 
expression (Eric Hanson)


Changes for Build #584
[thejas] HIVE-5550 : Import fails for tables created with default text, 
sequence and orc file formats using HCatalog API (Sushanth Sowmyan via Thejas 
Nair)

[ehans] HIVE-5895: vectorization handles division by zero differently from 
normal execution (Sergey Shelukhin via Eric Hanson)

[hashutosh] HIVE-5938 : Remove apache.mina dependency for test (Navis via 
Ashutosh Chauhan)

[xuefu] HIVE-5912: Show partition command doesn't support db.table (Yu Zhao via 
Xuefu)

[brock] HIVE-5906 - TestGenericUDFPower should use delta to compare doubles 
(Szehon Ho via Brock Noland)

[brock] HIVE-5855 - Add deprecated methods back to ColumnProjectionUtils (Brock 
Noland reviewed by Navis)

[brock] HIVE-5915 - Shade Kryo dependency (Brock Noland reviewed by Ashutosh 
Chauhan)


Changes for Build #585
[hashutosh] HIVE-5916 : No need to aggregate statistics collected via counter 
mechanism (Ashutosh Chauhan via Navis)

[xuefu] HIVE-5947: Fix test failure in decimal_udf.q (reviewed by Brock)


Changes for Build #586
[hashutosh] HIVE-5935 : hive.query.string is not provided to FetchTask (Navis 
via Ashutosh Chauhan)

[navis] HIVE-3455 : ANSI CORR(X,Y) is incorrect (Maxim Bolotin via Navis)

[hashutosh] HIVE-5921 : Better heuristics for worst case statistics estimates 
for join, limit and filter operator (Prasanth J via Harish Butani)

[rhbutani] HIVE-5899 NPE during explain extended with char/varchar columns 
(Jason Dere via Harish Butani)


Changes for Build #587
[xuefu] HIVE-3181: getDatabaseMajor/Minor version does not return values 
(Szehon via Xuefu, reviewed by Navis)

[brock] HIVE-5641 - BeeLineOpts ignores Throwable (Brock Noland reviewed by 
Prasad and Thejas)

[hashutosh] HIVE-5909 : locate and instr throw 
java.nio.BufferUnderflowException when empty string as substring (Navis via 
Ashutosh Chauhan)

[hashutosh] HIVE-5686 : partition column type validation doesn't quite work for 
dates (Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5887 : metastore direct sql doesn't work with oracle (Sergey 
Shelukhin via Ashutosh Chauhan)


Changes for Build #588

Changes for Build #589

Changes for Build #590
[brock] HIVE-5981 - Add hive-unit back to itests pom (Brock Noland reviewed by 
Prasad)


Changes for Build #591
[xuefu] HIVE-5872: Make UDAFs such as GenericUDAFSum report accurat

[jira] [Commented] (HIVE-6017) Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13850047#comment-13850047
 ] 

Hive QA commented on HIVE-6017:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12619021/HIVE-6017.04.patch

{color:green}SUCCESS:{color} +1 4840 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/662/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/662/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12619021

> Contribute Decimal128 high-performance decimal(p, s) package from Microsoft 
> to Hive
> ---
>
> Key: HIVE-6017
> URL: https://issues.apache.org/jira/browse/HIVE-6017
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 0.13.0
>Reporter: Eric Hanson
>Assignee: Eric Hanson
> Attachments: HIVE-6017.01.patch, HIVE-6017.02.patch, 
> HIVE-6017.03.patch, HIVE-6017.04.patch
>
>
> Contribute the Decimal128 high-performance decimal package developed by 
> Microsoft to Hive. This was originally written for Microsoft PolyBase by 
> Hideaki Kimura.
> This code is about 8X more efficient than Java BigDecimal for typical 
> operations. It uses a finite (128 bit) precision and can handle up to 
> decimal(38, X). It is also "mutable" so you can change the contents of an 
> existing object. This helps reduce the cost of new() and garbage collection.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5230) Better error reporting by async threads in HiveServer2

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13850007#comment-13850007
 ] 

Hive QA commented on HIVE-5230:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12618991/HIVE-5230.10.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 4791 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/661/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/661/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12618991

> Better error reporting by async threads in HiveServer2
> --
>
> Key: HIVE-5230
> URL: https://issues.apache.org/jira/browse/HIVE-5230
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Fix For: 0.13.0
>
> Attachments: HIVE-5230.1.patch, HIVE-5230.1.patch, 
> HIVE-5230.10.patch, HIVE-5230.2.patch, HIVE-5230.3.patch, HIVE-5230.4.patch, 
> HIVE-5230.6.patch, HIVE-5230.7.patch, HIVE-5230.8.patch, HIVE-5230.9.patch
>
>
> [HIVE-4617|https://issues.apache.org/jira/browse/HIVE-4617] provides support 
> for async execution in HS2. When a background thread gets an error, currently 
> the client can only poll for the operation state and also the error with its 
> stacktrace is logged. However, it will be useful to provide a richer error 
> response like thrift API does with TStatus (which is constructed while 
> building a Thrift response object). 



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6017) Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive

[Eric Hanson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Hanson updated HIVE-6017:
--

Attachment: HIVE-6017.04.patch

remove trailing white space

> Contribute Decimal128 high-performance decimal(p, s) package from Microsoft 
> to Hive
> ---
>
> Key: HIVE-6017
> URL: https://issues.apache.org/jira/browse/HIVE-6017
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 0.13.0
>Reporter: Eric Hanson
>Assignee: Eric Hanson
> Attachments: HIVE-6017.01.patch, HIVE-6017.02.patch, 
> HIVE-6017.03.patch, HIVE-6017.04.patch
>
>
> Contribute the Decimal128 high-performance decimal package developed by 
> Microsoft to Hive. This was originally written for Microsoft PolyBase by 
> Hideaki Kimura.
> This code is about 8X more efficient than Java BigDecimal for typical 
> operations. It uses a finite (128 bit) precision and can handle up to 
> decimal(38, X). It is also "mutable" so you can change the contents of an 
> existing object. This helps reduce the cost of new() and garbage collection.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5521) Remove CommonRCFileInputFormat

[Jitendra Nath Pandey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jitendra Nath Pandey updated HIVE-5521:
---

Fix Version/s: 0.13.0

> Remove CommonRCFileInputFormat
> --
>
> Key: HIVE-5521
> URL: https://issues.apache.org/jira/browse/HIVE-5521
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats, Vectorization
>Affects Versions: 0.13.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.13.0
>
> Attachments: HIVE-5521.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 16307: Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive

[Eric Hanson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16307/
---

(Updated Dec. 17, 2013, 2:12 a.m.)


Review request for hive and Gunther Hagleitner.


Changes
---

removed trailing white space


Bugs: HIVE-6017
https://issues.apache.org/jira/browse/HIVE-6017


Repository: hive-git


Description
---

Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to 
Hive


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/type/Decimal128.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/type/SignedInt128.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/type/SqlMathUtil.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/type/UnsignedInt128.java 
PRE-CREATION 
  common/src/test/org/apache/hadoop/hive/common/type/TestDecimal128.java 
PRE-CREATION 
  common/src/test/org/apache/hadoop/hive/common/type/TestSignedInt128.java 
PRE-CREATION 
  common/src/test/org/apache/hadoop/hive/common/type/TestSqlMathUtil.java 
PRE-CREATION 
  common/src/test/org/apache/hadoop/hive/common/type/TestUnsignedInt128.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/16307/diff/


Testing
---


Thanks,

Eric Hanson

[jira] [Updated] (HIVE-5521) Remove CommonRCFileInputFormat

[Jitendra Nath Pandey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jitendra Nath Pandey updated HIVE-5521:
---

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks to Ashutosh!

> Remove CommonRCFileInputFormat
> --
>
> Key: HIVE-5521
> URL: https://issues.apache.org/jira/browse/HIVE-5521
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats, Vectorization
>Affects Versions: 0.13.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.13.0
>
> Attachments: HIVE-5521.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6017) Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive

[Eric Hanson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Hanson updated HIVE-6017:
--

Attachment: HIVE-6017.03.patch

> Contribute Decimal128 high-performance decimal(p, s) package from Microsoft 
> to Hive
> ---
>
> Key: HIVE-6017
> URL: https://issues.apache.org/jira/browse/HIVE-6017
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 0.13.0
>Reporter: Eric Hanson
>Assignee: Eric Hanson
> Attachments: HIVE-6017.01.patch, HIVE-6017.02.patch, 
> HIVE-6017.03.patch
>
>
> Contribute the Decimal128 high-performance decimal package developed by 
> Microsoft to Hive. This was originally written for Microsoft PolyBase by 
> Hideaki Kimura.
> This code is about 8X more efficient than Java BigDecimal for typical 
> operations. It uses a finite (128 bit) precision and can handle up to 
> decimal(38, X). It is also "mutable" so you can change the contents of an 
> existing object. This helps reduce the cost of new() and garbage collection.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 16307: Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive

[Eric Hanson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16307/
---

(Updated Dec. 17, 2013, 1:52 a.m.)


Review request for hive and Gunther Hagleitner.


Changes
---

Tweaked comment spacing and copyright.


Bugs: HIVE-6017
https://issues.apache.org/jira/browse/HIVE-6017


Repository: hive-git


Description
---

Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to 
Hive


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/type/Decimal128.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/type/SignedInt128.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/type/SqlMathUtil.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/type/UnsignedInt128.java 
PRE-CREATION 
  common/src/test/org/apache/hadoop/hive/common/type/TestDecimal128.java 
PRE-CREATION 
  common/src/test/org/apache/hadoop/hive/common/type/TestSignedInt128.java 
PRE-CREATION 
  common/src/test/org/apache/hadoop/hive/common/type/TestSqlMathUtil.java 
PRE-CREATION 
  common/src/test/org/apache/hadoop/hive/common/type/TestUnsignedInt128.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/16307/diff/


Testing
---


Thanks,

Eric Hanson

[jira] [Updated] (HIVE-6021) Problem in GroupByOperator for handling distinct aggrgations

[Sun Rui (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sun Rui updated HIVE-6021:
--

Attachment: HIVE-6021.2.patch

attach the updated patch.

> Problem in GroupByOperator for handling distinct aggrgations
> 
>
> Key: HIVE-6021
> URL: https://issues.apache.org/jira/browse/HIVE-6021
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.12.0
>Reporter: Sun Rui
>Assignee: Sun Rui
> Attachments: HIVE-6021.1.patch, HIVE-6021.2.patch
>
>
> Use the following test case with HIVE 0.12:
> {code:sql}
> create table src(key int, value string);
> load data local inpath 'src/data/files/kv1.txt' overwrite into table src;
> set hive.map.aggr=false; 
> select count(key),count(distinct value) from src group by key;
> {code}
> We will get an ArrayIndexOutOfBoundsException from GroupByOperator:
> {code}
> java.lang.RuntimeException: Error in configuring object
>   at 
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>   at 
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>   at 
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>   at 
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:485)
>   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
>   at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:260)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>   ... 5 more
> Caused by: java.lang.RuntimeException: Reduce operator initialization failed
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:159)
>   ... 10 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.initializeOp(GroupByOperator.java:281)
>   at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:377)
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:152)
>   ... 10 more
> {code}
> explain select count(key),count(distinct value) from src group by key;
> {code}
> STAGE PLANS:
>   Stage: Stage-1
> Map Reduce
>   Alias -> Map Operator Tree:
> src 
>   TableScan
> alias: src
> Select Operator
>   expressions:
> expr: key
> type: int
> expr: value
> type: string
>   outputColumnNames: key, value
>   Reduce Output Operator
> key expressions:
>   expr: key
>   type: int
>   expr: value
>   type: string
> sort order: ++
> Map-reduce partition columns:
>   expr: key
>   type: int
> tag: -1
>   Reduce Operator Tree:
> Group By Operator
>   aggregations:
> expr: count(KEY._col0)   // The parameter causes this problem
>^^^
> expr: count(DISTINCT KEY._col1:0._col0)
>   bucketGroup: false
>   keys:
> expr: KEY._col0
> type: int
>   mode: complete
>   outputColumnNames: _col0, _col1, _col2
>   Select Operator
> expressions:
>   expr: _col1
>   type: bigint
>   expr: _col2
>   type: bigint
> outputColumnNames: _col0, _col1
> File Output Operator
>   compressed: false
>   GlobalTableId: 0
>   table:
>   input format: org.apache.hadoop.mapred.TextInputFormat
>   output format: 
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
>   Stage: Stage-0
> Fetch Operator
>   limit: -1
> {code}
> The root cause is within GroupByOperator.initializeOp(). The method forgets 
> to handle the case:
> For a query has distinct aggregations, there is an aggregation function has a 
> parameter which is a groupby key column but not distinct key column.
> {code}
> if (unionExprEval != null) {
>   String[] names = parameters.get(j).getExprString().split("\\.");
>   

[jira] [Updated] (HIVE-6017) Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive

[Eric Hanson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Hanson updated HIVE-6017:
--

Attachment: HIVE-6017.02.patch

Renamed new test classes to start with Test rather than end with Test per 
instructions from Brock Noland.

> Contribute Decimal128 high-performance decimal(p, s) package from Microsoft 
> to Hive
> ---
>
> Key: HIVE-6017
> URL: https://issues.apache.org/jira/browse/HIVE-6017
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 0.13.0
>Reporter: Eric Hanson
>Assignee: Eric Hanson
> Attachments: HIVE-6017.01.patch, HIVE-6017.02.patch
>
>
> Contribute the Decimal128 high-performance decimal package developed by 
> Microsoft to Hive. This was originally written for Microsoft PolyBase by 
> Hideaki Kimura.
> This code is about 8X more efficient than Java BigDecimal for typical 
> operations. It uses a finite (128 bit) precision and can handle up to 
> decimal(38, X). It is also "mutable" so you can change the contents of an 
> existing object. This helps reduce the cost of new() and garbage collection.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 16307: Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive

[Eric Hanson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16307/
---

(Updated Dec. 17, 2013, 1:03 a.m.)


Review request for hive and Gunther Hagleitner.


Changes
---

Fixed test class names to start with Test.


Bugs: HIVE-6017
https://issues.apache.org/jira/browse/HIVE-6017


Repository: hive-git


Description
---

Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to 
Hive


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/type/Decimal128.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/type/SignedInt128.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/type/SqlMathUtil.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/type/UnsignedInt128.java 
PRE-CREATION 
  common/src/test/org/apache/hadoop/hive/common/type/TestDecimal128.java 
PRE-CREATION 
  common/src/test/org/apache/hadoop/hive/common/type/TestSignedInt128.java 
PRE-CREATION 
  common/src/test/org/apache/hadoop/hive/common/type/TestSqlMathUtil.java 
PRE-CREATION 
  common/src/test/org/apache/hadoop/hive/common/type/TestUnsignedInt128.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/16307/diff/


Testing
---


Thanks,

Eric Hanson

[jira] [Updated] (HIVE-5973) SMB joins produce incorrect results with multiple partitions and buckets

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-5973:


Resolution: Fixed
Status: Resolved  (was: Patch Available)

thanks Vikram

> SMB joins produce incorrect results with multiple partitions and buckets
> 
>
> Key: HIVE-5973
> URL: https://issues.apache.org/jira/browse/HIVE-5973
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Fix For: 0.13.0
>
> Attachments: HIVE-5973.1.patch, HIVE-5973.2.patch
>
>
> It looks like there is an issue with re-using the output object array in the 
> select operator. When we read rows of the non-big tables, we hold on to the 
> output object in the priority queue. This causes hive to produce incorrect 
> results because all the elements in the priority queue refer to the same 
> object and the join happens on only one of the buckets.
> {noformat}
> output[i] = eval[i].evaluate(row);
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6015) vectorized logarithm produces results for 0 that are different from a non-vectorized one

[Eric Hanson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Hanson updated HIVE-6015:
--

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

> vectorized logarithm produces results for 0 that are different from a 
> non-vectorized one
> 
>
> Key: HIVE-6015
> URL: https://issues.apache.org/jira/browse/HIVE-6015
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>  Labels: vectorization
> Fix For: 0.13.0
>
> Attachments: HIVE-6015.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6010) create a test that would ensure vectorization produces same results as non-vectorized execution

[Jitendra Nath Pandey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849926#comment-13849926
 ] 

Jitendra Nath Pandey commented on HIVE-6010:


+1

> create a test that would ensure vectorization produces same results as 
> non-vectorized execution
> ---
>
> Key: HIVE-6010
> URL: https://issues.apache.org/jira/browse/HIVE-6010
> Project: Hive
>  Issue Type: Test
>  Components: Tests, Vectorization
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-6010.01.patch, HIVE-6010.02.patch, HIVE-6010.patch
>
>
> So as to ensure that vectorization is not forgotten when changes are made to 
> things. Obviously it would not be viable to have a bulletproof test, but at 
> least a subset of operations can be verified.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6015) vectorized logarithm produces results for 0 that are different from a non-vectorized one

[Eric Hanson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849927#comment-13849927
 ] 

Eric Hanson commented on HIVE-6015:
---

Committed to trunk. Thanks Sergey!

> vectorized logarithm produces results for 0 that are different from a 
> non-vectorized one
> 
>
> Key: HIVE-6015
> URL: https://issues.apache.org/jira/browse/HIVE-6015
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>  Labels: vectorization
> Fix For: 0.13.0
>
> Attachments: HIVE-6015.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 16229: HIVE-6010 create a test that would ensure vectorization produces same results as non-vectorized execution

[Jitendra Pandey]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16229/#review30506
---

Ship it!


Ship It!

- Jitendra Pandey


On Dec. 13, 2013, 7:15 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16229/
> ---
> 
> (Updated Dec. 13, 2013, 7:15 p.m.)
> 
> 
> Review request for hive and Jitendra Pandey.
> 
> 
> Bugs: HIVE-6010
> https://issues.apache.org/jira/browse/HIVE-6010
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See jira.
> 
> 
> Diffs
> -
> 
>   ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 85351aa 
>   itests/qtest/pom.xml 8c249a0 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java c16e82d 
>   ql/src/test/queries/clientcompare/vectorized_math_funcs.q PRE-CREATION 
>   ql/src/test/queries/clientcompare/vectorized_math_funcs_00.qv PRE-CREATION 
>   ql/src/test/queries/clientcompare/vectorized_math_funcs_01.qv PRE-CREATION 
>   ql/src/test/templates/TestCompareCliDriver.vm PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/16229/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Commented] (HIVE-6013) Supporting Quoted Identifiers in Column Names

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849923#comment-13849923
 ] 

Hive QA commented on HIVE-6013:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12618984/HIVE-6013.3.patch

{color:green}SUCCESS:{color} +1 4793 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/659/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/659/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12618984

> Supporting Quoted Identifiers in Column Names
> -
>
> Key: HIVE-6013
> URL: https://issues.apache.org/jira/browse/HIVE-6013
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Harish Butani
>Assignee: Harish Butani
> Fix For: 0.13.0
>
> Attachments: HIVE-6013.1.patch, HIVE-6013.2.patch, HIVE-6013.3.patch, 
> QuotedIdentifier.html
>
>
> Hive's current behavior on Quoted Identifiers is different from the normal 
> interpretation. Quoted Identifier (using backticks) has a special 
> interpretation for Select expressions(as Regular Expressions). Have 
> documented current behavior and proposed a solution in attached doc.
> Summary of solution is:
> - Introduce 'standard' quoted identifiers for columns only. 
> - At the langauage level this is turned on by a flag.
> - At the metadata level we relax the constraint on column names.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 16307: Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive

[Brock Noland]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16307/#review30503
---


FYI PTest required tests start with Test not end with it.

- Brock Noland


On Dec. 17, 2013, 12:01 a.m., Eric Hanson wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16307/
> ---
> 
> (Updated Dec. 17, 2013, 12:01 a.m.)
> 
> 
> Review request for hive and Gunther Hagleitner.
> 
> 
> Bugs: HIVE-6017
> https://issues.apache.org/jira/browse/HIVE-6017
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Contribute Decimal128 high-performance decimal(p, s) package from Microsoft 
> to Hive
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/type/Decimal128.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/type/SignedInt128.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/type/SqlMathUtil.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/type/UnsignedInt128.java 
> PRE-CREATION 
>   common/src/test/org/apache/hadoop/hive/common/type/Decimal128Test.java 
> PRE-CREATION 
>   common/src/test/org/apache/hadoop/hive/common/type/SignedInt128Test.java 
> PRE-CREATION 
>   common/src/test/org/apache/hadoop/hive/common/type/SqlMathUtilTest.java 
> PRE-CREATION 
>   common/src/test/org/apache/hadoop/hive/common/type/UnsignedInt128Test.java 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/16307/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Eric Hanson
> 
>

[jira] [Commented] (HIVE-6017) Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive

[Eric Hanson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849901#comment-13849901
 ] 

Eric Hanson commented on HIVE-6017:
---

Code review available at https://reviews.apache.org/r/16307/

> Contribute Decimal128 high-performance decimal(p, s) package from Microsoft 
> to Hive
> ---
>
> Key: HIVE-6017
> URL: https://issues.apache.org/jira/browse/HIVE-6017
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 0.13.0
>Reporter: Eric Hanson
>Assignee: Eric Hanson
> Attachments: HIVE-6017.01.patch
>
>
> Contribute the Decimal128 high-performance decimal package developed by 
> Microsoft to Hive. This was originally written for Microsoft PolyBase by 
> Hideaki Kimura.
> This code is about 8X more efficient than Java BigDecimal for typical 
> operations. It uses a finite (128 bit) precision and can handle up to 
> decimal(38, X). It is also "mutable" so you can change the contents of an 
> existing object. This helps reduce the cost of new() and garbage collection.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6015) vectorized logarithm produces results for 0 that are different from a non-vectorized one

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849896#comment-13849896
 ] 

Sergey Shelukhin commented on HIVE-6015:


[~ehans] can you commit? I don't have access

> vectorized logarithm produces results for 0 that are different from a 
> non-vectorized one
> 
>
> Key: HIVE-6015
> URL: https://issues.apache.org/jira/browse/HIVE-6015
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>  Labels: vectorization
> Attachments: HIVE-6015.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Review Request 16307: Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive

[Eric Hanson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16307/
---

Review request for hive and Gunther Hagleitner.


Bugs: HIVE-6017
https://issues.apache.org/jira/browse/HIVE-6017


Repository: hive-git


Description
---

Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to 
Hive


Diffs
-

  common/src/java/org/apache/hadoop/hive/common/type/Decimal128.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/type/SignedInt128.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/type/SqlMathUtil.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/type/UnsignedInt128.java 
PRE-CREATION 
  common/src/test/org/apache/hadoop/hive/common/type/Decimal128Test.java 
PRE-CREATION 
  common/src/test/org/apache/hadoop/hive/common/type/SignedInt128Test.java 
PRE-CREATION 
  common/src/test/org/apache/hadoop/hive/common/type/SqlMathUtilTest.java 
PRE-CREATION 
  common/src/test/org/apache/hadoop/hive/common/type/UnsignedInt128Test.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/16307/diff/


Testing
---


Thanks,

Eric Hanson

[jira] [Updated] (HIVE-5829) Rewrite Trim and Pad UDFs based on GenericUDF

[Mohammad Kamrul Islam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mohammad Kamrul Islam updated HIVE-5829:


Attachment: HIVE-5829.2.patch
tmp.HIVE-5829.patch

Addressed the failed test case and rebased with latest code base.

> Rewrite Trim and Pad UDFs based on GenericUDF
> -
>
> Key: HIVE-5829
> URL: https://issues.apache.org/jira/browse/HIVE-5829
> Project: Hive
>  Issue Type: Bug
>Reporter: Mohammad Kamrul Islam
>Assignee: Mohammad Kamrul Islam
> Attachments: HIVE-5829.1.patch, HIVE-5829.2.patch, tmp.HIVE-5829.patch
>
>
> This JIRA includes following UDFs:
> 1. trim()
> 2. ltrim()
> 3. rtrim()
> 4. lpad()
> 5. rpad()



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 15654: Rewrite Trim and Pad UDFs based on GenericUDF

[Mohammad Islam]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15654/
---

(Updated Dec. 17, 2013, midnight)


Review request for hive, Ashutosh Chauhan, Carl Steinbach, and Jitendra Pandey.


Changes
---

Fix the failed test case.


Bugs: HIVE-5829
https://issues.apache.org/jira/browse/HIVE-5829


Repository: hive-git


Description
---

Rewrite the UDFS *pads and *trim using GenericUDF.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java a895d65 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
bca1f26 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLTrim.java dc00cf9 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLpad.java d1da19a 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFRTrim.java 2bcc5fa 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFRpad.java 9652ce2 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFTrim.java 490886d 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBasePad.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseTrim.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLTrim.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLpad.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRTrim.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRpad.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTrim.java 
PRE-CREATION 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorizationContext.java 
eff251f 
  ql/src/test/org/apache/hadoop/hive/ql/udf/TestGenericUDFLTrim.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/udf/TestGenericUDFLpad.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/udf/TestGenericUDFRTrim.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/udf/TestGenericUDFRpad.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/udf/TestGenericUDFTrim.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/15654/diff/


Testing
---


Thanks,

Mohammad Islam

[jira] [Updated] (HIVE-6017) Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive

[Eric Hanson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Hanson updated HIVE-6017:
--

Affects Version/s: 0.13.0
   Status: Patch Available  (was: In Progress)

Added Decimal128 package into Hive package org.apache.hadoop.hive.common.type. 
Verified that it compiles and unit tests pass. 

> Contribute Decimal128 high-performance decimal(p, s) package from Microsoft 
> to Hive
> ---
>
> Key: HIVE-6017
> URL: https://issues.apache.org/jira/browse/HIVE-6017
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 0.13.0
>Reporter: Eric Hanson
>Assignee: Eric Hanson
> Attachments: HIVE-6017.01.patch
>
>
> Contribute the Decimal128 high-performance decimal package developed by 
> Microsoft to Hive. This was originally written for Microsoft PolyBase by 
> Hideaki Kimura.
> This code is about 8X more efficient than Java BigDecimal for typical 
> operations. It uses a finite (128 bit) precision and can handle up to 
> decimal(38, X). It is also "mutable" so you can change the contents of an 
> existing object. This helps reduce the cost of new() and garbage collection.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6017) Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive

[Eric Hanson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Hanson updated HIVE-6017:
--

Attachment: HIVE-6017.01.patch

> Contribute Decimal128 high-performance decimal(p, s) package from Microsoft 
> to Hive
> ---
>
> Key: HIVE-6017
> URL: https://issues.apache.org/jira/browse/HIVE-6017
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Eric Hanson
>Assignee: Eric Hanson
> Attachments: HIVE-6017.01.patch
>
>
> Contribute the Decimal128 high-performance decimal package developed by 
> Microsoft to Hive. This was originally written for Microsoft PolyBase by 
> Hideaki Kimura.
> This code is about 8X more efficient than Java BigDecimal for typical 
> operations. It uses a finite (128 bit) precision and can handle up to 
> decimal(38, X). It is also "mutable" so you can change the contents of an 
> existing object. This helps reduce the cost of new() and garbage collection.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6039) Round, AVG and SUM functions reject char/varch input while accepting string input

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849869#comment-13849869
 ] 

Hive QA commented on HIVE-6039:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12618986/HIVE-6039.patch

{color:green}SUCCESS:{color} +1 4790 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/658/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/658/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12618986

> Round, AVG and SUM functions reject char/varch input while accepting string 
> input
> -
>
> Key: HIVE-6039
> URL: https://issues.apache.org/jira/browse/HIVE-6039
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Fix For: 0.13.0
>
> Attachments: HIVE-6039.patch
>
>
> An error similar to the following will occur:
> {code}
> hive> create table tabs (c char(8), vc varchar(10)) ROW FORMAT DELIMITED 
> FIELDS TERMINATED BY '\t';
> hive> select sum(c), avg(c), sum(vc), avg(vc) from tabs;
> FAILED: UDFArgumentTypeException Only numeric or string type arguments are 
> accepted but char(8) is passed.
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 16229: HIVE-6010 create a test that would ensure vectorization produces same results as non-vectorized execution

[Sergey Shelukhin]

> On Dec. 16, 2013, 9:19 p.m., Jitendra Pandey wrote:
> > ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java, line 453
> > 
> >
> > Is it ok to write to stdout?

yes, it ends up in build output


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16229/#review30464
---


On Dec. 13, 2013, 7:15 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16229/
> ---
> 
> (Updated Dec. 13, 2013, 7:15 p.m.)
> 
> 
> Review request for hive and Jitendra Pandey.
> 
> 
> Bugs: HIVE-6010
> https://issues.apache.org/jira/browse/HIVE-6010
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See jira.
> 
> 
> Diffs
> -
> 
>   ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 85351aa 
>   itests/qtest/pom.xml 8c249a0 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java c16e82d 
>   ql/src/test/queries/clientcompare/vectorized_math_funcs.q PRE-CREATION 
>   ql/src/test/queries/clientcompare/vectorized_math_funcs_00.qv PRE-CREATION 
>   ql/src/test/queries/clientcompare/vectorized_math_funcs_01.qv PRE-CREATION 
>   ql/src/test/templates/TestCompareCliDriver.vm PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/16229/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 16229: HIVE-6010 create a test that would ensure vectorization produces same results as non-vectorized execution

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16229/#review30486
---



ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java


that is because the new test has no result directory, it compares outputs 
of multiple queries



ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java


yeah, it ends up being written to build output. Do you think it should be 
written elsewhere?



ql/src/test/queries/clientcompare/vectorized_math_funcs_00.qv


this will muddy the naming convention for versions and potentially cause 
ambiguity; discussed the details in person


- Sergey Shelukhin


On Dec. 13, 2013, 7:15 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16229/
> ---
> 
> (Updated Dec. 13, 2013, 7:15 p.m.)
> 
> 
> Review request for hive and Jitendra Pandey.
> 
> 
> Bugs: HIVE-6010
> https://issues.apache.org/jira/browse/HIVE-6010
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See jira.
> 
> 
> Diffs
> -
> 
>   ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 85351aa 
>   itests/qtest/pom.xml 8c249a0 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java c16e82d 
>   ql/src/test/queries/clientcompare/vectorized_math_funcs.q PRE-CREATION 
>   ql/src/test/queries/clientcompare/vectorized_math_funcs_00.qv PRE-CREATION 
>   ql/src/test/queries/clientcompare/vectorized_math_funcs_01.qv PRE-CREATION 
>   ql/src/test/templates/TestCompareCliDriver.vm PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/16229/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Commented] (HIVE-5812) HiveServer2 SSL connection transport binds to loopback address by default

[Brock Noland (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849843#comment-13849843
 ] 

Brock Noland commented on HIVE-5812:


+1

> HiveServer2 SSL connection transport binds to loopback address by default
> -
>
> Key: HIVE-5812
> URL: https://issues.apache.org/jira/browse/HIVE-5812
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.13.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Attachments: HIVE-5812.1.patch, HIVE-5812.2.patch
>
>
> The secure socket transport implemented as part of HIVE-5351, binds to 
> loopback address by default. If the bind interface gets used only if its 
> explicitly defined in the hive-site or via environment.
> This behavior should be same as non-SSL transport.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5230) Better error reporting by async threads in HiveServer2

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-5230:
---

Status: Patch Available  (was: Open)

> Better error reporting by async threads in HiveServer2
> --
>
> Key: HIVE-5230
> URL: https://issues.apache.org/jira/browse/HIVE-5230
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Fix For: 0.13.0
>
> Attachments: HIVE-5230.1.patch, HIVE-5230.1.patch, 
> HIVE-5230.10.patch, HIVE-5230.2.patch, HIVE-5230.3.patch, HIVE-5230.4.patch, 
> HIVE-5230.6.patch, HIVE-5230.7.patch, HIVE-5230.8.patch, HIVE-5230.9.patch
>
>
> [HIVE-4617|https://issues.apache.org/jira/browse/HIVE-4617] provides support 
> for async execution in HS2. When a background thread gets an error, currently 
> the client can only poll for the operation state and also the error with its 
> stacktrace is logged. However, it will be useful to provide a richer error 
> response like thrift API does with TStatus (which is constructed while 
> building a Thrift response object). 



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5230) Better error reporting by async threads in HiveServer2

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849827#comment-13849827
 ] 

Vaibhav Gumashta commented on HIVE-5230:


[~thejas] Uploaded a rebased patch which contains the thrift generated code as 
well. Thanks!

> Better error reporting by async threads in HiveServer2
> --
>
> Key: HIVE-5230
> URL: https://issues.apache.org/jira/browse/HIVE-5230
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Fix For: 0.13.0
>
> Attachments: HIVE-5230.1.patch, HIVE-5230.1.patch, 
> HIVE-5230.10.patch, HIVE-5230.2.patch, HIVE-5230.3.patch, HIVE-5230.4.patch, 
> HIVE-5230.6.patch, HIVE-5230.7.patch, HIVE-5230.8.patch, HIVE-5230.9.patch
>
>
> [HIVE-4617|https://issues.apache.org/jira/browse/HIVE-4617] provides support 
> for async execution in HS2. When a background thread gets an error, currently 
> the client can only poll for the operation state and also the error with its 
> stacktrace is logged. However, it will be useful to provide a richer error 
> response like thrift API does with TStatus (which is constructed while 
> building a Thrift response object). 



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5230) Better error reporting by async threads in HiveServer2

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-5230:
---

Attachment: HIVE-5230.10.patch

Patch rebased on latest trunk.

> Better error reporting by async threads in HiveServer2
> --
>
> Key: HIVE-5230
> URL: https://issues.apache.org/jira/browse/HIVE-5230
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Fix For: 0.13.0
>
> Attachments: HIVE-5230.1.patch, HIVE-5230.1.patch, 
> HIVE-5230.10.patch, HIVE-5230.2.patch, HIVE-5230.3.patch, HIVE-5230.4.patch, 
> HIVE-5230.6.patch, HIVE-5230.7.patch, HIVE-5230.8.patch, HIVE-5230.9.patch
>
>
> [HIVE-4617|https://issues.apache.org/jira/browse/HIVE-4617] provides support 
> for async execution in HS2. When a background thread gets an error, currently 
> the client can only poll for the operation state and also the error with its 
> stacktrace is logged. However, it will be useful to provide a richer error 
> response like thrift API does with TStatus (which is constructed while 
> building a Thrift response object). 



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5992) Hive inconsistently converts timestamp in AVG and SUM UDAF's

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849819#comment-13849819
 ] 

Hive QA commented on HIVE-5992:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12618977/HIVE-5992.patch

{color:green}SUCCESS:{color} +1 4789 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/656/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/656/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12618977

> Hive inconsistently converts timestamp in AVG and SUM UDAF's
> 
>
> Key: HIVE-5992
> URL: https://issues.apache.org/jira/browse/HIVE-5992
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 0.12.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-5992.patch
>
>
> {code}
> hive> select t, sum(t), count(*), sum(t)/count(*), avg(t) from ts group by t;
> ...
> OK
> 1977-03-15 12:34:22.345678 227306062  1  227306062
> 2.27306062345678E8
> {code}
> As it can be seen, timestamp value (1977-03-15 12:34:22.345678) is converted 
> with fractional part ignored in sum, while preserved in avg. As a further 
> result, sum()/count() is not equivalent to avg.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6039) Round, AVG and SUM functions reject char/varch input while accepting string input

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-6039:
--

Attachment: HIVE-6039.patch

> Round, AVG and SUM functions reject char/varch input while accepting string 
> input
> -
>
> Key: HIVE-6039
> URL: https://issues.apache.org/jira/browse/HIVE-6039
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Fix For: 0.13.0
>
> Attachments: HIVE-6039.patch
>
>
> An error similar to the following will occur:
> {code}
> hive> create table tabs (c char(8), vc varchar(10)) ROW FORMAT DELIMITED 
> FIELDS TERMINATED BY '\t';
> hive> select sum(c), avg(c), sum(vc), avg(vc) from tabs;
> FAILED: UDFArgumentTypeException Only numeric or string type arguments are 
> accepted but char(8) is passed.
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6039) Round, AVG and SUM functions reject char/varch input while accepting string input

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-6039:
--

Status: Patch Available  (was: Open)

> Round, AVG and SUM functions reject char/varch input while accepting string 
> input
> -
>
> Key: HIVE-6039
> URL: https://issues.apache.org/jira/browse/HIVE-6039
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Fix For: 0.13.0
>
> Attachments: HIVE-6039.patch
>
>
> An error similar to the following will occur:
> {code}
> hive> create table tabs (c char(8), vc varchar(10)) ROW FORMAT DELIMITED 
> FIELDS TERMINATED BY '\t';
> hive> select sum(c), avg(c), sum(vc), avg(vc) from tabs;
> FAILED: UDFArgumentTypeException Only numeric or string type arguments are 
> accepted but char(8) is passed.
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6013) Supporting Quoted Identifiers in Column Names

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849809#comment-13849809
 ] 

Sergey Shelukhin commented on HIVE-6013:


The fun fact is that double quotes are ANSI way of specifying quoted 
identifiers :)

> Supporting Quoted Identifiers in Column Names
> -
>
> Key: HIVE-6013
> URL: https://issues.apache.org/jira/browse/HIVE-6013
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Harish Butani
>Assignee: Harish Butani
> Fix For: 0.13.0
>
> Attachments: HIVE-6013.1.patch, HIVE-6013.2.patch, HIVE-6013.3.patch, 
> QuotedIdentifier.html
>
>
> Hive's current behavior on Quoted Identifiers is different from the normal 
> interpretation. Quoted Identifier (using backticks) has a special 
> interpretation for Select expressions(as Regular Expressions). Have 
> documented current behavior and proposed a solution in attached doc.
> Summary of solution is:
> - Introduce 'standard' quoted identifiers for columns only. 
> - At the langauage level this is turned on by a flag.
> - At the metadata level we relax the constraint on column names.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6013) Supporting Quoted Identifiers in Column Names

[Eric Hanson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849802#comment-13849802
 ] 

Eric Hanson commented on HIVE-6013:
---

Okay, sounds fine then.

> Supporting Quoted Identifiers in Column Names
> -
>
> Key: HIVE-6013
> URL: https://issues.apache.org/jira/browse/HIVE-6013
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Harish Butani
>Assignee: Harish Butani
> Fix For: 0.13.0
>
> Attachments: HIVE-6013.1.patch, HIVE-6013.2.patch, HIVE-6013.3.patch, 
> QuotedIdentifier.html
>
>
> Hive's current behavior on Quoted Identifiers is different from the normal 
> interpretation. Quoted Identifier (using backticks) has a special 
> interpretation for Select expressions(as Regular Expressions). Have 
> documented current behavior and proposed a solution in attached doc.
> Summary of solution is:
> - Introduce 'standard' quoted identifiers for columns only. 
> - At the langauage level this is turned on by a flag.
> - At the metadata level we relax the constraint on column names.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5795) Hive should be able to skip header and footer rows when reading data file for a table

[Eric Hanson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849801#comment-13849801
 ] 

Eric Hanson commented on HIVE-5795:
---

Looks good, but there are a bunch of places where you need to modify your 
comments to obey Hive coding conventions -- 
https://cwiki.apache.org/confluence/display/Hive/HowToContribute --, especially 
for putting blank lines before comments, and also in a small number of places 
putting blanks between tokens.

> Hive should be able to skip header and footer rows when reading data file for 
> a table
> -
>
> Key: HIVE-5795
> URL: https://issues.apache.org/jira/browse/HIVE-5795
> Project: Hive
>  Issue Type: Bug
>Reporter: Shuaishuai Nie
>Assignee: Shuaishuai Nie
> Attachments: HIVE-5795.1.patch, HIVE-5795.2.patch
>
>
> Hive should be able to skip header and footer lines when reading data file 
> from table. In this way, user don't need to processing data which generated 
> by other application with a header or footer and directly use the file for 
> table operations.
> To implement this, the idea is adding new properties in table descriptions to 
> define the number of lines in header and footer and skip them when reading 
> the record from record reader. An DDL example for creating a table with 
> header and footer should be like this:
> {code}
> Create external table testtable (name string, message string) row format 
> delimited fields terminated by '\t' lines terminated by '\n' location 
> '/testtable' tblproperties ("skip.header.number"="1", 
> "skip.footer.number"="2");
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 16184: Hive should be able to skip header and footer rows when reading data file for a table (HIVE-5795)

[Eric Hanson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16184/#review30489
---



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java


fix comment to say "max number of lines of footer user can set for a table 
file"



ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java


Please use block comment style for multi-line comments. Please use full 
sentences and proper punctuation (start with capital letter, end with period).


- Eric Hanson


On Dec. 11, 2013, 9:19 p.m., Shuaishuai Nie wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16184/
> ---
> 
> (Updated Dec. 11, 2013, 9:19 p.m.)
> 
> 
> Review request for hive, Eric Hanson and Thejas Nair.
> 
> 
> Bugs: hive-5795
> https://issues.apache.org/jira/browse/hive-5795
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Hive should be able to skip header and footer rows when reading data file for 
> a table
> (follow up with review https://reviews.apache.org/r/15663/diff/#index_header)
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fa3e048 
>   conf/hive-default.xml.template c61a0bb 
>   data/files/header_footer_table_1/0001.txt PRE-CREATION 
>   data/files/header_footer_table_1/0002.txt PRE-CREATION 
>   data/files/header_footer_table_1/0003.txt PRE-CREATION 
>   data/files/header_footer_table_2/2012/01/01/0001.txt PRE-CREATION 
>   data/files/header_footer_table_2/2012/01/02/0002.txt PRE-CREATION 
>   data/files/header_footer_table_2/2012/01/03/0003.txt PRE-CREATION 
>   itests/qtest/pom.xml c3cbb89 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java d2b2526 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java 
> dd5cb6b 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 974a5d6 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/TestHiveBinarySearchRecordReader.java
>  85dd975 
>   ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java 
> 0686d9b 
>   ql/src/test/queries/clientnegative/file_with_header_footer_negative.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/file_with_header_footer.q PRE-CREATION 
>   ql/src/test/results/clientnegative/file_with_header_footer_negative.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/file_with_header_footer.q.out 
> PRE-CREATION 
>   serde/if/serde.thrift 2ceb572 
>   
> serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/serdeConstants.java
>  22a6168 
> 
> Diff: https://reviews.apache.org/r/16184/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Shuaishuai Nie
> 
>

[jira] [Updated] (HIVE-6013) Supporting Quoted Identifiers in Column Names

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6013:


Status: Patch Available  (was: Open)

> Supporting Quoted Identifiers in Column Names
> -
>
> Key: HIVE-6013
> URL: https://issues.apache.org/jira/browse/HIVE-6013
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Harish Butani
>Assignee: Harish Butani
> Fix For: 0.13.0
>
> Attachments: HIVE-6013.1.patch, HIVE-6013.2.patch, HIVE-6013.3.patch, 
> QuotedIdentifier.html
>
>
> Hive's current behavior on Quoted Identifiers is different from the normal 
> interpretation. Quoted Identifier (using backticks) has a special 
> interpretation for Select expressions(as Regular Expressions). Have 
> documented current behavior and proposed a solution in attached doc.
> Summary of solution is:
> - Introduce 'standard' quoted identifiers for columns only. 
> - At the langauage level this is turned on by a flag.
> - At the metadata level we relax the constraint on column names.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6013) Supporting Quoted Identifiers in Column Names

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6013:


Attachment: HIVE-6013.3.patch

> Supporting Quoted Identifiers in Column Names
> -
>
> Key: HIVE-6013
> URL: https://issues.apache.org/jira/browse/HIVE-6013
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Harish Butani
>Assignee: Harish Butani
> Fix For: 0.13.0
>
> Attachments: HIVE-6013.1.patch, HIVE-6013.2.patch, HIVE-6013.3.patch, 
> QuotedIdentifier.html
>
>
> Hive's current behavior on Quoted Identifiers is different from the normal 
> interpretation. Quoted Identifier (using backticks) has a special 
> interpretation for Select expressions(as Regular Expressions). Have 
> documented current behavior and proposed a solution in attached doc.
> Summary of solution is:
> - Introduce 'standard' quoted identifiers for columns only. 
> - At the langauage level this is turned on by a flag.
> - At the metadata level we relax the constraint on column names.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 16299: HIVE-6013: Supporting Quoted Identifiers in Column Names

[Harish Butani]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16299/
---

(Updated Dec. 16, 2013, 10:22 p.m.)


Review request for hive, Ashutosh Chauhan and Alan Gates.


Changes
---

fix paths in quotedid_skew.q


Bugs: HIVE-6013
https://issues.apache.org/jira/browse/HIVE-6013


Repository: hive-git


Description
---

Hive's current behavior on Quoted Identifiers is different from the normal 
interpretation. Quoted Identifier (using backticks) has a special 
interpretation for Select expressions(as Regular Expressions). Have documented 
current behavior and proposed a solution in attached doc.
Summary of solution is:
Introduce 'standard' quoted identifiers for columns only.
At the langauage level this is turned on by a flag.
At the metadata level we relax the constraint on column names.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fa3e048 
  itests/qtest/pom.xml 8c249a0 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
3deed45 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveUtils.java eb26e7f 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 321759b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 17e6aad 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g ed9917d 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 1e6826f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java d18ea03 
  ql/src/java/org/apache/hadoop/hive/ql/parse/UnparseTranslator.java 8fe2262 
  ql/src/test/queries/clientnegative/invalid_columns.q f8be8c8 
  ql/src/test/queries/clientpositive/quotedid_alter.q PRE-CREATION 
  ql/src/test/queries/clientpositive/quotedid_basic.q PRE-CREATION 
  ql/src/test/queries/clientpositive/quotedid_partition.q PRE-CREATION 
  ql/src/test/queries/clientpositive/quotedid_skew.q PRE-CREATION 
  ql/src/test/queries/clientpositive/quotedid_smb.q PRE-CREATION 
  ql/src/test/queries/clientpositive/quotedid_tblproperty.q PRE-CREATION 
  ql/src/test/results/clientnegative/invalid_columns.q.out 3311b0a 
  ql/src/test/results/clientpositive/quotedid_alter.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/quotedid_basic.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/quotedid_partition.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/quotedid_skew.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/quotedid_smb.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/quotedid_tblproperty.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/16299/diff/


Testing
---

added new tests for create, alter, delete, query with columns containing 
special characters.
Tests start with quotedid


Thanks,

Harish Butani

[jira] [Updated] (HIVE-6013) Supporting Quoted Identifiers in Column Names

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6013:


Status: Open  (was: Patch Available)

> Supporting Quoted Identifiers in Column Names
> -
>
> Key: HIVE-6013
> URL: https://issues.apache.org/jira/browse/HIVE-6013
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Harish Butani
>Assignee: Harish Butani
> Fix For: 0.13.0
>
> Attachments: HIVE-6013.1.patch, HIVE-6013.2.patch, 
> QuotedIdentifier.html
>
>
> Hive's current behavior on Quoted Identifiers is different from the normal 
> interpretation. Quoted Identifier (using backticks) has a special 
> interpretation for Select expressions(as Regular Expressions). Have 
> documented current behavior and proposed a solution in attached doc.
> Summary of solution is:
> - Introduce 'standard' quoted identifiers for columns only. 
> - At the langauage level this is turned on by a flag.
> - At the metadata level we relax the constraint on column names.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Hive-trunk-hadoop2 - Build # 605 - Still Failing

[Apache Jenkins Server]

Changes for Build #570
[rhbutani] HIVE-5849 Improve the stats of operators based on heuristics in the 
absence of any column statistics (Prasanth Jayachandran via Harish Butani)

[hashutosh] HIVE-5793 : Update hive-default.xml.template for HIVE4002 (Navis 
via Ashutosh Chauhan)


Changes for Build #571
[navis] HIVE-4518 : Should be removed files (OptrStatsGroupByHook, etc.)

[navis] HIVE-5839 : BytesRefArrayWritable compareTo violates contract (Xuefu 
Zhang via Navis)

[navis] HIVE-4518 : Missing file (HiveFatalException)

[navis] HIVE-4518 : Counter Strike: Operation Operator (Gunther Hagleitner and 
Jason Dere via Navis)


Changes for Build #572
[brock] HIVE-4741 - Add Hive config API to modify the restrict list (Prasad 
Mujumdar, Navis via Brock Noland)


Changes for Build #573
[navis] HIVE-5827 : Incorrect location of logs for failed tests (Vikram Dixit K 
and Szehon Ho via Navis)

[thejas] HIVE-4485 : beeline prints null as empty strings (Thejas Nair reviewed 
by Ashutosh Chauhan)

[brock] HIVE-5704 - A couple of generic UDFs are not in the right 
folder/package (Xuefu Zhang via Brock Noland)

[brock] HIVE-5706 - Move a few numeric UDFs to generic implementations (Xuefu 
Zhang via Brock Noland)

[hashutosh] HIVE-5817 : column name to index mapping in VectorizationContext is 
broken (Remus Rusanu, Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5876 : Split elimination in ORC breaks for partitioned tables 
(Prasanth J via Ashutosh Chauhan)

[hashutosh] HIVE-5886 : [Refactor] Remove unused class JobCloseFeedback 
(Ashutosh Chauhan via Thejas Nair)

[brock] HIVE-5894 - Fix minor PTest2 issues (Brock Noland)


Changes for Build #574
[brock] HIVE-5755 - Fix hadoop2 execution environment Milestone 1 (Vikram Dixit 
K via Brock Noland)


Changes for Build #575
[xuefu] HIVE-5893: hive-schema-0.13.0.mysql.sql contains reference to 
nonexistent column (Carl via Xuefu)

[xuefu] HIVE-5684: Serde support for char (Jason via Xuefu)


Changes for Build #576

Changes for Build #577

Changes for Build #578

Changes for Build #579
[brock] HIVE-5441 - Async query execution doesn't return resultset status 
(Prasad Mujumdar via Thejas M Nair)

[brock] HIVE-5880 - Rename HCatalog HBase Storage Handler artifact id (Brock 
Noland reviewed by Prasad Mujumdar)


Changes for Build #580
[ehans] HIVE-5581: Implement vectorized year/month/day... etc. for string 
arguments (Teddy Choi via Eric Hanson)


Changes for Build #581
[rhbutani] HIVE-5898 Make fetching of column statistics configurable (Prasanth 
Jayachandran via Harish Butani)


Changes for Build #582
[brock] HIVE-5880 - (Rename HCatalog HBase Storage Handler artifact id) breaks 
packaging (Xuefu Zhang via Brock Noland)


Changes for Build #583
[xuefu] HIVE-5866: Hive divide operator generates wrong results in certain 
cases (reviewed by Prasad)

[ehans] HIVE-5877: Implement vectorized support for IN as boolean-valued 
expression (Eric Hanson)


Changes for Build #584
[thejas] HIVE-5550 : Import fails for tables created with default text, 
sequence and orc file formats using HCatalog API (Sushanth Sowmyan via Thejas 
Nair)

[ehans] HIVE-5895: vectorization handles division by zero differently from 
normal execution (Sergey Shelukhin via Eric Hanson)

[hashutosh] HIVE-5938 : Remove apache.mina dependency for test (Navis via 
Ashutosh Chauhan)

[xuefu] HIVE-5912: Show partition command doesn't support db.table (Yu Zhao via 
Xuefu)

[brock] HIVE-5906 - TestGenericUDFPower should use delta to compare doubles 
(Szehon Ho via Brock Noland)

[brock] HIVE-5855 - Add deprecated methods back to ColumnProjectionUtils (Brock 
Noland reviewed by Navis)

[brock] HIVE-5915 - Shade Kryo dependency (Brock Noland reviewed by Ashutosh 
Chauhan)


Changes for Build #585
[hashutosh] HIVE-5916 : No need to aggregate statistics collected via counter 
mechanism (Ashutosh Chauhan via Navis)

[xuefu] HIVE-5947: Fix test failure in decimal_udf.q (reviewed by Brock)


Changes for Build #586
[hashutosh] HIVE-5935 : hive.query.string is not provided to FetchTask (Navis 
via Ashutosh Chauhan)

[navis] HIVE-3455 : ANSI CORR(X,Y) is incorrect (Maxim Bolotin via Navis)

[hashutosh] HIVE-5921 : Better heuristics for worst case statistics estimates 
for join, limit and filter operator (Prasanth J via Harish Butani)

[rhbutani] HIVE-5899 NPE during explain extended with char/varchar columns 
(Jason Dere via Harish Butani)


Changes for Build #587
[xuefu] HIVE-3181: getDatabaseMajor/Minor version does not return values 
(Szehon via Xuefu, reviewed by Navis)

[brock] HIVE-5641 - BeeLineOpts ignores Throwable (Brock Noland reviewed by 
Prasad and Thejas)

[hashutosh] HIVE-5909 : locate and instr throw 
java.nio.BufferUnderflowException when empty string as substring (Navis via 
Ashutosh Chauhan)

[hashutosh] HIVE-5686 : partition column type validation doesn't quite work for 
dates (Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5887 : metastore direct sql doesn't work with oracle (Sergey 
Shel

[jira] [Created] (HIVE-6039) Round, AVG and SUM functions reject char/varch input while accepting string input

[Xuefu Zhang (JIRA)]

Xuefu Zhang created HIVE-6039:
-

 Summary: Round, AVG and SUM functions reject char/varch input 
while accepting string input
 Key: HIVE-6039
 URL: https://issues.apache.org/jira/browse/HIVE-6039
 Project: Hive
  Issue Type: Bug
  Components: UDF
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
 Fix For: 0.13.0


An error similar to the following will occur:
{code}
hive> create table tabs (c char(8), vc varchar(10)) ROW FORMAT DELIMITED FIELDS 
TERMINATED BY '\t';
hive> select sum(c), avg(c), sum(vc), avg(vc) from tabs;
FAILED: UDFArgumentTypeException Only numeric or string type arguments are 
accepted but char(8) is passed.
{code}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5795) Hive should be able to skip header and footer rows when reading data file for a table

[Shuaishuai Nie (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849774#comment-13849774
 ] 

Shuaishuai Nie commented on HIVE-5795:
--

Sorry, wrong link to the new review board. Update the link here: 
https://reviews.apache.org/r/16184/

> Hive should be able to skip header and footer rows when reading data file for 
> a table
> -
>
> Key: HIVE-5795
> URL: https://issues.apache.org/jira/browse/HIVE-5795
> Project: Hive
>  Issue Type: Bug
>Reporter: Shuaishuai Nie
>Assignee: Shuaishuai Nie
> Attachments: HIVE-5795.1.patch, HIVE-5795.2.patch
>
>
> Hive should be able to skip header and footer lines when reading data file 
> from table. In this way, user don't need to processing data which generated 
> by other application with a header or footer and directly use the file for 
> table operations.
> To implement this, the idea is adding new properties in table descriptions to 
> define the number of lines in header and footer and skip them when reading 
> the record from record reader. An DDL example for creating a table with 
> header and footer should be like this:
> {code}
> Create external table testtable (name string, message string) row format 
> delimited fields terminated by '\t' lines terminated by '\n' location 
> '/testtable' tblproperties ("skip.header.number"="1", 
> "skip.footer.number"="2");
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5812) HiveServer2 SSL connection transport binds to loopback address by default

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849770#comment-13849770
 ] 

Hive QA commented on HIVE-5812:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12618972/HIVE-5812.2.patch

{color:green}SUCCESS:{color} +1 4789 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/655/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/655/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12618972

> HiveServer2 SSL connection transport binds to loopback address by default
> -
>
> Key: HIVE-5812
> URL: https://issues.apache.org/jira/browse/HIVE-5812
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.13.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Attachments: HIVE-5812.1.patch, HIVE-5812.2.patch
>
>
> The secure socket transport implemented as part of HIVE-5351, binds to 
> loopback address by default. If the bind interface gets used only if its 
> explicitly defined in the hive-site or via environment.
> This behavior should be same as non-SSL transport.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6013) Supporting Quoted Identifiers in Column Names

[Harish Butani (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849765#comment-13849765
 ] 

Harish Butani commented on HIVE-6013:
-

Back-ticks are already there (sort of).  For e.g: today view definitions are 
unparsed(and persisted) to wrap all identifiers with back-ticks. 
Moving to square brackets would either require a migration step, or supporting 
both back-ticks and square brackets. Don't want to take on either option in 
this first step.
Besides not clear which is the preferred style for the hive community: 
back-ticks seems to have come from the mysql way of specifying quoted 
identifiers.



> Supporting Quoted Identifiers in Column Names
> -
>
> Key: HIVE-6013
> URL: https://issues.apache.org/jira/browse/HIVE-6013
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Harish Butani
>Assignee: Harish Butani
> Fix For: 0.13.0
>
> Attachments: HIVE-6013.1.patch, HIVE-6013.2.patch, 
> QuotedIdentifier.html
>
>
> Hive's current behavior on Quoted Identifiers is different from the normal 
> interpretation. Quoted Identifier (using backticks) has a special 
> interpretation for Select expressions(as Regular Expressions). Have 
> documented current behavior and proposed a solution in attached doc.
> Summary of solution is:
> - Introduce 'standard' quoted identifiers for columns only. 
> - At the langauage level this is turned on by a flag.
> - At the metadata level we relax the constraint on column names.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5992) Hive inconsistently converts timestamp in AVG and SUM UDAF's

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-5992:
--

Status: Patch Available  (was: Open)

> Hive inconsistently converts timestamp in AVG and SUM UDAF's
> 
>
> Key: HIVE-5992
> URL: https://issues.apache.org/jira/browse/HIVE-5992
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 0.12.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-5992.patch
>
>
> {code}
> hive> select t, sum(t), count(*), sum(t)/count(*), avg(t) from ts group by t;
> ...
> OK
> 1977-03-15 12:34:22.345678 227306062  1  227306062
> 2.27306062345678E8
> {code}
> As it can be seen, timestamp value (1977-03-15 12:34:22.345678) is converted 
> with fractional part ignored in sum, while preserved in avg. As a further 
> result, sum()/count() is not equivalent to avg.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5992) Hive inconsistently converts timestamp in AVG and SUM UDAF's

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-5992:
--

Attachment: HIVE-5992.patch

> Hive inconsistently converts timestamp in AVG and SUM UDAF's
> 
>
> Key: HIVE-5992
> URL: https://issues.apache.org/jira/browse/HIVE-5992
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 0.12.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-5992.patch
>
>
> {code}
> hive> select t, sum(t), count(*), sum(t)/count(*), avg(t) from ts group by t;
> ...
> OK
> 1977-03-15 12:34:22.345678 227306062  1  227306062
> 2.27306062345678E8
> {code}
> As it can be seen, timestamp value (1977-03-15 12:34:22.345678) is converted 
> with fractional part ignored in sum, while preserved in avg. As a further 
> result, sum()/count() is not equivalent to avg.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 16229: HIVE-6010 create a test that would ensure vectorization produces same results as non-vectorized execution

[Jitendra Pandey]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16229/#review30464
---



ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java


We should throw an exception if it is null. What is the reason for the 
change in behavior?



ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java


Is it ok to write to stdout?



ql/src/test/queries/clientcompare/vectorized_math_funcs_00.qv


The name of the file could be suffixed to indicate vectorization disabled 
or enabled.


- Jitendra Pandey


On Dec. 13, 2013, 7:15 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16229/
> ---
> 
> (Updated Dec. 13, 2013, 7:15 p.m.)
> 
> 
> Review request for hive and Jitendra Pandey.
> 
> 
> Bugs: HIVE-6010
> https://issues.apache.org/jira/browse/HIVE-6010
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See jira.
> 
> 
> Diffs
> -
> 
>   ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 85351aa 
>   itests/qtest/pom.xml 8c249a0 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java c16e82d 
>   ql/src/test/queries/clientcompare/vectorized_math_funcs.q PRE-CREATION 
>   ql/src/test/queries/clientcompare/vectorized_math_funcs_00.qv PRE-CREATION 
>   ql/src/test/queries/clientcompare/vectorized_math_funcs_01.qv PRE-CREATION 
>   ql/src/test/templates/TestCompareCliDriver.vm PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/16229/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Updated] (HIVE-5812) HiveServer2 SSL connection transport binds to loopback address by default

[Prasad Mujumdar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad Mujumdar updated HIVE-5812:
--

Attachment: HIVE-5812.2.patch

Rebased patch

[~brocknoland] would you like to take another look ?

> HiveServer2 SSL connection transport binds to loopback address by default
> -
>
> Key: HIVE-5812
> URL: https://issues.apache.org/jira/browse/HIVE-5812
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.13.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Attachments: HIVE-5812.1.patch, HIVE-5812.2.patch
>
>
> The secure socket transport implemented as part of HIVE-5351, binds to 
> loopback address by default. If the bind interface gets used only if its 
> explicitly defined in the hive-site or via environment.
> This behavior should be same as non-SSL transport.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6034) vectorized % doesn't handle zeroes the same way as non-vectorized

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849714#comment-13849714
 ] 

Hive QA commented on HIVE-6034:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12618952/HIVE-6034.01.patch

{color:green}SUCCESS:{color} +1 4789 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/654/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/654/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12618952

> vectorized % doesn't handle zeroes the same way as non-vectorized
> -
>
> Key: HIVE-6034
> URL: https://issues.apache.org/jira/browse/HIVE-6034
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Minor
> Attachments: HIVE-6034.01.patch, HIVE-6034.patch
>
>
> % 0 is NULL, but if vectorized it's NaN



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Hive-trunk-h0.21 - Build # 2506 - Still Failing

[Apache Jenkins Server]

Changes for Build #2471
[rhbutani] HIVE-5849 Improve the stats of operators based on heuristics in the 
absence of any column statistics (Prasanth Jayachandran via Harish Butani)

[hashutosh] HIVE-5793 : Update hive-default.xml.template for HIVE4002 (Navis 
via Ashutosh Chauhan)


Changes for Build #2472
[navis] HIVE-4518 : Should be removed files (OptrStatsGroupByHook, etc.)

[navis] HIVE-5839 : BytesRefArrayWritable compareTo violates contract (Xuefu 
Zhang via Navis)

[navis] HIVE-4518 : Missing file (HiveFatalException)

[navis] HIVE-4518 : Counter Strike: Operation Operator (Gunther Hagleitner and 
Jason Dere via Navis)


Changes for Build #2473
[brock] HIVE-4741 - Add Hive config API to modify the restrict list (Prasad 
Mujumdar, Navis via Brock Noland)


Changes for Build #2474
[navis] HIVE-5827 : Incorrect location of logs for failed tests (Vikram Dixit K 
and Szehon Ho via Navis)

[thejas] HIVE-4485 : beeline prints null as empty strings (Thejas Nair reviewed 
by Ashutosh Chauhan)

[brock] HIVE-5704 - A couple of generic UDFs are not in the right 
folder/package (Xuefu Zhang via Brock Noland)

[brock] HIVE-5706 - Move a few numeric UDFs to generic implementations (Xuefu 
Zhang via Brock Noland)

[hashutosh] HIVE-5817 : column name to index mapping in VectorizationContext is 
broken (Remus Rusanu, Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5876 : Split elimination in ORC breaks for partitioned tables 
(Prasanth J via Ashutosh Chauhan)

[hashutosh] HIVE-5886 : [Refactor] Remove unused class JobCloseFeedback 
(Ashutosh Chauhan via Thejas Nair)

[brock] HIVE-5894 - Fix minor PTest2 issues (Brock Noland)


Changes for Build #2475
[brock] HIVE-5755 - Fix hadoop2 execution environment Milestone 1 (Vikram Dixit 
K via Brock Noland)


Changes for Build #2476
[xuefu] HIVE-5893: hive-schema-0.13.0.mysql.sql contains reference to 
nonexistent column (Carl via Xuefu)

[xuefu] HIVE-5684: Serde support for char (Jason via Xuefu)


Changes for Build #2477

Changes for Build #2478

Changes for Build #2479

Changes for Build #2480
[brock] HIVE-5441 - Async query execution doesn't return resultset status 
(Prasad Mujumdar via Thejas M Nair)

[brock] HIVE-5880 - Rename HCatalog HBase Storage Handler artifact id (Brock 
Noland reviewed by Prasad Mujumdar)


Changes for Build #2481

Changes for Build #2482
[ehans] HIVE-5581: Implement vectorized year/month/day... etc. for string 
arguments (Teddy Choi via Eric Hanson)


Changes for Build #2483
[rhbutani] HIVE-5898 Make fetching of column statistics configurable (Prasanth 
Jayachandran via Harish Butani)


Changes for Build #2484
[brock] HIVE-5880 - (Rename HCatalog HBase Storage Handler artifact id) breaks 
packaging (Xuefu Zhang via Brock Noland)


Changes for Build #2485
[xuefu] HIVE-5866: Hive divide operator generates wrong results in certain 
cases (reviewed by Prasad)

[ehans] HIVE-5877: Implement vectorized support for IN as boolean-valued 
expression (Eric Hanson)


Changes for Build #2486
[ehans] HIVE-5895: vectorization handles division by zero differently from 
normal execution (Sergey Shelukhin via Eric Hanson)

[hashutosh] HIVE-5938 : Remove apache.mina dependency for test (Navis via 
Ashutosh Chauhan)

[xuefu] HIVE-5912: Show partition command doesn't support db.table (Yu Zhao via 
Xuefu)

[brock] HIVE-5906 - TestGenericUDFPower should use delta to compare doubles 
(Szehon Ho via Brock Noland)

[brock] HIVE-5855 - Add deprecated methods back to ColumnProjectionUtils (Brock 
Noland reviewed by Navis)

[brock] HIVE-5915 - Shade Kryo dependency (Brock Noland reviewed by Ashutosh 
Chauhan)


Changes for Build #2487
[hashutosh] HIVE-5916 : No need to aggregate statistics collected via counter 
mechanism (Ashutosh Chauhan via Navis)

[xuefu] HIVE-5947: Fix test failure in decimal_udf.q (reviewed by Brock)

[thejas] HIVE-5550 : Import fails for tables created with default text, 
sequence and orc file formats using HCatalog API (Sushanth Sowmyan via Thejas 
Nair)


Changes for Build #2488
[hashutosh] HIVE-5935 : hive.query.string is not provided to FetchTask (Navis 
via Ashutosh Chauhan)

[navis] HIVE-3455 : ANSI CORR(X,Y) is incorrect (Maxim Bolotin via Navis)

[hashutosh] HIVE-5921 : Better heuristics for worst case statistics estimates 
for join, limit and filter operator (Prasanth J via Harish Butani)

[rhbutani] HIVE-5899 NPE during explain extended with char/varchar columns 
(Jason Dere via Harish Butani)


Changes for Build #2489
[xuefu] HIVE-3181: getDatabaseMajor/Minor version does not return values 
(Szehon via Xuefu, reviewed by Navis)

[brock] HIVE-5641 - BeeLineOpts ignores Throwable (Brock Noland reviewed by 
Prasad and Thejas)

[hashutosh] HIVE-5909 : locate and instr throw 
java.nio.BufferUnderflowException when empty string as substring (Navis via 
Ashutosh Chauhan)

[hashutosh] HIVE-5686 : partition column type validation doesn't quite work for 
dates (Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5887 : metastore direct

[jira] [Commented] (HIVE-5414) The result of show grant is not visible via JDBC

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849706#comment-13849706
 ] 

Phabricator commented on HIVE-5414:
---

thejas has commented on the revision "HIVE-5414 [jira] The result of show grant 
is not visible via JDBC".

  The patch does not apply on trunk anymore. Can you also please rebase ?
  Sorry about the delay in reviewing this patch.

INLINE COMMENTS
  ql/src/java/org/apache/hadoop/hive/ql/plan/ShowGrantDesc.java:36 I think 
"property" would be a better name for column, than "key".
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java:1816 
can you also add test for column name and column type of result ?
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java:1814 
can you also add a test for 'show roles' ?

REVISION DETAIL
  https://reviews.facebook.net/D13209

To: JIRA, navis
Cc: Unknown Object (Phabricator User), thejas


> The result of show grant is not visible via JDBC
> 
>
> Key: HIVE-5414
> URL: https://issues.apache.org/jira/browse/HIVE-5414
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, JDBC
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: D13209.1.patch, D13209.2.patch, D13209.3.patch, 
> HIVE-5414.4.patch.txt
>
>
> Currently, show grant / show role grant does not make fetch task, which 
> provides the result schema for jdbc clients.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6013) Supporting Quoted Identifiers in Column Names

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849659#comment-13849659
 ] 

Hive QA commented on HIVE-6013:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12618945/HIVE-6013.2.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 4790 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_quotedid_skew
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/653/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/653/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12618945

> Supporting Quoted Identifiers in Column Names
> -
>
> Key: HIVE-6013
> URL: https://issues.apache.org/jira/browse/HIVE-6013
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Harish Butani
>Assignee: Harish Butani
> Fix For: 0.13.0
>
> Attachments: HIVE-6013.1.patch, HIVE-6013.2.patch, 
> QuotedIdentifier.html
>
>
> Hive's current behavior on Quoted Identifiers is different from the normal 
> interpretation. Quoted Identifier (using backticks) has a special 
> interpretation for Select expressions(as Regular Expressions). Have 
> documented current behavior and proposed a solution in attached doc.
> Summary of solution is:
> - Introduce 'standard' quoted identifiers for columns only. 
> - At the langauage level this is turned on by a flag.
> - At the metadata level we relax the constraint on column names.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-1466) Add NULL DEFINED AS to ROW FORMAT specification

[Prasad Mujumdar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad Mujumdar updated HIVE-1466:
--

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Patch committed to trunk.

> Add NULL DEFINED AS to ROW FORMAT specification
> ---
>
> Key: HIVE-1466
> URL: https://issues.apache.org/jira/browse/HIVE-1466
> Project: Hive
>  Issue Type: Improvement
>Reporter: Adam Kramer
>Assignee: Prasad Mujumdar
> Attachments: HIVE-1466.1.patch, HIVE-1466.2.patch
>
>
> NULL values are passed to transformers as a literal backslash and a literal 
> N. NULL values are saved when INSERT OVERWRITing LOCAL DIRECTORies as "NULL". 
> This is inconsistent.
> The ROW FORMAT specification of tables should be able to specify the manner 
> in which a null character is represented. ROW FORMAT NULL DEFINED AS '\N' or 
> '\003' or whatever should apply to all instances of table export and saving.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6036) A test case for embedded beeline - with URL jdbc:hive2:///default

[Prasad Mujumdar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849632#comment-13849632
 ] 

Prasad Mujumdar commented on HIVE-6036:
---

+1
Thanks for the addressing the issue!

> A test case for embedded beeline  -  with URL jdbc:hive2:///default
> ---
>
> Key: HIVE-6036
> URL: https://issues.apache.org/jira/browse/HIVE-6036
> Project: Hive
>  Issue Type: Bug
>Reporter: Anandha L Ranganathan
>Assignee: Anandha L Ranganathan
> Attachments: HIVE-6036.patch
>
>
> A test case for embedded beeline would have been helpful. ie, with URL 
> jdbc:hive2:///default
> This causes beeline (JDBC driver) to invoken embedded hive.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Hive-trunk-h0.21 - Build # 2505 - Still Failing

[Apache Jenkins Server]

Changes for Build #2471
[rhbutani] HIVE-5849 Improve the stats of operators based on heuristics in the 
absence of any column statistics (Prasanth Jayachandran via Harish Butani)

[hashutosh] HIVE-5793 : Update hive-default.xml.template for HIVE4002 (Navis 
via Ashutosh Chauhan)


Changes for Build #2472
[navis] HIVE-4518 : Should be removed files (OptrStatsGroupByHook, etc.)

[navis] HIVE-5839 : BytesRefArrayWritable compareTo violates contract (Xuefu 
Zhang via Navis)

[navis] HIVE-4518 : Missing file (HiveFatalException)

[navis] HIVE-4518 : Counter Strike: Operation Operator (Gunther Hagleitner and 
Jason Dere via Navis)


Changes for Build #2473
[brock] HIVE-4741 - Add Hive config API to modify the restrict list (Prasad 
Mujumdar, Navis via Brock Noland)


Changes for Build #2474
[navis] HIVE-5827 : Incorrect location of logs for failed tests (Vikram Dixit K 
and Szehon Ho via Navis)

[thejas] HIVE-4485 : beeline prints null as empty strings (Thejas Nair reviewed 
by Ashutosh Chauhan)

[brock] HIVE-5704 - A couple of generic UDFs are not in the right 
folder/package (Xuefu Zhang via Brock Noland)

[brock] HIVE-5706 - Move a few numeric UDFs to generic implementations (Xuefu 
Zhang via Brock Noland)

[hashutosh] HIVE-5817 : column name to index mapping in VectorizationContext is 
broken (Remus Rusanu, Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5876 : Split elimination in ORC breaks for partitioned tables 
(Prasanth J via Ashutosh Chauhan)

[hashutosh] HIVE-5886 : [Refactor] Remove unused class JobCloseFeedback 
(Ashutosh Chauhan via Thejas Nair)

[brock] HIVE-5894 - Fix minor PTest2 issues (Brock Noland)


Changes for Build #2475
[brock] HIVE-5755 - Fix hadoop2 execution environment Milestone 1 (Vikram Dixit 
K via Brock Noland)


Changes for Build #2476
[xuefu] HIVE-5893: hive-schema-0.13.0.mysql.sql contains reference to 
nonexistent column (Carl via Xuefu)

[xuefu] HIVE-5684: Serde support for char (Jason via Xuefu)


Changes for Build #2477

Changes for Build #2478

Changes for Build #2479

Changes for Build #2480
[brock] HIVE-5441 - Async query execution doesn't return resultset status 
(Prasad Mujumdar via Thejas M Nair)

[brock] HIVE-5880 - Rename HCatalog HBase Storage Handler artifact id (Brock 
Noland reviewed by Prasad Mujumdar)


Changes for Build #2481

Changes for Build #2482
[ehans] HIVE-5581: Implement vectorized year/month/day... etc. for string 
arguments (Teddy Choi via Eric Hanson)


Changes for Build #2483
[rhbutani] HIVE-5898 Make fetching of column statistics configurable (Prasanth 
Jayachandran via Harish Butani)


Changes for Build #2484
[brock] HIVE-5880 - (Rename HCatalog HBase Storage Handler artifact id) breaks 
packaging (Xuefu Zhang via Brock Noland)


Changes for Build #2485
[xuefu] HIVE-5866: Hive divide operator generates wrong results in certain 
cases (reviewed by Prasad)

[ehans] HIVE-5877: Implement vectorized support for IN as boolean-valued 
expression (Eric Hanson)


Changes for Build #2486
[ehans] HIVE-5895: vectorization handles division by zero differently from 
normal execution (Sergey Shelukhin via Eric Hanson)

[hashutosh] HIVE-5938 : Remove apache.mina dependency for test (Navis via 
Ashutosh Chauhan)

[xuefu] HIVE-5912: Show partition command doesn't support db.table (Yu Zhao via 
Xuefu)

[brock] HIVE-5906 - TestGenericUDFPower should use delta to compare doubles 
(Szehon Ho via Brock Noland)

[brock] HIVE-5855 - Add deprecated methods back to ColumnProjectionUtils (Brock 
Noland reviewed by Navis)

[brock] HIVE-5915 - Shade Kryo dependency (Brock Noland reviewed by Ashutosh 
Chauhan)


Changes for Build #2487
[hashutosh] HIVE-5916 : No need to aggregate statistics collected via counter 
mechanism (Ashutosh Chauhan via Navis)

[xuefu] HIVE-5947: Fix test failure in decimal_udf.q (reviewed by Brock)

[thejas] HIVE-5550 : Import fails for tables created with default text, 
sequence and orc file formats using HCatalog API (Sushanth Sowmyan via Thejas 
Nair)


Changes for Build #2488
[hashutosh] HIVE-5935 : hive.query.string is not provided to FetchTask (Navis 
via Ashutosh Chauhan)

[navis] HIVE-3455 : ANSI CORR(X,Y) is incorrect (Maxim Bolotin via Navis)

[hashutosh] HIVE-5921 : Better heuristics for worst case statistics estimates 
for join, limit and filter operator (Prasanth J via Harish Butani)

[rhbutani] HIVE-5899 NPE during explain extended with char/varchar columns 
(Jason Dere via Harish Butani)


Changes for Build #2489
[xuefu] HIVE-3181: getDatabaseMajor/Minor version does not return values 
(Szehon via Xuefu, reviewed by Navis)

[brock] HIVE-5641 - BeeLineOpts ignores Throwable (Brock Noland reviewed by 
Prasad and Thejas)

[hashutosh] HIVE-5909 : locate and instr throw 
java.nio.BufferUnderflowException when empty string as substring (Navis via 
Ashutosh Chauhan)

[hashutosh] HIVE-5686 : partition column type validation doesn't quite work for 
dates (Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5887 : metastore direct

[jira] [Commented] (HIVE-6034) vectorized % doesn't handle zeroes the same way as non-vectorized

[Eric Hanson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849619#comment-13849619
 ] 

Eric Hanson commented on HIVE-6034:
---

+1


> vectorized % doesn't handle zeroes the same way as non-vectorized
> -
>
> Key: HIVE-6034
> URL: https://issues.apache.org/jira/browse/HIVE-6034
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Minor
> Attachments: HIVE-6034.01.patch, HIVE-6034.patch
>
>
> % 0 is NULL, but if vectorized it's NaN



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6013) Supporting Quoted Identifiers in Column Names

[Eric Hanson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849616#comment-13849616
 ] 

Eric Hanson commented on HIVE-6013:
---

The spec sounds reasonable. I'm fine with it. I'm curious though, did you 
consider allowing square brackets to delimit an identifier, like SQL Server 
allows? E.g. create table sale([Shipment Date] date, ); That seems a tad 
more readable than the back-quotes.

> Supporting Quoted Identifiers in Column Names
> -
>
> Key: HIVE-6013
> URL: https://issues.apache.org/jira/browse/HIVE-6013
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Harish Butani
>Assignee: Harish Butani
> Fix For: 0.13.0
>
> Attachments: HIVE-6013.1.patch, HIVE-6013.2.patch, 
> QuotedIdentifier.html
>
>
> Hive's current behavior on Quoted Identifiers is different from the normal 
> interpretation. Quoted Identifier (using backticks) has a special 
> interpretation for Select expressions(as Regular Expressions). Have 
> documented current behavior and proposed a solution in attached doc.
> Summary of solution is:
> - Introduce 'standard' quoted identifiers for columns only. 
> - At the langauage level this is turned on by a flag.
> - At the metadata level we relax the constraint on column names.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-4256) JDBC2 HiveConnection does not use the specified database

[Prasad Mujumdar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849613#comment-13849613
 ] 

Prasad Mujumdar commented on HIVE-4256:
---

+1


> JDBC2 HiveConnection does not use the specified database
> 
>
> Key: HIVE-4256
> URL: https://issues.apache.org/jira/browse/HIVE-4256
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Affects Versions: 0.11.0
>Reporter: Chris Drome
>Assignee: Anandha L Ranganathan
> Attachments: HIVE-4256.1.patch, HIVE-4256.2.patch, HIVE-4256.3.patch, 
> HIVE-4256.4.patch, HIVE-4256.5.patch, HIVE-4256.6.patch, HIVE-4256.patch
>
>
> HiveConnection ignores the database specified in the connection string when 
> configuring the connection.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6034) vectorized % doesn't handle zeroes the same way as non-vectorized

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6034:
---

Attachment: HIVE-6034.01.patch

Address RB feedback about javadocs

> vectorized % doesn't handle zeroes the same way as non-vectorized
> -
>
> Key: HIVE-6034
> URL: https://issues.apache.org/jira/browse/HIVE-6034
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Minor
> Attachments: HIVE-6034.01.patch, HIVE-6034.patch
>
>
> % 0 is NULL, but if vectorized it's NaN



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 16269: HIVE-6034 vectorized % doesn't handle zeroes the same way as non-vectorized

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16269/
---

(Updated Dec. 16, 2013, 7:18 p.m.)


Review request for hive and Jitendra Pandey.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  ant/src/org/apache/hadoop/hive/ant/GenVectorCode.java a286024 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnDivideColumn.txt 218ab6f 
  ql/src/gen/vectorization/ExpressionTemplates/ScalarDivideColumn.txt c969b2f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/NullUtil.java 
e4ee372 
  ql/src/test/queries/clientpositive/vectorization_12.q 78af086 
  ql/src/test/queries/clientpositive/vectorization_14.q bc7730d 
  ql/src/test/results/clientpositive/vectorization_12.q.out 01e2506 
  ql/src/test/results/clientpositive/vectorization_14.q.out 247197a 

Diff: https://reviews.apache.org/r/16269/diff/


Testing
---


Thanks,

Sergey Shelukhin

Hive-trunk-hadoop2 - Build # 604 - Still Failing

[Apache Jenkins Server]

Changes for Build #570
[rhbutani] HIVE-5849 Improve the stats of operators based on heuristics in the 
absence of any column statistics (Prasanth Jayachandran via Harish Butani)

[hashutosh] HIVE-5793 : Update hive-default.xml.template for HIVE4002 (Navis 
via Ashutosh Chauhan)


Changes for Build #571
[navis] HIVE-4518 : Should be removed files (OptrStatsGroupByHook, etc.)

[navis] HIVE-5839 : BytesRefArrayWritable compareTo violates contract (Xuefu 
Zhang via Navis)

[navis] HIVE-4518 : Missing file (HiveFatalException)

[navis] HIVE-4518 : Counter Strike: Operation Operator (Gunther Hagleitner and 
Jason Dere via Navis)


Changes for Build #572
[brock] HIVE-4741 - Add Hive config API to modify the restrict list (Prasad 
Mujumdar, Navis via Brock Noland)


Changes for Build #573
[navis] HIVE-5827 : Incorrect location of logs for failed tests (Vikram Dixit K 
and Szehon Ho via Navis)

[thejas] HIVE-4485 : beeline prints null as empty strings (Thejas Nair reviewed 
by Ashutosh Chauhan)

[brock] HIVE-5704 - A couple of generic UDFs are not in the right 
folder/package (Xuefu Zhang via Brock Noland)

[brock] HIVE-5706 - Move a few numeric UDFs to generic implementations (Xuefu 
Zhang via Brock Noland)

[hashutosh] HIVE-5817 : column name to index mapping in VectorizationContext is 
broken (Remus Rusanu, Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5876 : Split elimination in ORC breaks for partitioned tables 
(Prasanth J via Ashutosh Chauhan)

[hashutosh] HIVE-5886 : [Refactor] Remove unused class JobCloseFeedback 
(Ashutosh Chauhan via Thejas Nair)

[brock] HIVE-5894 - Fix minor PTest2 issues (Brock Noland)


Changes for Build #574
[brock] HIVE-5755 - Fix hadoop2 execution environment Milestone 1 (Vikram Dixit 
K via Brock Noland)


Changes for Build #575
[xuefu] HIVE-5893: hive-schema-0.13.0.mysql.sql contains reference to 
nonexistent column (Carl via Xuefu)

[xuefu] HIVE-5684: Serde support for char (Jason via Xuefu)


Changes for Build #576

Changes for Build #577

Changes for Build #578

Changes for Build #579
[brock] HIVE-5441 - Async query execution doesn't return resultset status 
(Prasad Mujumdar via Thejas M Nair)

[brock] HIVE-5880 - Rename HCatalog HBase Storage Handler artifact id (Brock 
Noland reviewed by Prasad Mujumdar)


Changes for Build #580
[ehans] HIVE-5581: Implement vectorized year/month/day... etc. for string 
arguments (Teddy Choi via Eric Hanson)


Changes for Build #581
[rhbutani] HIVE-5898 Make fetching of column statistics configurable (Prasanth 
Jayachandran via Harish Butani)


Changes for Build #582
[brock] HIVE-5880 - (Rename HCatalog HBase Storage Handler artifact id) breaks 
packaging (Xuefu Zhang via Brock Noland)


Changes for Build #583
[xuefu] HIVE-5866: Hive divide operator generates wrong results in certain 
cases (reviewed by Prasad)

[ehans] HIVE-5877: Implement vectorized support for IN as boolean-valued 
expression (Eric Hanson)


Changes for Build #584
[thejas] HIVE-5550 : Import fails for tables created with default text, 
sequence and orc file formats using HCatalog API (Sushanth Sowmyan via Thejas 
Nair)

[ehans] HIVE-5895: vectorization handles division by zero differently from 
normal execution (Sergey Shelukhin via Eric Hanson)

[hashutosh] HIVE-5938 : Remove apache.mina dependency for test (Navis via 
Ashutosh Chauhan)

[xuefu] HIVE-5912: Show partition command doesn't support db.table (Yu Zhao via 
Xuefu)

[brock] HIVE-5906 - TestGenericUDFPower should use delta to compare doubles 
(Szehon Ho via Brock Noland)

[brock] HIVE-5855 - Add deprecated methods back to ColumnProjectionUtils (Brock 
Noland reviewed by Navis)

[brock] HIVE-5915 - Shade Kryo dependency (Brock Noland reviewed by Ashutosh 
Chauhan)


Changes for Build #585
[hashutosh] HIVE-5916 : No need to aggregate statistics collected via counter 
mechanism (Ashutosh Chauhan via Navis)

[xuefu] HIVE-5947: Fix test failure in decimal_udf.q (reviewed by Brock)


Changes for Build #586
[hashutosh] HIVE-5935 : hive.query.string is not provided to FetchTask (Navis 
via Ashutosh Chauhan)

[navis] HIVE-3455 : ANSI CORR(X,Y) is incorrect (Maxim Bolotin via Navis)

[hashutosh] HIVE-5921 : Better heuristics for worst case statistics estimates 
for join, limit and filter operator (Prasanth J via Harish Butani)

[rhbutani] HIVE-5899 NPE during explain extended with char/varchar columns 
(Jason Dere via Harish Butani)


Changes for Build #587
[xuefu] HIVE-3181: getDatabaseMajor/Minor version does not return values 
(Szehon via Xuefu, reviewed by Navis)

[brock] HIVE-5641 - BeeLineOpts ignores Throwable (Brock Noland reviewed by 
Prasad and Thejas)

[hashutosh] HIVE-5909 : locate and instr throw 
java.nio.BufferUnderflowException when empty string as substring (Navis via 
Ashutosh Chauhan)

[hashutosh] HIVE-5686 : partition column type validation doesn't quite work for 
dates (Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5887 : metastore direct sql doesn't work with oracle (Sergey 
Shel

[jira] [Commented] (HIVE-6021) Problem in GroupByOperator for handling distinct aggrgations

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849593#comment-13849593
 ] 

Xuefu Zhang commented on HIVE-6021:
---

[~sunrui] Thanks for the update. Could you please attach your updated patch 
here? The review board has the latest thought.

> Problem in GroupByOperator for handling distinct aggrgations
> 
>
> Key: HIVE-6021
> URL: https://issues.apache.org/jira/browse/HIVE-6021
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.12.0
>Reporter: Sun Rui
>Assignee: Sun Rui
> Attachments: HIVE-6021.1.patch
>
>
> Use the following test case with HIVE 0.12:
> {code:sql}
> create table src(key int, value string);
> load data local inpath 'src/data/files/kv1.txt' overwrite into table src;
> set hive.map.aggr=false; 
> select count(key),count(distinct value) from src group by key;
> {code}
> We will get an ArrayIndexOutOfBoundsException from GroupByOperator:
> {code}
> java.lang.RuntimeException: Error in configuring object
>   at 
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>   at 
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>   at 
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>   at 
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:485)
>   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
>   at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:260)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>   ... 5 more
> Caused by: java.lang.RuntimeException: Reduce operator initialization failed
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:159)
>   ... 10 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.initializeOp(GroupByOperator.java:281)
>   at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:377)
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:152)
>   ... 10 more
> {code}
> explain select count(key),count(distinct value) from src group by key;
> {code}
> STAGE PLANS:
>   Stage: Stage-1
> Map Reduce
>   Alias -> Map Operator Tree:
> src 
>   TableScan
> alias: src
> Select Operator
>   expressions:
> expr: key
> type: int
> expr: value
> type: string
>   outputColumnNames: key, value
>   Reduce Output Operator
> key expressions:
>   expr: key
>   type: int
>   expr: value
>   type: string
> sort order: ++
> Map-reduce partition columns:
>   expr: key
>   type: int
> tag: -1
>   Reduce Operator Tree:
> Group By Operator
>   aggregations:
> expr: count(KEY._col0)   // The parameter causes this problem
>^^^
> expr: count(DISTINCT KEY._col1:0._col0)
>   bucketGroup: false
>   keys:
> expr: KEY._col0
> type: int
>   mode: complete
>   outputColumnNames: _col0, _col1, _col2
>   Select Operator
> expressions:
>   expr: _col1
>   type: bigint
>   expr: _col2
>   type: bigint
> outputColumnNames: _col0, _col1
> File Output Operator
>   compressed: false
>   GlobalTableId: 0
>   table:
>   input format: org.apache.hadoop.mapred.TextInputFormat
>   output format: 
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
>   Stage: Stage-0
> Fetch Operator
>   limit: -1
> {code}
> The root cause is within GroupByOperator.initializeOp(). The method forgets 
> to handle the case:
> For a query has distinct aggregations, there is an aggregation function has a 
> parameter which is a groupby key column but not distinct key column.
> {code}
> if (u

[jira] [Updated] (HIVE-6013) Supporting Quoted Identifiers in Column Names

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6013:


Attachment: HIVE-6013.2.patch

> Supporting Quoted Identifiers in Column Names
> -
>
> Key: HIVE-6013
> URL: https://issues.apache.org/jira/browse/HIVE-6013
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Harish Butani
>Assignee: Harish Butani
> Fix For: 0.13.0
>
> Attachments: HIVE-6013.1.patch, HIVE-6013.2.patch, 
> QuotedIdentifier.html
>
>
> Hive's current behavior on Quoted Identifiers is different from the normal 
> interpretation. Quoted Identifier (using backticks) has a special 
> interpretation for Select expressions(as Regular Expressions). Have 
> documented current behavior and proposed a solution in attached doc.
> Summary of solution is:
> - Introduce 'standard' quoted identifiers for columns only. 
> - At the langauage level this is turned on by a flag.
> - At the metadata level we relax the constraint on column names.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6013) Supporting Quoted Identifiers in Column Names

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6013:


Status: Patch Available  (was: Open)

> Supporting Quoted Identifiers in Column Names
> -
>
> Key: HIVE-6013
> URL: https://issues.apache.org/jira/browse/HIVE-6013
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Harish Butani
>Assignee: Harish Butani
> Fix For: 0.13.0
>
> Attachments: HIVE-6013.1.patch, HIVE-6013.2.patch, 
> QuotedIdentifier.html
>
>
> Hive's current behavior on Quoted Identifiers is different from the normal 
> interpretation. Quoted Identifier (using backticks) has a special 
> interpretation for Select expressions(as Regular Expressions). Have 
> documented current behavior and proposed a solution in attached doc.
> Summary of solution is:
> - Introduce 'standard' quoted identifiers for columns only. 
> - At the langauage level this is turned on by a flag.
> - At the metadata level we relax the constraint on column names.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6013) Supporting Quoted Identifiers in Column Names

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6013:


Status: Open  (was: Patch Available)

> Supporting Quoted Identifiers in Column Names
> -
>
> Key: HIVE-6013
> URL: https://issues.apache.org/jira/browse/HIVE-6013
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Harish Butani
>Assignee: Harish Butani
> Fix For: 0.13.0
>
> Attachments: HIVE-6013.1.patch, HIVE-6013.2.patch, 
> QuotedIdentifier.html
>
>
> Hive's current behavior on Quoted Identifiers is different from the normal 
> interpretation. Quoted Identifier (using backticks) has a special 
> interpretation for Select expressions(as Regular Expressions). Have 
> documented current behavior and proposed a solution in attached doc.
> Summary of solution is:
> - Introduce 'standard' quoted identifiers for columns only. 
> - At the langauage level this is turned on by a flag.
> - At the metadata level we relax the constraint on column names.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Review Request 16299: HIVE-6013: Supporting Quoted Identifiers in Column Names

[Harish Butani]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16299/
---

Review request for hive, Ashutosh Chauhan and Alan Gates.


Bugs: HIVE-6013
https://issues.apache.org/jira/browse/HIVE-6013


Repository: hive-git


Description
---

Hive's current behavior on Quoted Identifiers is different from the normal 
interpretation. Quoted Identifier (using backticks) has a special 
interpretation for Select expressions(as Regular Expressions). Have documented 
current behavior and proposed a solution in attached doc.
Summary of solution is:
Introduce 'standard' quoted identifiers for columns only.
At the langauage level this is turned on by a flag.
At the metadata level we relax the constraint on column names.


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fa3e048 
  itests/qtest/pom.xml 8c249a0 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
3deed45 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveUtils.java eb26e7f 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 321759b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 17e6aad 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 366b714 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 1e6826f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ace1df9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/UnparseTranslator.java 8fe2262 
  ql/src/test/queries/clientnegative/invalid_columns.q f8be8c8 
  ql/src/test/queries/clientpositive/quotedid_alter.q PRE-CREATION 
  ql/src/test/queries/clientpositive/quotedid_basic.q PRE-CREATION 
  ql/src/test/queries/clientpositive/quotedid_partition.q PRE-CREATION 
  ql/src/test/queries/clientpositive/quotedid_skew.q PRE-CREATION 
  ql/src/test/queries/clientpositive/quotedid_smb.q PRE-CREATION 
  ql/src/test/queries/clientpositive/quotedid_tblproperty.q PRE-CREATION 
  ql/src/test/results/clientnegative/invalid_columns.q.out 3311b0a 
  ql/src/test/results/clientpositive/quotedid_alter.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/quotedid_basic.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/quotedid_partition.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/quotedid_skew.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/quotedid_smb.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/quotedid_tblproperty.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/16299/diff/


Testing
---

added new tests for create, alter, delete, query with columns containing 
special characters.
Tests start with quotedid


Thanks,

Harish Butani

[jira] [Updated] (HIVE-5975) [WebHCat] templeton mapreduce job failed if provide "define" parameters

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-5975:


   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

Patch committed to trunk.
Thanks for the patch Shanyu, and thanks for the review Eugene!


> [WebHCat] templeton mapreduce job failed if provide "define" parameters
> ---
>
> Key: HIVE-5975
> URL: https://issues.apache.org/jira/browse/HIVE-5975
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.12.0, 0.13.0
>Reporter: shanyu zhao
>Assignee: shanyu zhao
> Fix For: 0.13.0
>
> Attachments: hive-5975.2.patch, hive-5975.patch
>
>
> Trying to submit a mapreduce job through templeton failed:
> curl -k -u user:pass -d user.name=user -d define=JobName=MRPiJob -d class=pi 
> -d arg=16 -d arg=100 -d jar="hadoop-mapreduce-examples.jar" 
> https://xxx/templeton/v1/mapreduce/jar
> The error message is:
> "Usage: org.apache.hadoop.examples.QuasiMonteCarlo  
>  Generic options supported are
>  -conf  specify an application configuration file
>  -D  use value for given property
>  -fs  specify a namenode
>  -jt  specify a job tracker
>  -files  specify comma separated files to be 
> copied to the map reduce cluster
>  -libjars  specify comma separated jar files to 
> include in the classpath.
>  -archives  specify comma separated 
> archives to be unarchived on the compute machines.
> The general command line syntax is
>  bin/hadoop command [genericOptions] [commandOptions]
> templeton: job failed with exit code 2"
> Note that if we remove the "define" parameter it works fine.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6034) vectorized % doesn't handle zeroes the same way as non-vectorized

[Eric Hanson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849411#comment-13849411
 ] 

Eric Hanson commented on HIVE-6034:
---

Please see my comments on ReviewBoard

> vectorized % doesn't handle zeroes the same way as non-vectorized
> -
>
> Key: HIVE-6034
> URL: https://issues.apache.org/jira/browse/HIVE-6034
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Minor
> Attachments: HIVE-6034.patch
>
>
> % 0 is NULL, but if vectorized it's NaN



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 16269: HIVE-6034 vectorized % doesn't handle zeroes the same way as non-vectorized

[Eric Hanson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16269/#review30454
---



ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/NullUtil.java


Instead of NaN say "the null data value". NaN is only for double.



ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/NullUtil.java


Can we remove this statement about infinities? The code does not check for 
infinities, just 0 denoms. The point here is that zero divide is supposed to 
yield NULL.

It is worth thinking about whether infinity, NaN, etc. can be found in the 
stored data and if that causes any correctness issues elsewhere. 



ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/NullUtil.java


Again, NaN is only for double, not long


- Eric Hanson


On Dec. 14, 2013, 1:38 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16269/
> ---
> 
> (Updated Dec. 14, 2013, 1:38 a.m.)
> 
> 
> Review request for hive and Jitendra Pandey.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   ant/src/org/apache/hadoop/hive/ant/GenVectorCode.java a286024 
>   ql/src/gen/vectorization/ExpressionTemplates/ColumnDivideColumn.txt 218ab6f 
>   ql/src/gen/vectorization/ExpressionTemplates/ScalarDivideColumn.txt c969b2f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/NullUtil.java 
> e4ee372 
>   ql/src/test/queries/clientpositive/vectorization_12.q 78af086 
>   ql/src/test/queries/clientpositive/vectorization_14.q bc7730d 
>   ql/src/test/results/clientpositive/vectorization_12.q.out 01e2506 
>   ql/src/test/results/clientpositive/vectorization_14.q.out 247197a 
> 
> Diff: https://reviews.apache.org/r/16269/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: adding ANSI flag for hive

[Alan Gates]

A couple of thoughts on this:

1) If we did this I think we should have one flag, not many.  As Thejas points 
out, your test matrix goes insane when you have too many flags and hence things 
don't get properly tested.

2) We could do this in an incremental way, where we create this new ANSI flag 
and are clear with users that for a while this will be evolving.  That is, as 
we find new issues with data types, semantics, whatever, we will continue to 
change the behavior of this flag.  At some point in the future (as Thejas 
suggests, at a 1.0 release) we could make this the default behavior.  This 
avoids having to do a full sweep now and find everything that we want to change 
and make ANSI compliant and living with whatever we miss.

Alan.

On Dec 11, 2013, at 5:14 PM, Thejas Nair wrote:

> Having too many configs complicates things for the user, and also
> complicates the code, and you also end up having many untested
> combinations of config flags.
> I think we should identify a bunch of non compatible changes that we
> think are important, fix it in a branch and make a major version
> release (say 1.x).
> 
> This is also related to HIVE-5875, where there is a discussion on
> switching the defaults for some of the configs to more desirable
> values, but non backward compatible values.
> 
> On Wed, Dec 11, 2013 at 4:33 PM, Sergey Shelukhin
>  wrote:
>> Hi.
>> 
>> There's recently been some discussion about data type changes in Hive
>> (double to decimal), and result changes for special cases like division by
>> zero, etc., to bring it in compliance with MySQL (that's what JIRAs use an
>> example; I am assuming ANSI SQL is meant).
>> The latter are non-controversial (I guess), but for the former, performance
>> may suffer and/or backward compat may be broken if Hive is brought in
>> compliance.
>> If fuller ANSI compat is sought in the future, there may be some even
>> hairier issues such as double-quoted identifiers.
>> 
>> In light of that, and also following MySQL, I wonder if we should add a
>> flag, or set of flags, to HIVE to be able to force ANSI compliance.
>> When this/ese flag/s is/are not set, for example, int/int division could
>> return double for backward compat/perf, vectorization can skip the special
>> case handling for division by zero/etc., etc.
>> Wdyt?
>> 
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
> 
> -- 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to 
> which it is addressed and may contain information that is confidential, 
> privileged and exempt from disclosure under applicable law. If the reader 
> of this message is not the intended recipient, you are hereby notified that 
> any printing, copying, dissemination, distribution, disclosure or 
> forwarding of this communication is strictly prohibited. If you have 
> received this communication in error, please contact the sender immediately 
> and delete it from your system. Thank You.


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Hive-trunk-h0.21 - Build # 2504 - Still Failing

[Apache Jenkins Server]

Changes for Build #2471
[rhbutani] HIVE-5849 Improve the stats of operators based on heuristics in the 
absence of any column statistics (Prasanth Jayachandran via Harish Butani)

[hashutosh] HIVE-5793 : Update hive-default.xml.template for HIVE4002 (Navis 
via Ashutosh Chauhan)


Changes for Build #2472
[navis] HIVE-4518 : Should be removed files (OptrStatsGroupByHook, etc.)

[navis] HIVE-5839 : BytesRefArrayWritable compareTo violates contract (Xuefu 
Zhang via Navis)

[navis] HIVE-4518 : Missing file (HiveFatalException)

[navis] HIVE-4518 : Counter Strike: Operation Operator (Gunther Hagleitner and 
Jason Dere via Navis)


Changes for Build #2473
[brock] HIVE-4741 - Add Hive config API to modify the restrict list (Prasad 
Mujumdar, Navis via Brock Noland)


Changes for Build #2474
[navis] HIVE-5827 : Incorrect location of logs for failed tests (Vikram Dixit K 
and Szehon Ho via Navis)

[thejas] HIVE-4485 : beeline prints null as empty strings (Thejas Nair reviewed 
by Ashutosh Chauhan)

[brock] HIVE-5704 - A couple of generic UDFs are not in the right 
folder/package (Xuefu Zhang via Brock Noland)

[brock] HIVE-5706 - Move a few numeric UDFs to generic implementations (Xuefu 
Zhang via Brock Noland)

[hashutosh] HIVE-5817 : column name to index mapping in VectorizationContext is 
broken (Remus Rusanu, Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5876 : Split elimination in ORC breaks for partitioned tables 
(Prasanth J via Ashutosh Chauhan)

[hashutosh] HIVE-5886 : [Refactor] Remove unused class JobCloseFeedback 
(Ashutosh Chauhan via Thejas Nair)

[brock] HIVE-5894 - Fix minor PTest2 issues (Brock Noland)


Changes for Build #2475
[brock] HIVE-5755 - Fix hadoop2 execution environment Milestone 1 (Vikram Dixit 
K via Brock Noland)


Changes for Build #2476
[xuefu] HIVE-5893: hive-schema-0.13.0.mysql.sql contains reference to 
nonexistent column (Carl via Xuefu)

[xuefu] HIVE-5684: Serde support for char (Jason via Xuefu)


Changes for Build #2477

Changes for Build #2478

Changes for Build #2479

Changes for Build #2480
[brock] HIVE-5441 - Async query execution doesn't return resultset status 
(Prasad Mujumdar via Thejas M Nair)

[brock] HIVE-5880 - Rename HCatalog HBase Storage Handler artifact id (Brock 
Noland reviewed by Prasad Mujumdar)


Changes for Build #2481

Changes for Build #2482
[ehans] HIVE-5581: Implement vectorized year/month/day... etc. for string 
arguments (Teddy Choi via Eric Hanson)


Changes for Build #2483
[rhbutani] HIVE-5898 Make fetching of column statistics configurable (Prasanth 
Jayachandran via Harish Butani)


Changes for Build #2484
[brock] HIVE-5880 - (Rename HCatalog HBase Storage Handler artifact id) breaks 
packaging (Xuefu Zhang via Brock Noland)


Changes for Build #2485
[xuefu] HIVE-5866: Hive divide operator generates wrong results in certain 
cases (reviewed by Prasad)

[ehans] HIVE-5877: Implement vectorized support for IN as boolean-valued 
expression (Eric Hanson)


Changes for Build #2486
[ehans] HIVE-5895: vectorization handles division by zero differently from 
normal execution (Sergey Shelukhin via Eric Hanson)

[hashutosh] HIVE-5938 : Remove apache.mina dependency for test (Navis via 
Ashutosh Chauhan)

[xuefu] HIVE-5912: Show partition command doesn't support db.table (Yu Zhao via 
Xuefu)

[brock] HIVE-5906 - TestGenericUDFPower should use delta to compare doubles 
(Szehon Ho via Brock Noland)

[brock] HIVE-5855 - Add deprecated methods back to ColumnProjectionUtils (Brock 
Noland reviewed by Navis)

[brock] HIVE-5915 - Shade Kryo dependency (Brock Noland reviewed by Ashutosh 
Chauhan)


Changes for Build #2487
[hashutosh] HIVE-5916 : No need to aggregate statistics collected via counter 
mechanism (Ashutosh Chauhan via Navis)

[xuefu] HIVE-5947: Fix test failure in decimal_udf.q (reviewed by Brock)

[thejas] HIVE-5550 : Import fails for tables created with default text, 
sequence and orc file formats using HCatalog API (Sushanth Sowmyan via Thejas 
Nair)


Changes for Build #2488
[hashutosh] HIVE-5935 : hive.query.string is not provided to FetchTask (Navis 
via Ashutosh Chauhan)

[navis] HIVE-3455 : ANSI CORR(X,Y) is incorrect (Maxim Bolotin via Navis)

[hashutosh] HIVE-5921 : Better heuristics for worst case statistics estimates 
for join, limit and filter operator (Prasanth J via Harish Butani)

[rhbutani] HIVE-5899 NPE during explain extended with char/varchar columns 
(Jason Dere via Harish Butani)


Changes for Build #2489
[xuefu] HIVE-3181: getDatabaseMajor/Minor version does not return values 
(Szehon via Xuefu, reviewed by Navis)

[brock] HIVE-5641 - BeeLineOpts ignores Throwable (Brock Noland reviewed by 
Prasad and Thejas)

[hashutosh] HIVE-5909 : locate and instr throw 
java.nio.BufferUnderflowException when empty string as substring (Navis via 
Ashutosh Chauhan)

[hashutosh] HIVE-5686 : partition column type validation doesn't quite work for 
dates (Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5887 : metastore direct

Hive-trunk-hadoop2 - Build # 603 - Still Failing

[Apache Jenkins Server]

Changes for Build #570
[rhbutani] HIVE-5849 Improve the stats of operators based on heuristics in the 
absence of any column statistics (Prasanth Jayachandran via Harish Butani)

[hashutosh] HIVE-5793 : Update hive-default.xml.template for HIVE4002 (Navis 
via Ashutosh Chauhan)


Changes for Build #571
[navis] HIVE-4518 : Should be removed files (OptrStatsGroupByHook, etc.)

[navis] HIVE-5839 : BytesRefArrayWritable compareTo violates contract (Xuefu 
Zhang via Navis)

[navis] HIVE-4518 : Missing file (HiveFatalException)

[navis] HIVE-4518 : Counter Strike: Operation Operator (Gunther Hagleitner and 
Jason Dere via Navis)


Changes for Build #572
[brock] HIVE-4741 - Add Hive config API to modify the restrict list (Prasad 
Mujumdar, Navis via Brock Noland)


Changes for Build #573
[navis] HIVE-5827 : Incorrect location of logs for failed tests (Vikram Dixit K 
and Szehon Ho via Navis)

[thejas] HIVE-4485 : beeline prints null as empty strings (Thejas Nair reviewed 
by Ashutosh Chauhan)

[brock] HIVE-5704 - A couple of generic UDFs are not in the right 
folder/package (Xuefu Zhang via Brock Noland)

[brock] HIVE-5706 - Move a few numeric UDFs to generic implementations (Xuefu 
Zhang via Brock Noland)

[hashutosh] HIVE-5817 : column name to index mapping in VectorizationContext is 
broken (Remus Rusanu, Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5876 : Split elimination in ORC breaks for partitioned tables 
(Prasanth J via Ashutosh Chauhan)

[hashutosh] HIVE-5886 : [Refactor] Remove unused class JobCloseFeedback 
(Ashutosh Chauhan via Thejas Nair)

[brock] HIVE-5894 - Fix minor PTest2 issues (Brock Noland)


Changes for Build #574
[brock] HIVE-5755 - Fix hadoop2 execution environment Milestone 1 (Vikram Dixit 
K via Brock Noland)


Changes for Build #575
[xuefu] HIVE-5893: hive-schema-0.13.0.mysql.sql contains reference to 
nonexistent column (Carl via Xuefu)

[xuefu] HIVE-5684: Serde support for char (Jason via Xuefu)


Changes for Build #576

Changes for Build #577

Changes for Build #578

Changes for Build #579
[brock] HIVE-5441 - Async query execution doesn't return resultset status 
(Prasad Mujumdar via Thejas M Nair)

[brock] HIVE-5880 - Rename HCatalog HBase Storage Handler artifact id (Brock 
Noland reviewed by Prasad Mujumdar)


Changes for Build #580
[ehans] HIVE-5581: Implement vectorized year/month/day... etc. for string 
arguments (Teddy Choi via Eric Hanson)


Changes for Build #581
[rhbutani] HIVE-5898 Make fetching of column statistics configurable (Prasanth 
Jayachandran via Harish Butani)


Changes for Build #582
[brock] HIVE-5880 - (Rename HCatalog HBase Storage Handler artifact id) breaks 
packaging (Xuefu Zhang via Brock Noland)


Changes for Build #583
[xuefu] HIVE-5866: Hive divide operator generates wrong results in certain 
cases (reviewed by Prasad)

[ehans] HIVE-5877: Implement vectorized support for IN as boolean-valued 
expression (Eric Hanson)


Changes for Build #584
[thejas] HIVE-5550 : Import fails for tables created with default text, 
sequence and orc file formats using HCatalog API (Sushanth Sowmyan via Thejas 
Nair)

[ehans] HIVE-5895: vectorization handles division by zero differently from 
normal execution (Sergey Shelukhin via Eric Hanson)

[hashutosh] HIVE-5938 : Remove apache.mina dependency for test (Navis via 
Ashutosh Chauhan)

[xuefu] HIVE-5912: Show partition command doesn't support db.table (Yu Zhao via 
Xuefu)

[brock] HIVE-5906 - TestGenericUDFPower should use delta to compare doubles 
(Szehon Ho via Brock Noland)

[brock] HIVE-5855 - Add deprecated methods back to ColumnProjectionUtils (Brock 
Noland reviewed by Navis)

[brock] HIVE-5915 - Shade Kryo dependency (Brock Noland reviewed by Ashutosh 
Chauhan)


Changes for Build #585
[hashutosh] HIVE-5916 : No need to aggregate statistics collected via counter 
mechanism (Ashutosh Chauhan via Navis)

[xuefu] HIVE-5947: Fix test failure in decimal_udf.q (reviewed by Brock)


Changes for Build #586
[hashutosh] HIVE-5935 : hive.query.string is not provided to FetchTask (Navis 
via Ashutosh Chauhan)

[navis] HIVE-3455 : ANSI CORR(X,Y) is incorrect (Maxim Bolotin via Navis)

[hashutosh] HIVE-5921 : Better heuristics for worst case statistics estimates 
for join, limit and filter operator (Prasanth J via Harish Butani)

[rhbutani] HIVE-5899 NPE during explain extended with char/varchar columns 
(Jason Dere via Harish Butani)


Changes for Build #587
[xuefu] HIVE-3181: getDatabaseMajor/Minor version does not return values 
(Szehon via Xuefu, reviewed by Navis)

[brock] HIVE-5641 - BeeLineOpts ignores Throwable (Brock Noland reviewed by 
Prasad and Thejas)

[hashutosh] HIVE-5909 : locate and instr throw 
java.nio.BufferUnderflowException when empty string as substring (Navis via 
Ashutosh Chauhan)

[hashutosh] HIVE-5686 : partition column type validation doesn't quite work for 
dates (Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5887 : metastore direct sql doesn't work with oracle (Sergey 
Shel

Hive-trunk-h0.21 - Build # 2503 - Still Failing

[Apache Jenkins Server]

Changes for Build #2471
[rhbutani] HIVE-5849 Improve the stats of operators based on heuristics in the 
absence of any column statistics (Prasanth Jayachandran via Harish Butani)

[hashutosh] HIVE-5793 : Update hive-default.xml.template for HIVE4002 (Navis 
via Ashutosh Chauhan)


Changes for Build #2472
[navis] HIVE-4518 : Should be removed files (OptrStatsGroupByHook, etc.)

[navis] HIVE-5839 : BytesRefArrayWritable compareTo violates contract (Xuefu 
Zhang via Navis)

[navis] HIVE-4518 : Missing file (HiveFatalException)

[navis] HIVE-4518 : Counter Strike: Operation Operator (Gunther Hagleitner and 
Jason Dere via Navis)


Changes for Build #2473
[brock] HIVE-4741 - Add Hive config API to modify the restrict list (Prasad 
Mujumdar, Navis via Brock Noland)


Changes for Build #2474
[navis] HIVE-5827 : Incorrect location of logs for failed tests (Vikram Dixit K 
and Szehon Ho via Navis)

[thejas] HIVE-4485 : beeline prints null as empty strings (Thejas Nair reviewed 
by Ashutosh Chauhan)

[brock] HIVE-5704 - A couple of generic UDFs are not in the right 
folder/package (Xuefu Zhang via Brock Noland)

[brock] HIVE-5706 - Move a few numeric UDFs to generic implementations (Xuefu 
Zhang via Brock Noland)

[hashutosh] HIVE-5817 : column name to index mapping in VectorizationContext is 
broken (Remus Rusanu, Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5876 : Split elimination in ORC breaks for partitioned tables 
(Prasanth J via Ashutosh Chauhan)

[hashutosh] HIVE-5886 : [Refactor] Remove unused class JobCloseFeedback 
(Ashutosh Chauhan via Thejas Nair)

[brock] HIVE-5894 - Fix minor PTest2 issues (Brock Noland)


Changes for Build #2475
[brock] HIVE-5755 - Fix hadoop2 execution environment Milestone 1 (Vikram Dixit 
K via Brock Noland)


Changes for Build #2476
[xuefu] HIVE-5893: hive-schema-0.13.0.mysql.sql contains reference to 
nonexistent column (Carl via Xuefu)

[xuefu] HIVE-5684: Serde support for char (Jason via Xuefu)


Changes for Build #2477

Changes for Build #2478

Changes for Build #2479

Changes for Build #2480
[brock] HIVE-5441 - Async query execution doesn't return resultset status 
(Prasad Mujumdar via Thejas M Nair)

[brock] HIVE-5880 - Rename HCatalog HBase Storage Handler artifact id (Brock 
Noland reviewed by Prasad Mujumdar)


Changes for Build #2481

Changes for Build #2482
[ehans] HIVE-5581: Implement vectorized year/month/day... etc. for string 
arguments (Teddy Choi via Eric Hanson)


Changes for Build #2483
[rhbutani] HIVE-5898 Make fetching of column statistics configurable (Prasanth 
Jayachandran via Harish Butani)


Changes for Build #2484
[brock] HIVE-5880 - (Rename HCatalog HBase Storage Handler artifact id) breaks 
packaging (Xuefu Zhang via Brock Noland)


Changes for Build #2485
[xuefu] HIVE-5866: Hive divide operator generates wrong results in certain 
cases (reviewed by Prasad)

[ehans] HIVE-5877: Implement vectorized support for IN as boolean-valued 
expression (Eric Hanson)


Changes for Build #2486
[ehans] HIVE-5895: vectorization handles division by zero differently from 
normal execution (Sergey Shelukhin via Eric Hanson)

[hashutosh] HIVE-5938 : Remove apache.mina dependency for test (Navis via 
Ashutosh Chauhan)

[xuefu] HIVE-5912: Show partition command doesn't support db.table (Yu Zhao via 
Xuefu)

[brock] HIVE-5906 - TestGenericUDFPower should use delta to compare doubles 
(Szehon Ho via Brock Noland)

[brock] HIVE-5855 - Add deprecated methods back to ColumnProjectionUtils (Brock 
Noland reviewed by Navis)

[brock] HIVE-5915 - Shade Kryo dependency (Brock Noland reviewed by Ashutosh 
Chauhan)


Changes for Build #2487
[hashutosh] HIVE-5916 : No need to aggregate statistics collected via counter 
mechanism (Ashutosh Chauhan via Navis)

[xuefu] HIVE-5947: Fix test failure in decimal_udf.q (reviewed by Brock)

[thejas] HIVE-5550 : Import fails for tables created with default text, 
sequence and orc file formats using HCatalog API (Sushanth Sowmyan via Thejas 
Nair)


Changes for Build #2488
[hashutosh] HIVE-5935 : hive.query.string is not provided to FetchTask (Navis 
via Ashutosh Chauhan)

[navis] HIVE-3455 : ANSI CORR(X,Y) is incorrect (Maxim Bolotin via Navis)

[hashutosh] HIVE-5921 : Better heuristics for worst case statistics estimates 
for join, limit and filter operator (Prasanth J via Harish Butani)

[rhbutani] HIVE-5899 NPE during explain extended with char/varchar columns 
(Jason Dere via Harish Butani)


Changes for Build #2489
[xuefu] HIVE-3181: getDatabaseMajor/Minor version does not return values 
(Szehon via Xuefu, reviewed by Navis)

[brock] HIVE-5641 - BeeLineOpts ignores Throwable (Brock Noland reviewed by 
Prasad and Thejas)

[hashutosh] HIVE-5909 : locate and instr throw 
java.nio.BufferUnderflowException when empty string as substring (Navis via 
Ashutosh Chauhan)

[hashutosh] HIVE-5686 : partition column type validation doesn't quite work for 
dates (Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5887 : metastore direct

Hive-trunk-hadoop2 - Build # 602 - Still Failing

[Apache Jenkins Server]

Changes for Build #570
[rhbutani] HIVE-5849 Improve the stats of operators based on heuristics in the 
absence of any column statistics (Prasanth Jayachandran via Harish Butani)

[hashutosh] HIVE-5793 : Update hive-default.xml.template for HIVE4002 (Navis 
via Ashutosh Chauhan)


Changes for Build #571
[navis] HIVE-4518 : Should be removed files (OptrStatsGroupByHook, etc.)

[navis] HIVE-5839 : BytesRefArrayWritable compareTo violates contract (Xuefu 
Zhang via Navis)

[navis] HIVE-4518 : Missing file (HiveFatalException)

[navis] HIVE-4518 : Counter Strike: Operation Operator (Gunther Hagleitner and 
Jason Dere via Navis)


Changes for Build #572
[brock] HIVE-4741 - Add Hive config API to modify the restrict list (Prasad 
Mujumdar, Navis via Brock Noland)


Changes for Build #573
[navis] HIVE-5827 : Incorrect location of logs for failed tests (Vikram Dixit K 
and Szehon Ho via Navis)

[thejas] HIVE-4485 : beeline prints null as empty strings (Thejas Nair reviewed 
by Ashutosh Chauhan)

[brock] HIVE-5704 - A couple of generic UDFs are not in the right 
folder/package (Xuefu Zhang via Brock Noland)

[brock] HIVE-5706 - Move a few numeric UDFs to generic implementations (Xuefu 
Zhang via Brock Noland)

[hashutosh] HIVE-5817 : column name to index mapping in VectorizationContext is 
broken (Remus Rusanu, Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5876 : Split elimination in ORC breaks for partitioned tables 
(Prasanth J via Ashutosh Chauhan)

[hashutosh] HIVE-5886 : [Refactor] Remove unused class JobCloseFeedback 
(Ashutosh Chauhan via Thejas Nair)

[brock] HIVE-5894 - Fix minor PTest2 issues (Brock Noland)


Changes for Build #574
[brock] HIVE-5755 - Fix hadoop2 execution environment Milestone 1 (Vikram Dixit 
K via Brock Noland)


Changes for Build #575
[xuefu] HIVE-5893: hive-schema-0.13.0.mysql.sql contains reference to 
nonexistent column (Carl via Xuefu)

[xuefu] HIVE-5684: Serde support for char (Jason via Xuefu)


Changes for Build #576

Changes for Build #577

Changes for Build #578

Changes for Build #579
[brock] HIVE-5441 - Async query execution doesn't return resultset status 
(Prasad Mujumdar via Thejas M Nair)

[brock] HIVE-5880 - Rename HCatalog HBase Storage Handler artifact id (Brock 
Noland reviewed by Prasad Mujumdar)


Changes for Build #580
[ehans] HIVE-5581: Implement vectorized year/month/day... etc. for string 
arguments (Teddy Choi via Eric Hanson)


Changes for Build #581
[rhbutani] HIVE-5898 Make fetching of column statistics configurable (Prasanth 
Jayachandran via Harish Butani)


Changes for Build #582
[brock] HIVE-5880 - (Rename HCatalog HBase Storage Handler artifact id) breaks 
packaging (Xuefu Zhang via Brock Noland)


Changes for Build #583
[xuefu] HIVE-5866: Hive divide operator generates wrong results in certain 
cases (reviewed by Prasad)

[ehans] HIVE-5877: Implement vectorized support for IN as boolean-valued 
expression (Eric Hanson)


Changes for Build #584
[thejas] HIVE-5550 : Import fails for tables created with default text, 
sequence and orc file formats using HCatalog API (Sushanth Sowmyan via Thejas 
Nair)

[ehans] HIVE-5895: vectorization handles division by zero differently from 
normal execution (Sergey Shelukhin via Eric Hanson)

[hashutosh] HIVE-5938 : Remove apache.mina dependency for test (Navis via 
Ashutosh Chauhan)

[xuefu] HIVE-5912: Show partition command doesn't support db.table (Yu Zhao via 
Xuefu)

[brock] HIVE-5906 - TestGenericUDFPower should use delta to compare doubles 
(Szehon Ho via Brock Noland)

[brock] HIVE-5855 - Add deprecated methods back to ColumnProjectionUtils (Brock 
Noland reviewed by Navis)

[brock] HIVE-5915 - Shade Kryo dependency (Brock Noland reviewed by Ashutosh 
Chauhan)


Changes for Build #585
[hashutosh] HIVE-5916 : No need to aggregate statistics collected via counter 
mechanism (Ashutosh Chauhan via Navis)

[xuefu] HIVE-5947: Fix test failure in decimal_udf.q (reviewed by Brock)


Changes for Build #586
[hashutosh] HIVE-5935 : hive.query.string is not provided to FetchTask (Navis 
via Ashutosh Chauhan)

[navis] HIVE-3455 : ANSI CORR(X,Y) is incorrect (Maxim Bolotin via Navis)

[hashutosh] HIVE-5921 : Better heuristics for worst case statistics estimates 
for join, limit and filter operator (Prasanth J via Harish Butani)

[rhbutani] HIVE-5899 NPE during explain extended with char/varchar columns 
(Jason Dere via Harish Butani)


Changes for Build #587
[xuefu] HIVE-3181: getDatabaseMajor/Minor version does not return values 
(Szehon via Xuefu, reviewed by Navis)

[brock] HIVE-5641 - BeeLineOpts ignores Throwable (Brock Noland reviewed by 
Prasad and Thejas)

[hashutosh] HIVE-5909 : locate and instr throw 
java.nio.BufferUnderflowException when empty string as substring (Navis via 
Ashutosh Chauhan)

[hashutosh] HIVE-5686 : partition column type validation doesn't quite work for 
dates (Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5887 : metastore direct sql doesn't work with oracle (Sergey 
Shel

[jira] [Resolved] (HIVE-6038) Fix Tez branch to properly compile against hadoop-1 profile

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner resolved HIVE-6038.
--

Resolution: Fixed

Committed to branch.

> Fix Tez branch to properly compile against hadoop-1 profile
> ---
>
> Key: HIVE-6038
> URL: https://issues.apache.org/jira/browse/HIVE-6038
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Fix For: tez-branch
>
> Attachments: HIVE-6038.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6038) Fix Tez branch to properly compile against hadoop-1 profile

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-6038:
-

Attachment: HIVE-6038.1.patch

> Fix Tez branch to properly compile against hadoop-1 profile
> ---
>
> Key: HIVE-6038
> URL: https://issues.apache.org/jira/browse/HIVE-6038
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Fix For: tez-branch
>
> Attachments: HIVE-6038.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-6038) Fix Tez branch to properly compile against hadoop-1 profile

[Gunther Hagleitner (JIRA)]

Gunther Hagleitner created HIVE-6038:


 Summary: Fix Tez branch to properly compile against hadoop-1 
profile
 Key: HIVE-6038
 URL: https://issues.apache.org/jira/browse/HIVE-6038
 Project: Hive
  Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
 Fix For: tez-branch






--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6022) Load statements with incorrect order of partitions put input files to unreadable places

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-6022:
--

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

Patch committed to trunk. Thanks to Teruyoshi for the contribution.

> Load statements with incorrect order of partitions put input files to 
> unreadable places
> ---
>
> Key: HIVE-6022
> URL: https://issues.apache.org/jira/browse/HIVE-6022
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Teruyoshi Zenmyo
>Assignee: Teruyoshi Zenmyo
> Fix For: 0.13.0
>
> Attachments: HIVE-6022.1.patch, HIVE-6022.1.patch.txt, 
> HIVE-6022.2.patch.txt
>
>
> Load statements with incorrect order of partitions put input files to 
> incorrect paths. 
> {code}
> CREATE TABLE test_parts (c1 string, c2 int) PARTITIONED BY (p1 string,p2 
> string);
> LOAD DATA LOCAL INPATH '/opt/hive/examples/files/kv1.txt' OVERWRITE INTO 
> TABLE test_parts PARTITION (p2='p1', p1='p2')"
> {code}
> The input file is located as below and the data is not readable.
> {code}
> % find /user/hive/warehouse/test_parts/
> /user/hive/warehouse/test_parts/
> /user/hive/warehouse/test_parts//p1=p2
> /user/hive/warehouse/test_parts//p1=p2/p2=p1
> /user/hive/warehouse/test_parts//p2=p1
> /user/hive/warehouse/test_parts//p2=p1/p1=p2
> /user/hive/warehouse/test_parts//p2=p1/p1=p2/.kv1.txt.crc
> /user/hive/warehouse/test_parts//p2=p1/p1=p2/kv1.txt
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-3286) Explicit skew join on user provided condition

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849088#comment-13849088
 ] 

Hive QA commented on HIVE-3286:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12618867/HIVE-3286.16.patch.txt

{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 4790 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_explicit
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_skewjoin_explicit_invalid1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_skewjoin_explicit_invalid2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_skewjoin_explicit_invalid3
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/652/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/652/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12618867

> Explicit skew join on user provided condition
> -
>
> Key: HIVE-3286
> URL: https://issues.apache.org/jira/browse/HIVE-3286
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D4287.11.patch, HIVE-3286.12.patch.txt, 
> HIVE-3286.13.patch.txt, HIVE-3286.14.patch.txt, HIVE-3286.15.patch.txt, 
> HIVE-3286.16.patch.txt, HIVE-3286.D4287.10.patch, HIVE-3286.D4287.5.patch, 
> HIVE-3286.D4287.6.patch, HIVE-3286.D4287.7.patch, HIVE-3286.D4287.8.patch, 
> HIVE-3286.D4287.9.patch
>
>
> Join operation on table with skewed data takes most of execution time 
> handling the skewed keys. But mostly we already know about that and even know 
> what is look like the skewed keys.
> If we can explicitly assign reducer slots for the skewed keys, total 
> execution time could be greatly shortened.
> As for a start, I've extended join grammar something like this.
> {code}
> select * from src a join src b on a.key=b.key skew on (a.key+1 < 50, a.key+1 
> < 100, a.key < 150);
> {code}
> which means if above query is executed by 20 reducers, one reducer for 
> a.key+1 < 50, one reducer for 50 <= a.key+1 < 100, one reducer for 99 <= 
> a.key < 150, and 17 reducers for others (could be extended to assign more 
> than one reducer later)
> This can be only used with common-inner-equi joins. And skew condition should 
> be composed of join keys only.
> Work till done now will be updated shortly after code cleanup.
> 
> Skew expressions* in "SKEW ON (expr, expr, ...)" are evaluated sequentially 
> at runtime, and first 'true' one decides skew group for the row. Each skew 
> group has reserved partition slot(s), to which all rows in a group would be 
> assigned. 
> The number of partition slot reserved for each group is decided also at 
> runtime by simple calculation of percentage. If a skew group is "CLUSTER BY 
> 20 PERCENT" and total partition slot (=number of reducer) is 20, that group 
> will reserve 4 partition slots, etc.
> "DISTRIBUTE BY" decides how the rows in a group is dispersed in the range of 
> reserved slots (If there is only one slot for a group, this is meaningless). 
> Currently, three distribution policies are available: RANDOM, KEYS, 
> . 
> 1. RANDOM : rows of driver** alias are dispersed by random and rows of 
> non-driver alias are duplicated for all the slots (default if not specified)
> 2. KEYS : determined by hash value of keys (same with previous)
> 3. expression : determined by hash of object evaluated by user-provided 
> expression
> Only possible with inner, equi, common-joins. Not yet supports join tree 
> merging.
> Might be used by other RS users like "SORT BY" or "GROUP BY"
> If there exists column statistics for the key, it could be possible to apply 
> automatically.
> For example, if 20 reducers are used for the query below,
> {code}
> select count(*) from src a join src b on a.key=b.key skew on (
>a.key = '0' CLUSTER BY 10 PERCENT,
>b.key < '100' CLUSTER BY 20 PERCENT DISTRIBUTE BY upper(b.key),
>cast(a.key as int) > 300 CLUSTER BY 40 PERCENT DISTRIBUTE BY KEYS);
> {code}
> group-0 will reserve slots 6~7, group-1 8~11, group-2 12~19 and others will 
> reserve slots 0~5.
> For a row with key='0' from alias a, the row is randomly assigned in the 
> range of 6~7 (driver alias) : 6 or 7
> For a row with key='0' from alias b, the row is di

Re: doc on predicate pushdown in joins

[Lefty Leverenz]

>
> This is old code, hard to attribute to  jiras
>

Okay, HIVE-2337 was way back in release 0.8.0 so I think we can safely omit
version information.  If anyone objects, I can link the wikidoc to that
ticket.  (And I've marked HIVE-2337 as being related to HIVE-279 in the
JIRA.)

Thanks for all the help.

-- Lefty


On Wed, Dec 11, 2013 at 7:03 PM, Harish Butani wrote:

> I see.  Let's leave it in.
>
> This is old code, hard to attribute to  jiras:
> - The PPD code comes from: HIVE-279, HIVE-2337
> - I cannot tell when the join condition parsing code was added.
>
> regards,
> Harish.
>
> On Dec 11, 2013, at 5:17 PM, Lefty Leverenz 
> wrote:
>
> > Maybe we should remove the section on Hive Implementation here.
> > It is in the Design doc; this information only concerns developers.
>
> But this is the Design doc (unless there's another one somewhere -- maybe
> attached to a JIRA ticket?) and it's in the Resources for Contributors part
> of the wiki, so it seems appropriate to me.  I'll delete the implementation
> section if that's your preference.
>
> Here are the links again, with fixes:
>
>- Design Docs 
> (bottom
>of list)
>- Predicate Pushdown 
> Rules
>
>
> Speaking of JIRA tickets, is there one for this and should I add any
> version information?
>
> -- Lefty
>
>
> On Wed, Dec 11, 2013 at 7:59 AM, Harish Butani wrote:
>
>> getQualifiedAliases is a private method in JoinPPD.
>>
>> Maybe we should remove the section on Hive Implementation here. It is in
>> the Design doc; this information only concerns developers.
>>
>> regards,
>> Harish.
>>
>>
>> On Dec 11, 2013, at 3:05 AM, Lefty Leverenz 
>> wrote:
>>
>> Happy to fix the sentence and the link.  I pointed out the name change
>> just so you would review it, so please don't apologize!
>>
>> One more question:  why am I not finding getQualifiedAliases() in the
>> SemanticAnalyzer class?  It turns up in OpProcFactory.java with javadoc
>> comments, but I can't find it anywhere in the API docs -- not even in the
>> index (Hive 0.12.0 API ):
>>
>> *getQMap()*
>>  -
>> Method in class 
>> org.apache.hadoop.hive.ql.QTestUtil
>>  
>> *getQualifiedName()*
>>  -
>> Method in class 
>> org.apache.hadoop.hive.serde2.typeinfo.TypeInfo
>>  String
>> representing the qualified type 
>> name.*getQualifiers()*
>>  -
>> Method in class 
>> org.apache.hive.service.cli.thrift.TTypeQualifiers
>>  
>> *getQualifiersSize()*
>>  -
>> Method in class 
>> org.apache.hive.service.cli.thrift.TTypeQualifiers
>> Most mysterious.
>>
>> -- Lefty
>>
>>
>> On Tue, Dec 10, 2013 at 2:35 PM, Harish Butani 
>> wrote:
>>
>>> I can see why you would rename.
>>>
>>> But this sentence is not correct:
>>> 'Hive enforces the predicate pushdown rules by these methods in the
>>> SemanticAnalyzer and JoinPPD classes:'
>>>
>>> It should be:
>>> Hive enforces the rules by these methods in the SemanticAnalyzer and
>>> JoinPPD classes:
>>>
>>> (The implementation involves both predicate pushdown and analyzing join
>>> conditions)
>>> Sorry about this.
>>>
>>> So the link should say 'Hive Outer Join Behavior'
>>>
>>> regards,
>>> Harish.
>>>
>>>
>>> On Dec 10, 2013, at 2:01 PM, Lefty Leverenz 
>>> wrote:
>>>
>>> How's this?  Hive 
>>> Implementation
>>>
>>> Also, I moved the link on the Design Docs 
>>> pagefrom
>>> *Proposed* to *Other*.  (It's called SQL Outer Join Predicate Pushdown
>>> Rules 
>>> which
>>> doesn't match the title, but seems okay because it's more descriptive.)
>>>
>>> -- Lefty
>>>
>>>
>>> On Tue, Dec 10, 2013 at 7:27 AM, Harish Butani 
>>> wrote:
>>>
 You are correct, it is plural.

 regards,
 Harish.

 On Dec 10, 2013, at 4:03 AM, Lefty Leverenz 
 wrote:

 Okay, then monospace with "()" after the method name 

[jira] [Commented] (HIVE-5975) [WebHCat] templeton mapreduce job failed if provide "define" parameters

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849044#comment-13849044
 ] 

Hive QA commented on HIVE-5975:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12618157/hive-5975.2.patch

{color:green}SUCCESS:{color} +1 4785 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/651/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/651/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12618157

> [WebHCat] templeton mapreduce job failed if provide "define" parameters
> ---
>
> Key: HIVE-5975
> URL: https://issues.apache.org/jira/browse/HIVE-5975
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.12.0, 0.13.0
>Reporter: shanyu zhao
>Assignee: shanyu zhao
> Attachments: hive-5975.2.patch, hive-5975.patch
>
>
> Trying to submit a mapreduce job through templeton failed:
> curl -k -u user:pass -d user.name=user -d define=JobName=MRPiJob -d class=pi 
> -d arg=16 -d arg=100 -d jar="hadoop-mapreduce-examples.jar" 
> https://xxx/templeton/v1/mapreduce/jar
> The error message is:
> "Usage: org.apache.hadoop.examples.QuasiMonteCarlo  
>  Generic options supported are
>  -conf  specify an application configuration file
>  -D  use value for given property
>  -fs  specify a namenode
>  -jt  specify a job tracker
>  -files  specify comma separated files to be 
> copied to the map reduce cluster
>  -libjars  specify comma separated jar files to 
> include in the classpath.
>  -archives  specify comma separated 
> archives to be unarchived on the compute machines.
> The general command line syntax is
>  bin/hadoop command [genericOptions] [commandOptions]
> templeton: job failed with exit code 2"
> Note that if we remove the "define" parameter it works fine.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5879) Fix spelling errors in hive-default.xml

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849013#comment-13849013
 ] 

Lefty Leverenz commented on HIVE-5879:
--

bq. Beside, I've made a patch pulling description and default values from 
hive-default.xml.template into HiveConf.ConfVars

Sweet!

The [Configuration 
Properties|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties]
 wikidoc also has release information for some properties.  A handful have 
"Added in:" and two have "Removed in:" plus one with a behavior change.  Should 
this release information be merged with the descriptions?

 Here's the list: 

* mapred.reduce.tasks   Added In: 0.1
* hive.metastore.schema.verification   Added In: Hive 0.12.0 with HIVE-3764
* hive.security.command.whitelist   Added In: Hive 0.13.0 with HIVE-5400
* hive.security.metastore.authorization.manager   Added In: Hive 0.10.0
* hive.security.metastore.authenticator.manager   Added In: Hive 0.10.0
* hive.task.progress   Removed in: Hive 0.13.0 with HIVE-4518
* hive.metastore.local   Removed in: Hive 0.10 with HIVE-2585
* datanucleus.autoCreateSchema   In Hive 0.12.0 and later releases, 
datanucleus.autoCreateSchema is disabled if hive.metastore.schema.verification 
is true.
* __  hive.exec.parallel  added in 0.5.0 
(HIVE-549); default → true in 0.13.0 (HIVE-1033)

> Fix spelling errors in hive-default.xml
> ---
>
> Key: HIVE-5879
> URL: https://issues.apache.org/jira/browse/HIVE-5879
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 0.12.0
>Reporter: Brock Noland
>Assignee: Lefty Leverenz
>Priority: Trivial
>  Labels: documentation
> Fix For: 0.13.0
>
> Attachments: HIVE-5879.2.patch.txt, HIVE-5879.patch
>
>
> See 
> https://issues.apache.org/jira/browse/HIVE-5400?focusedCommentId=13830626&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13830626



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6037) Synchronize HiveConf with hive-default.xml.template and support show conf

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849000#comment-13849000
 ] 

Hive QA commented on HIVE-6037:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12618865/HIVE-6037.1.patch.txt

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/650/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/650/console

Messages:
{noformat}
 This message was trimmed, see log for full details 
[INFO] Executing tasks

main:
[mkdir] Created dir: 
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/tmp
[mkdir] Created dir: 
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/warehouse
[mkdir] Created dir: 
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/tmp/conf
 [copy] Copying 4 files to 
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/tmp/conf
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
hive-hcatalog-it-unit ---
[INFO] Compiling 7 source files to 
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/test-classes
[WARNING] Note: Some input files use or override a deprecated API.
[WARNING] Note: Recompile with -Xlint:deprecation for details.
[INFO] 
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ 
hive-hcatalog-it-unit ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-hcatalog-it-unit ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: 
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/hive-hcatalog-it-unit-0.13.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-jar-plugin:2.2:test-jar (default) @ hive-hcatalog-it-unit ---
[INFO] Building jar: 
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/hive-hcatalog-it-unit-0.13.0-SNAPSHOT-tests.jar
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ 
hive-hcatalog-it-unit ---
[INFO] Installing 
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/hive-hcatalog-it-unit-0.13.0-SNAPSHOT.jar
 to 
/data/hive-ptest/working/maven/org/apache/hive/hive-hcatalog-it-unit/0.13.0-SNAPSHOT/hive-hcatalog-it-unit-0.13.0-SNAPSHOT.jar
[INFO] Installing 
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/pom.xml 
to 
/data/hive-ptest/working/maven/org/apache/hive/hive-hcatalog-it-unit/0.13.0-SNAPSHOT/hive-hcatalog-it-unit-0.13.0-SNAPSHOT.pom
[INFO] Installing 
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/hive-hcatalog-it-unit-0.13.0-SNAPSHOT-tests.jar
 to 
/data/hive-ptest/working/maven/org/apache/hive/hive-hcatalog-it-unit/0.13.0-SNAPSHOT/hive-hcatalog-it-unit-0.13.0-SNAPSHOT-tests.jar
[INFO] 
[INFO] 
[INFO] Building Hive Integration - Testing Utilities 0.13.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-it-util ---
[INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/itests/util 
(includes = [datanucleus.log, derby.log], excludes = [])
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
hive-it-util ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 
/data/hive-ptest/working/apache-svn-trunk-source/itests/util/src/main/resources
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-it-util ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-it-util 
---
[INFO] Compiling 41 source files to 
/data/hive-ptest/working/apache-svn-trunk-source/itests/util/target/classes
[WARNING] Note: Some input files use or override a deprecated API.
[WARNING] Note: Recompile with -Xlint:deprecation for details.
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
hive-it-util ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 
/data/hive-ptest/working/apache-svn-trunk-source/itests/util/src/test/resources
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-it-util ---
[INFO] Executing tasks

main:
[mkdir] Created dir: 
/data/hive-ptest/working/apache-svn-trunk-source/itests/util/target/tmp
[mkdir] Created dir: 
/data/hive-ptest/working/apache-svn-trunk-source/itests/ut