[jira] [Created] (HIVE-13859) mask() UDF not retaining day and month field values

[Madhan Neethiraj (JIRA)]

Madhan Neethiraj created HIVE-13859:
---

 Summary: mask() UDF not retaining day and month field values
 Key: HIVE-13859
 URL: https://issues.apache.org/jira/browse/HIVE-13859
 Project: Hive
  Issue Type: Bug
  Components: UDF
Affects Versions: 2.1.0
Reporter: Madhan Neethiraj
 Fix For: 2.1.0


For date type parameters, mask() UDF replaces year/month/day field values with 
the values given in arguments to the UDF. Argument value -1 is treated as 
special, to specify that mask() should retain the value in the parameter. This 
allows to selectively mask only year/month/day fields.

Specifying "-1" does not retain the values for day/month fields; however the 
year value is retained, as shown below.

{code}
0: jdbc:hive2://localhost:1> select id, join_date from employee where id < 
4;
+-+-+--+
| id  |  join_date  |
+-+-+--+
| 1   | 2012-01-01  |
| 2   | 2014-02-01  |
| 3   | 2013-03-01  |
+-+-+--+
3 rows selected (0.435 seconds)
0: jdbc:hive2://localhost:1> select id, mask(join_date, -1, -1, -1, -1,-1, 
-1,-1,-1) join_date from employee where id < 4;
+-+-+--+
| id  |  join_date  |
+-+-+--+
| 1   | 2012-01-01  |
| 2   | 2014-01-01  |
| 3   | 2013-01-01  |
+-+-+--+
3 rows selected (0.344 seconds)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13858) LLAP: A preempted task can end up waiting on completeInitialization if some part of the executing code suppressed the interrupt

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-13858:
-

 Summary: LLAP: A preempted task can end up waiting on 
completeInitialization if some part of the executing code suppressed the 
interrupt
 Key: HIVE-13858
 URL: https://issues.apache.org/jira/browse/HIVE-13858
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Priority: Critical


An interrupt along with a HiveProcessor.abort call is made when attempting to 
preempt a task.

In this specific case, the task was in the middle of HDFS IO - which 'handled' 
the interrupt by retrying. As a result the interrupt status on the thread was 
reset - so instead of skipping the future.get in completeInitialization - the 
task ended up blocking there.

End result - a single executor slot permanently blocked in LLAP. Depending on 
what else is running - this can cause a cluster level deadlock.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13857) insert overwrite select from some table fails throwing org.apache.hadoop.security.AccessControlException - II

[Hari Sankar Sivarama Subramaniyan (JIRA)]

Hari Sankar Sivarama Subramaniyan created HIVE-13857:


 Summary: insert overwrite select from some table fails throwing 
org.apache.hadoop.security.AccessControlException - II
 Key: HIVE-13857
 URL: https://issues.apache.org/jira/browse/HIVE-13857
 Project: Hive
  Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan


HIVE-13810 missed a fix, tracking it here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13856) Fetching transaction batches during ACID streaming against Hive Metastore using Oracle DB fails

[Deepesh Khandelwal (JIRA)]

Deepesh Khandelwal created HIVE-13856:
-

 Summary: Fetching transaction batches during ACID streaming 
against Hive Metastore using Oracle DB fails
 Key: HIVE-13856
 URL: https://issues.apache.org/jira/browse/HIVE-13856
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Reporter: Deepesh Khandelwal


{noformat}
2016-05-25 00:43:49,682 INFO  [pool-4-thread-5]: txn.TxnHandler 
(TxnHandler.java:checkRetryable(1585)) - Non-retryable error: ORA-00933: SQL 
command not properly ended
 (SQLState=42000, ErrorCode=933)
2016-05-25 00:43:49,685 ERROR [pool-4-thread-5]: metastore.RetryingHMSHandler 
(RetryingHMSHandler.java:invoke(159)) - MetaException(message:Unable to select 
from transaction database java.sql.SQLSyntaxErrorException: ORA-00933: SQL 
command not properly ended

at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:837)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:445)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:193)
at oracle.jdbc.driver.T4CStatement.executeForRows(T4CStatement.java:999)
at 
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1315)
at 
oracle.jdbc.driver.OracleStatement.executeInternal(OracleStatement.java:1890)
at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:1855)
at 
oracle.jdbc.driver.OracleStatementWrapper.execute(OracleStatementWrapper.java:304)
at com.jolbox.bonecp.StatementHandle.execute(StatementHandle.java:254)
at 
org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:429)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.open_txns(HiveMetaStore.java:5647)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy15.open_txns(Unknown Source)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$open_txns.getResult(ThriftHiveMetastore.java:11604)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$open_txns.getResult(ThriftHiveMetastore.java:11589)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at 
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
at 
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at 
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
)
at 
org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:438)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.open_txns(HiveMetaStore.java:5647)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy15.open_txns(Unknown Source)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$open_txns.getResult(ThriftHiveMetastore.java:11604)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$open_txns.getResult(ThriftHiveMetastore.java:11589)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at 
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
at 

Review Request 47854: Set basic stats as inaccurate for all ACID tables

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47854/
---

Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-13838


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java f3c7e99 
  ql/src/test/queries/clientpositive/acid_table_stats.q PRE-CREATION 
  ql/src/test/results/clientpositive/acid_table_stats.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/47854/diff/


Testing
---


Thanks,

pengcheng xiong

Hive and using Pooled Connections

[Mich Talebzadeh]

Hi,


I am sure someone knows the answer to this question.

Does Hive 2.0 use connection pool to connect to its metastore? I see a lot
of open and closed connections to the metastore that may not be necessary.

A *connection pool* is a cache of database connection objects. Connection
pools promote the reuse of connection objects and reduce the number of
times that connection objects are created. Connection pools significantly
improve performance for database-intensive applications because creating
connection objects is costly both in terms of time and resources.
Thanks

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com

[jira] [Created] (HIVE-13855) select INPUT__FILE__NAME throws NPE exception

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-13855:
---

 Summary: select INPUT__FILE__NAME throws NPE exception
 Key: HIVE-13855
 URL: https://issues.apache.org/jira/browse/HIVE-13855
 Project: Hive
  Issue Type: Bug
  Components: Query Planning
Reporter: Aihua Xu
Assignee: Aihua Xu


The following query executes successfully

select INPUT__FILE__NAME from src limit 1;

But the following NPE is thrown

{noformat}
16/05/25 16:49:49 ERROR exec.Utilities: Failed to load plan: null: 
java.lang.NullPointerException
java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:407)
at 
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:299)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:315)
at 
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
at 
org.apache.hadoop.hive.ql.exec.FetchOperator$1.doNext(FetchOperator.java:340)
at 
org.apache.hadoop.hive.ql.exec.FetchOperator$1.doNext(FetchOperator.java:331)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:484)
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:424)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:144)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1884)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:252)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13854) select INPUT__FILE__NAME fails from the join query

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-13854:
---

 Summary: select INPUT__FILE__NAME fails from the join query
 Key: HIVE-13854
 URL: https://issues.apache.org/jira/browse/HIVE-13854
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Reporter: Aihua Xu
Assignee: Aihua Xu


The query {{select t1.INPUT__FILE__NAME from src t1 join src t2 limit 1;}} 
fails with the following exception
{noformat}
java.lang.RuntimeException: cannot find field input__file__name from [0:key, 
1:value]
at 
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
at 
org.apache.hadoop.hive.serde2.BaseStructObjectInspector.getStructFieldRef(BaseStructObjectInspector.java:133)
at 
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:55)
at 
org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:973)
at 
org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:999)
at 
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:75)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:355)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:504)
at 
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:457)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:365)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:499)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:403)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:383)
at 
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:751)
{noformat}
when hive.auto.convert.join is set to true.

The workaround is to set hive.auto.convert.join to false.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13853) Add X-XSRF-Header filter to HS2 HTTP mode and WebHCat

[Sushanth Sowmyan (JIRA)]

Sushanth Sowmyan created HIVE-13853:
---

 Summary: Add X-XSRF-Header filter to HS2 HTTP mode and WebHCat
 Key: HIVE-13853
 URL: https://issues.apache.org/jira/browse/HIVE-13853
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2, WebHCat
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan


There is a possibility that there may be a CSRF-based attack on various hadoop 
components, and thus, there is an effort to add a block for all incoming http 
requests if they do not contain a X-XSRF-Header header. (See HADOOP-12691 for 
motivation)

This has potential to affect HS2 when running on thrift-over-http mode(if 
cookie-based-auth is used), and webhcat.

We introduce new flags to determine whether or not we're using the filter, and 
if we are, we will automatically reject any http requests which do not contain 
this header.

To allow this to work, we also need to make changes to our JDBC driver to 
automatically inject this header into any requests it makes. Also, any 
client-side programs/api not using the JDBC driver directly will need to make 
changes to add a X-XSRF-Header header to the request to make calls to 
HS2/WebHCat if this filter is enabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 47792: HIVE-13248 Change date_add/date_sub/to_date functions to return Date type rather than String

[Matt McCline]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47792/#review134806
---




ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CastStringToDate.java
 (line 121)


Nit -- Prasanth seems to think the new String form that takes the Charset 
enumeration instead of the charset String name "UTF-8" was better because it 
doesn't throw a UnsupportedEncodingException...

So, perhaps you don't need the try/catch...



ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorUDFDateAddScalarCol.java
 (line 94)


Ditto earlier comment.



ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorGenericDateExpressions.java
 (line 290)


Nit -- I think the Charset enumeration version of getBytes is also 
available.


- Matt McCline


On May 24, 2016, 8:33 p.m., Jason Dere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47792/
> ---
> 
> (Updated May 24, 2016, 8:33 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Matt McCline.
> 
> 
> Bugs: HIVE-13248
> https://issues.apache.org/jira/browse/HIVE-13248
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> - Change date_add/date_sub/to_date functions to return Date type rather than 
> String (vectorized and non-vectorized path)
> - Create date parser class for use by date_add/date_sub/to_date.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hive/common/util/DateParser.java PRE-CREATION 
>   common/src/test/org/apache/hive/common/util/TestDateParser.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CastStringToDate.java
>  98fdf4a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorUDFDateAddColCol.java
>  05dd93e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorUDFDateAddColScalar.java
>  59ca61e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorUDFDateAddScalarCol.java
>  2d0a28a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorUDFDateLong.java
>  a58bfb5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorUDFDateString.java
>  e27ac6a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorUDFDateTimestamp.java
>  cde0be4 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDate.java 
> 8c376a0 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDateAdd.java 
> 92a72bf 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDateSub.java 
> db0f4cd 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorGenericDateExpressions.java
>  3f2b031 
>   ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFDate.java 
> c9f566d 
>   
> ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFDateAdd.java 
> 08b9604 
>   
> ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFDateSub.java 
> ab8d109 
>   ql/src/test/queries/clientpositive/union_offcbo.q 66d4bee 
>   ql/src/test/results/clientpositive/fold_eq_with_case_when.q.out 13f6ab4 
>   ql/src/test/results/clientpositive/udf5.q.out 860ebcb 
>   ql/src/test/results/clientpositive/udf9.q.out ddd4a5d 
>   ql/src/test/results/clientpositive/union_offcbo.q.out 71c3bfc 
>   ql/src/test/results/clientpositive/vectorized_date_funcs.q.out b8023e2 
> 
> Diff: https://reviews.apache.org/r/47792/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jason Dere
> 
>

[jira] [Created] (HIVE-13852) NPE in TaskLocationHints during LLAP GetSplits request

[Jason Dere (JIRA)]

Jason Dere created HIVE-13852:
-

 Summary: NPE in TaskLocationHints during LLAP GetSplits request
 Key: HIVE-13852
 URL: https://issues.apache.org/jira/browse/HIVE-13852
 Project: Hive
  Issue Type: Bug
  Components: llap
Reporter: Jason Dere
Assignee: Jason Dere


{noformat}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
java.io.IOException: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:194)
at org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
at 
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:434) 
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:426) 
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:144)
... 15 more
Caused by: java.io.IOException: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.getSplits(GenericUDTFGetSplits.java:366)
at 
org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:185)
... 23 more
Caused by: java.lang.NullPointerException: null
at 
org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.getSplits(GenericUDTFGetSplits.java:344)
... 24 more
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 47838: HIVE-13831

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47838/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-13831
https://issues.apache.org/jira/browse/HIVE-13831


Repository: hive-git


Description
---

HIVE-13831


Diffs
-

  
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/AccumuloPredicateHandler.java
 d5cc9a5491edddc4fae840e087b384587ac17ddc 
  
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/PushdownTuple.java
 f326d522daf694a788b7b52f23da4134bc4251e6 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
42ea24e41883d8c02b49848020a80fcc7f179c50 
  hbase-handler/src/test/results/positive/external_table_ppd.q.out 
590906b63e18c64cbadc5b02d978cbad41c8220f 
  hbase-handler/src/test/results/positive/hbase_pushdown.q.out 
a42e36f0132457d22240d3ef4b8a52375a14fe34 
  hbase-handler/src/test/results/positive/hbase_timestamp.q.out 
29a04f9167803c896f998ed75a89e8f20ef4799e 
  hbase-handler/src/test/results/positive/ppd_key_ranges.q.out 
34c3b230bd64f4fe9aa92a0e874e0f0a4919d208 
  ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java 
2f0deae34ac42e5f6cbfffc6148473b62fb82d0c 
  ql/src/java/org/apache/hadoop/hive/ql/index/IndexSearchCondition.java 
3a2ecb7282a2eed0b1f7e49ca70b15f10c8e9675 

Diff: https://reviews.apache.org/r/47838/diff/


Testing
---


Thanks,

Jesús Camacho Rodríguez

[jira] [Created] (HIVE-13851) Change columns of textfile formatted table from float to double will drop some fractional digits

[Takahiko Saito (JIRA)]

Takahiko Saito created HIVE-13851:
-

 Summary: Change columns of textfile formatted table from float to 
double will drop some fractional digits
 Key: HIVE-13851
 URL: https://issues.apache.org/jira/browse/HIVE-13851
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.2.1
Reporter: Takahiko Saito


In a case of texfile formatted table, changing a column from float to double 
will drop some fractional digits:
{noformat}
0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> create table test(f float);
No rows affected (0.176 seconds)
0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> insert into table test 
values(-35664.76171875),(29497.349609375);
INFO  : Tez session hasn't been created yet. Opening session
INFO  : Dag name: insert into table tes...5),(29497.349609375)(Stage-1)
INFO  :

INFO  : Status: Running (Executing on YARN cluster with App id 
application_1464135801747_0400)

INFO  : Map 1: -/-
INFO  : Map 1: 0/1
INFO  : Map 1: 0/1
INFO  : Map 1: 0(+1)/1
INFO  : Map 1: 0(+1)/1
INFO  : Map 1: 1/1
INFO  : Loading data to table default.test from 
hdfs://os-r7-wbucju-hiveserver2-11-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-05-25_17-22-03_941_5459171987449455387-1/-ext-1
INFO  : Table default.test stats: [numFiles=1, numRows=2, totalSize=19, 
rawDataSize=17]
No rows affected (18.443 seconds)
0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> select * from test;
+--+--+
|  test.f  |
+--+--+
| -35664.76171875  |
| 29497.349609375  |
+--+--+
2 rows selected (0.231 seconds)
0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> alter table test change column f 
f double;
No rows affected (0.34 seconds)
0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> select * from test;
++--+
|   test.f   |
++--+
| -35664.76  |
| 29497.35   |
++--+
2 rows selected (0.128 seconds)
{noformat}

The issue is not seen with orc table:
{noformat}
0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> create table orc(f float) stored 
as orc;
No rows affected (0.162 seconds)
0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> insert into table orc 
values(-35664.76171875),(29497.349609375);
INFO  : Session is already open
INFO  : Dag name: insert into table orc...5),(29497.349609375)(Stage-1)
INFO  :

INFO  : Status: Running (Executing on YARN cluster with App id 
application_1464135801747_0400)

INFO  : Map 1: 0/1
INFO  : Map 1: 0/1
INFO  : Map 1: 0(+1)/1
INFO  : Map 1: 0(+1)/1
INFO  : Map 1: 1/1
INFO  : Loading data to table default.orc from 
hdfs://os-r7-wbucju-hiveserver2-11-5.openstacklocal:8020/apps/hive/warehouse/orc/.hive-staging_hive_2016-05-25_17-25-26_569_221649193772012279-1/-ext-1
INFO  : Table default.orc stats: [numFiles=1, numRows=2, totalSize=253, 
rawDataSize=8]
No rows affected (7.87 seconds)
0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> select * from orc;
+--+--+
|  orc.f   |
+--+--+
| -35664.76171875  |
| 29497.349609375  |
+--+--+
2 rows selected (0.117 seconds)
0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> alter table orc change column f 
f double;
No rows affected (0.85 seconds)
0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> select * from orc;
+--+--+
|  orc.f   |
+--+--+
| -35664.76171875  |
| 29497.349609375  |
+--+--+
2 rows selected (0.168 seconds)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13850) File name conflict when have multiple INSERT INTO queries running in parallel

[Bing Li (JIRA)]

Bing Li created HIVE-13850:
--

 Summary: File name conflict when have multiple INSERT INTO queries 
running in parallel
 Key: HIVE-13850
 URL: https://issues.apache.org/jira/browse/HIVE-13850
 Project: Hive
  Issue Type: Bug
Reporter: Bing Li
Assignee: Bing Li


We have an application which connect to HiveServer2 via JDBC.
In the application, it executes "INSERT INTO" query to the same table.

If there are a lot of users running the application at the same time. Some of 
the INSERT could fail.

In hive log,
org.apache.hadoop.hive.ql.metadata.HiveException: copyFiles: error  
while moving files!!! Cannot move hdfs://node:8020/apps/hive/warehouse/met  
  
adata.db/scalding_stats/.hive-staging_hive_2016-05-10_18-46-
23_642_2056172497900766879-3321/-ext-1/00_0 to 
hdfs://node:8020/apps/hive  
/warehouse/metadata.db/scalding_stats/00_0_copy_9014
at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java: 
2719)   
at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java: 
1645)  


In hadoop log, 
WARN  hdfs.StateChange (FSDirRenameOp.java: 
unprotectedRenameTo(174)) - DIR* FSDirectory.unprotectedRenameTo:   
failed to rename /apps/hive/warehouse/metadata.db/scalding_stats/.hive- 
staging_hive_2016-05-10_18-46-23_642_2056172497900766879-3321/-ext- 
1/00_0 to /apps/hive/warehouse/metadata.
db/scalding_stats/00_0_copy_9014 because destination exists



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13849) Wrong plan for hive.optimize.sort.dynamic.partition=true

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-13849:
--

 Summary: Wrong plan for hive.optimize.sort.dynamic.partition=true
 Key: HIVE-13849
 URL: https://issues.apache.org/jira/browse/HIVE-13849
 Project: Hive
  Issue Type: Bug
  Components: Physical Optimizer
Affects Versions: 2.1.0, 2.2.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez
Priority: Critical


To reproduce:

{noformat}
set hive.support.concurrency=true;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.optimize.sort.dynamic.partition=true;

CREATE TABLE non_acid(key string, value string) PARTITIONED BY(ds string, hr 
int) CLUSTERED BY(key) INTO 2 BUCKETS STORED AS ORC;

explain insert into table non_acid partition(ds,hr) select * from srcpart sort 
by value;
{noformat}

CC'ed [~ashutoshc], [~ekoifman]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 47787: HIVE-13453: Support ORDER BY and windowing clause in partitioning clause with distinct function

[Yongzhi Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47787/#review134784
---




ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCount.java (line 
169)


How do you handle countDistinct non windowing case?


- Yongzhi Chen


On May 24, 2016, 6:51 p.m., Aihua Xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47787/
> ---
> 
> (Updated May 24, 2016, 6:51 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13453: Support ORDER BY and windowing clause in partitioning clause with 
> distinct function
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
> 2f4a94c3796d3aff986eb638246248b75306183c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java 
> 3b54b4998c9efbf34bd9c5b08de55cd7062a0843 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingSpec.java 
> 5ce72004e03bc19a38bd87ae70f38a0d35c20927 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowFunctionDef.java 
> ed6c67156b93d6f9e4b76fb76dfa28c5dee6fd0c 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFAverage.java 
> 3c1ce26b26646a6075b3a661816e8d1b50ffc78e 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCount.java 
> 2825045890de1bcc414197ad3e06e723b9d212f3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFParameterInfo.java
>  6a62d7cc324286ae9aee95d2d71a688859f8c03f 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java 
> 7b1d6e545cdf35f3b2906621c7b0208bf0433731 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/SimpleGenericUDAFParameterInfo.java
>  1a1b570256afff46761daf4ebcf1da5e8f0e4f88 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java 
> 858b47ad43fa751e23482e4cb58f77bb9fb16a27 
>   ql/src/test/queries/clientpositive/windowing_distinct.q 
> bb192a7882fda592b3d2ba09a10c2f899aa5e165 
>   ql/src/test/results/clientpositive/windowing_distinct.q.out 
> 074a59498ebebc9e78553f68f59dd00bb51f4792 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
>  c58e8ed05453c78cbe2e4daf0b7afa51adbc0ce9 
> 
> Diff: https://reviews.apache.org/r/47787/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aihua Xu
> 
>

Fwd: Daily Status Update

[Subash Muthukumaraswamy]

Hello,  Tried out few ways and got some what near to the solution.

But either the class file is not available in the jar or its in a diff
name. Or

Kindly let me know if there is any one who tried the same.

Task : Hbase Coulmn has a String of JSON, Need to query the multiple level
in Hive using Array/Struts column

https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration


[image: Inline image 2]


[image: Inline image 1]

Why can't assign newly opened JIRA issue

[Svetozar Ivanov]

Hi all, I'm wondering why can't assign newly opened JIRA issue to myself 
anymore? There is no such field currently. Any ideas?


Best regards,
Svetozar Ivanov

[jira] [Created] (HIVE-13848) Hive SORT/ORDER BY regex_extract(expression) alias column does not work

[Tajul Bashar (JIRA)]

Tajul Bashar created HIVE-13848:
---

 Summary: Hive SORT/ORDER BY regex_extract(expression) alias column 
does not work
 Key: HIVE-13848
 URL: https://issues.apache.org/jira/browse/HIVE-13848
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 2.0.0
 Environment: Fedora Linux
Reporter: Tajul Bashar


Example column values:
---
b>$29 per month.  In addition  you must keep paying your Medicare Part B 
premium.
Additional $30.90 per month.  You must keep paying your Medicare Part B 
premium and your $29 monthly plan premium.
$59 per month.  In addition  you must keep paying your Medicare Part B 
premium.
$29 per month.  In addition  you must keep paying your Medicare Part B 
premium.
---
Query without SORT or ORDER BY:
hive> select CAST(regexp_extract(benefit, '\$?(\\d+)', 1) AS FLOAT) as premium 
from planservices where benefit like '%premium%' and benefit like '%%%' 
limit 10;   
OK
0.0
15.0
0.0
15.0
0.0
18.0
15.0
0.0
15.0
19.0
Time taken: 0.153 seconds, Fetched: 10 row(s)
-
Query with SORT or ORDER BY:
select CAST(regexp_extract(benefit, '\$?(\\d+)', 1) AS FLOAT) as premium from 
planservices where benefit like '%premium%' and benefit like '%%%' SORT 
BY premium limit 10;
OK
NULL
NULL
NULL
NULL
NULL
NULL
NULL
0.0
0.0
0.0
Time taken: 4.106 seconds, Fetched: 10 row(s)
--
The result is same irrespective of reducer counts set to 1 or more and whether 
query is SORT BY or ORDER BY [ running on Hive-on-MR ].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

branch-2.1 was created

[Jesus Camacho Rodriguez]

Hi,

I have created branch-2.1. From now on, please use 2.2 fix version for
master commits. In addition, backport patches that should go into 2.1
to the release branch.

I have also done a pass over the JIRAs that are not resolved and whose
target version is 2.1.
- Issues that are about to be committed still target version 2.1.
- I left comments for issues that seem important but whose thread has been
idle for a while; we will assess with the reporter/assignee whether they
will be pushed to 2.1 or deferred to a future release.
- For other issues, target version is not 2.1 anymore.

Please note that we have a few bugs to be pushed, thus there may be time
to get other JIRAs in even if they do not target 2.1.

Thanks,
Jesús

[jira] [Created] (HIVE-13847) Avoid file open call in RecordReaderUtils as the stream is already available

[Rajesh Balamohan (JIRA)]

Rajesh Balamohan created HIVE-13847:
---

 Summary: Avoid file open call in RecordReaderUtils as the stream 
is already available
 Key: HIVE-13847
 URL: https://issues.apache.org/jira/browse/HIVE-13847
 Project: Hive
  Issue Type: Improvement
  Components: ORC
Reporter: Rajesh Balamohan
Priority: Minor


File open call in RecordReaderUtils::readRowIndex can be avoided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13846) Issue when running SQL statement with where clause using Apache Hive when connecting to HDP 2.4 using HDP Hive ODBC Driver

[Ganesh Sundaralingam (JIRA)]

Ganesh Sundaralingam created HIVE-13846:
---

 Summary: Issue when running SQL statement with where clause using 
Apache Hive when connecting to HDP 2.4 using HDP Hive ODBC Driver
 Key: HIVE-13846
 URL: https://issues.apache.org/jira/browse/HIVE-13846
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 1.2.0
 Environment: Hortonworks 2.4
Reporter: Ganesh Sundaralingam
Priority: Minor


When I run the following query using Apache Hive ODBC connector the quer run 
okay. 

SELECT * from 
clean_staging_db.sales_transaction_orc_buck 



But if I run the following query then it doesn't run. 

SELECT sum(item_quantity) from clean_staging_db.sales_transaction_orc_buck; 


[ERROR MESSAGE] 
Query failed to execute. Execute Query failed. 
Error type: Odbc error. Odbc operation attempted: SQLExecDirect. [S1000:1: on 
HSTMT] [MicroStrategy][ODBC Apache Hive Wire Protocol driver][Apache Hive]Error 
while processing statement: FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.tez.TezTask 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13845) Delete beeline/pom.xml.orig

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-13845:
--

 Summary: Delete beeline/pom.xml.orig
 Key: HIVE-13845
 URL: https://issues.apache.org/jira/browse/HIVE-13845
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.1.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13844) Invalid index handler in org.apache.hadoop.hive.ql.index.HiveIndex class

[Svetozar Ivanov (JIRA)]

Svetozar Ivanov created HIVE-13844:
--

 Summary: Invalid index handler in 
org.apache.hadoop.hive.ql.index.HiveIndex class
 Key: HIVE-13844
 URL: https://issues.apache.org/jira/browse/HIVE-13844
 Project: Hive
  Issue Type: Bug
  Components: Indexing
Affects Versions: 2.0.0
Reporter: Svetozar Ivanov






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13843) Re-enable the HoS tests disabled in HIVE-13402

[Rui Li (JIRA)]

Rui Li created HIVE-13843:
-

 Summary: Re-enable the HoS tests disabled in HIVE-13402
 Key: HIVE-13843
 URL: https://issues.apache.org/jira/browse/HIVE-13843
 Project: Hive
  Issue Type: Test
Reporter: Rui Li
Assignee: Rui Li


With HIVE-13525, we can now fix and re-enable the tests for Spark.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)