[jira] [Created] (HIVE-13009) Fix add_jar_file.q on Windows

2016-02-04 Thread Jason Dere (JIRA) 
Jason Dere created HIVE-13009:
-

 Summary: Fix add_jar_file.q on Windows
 Key: HIVE-13009
 URL: https://issues.apache.org/jira/browse/HIVE-13009
 Project: Hive
  Issue Type: Bug
  Components: Tests, Windows
Reporter: Jason Dere
Assignee: Jason Dere


Forward slashes in the local file path don't work for Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 42281: Upgrade Hive to Calcite 1.6

2016-02-04 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42281/
---

(Updated Feb. 5, 2016, 3:11 a.m.)


Review request for hive, Ashutosh Chauhan and John Pullokkaran.


Repository: hive-git


Description
---

HIVE-12839


Diffs (updated)
-

  pom.xml 3c06188 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
1c15012 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java 
eeec44e 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveAlgorithmsUtil.java
 6840418 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveDefaultCostModel.java
 6669d32 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveOnTezCostModel.java
 61a3a64 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveRelMdCost.java 
90f6a7e 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java
 aae57e6 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java
 2f8cc7d 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveJoin.java
 c323564 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
 c5376bf 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSemiJoin.java
 4fac13e 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveTableScan.java
 5788805 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateJoinTransposeRule.java
 070c7ea 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExpandDistinctAggregatesRule.java
 7d7631b 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinPushTransitivePredicatesRule.java
 07928d8 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
 50e139b 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java
 9f7f8a4 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
 0af60e8 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java
 618c717 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
 0ec8bf1 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/FilterSelectivityEstimator.java
 c04060f 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdCollation.java
 84fa518 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdDistinctRowCount.java
 1220401 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdDistribution.java
 b83f240 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdMemory.java
 bea5943 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdParallelism.java
 2f51d3b 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdPredicates.java
 b7244fd 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
 caf8978 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java
 a0eb83d 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSize.java
 3224039 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdUniqueKeys.java
 7c22c33 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java
 a0e374c 

Diff: https://reviews.apache.org/r/42281/diff/


Testing
---


Thanks,

pengcheng xiong

[jira] [Created] (HIVE-13008) WebHcat DDL commands in secure mode NPE when default FileSystem doesn't support delegation tokens

2016-02-04 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-13008:
-

 Summary: WebHcat DDL commands in secure mode NPE when default 
FileSystem doesn't support delegation tokens
 Key: HIVE-13008
 URL: https://issues.apache.org/jira/browse/HIVE-13008
 Project: Hive
  Issue Type: Bug
  Components: WebHCat
Affects Versions: 1.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


{noformat}
ERROR | 11 Jan 2016 20:19:02,781 | 
org.apache.hive.hcatalog.templeton.CatchallExceptionMapper |
java.lang.NullPointerException
at 
org.apache.hive.hcatalog.templeton.SecureProxySupport$2.run(SecureProxySupport.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at 
org.apache.hive.hcatalog.templeton.SecureProxySupport.writeProxyDelegationTokens(SecureProxySupport.java:168)
at 
org.apache.hive.hcatalog.templeton.SecureProxySupport.open(SecureProxySupport.java:95)
at 
org.apache.hive.hcatalog.templeton.HcatDelegator.run(HcatDelegator.java:63)
at org.apache.hive.hcatalog.templeton.Server.ddl(Server.java:217)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at 
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at 
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at 
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at 
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at 
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at 
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at 
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1480)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1411)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1360)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1350)
at 
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at 
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1360)
at 
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:615)
at 
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:574)
at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:88)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at 
org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13007) add and API to force reload UDFs to LLAP (either force reload everything, or force a regular refresh)

2016-02-04 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-13007:
---

 Summary: add and API to force reload UDFs to LLAP (either force 
reload everything, or force a regular refresh)
 Key: HIVE-13007
 URL: https://issues.apache.org/jira/browse/HIVE-13007
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13006) LLAP: add finer-grained classloaders as an option to be able to block the usage of removed UDFs

2016-02-04 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-13006:
---

 Summary: LLAP: add finer-grained classloaders as an option to be 
able to block the usage of removed UDFs
 Key: HIVE-13006
 URL: https://issues.apache.org/jira/browse/HIVE-13006
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-02-04 Thread Takanobu Asanuma 


> On Feb. 3, 2016, 10:41 p.m., Sergey Shelukhin wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 530
> > 
> >
> > actually, do we expect more than one file here, should we handle this 
> > case?

I have some questions.
1. Is there any SELECT statement which creates multiple result files in the 
final directory? I tested some queries, but I wasn't able to find that query.
2. If there are multiple result files, can we assume that all those files' 
names are numbers and each file's content accords with the series?


- Takanobu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review117697
---


On Feb. 3, 2016, 9:32 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated Feb. 3, 2016, 9:32 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6678de6 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java b4dba44 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4c89812 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336 
>   service-rpc/if/TCLIService.thrift 0aa9d13 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 5364293 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp e62e8b7 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
>  31472c8 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
>  7101fa5 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
>  14d50ed 
>   service-rpc/src/gen/thrift/gen-php/Types.php 76805df 
>   service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
>   service-rpc/src/gen/thrift/gen-py/__init__.py PRE-CREATION 
>   service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
>   service/src/gen/thrift/gen-cpp/ThriftHive.h PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/ThriftHive.cpp PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/ThriftHive_server.skeleton.cpp PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/hive_service_constants.h PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/hive_service_constants.cpp PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/hive_service_types.h PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/hive_service_types.cpp PRE-CREATION 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveClusterStatus.java
>  PRE-CREATION 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveServerException.java
>  PRE-CREATION 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/JobTrackerState.java
>  PRE-CREATION 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/ThriftHive.java
>  PRE-CREATION 
>   service/src/gen/thrift/gen-php/ThriftHive.php PRE-CREATION 
>   service/src/gen/thrift/gen-php/Types.php PRE-CREATION 
>   service/src/gen/thrift/gen-py/__init__.py PRE-CREATION 
>   service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote PRE-CREATION 
>   service/src/gen/thrift/gen-py/hive_service/ThriftHive.py PRE-CREATION 
>   service/src/gen/thrift/gen-py/hive_service/__init__.py PRE-CREATION 
>   service/src/gen/thrift/gen-py/hive_service/constants.py PRE-CREATION 
>   service/src/gen/thrift/gen-py/hive_service/ttypes.py PRE-CREATION 
>   service/src/gen/thrift/gen-rb/hive_service_constants.rb PRE-CREATION 
>   service/src/gen/thrift/gen-rb/hive_service_types.rb PRE-CREATION 
>   service/src/gen/thrift/gen-rb/thrift_hive.rb PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/CLIService.java cc6e138 
>   service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 0c263cf 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> f1ce6f6 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 01b1d3d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessio

[jira] [Created] (HIVE-13005) CBO: Calcite Operator To Hive Operator (Calcite Return Path): RexNode convert(ExprNodeConstantDesc literal) decimal support bug

2016-02-04 Thread Hari Sankar Sivarama Subramaniyan (JIRA) 
Hari Sankar Sivarama Subramaniyan created HIVE-13005:


 Summary: CBO: Calcite Operator To Hive Operator (Calcite Return 
Path): RexNode convert(ExprNodeConstantDesc literal)  decimal support bug
 Key: HIVE-13005
 URL: https://issues.apache.org/jira/browse/HIVE-13005
 Project: Hive
  Issue Type: Sub-task
Reporter: Hari Sankar Sivarama Subramaniyan


HIVE-8064 seems to have introduced this code in RexNodeConverter::convert(), 
but the parameters look like they  are wrongly passed :

{code}
RelDataType relType = 
cluster.getTypeFactory().createSqlType(SqlTypeName.DECIMAL,
bd.scale(), unscaled.toString().length());
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13004) Remove encryption shims

2016-02-04 Thread Ashutosh Chauhan (JIRA) 
Ashutosh Chauhan created HIVE-13004:
---

 Summary: Remove encryption shims
 Key: HIVE-13004
 URL: https://issues.apache.org/jira/browse/HIVE-13004
 Project: Hive
  Issue Type: Task
  Components: Encryption
Reporter: Ashutosh Chauhan
Assignee: Sergio Peña


It has served its purpose. Now that we don't support hadoop-1, its no longer 
needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 43115: HIVE-12924 CBO: Calcite Operator To Hive Operator (Calcite Return Path): TestCliDriver groupby_ppr_multi_distinct.q failure

2016-02-04 Thread Hari Sankar Sivarama Subramaniyan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43115/
---

(Updated Feb. 4, 2016, 9:29 p.m.)


Review request for hive, Jesús Camacho Rodríguez and John Pullokkaran.


Changes
---

Thanks John for the review.

The naming convention for the Distinct UDAF field for the GBY in the reduce 
side : :._col_. It seems that currently we dont generate the colExprMap 
correctly for the above convention in HiveGBOpUtil.genMapSideRS(). The 
ReduceSide GBY pipeling looks good to me in the current return path code. Since 
we are not generating the entries for the correct columns in the MapSide Reduce 
Operator, we run into an exception when we look for an entry corresponding to a 
column in the reduce side aggreagation.

There is another optimization which can possibly done in the below 
scenario(after turning off mapside aggr):
explain FROM srcpart src SELECT count(DISTINCT src.value), count(DISTINCT 
src.key,src.key), sum(DISTINCT src.value) WHERE src.ds = '2008-04-08' GROUP BY 
substr(src.key,1,1);

The Reduce Operator Tree :
...
  Reduce Operator Tree:
Group By Operator
  aggregations: count(DISTINCT KEY._col1:0._col0), count(DISTINCT 
KEY._col1:1._col0, KEY._col1:1._col1), sum(DISTINCT KEY._col1:2._col0)
  keys: KEY._col0 (type: string)
  mode: complete
  outputColumnNames: _col0, _col1, _col2, _col3
  Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column 
stats: NONE
  Select Operator
  ..
As you can see, 
1. KEY._col1:1._col0, KEY._col1:1._col1 is mapped to the same column and hence 
we could have used the same column in the rowschema of the ReduceSink Operator 
pipeline
2. KEY._col1:2._col0,  KEY._col1:0._col0 is mapped to the same column and we 
can do the same thing mentioned in 1.

I verified that this happens even in the non-return path code and should be 
covered as a general change as a further optimization in a separate jira.

Thanks
Hari


Repository: hive-git


Description
---

CBO: Calcite Operator To Hive Operator (Calcite Return Path): TestCliDriver 
groupby_ppr_multi_distinct.q failure


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveGBOpConvUtil.java
 7fbf8cd 

Diff: https://reviews.apache.org/r/43115/diff/


Testing
---

Precommit runs


Thanks,

Hari Sankar Sivarama Subramaniyan

[jira] [Created] (HIVE-13003) remove the code to create emptyFile from Hive

2016-02-04 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-13003:
---

 Summary: remove the code to create emptyFile from Hive
 Key: HIVE-13003
 URL: https://issues.apache.org/jira/browse/HIVE-13003
 Project: Hive
  Issue Type: Improvement
Reporter: Sergey Shelukhin


After HIVE-12950, it would be nice to see if this code is needed anywhere any 
more.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 42948: HIVE-12950 support NullRowInputFormat in LLAP for nullscan cases

2016-02-04 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42948/
---

(Updated Feb. 4, 2016, 8:47 p.m.)


Review request for hive and Gopal V.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties fd2c924 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java 2b690f8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java dbe4f80 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ScriptOperator.java 9f311a4 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 5e0553d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java a1b7445 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java f4496df 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorGroupByOperator.java 
b7ce309 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 8b229af 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 1c0f4cd 
  ql/src/java/org/apache/hadoop/hive/ql/io/NullRowsInputFormat.java 9638f2a 
  ql/src/java/org/apache/hadoop/hive/ql/io/NullScanFileSystem.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/OneNullRowInputFormat.java 91a6d5a 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapDecider.java 
ae64749 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/NullScanTaskDispatcher.java
 0c4519c 
  ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 5914b1c 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 3ef50fc 
  ql/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem 
PRE-CREATION 
  ql/src/test/queries/clientpositive/llap_nullscan.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/llap_nullscan.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/metadataonly1.q.out 8da7564 
  ql/src/test/results/clientpositive/optimize_nullscan.q.out 1f515c8 
  ql/src/test/results/clientpositive/spark/optimize_nullscan.q.out 4f4a9c8 
  ql/src/test/results/clientpositive/tez/llap_nullscan.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez/metadataonly1.q.out 3c3a2ae 
  ql/src/test/results/clientpositive/tez/optimize_nullscan.q.out d57597c 
  
storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedRowBatch.java
 380300e 

Diff: https://reviews.apache.org/r/42948/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-13002) metastore call timing is not threadsafe

2016-02-04 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-13002:
---

 Summary: metastore call timing is not threadsafe
 Key: HIVE-13002
 URL: https://issues.apache.org/jira/browse/HIVE-13002
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


Discovered in some q test run:
{noformat}
 TestCliDriver.testCliDriver_insert_values_orig_table:123->runTest:199 
Unexpected exception java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
at java.util.HashMap$EntryIterator.next(HashMap.java:966)
at java.util.HashMap$EntryIterator.next(HashMap.java:964)
at 
org.apache.hadoop.hive.ql.metadata.Hive.dumpAndClearMetaCallTiming(Hive.java:3412)
at 
org.apache.hadoop.hive.ql.Driver.dumpMetaCallTimingWithoutEx(Driver.java:574)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1722)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1342)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13001) Hive pre-commits builds taking much longer than normal

2016-02-04 Thread Naveen Gangam (JIRA) 
Naveen Gangam created HIVE-13001:


 Summary: Hive pre-commits builds taking much longer than normal
 Key: HIVE-13001
 URL: https://issues.apache.org/jira/browse/HIVE-13001
 Project: Hive
  Issue Type: Test
  Components: Hive
Affects Versions: 1.3.0
Reporter: Naveen Gangam
Assignee: Sergio Peña


http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6863/

Build took 6+ hours to complete.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13000) Hive returns useless parsing error

2016-02-04 Thread Alina Abramova (JIRA) 
Alina Abramova created HIVE-13000:
-

 Summary: Hive returns useless parsing error 
 Key: HIVE-13000
 URL: https://issues.apache.org/jira/browse/HIVE-13000
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.2.1, 1.0.0, 0.13.0
Reporter: Alina Abramova
Assignee: Alina Abramova
Priority: Minor


When I run query like these I receive unclear exception

hive> SELECT record FROM ctest GROUP BY record.instance_id;
FAILED: SemanticException Error in parsing 

It will be clearer if it would be like:

hive> SELECT record FROM ctest GROUP BY record.instance_id;
FAILED: SemanticException  Expression not in GROUP BY key record



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 43192: HIVE-9534: incorrect result set for query that projects a windowed aggregate

2016-02-04 Thread Aihua Xu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43192/
---

(Updated Feb. 4, 2016, 4:31 p.m.)


Review request for hive, Chaoyu Tang, Szehon Ho, Xuefu Zhang, and Yongzhi Chen.


Repository: hive-git


Description
---

HIVE-9534: incorrect result set for query that projects a windowed aggregate


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/CanAggregateDistinct.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlCountAggFunction.java
 58191e5 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlSumAggFunction.java
 498cd0e 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
 19aa414 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFAverage.java 
cd2449f 
  ql/src/test/queries/clientpositive/windowing_distinct.q 9f6ddfd 
  ql/src/test/results/clientpositive/windowing_distinct.q.out 0858f0f 

Diff: https://reviews.apache.org/r/43192/diff/


Testing
---

Similar to sum(distinct) and count(distinct) supported, add avg(distinct) in 
this patch.


Thanks,

Aihua Xu

Review Request 43192: HIVE-9534: incorrect result set for query that projects a windowed aggregate

2016-02-04 Thread Aihua Xu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43192/
---

Review request for hive, Chaoyu Tang, Szehon Ho, Xuefu Zhang, and Yongzhi Chen.


Repository: hive-git


Description
---

HIVE-9534: incorrect result set for query that projects a windowed aggregate


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/CanAggregateDistinct.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlCountAggFunction.java
 58191e5e6e3d999f176884c6b71e6a781d9ba3ff 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlSumAggFunction.java
 498cd0ecaa52afa890e22a4f35754ad15dd4664d 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
 19aa414419bc0d3525ce3d05a59e2a17724b495e 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFAverage.java 
cd2449fdc2d1cdd595c764e9b3da4c0811b2e763 
  ql/src/test/queries/clientpositive/windowing_distinct.q 
9f6ddfd0934bc695f69576b5a7523fe7d49be4e6 
  ql/src/test/results/clientpositive/windowing_distinct.q.out 
0858f0f38c977c77fd0fc9919094f48ebf3a09f3 

Diff: https://reviews.apache.org/r/43192/diff/


Testing
---

Similar to sum(distinct) and count(distinct) supported, add avg(distinct) in 
this patch.


Thanks,

Aihua Xu

Re: Review Request 43176: HIVE-12965: Insert overwrite local directory should perserve the overwritten directory permission

2016-02-04 Thread Chaoyu Tang 


> On Feb. 4, 2016, 2:51 p.m., Xuefu Zhang wrote:
> >

Thanks Xuefu for review. I answered all the questions and please let me know if 
they make sense.


> On Feb. 4, 2016, 2:51 p.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java, line 152
> > 
> >
> > Instead of doing this, should we explicitly check the existance of the 
> > destination?

Yeah, if the code is like following and first explicitly checks the existence 
of target, it might be more readable. Instead I combined the exists and 
listStatus into one listStatus call which can save one trip to HDFS in case 
that targetPath exists.

if (dstFs.exists(targetPath)) {
   FileStatus[] destFiles = dstFs.listStatus(targetPath);
   if (dstFs.isDirectory(targetPath)) {
 for (FileStatus destFile : destFiles) {
   if (!dstFs.delete(destFile.getPath(), true)) {
 throw new IOException("Unable to clean the destination directory: " + 
targetPath);
   }
 }
   }
} else {
   if (!FileUtils.mkdir(dstFs, targetPath, false, conf)) {
 throw new HiveException("Failed to create local target directory " + 
targetPath);
}


> On Feb. 4, 2016, 2:51 p.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java, line 143
> > 
> >
> > Should we check if it's a directory before calling listStatus() on it?

By first calling FileStatus[] destFiles = dstFs.listStatus(targetPath), we can 
tell if the file with path targetPath exists or not. If yes, we then check if 
it is directory, otherwise it throws FileNotFoundException and in its catch 
block we create a directory with targetPath. (Please see explanation in next 
comment for the reason why it is implemented in this way).
But if we do dstFs.isDirectory(targetPath) first, for a non-existing directory, 
it returns false instead of throwing FileNotFoundException and going to 
FileNotFoundException block to create the target directory.
Yeah, I initally did in that way and found out that trick in the testing.


> On Feb. 4, 2016, 2:51 p.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java, line 137
> > 
> >
> > two nested try-catch block seem making the code hard to read.

Yeah, the outer try-catch
  } catch (IOException ioe) {
throw new HiveException("Unable to move source " + sourcePath + " to 
destination "
+ targetPath, ioe);
  }
Is mainly for logging purpose. Should we remove?


On Feb. 4, 2016, 2:51 p.m., Chaoyu Tang wrote:
> > On a high level, deleting file one by one is slower. Could we instead 
> > remember the original permission and set it to the new directory that we 
> > are going to replace?

Yeah, I initially implemented it in the way you suggested which is more 
performant. But it could not get the right permission to my local directory 
probably due to the defect in HDFS RawLocalFileSystem. It always return the 
HDFS default one (0666)to all local files/directory. I have put comment in code 
to explain why I delete the file one by one.
  // RawLocalFileSystem seems not able to get the right permissions for 
a local file, it
  // always returns hdfs default permission (00666). So we can not 
overwrite a directory
  // by deleting and recreating the directory and restoring its 
permissions. We should
  // delete all its files and subdirectories instead.


- Chaoyu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43176/#review117816
---


On Feb. 4, 2016, 4:07 a.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43176/
> ---
> 
> (Updated Feb. 4, 2016, 4:07 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-12965
> https://issues.apache.org/jira/browse/HIVE-12965
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> In Hive, "insert overwrite local directory" first deletes the overwritten 
> directory if exists, recreate a new one, then copy the files from src 
> directory to the new local directory. This process sometimes changes the 
> permissions of the to-be-overwritten local directory, therefore causing some 
> applications no more to be able to access its content.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java e9cd450 
> 
> Diff: https://reviews.apache.org/r/43176/diff/
> 
> 
> Testing
> ---
> 
> Manual tests
> Precommit tests
> 
> 
> Thanks,
> 
> Chaoyu Tang

Re: Review Request 43176: HIVE-12965: Insert overwrite local directory should perserve the overwritten directory permission

2016-02-04 Thread Xuefu Zhang 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43176/#review117816
---




ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java (line 129)


two nested try-catch block seem making the code hard to read.



ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java (line 135)


Should we check if it's a directory before calling listStatus() on it?



ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java (line 144)


Instead of doing this, should we explicitly check the existance of the 
destination?


On a high level, deleting file one by one is slower. Could we instead remember 
the original permission and set it to the new directory that we are going to 
replace?

- Xuefu Zhang


On Feb. 4, 2016, 4:07 a.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43176/
> ---
> 
> (Updated Feb. 4, 2016, 4:07 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-12965
> https://issues.apache.org/jira/browse/HIVE-12965
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> In Hive, "insert overwrite local directory" first deletes the overwritten 
> directory if exists, recreate a new one, then copy the files from src 
> directory to the new local directory. This process sometimes changes the 
> permissions of the to-be-overwritten local directory, therefore causing some 
> applications no more to be able to access its content.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java e9cd450 
> 
> Diff: https://reviews.apache.org/r/43176/diff/
> 
> 
> Testing
> ---
> 
> Manual tests
> Precommit tests
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>