Review request

2017-09-17 Thread Asokan, M 
Please review https://issues.apache.org/jira/browse/HIVE-6476

Review request: https://reviews.apache.org/r/61820/


Thanks.


-- Asokan






ATTENTION: -

The information contained in this message (including any files transmitted with 
this message) may contain proprietary, trade secret or other confidential 
and/or legally privileged information. Any pricing information contained in 
this message or in any files transmitted with this message is always 
confidential and cannot be shared with any third parties without prior written 
approval from Syncsort. This message is intended to be read only by the 
individual or entity to whom it is addressed or by their designee. If the 
reader of this message is not the intended recipient, you are on notice that 
any use, disclosure, copying or distribution of this message, in any form, is 
strictly prohibited. If you have received this message in error, please 
immediately notify the sender and/or Syncsort and destroy all copies of this 
message in your possession, custody or control.

Re: Review Request 62309: HIVE-17465 Fix filters cardinality for AND predicates

2017-09-17 Thread Vineet Garg 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62309/
---

(Updated Sept. 17, 2017, 11:44 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

Update to q files


Bugs: HIVE-17465
https://issues.apache.org/jira/browse/HIVE-17465


Repository: hive-git


Description
---

This patch fixes estimation of cardinality for filter with AND predicates.


Diffs (updated)
-

  data/files/filterCard.txt PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 423913b56b 
  ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out 
6e2975e671 
  ql/src/test/results/clientpositive/annotate_stats_join_pkfk.q.out e04c1c6bc5 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 48c02a24ad 
  ql/src/test/results/clientpositive/llap/vector_decimal_cast.q.out c45210e562 
  ql/src/test/results/clientpositive/llap/vector_if_expr.q.out 32d100180d 
  ql/src/test/results/clientpositive/llap/vectorization_0.q.out db76eaaf1b 
  ql/src/test/results/clientpositive/llap/vectorization_10.q.out f06c2dbcc6 
  ql/src/test/results/clientpositive/llap/vectorization_17.q.out 1c07962d83 
  ql/src/test/results/clientpositive/llap/vectorization_7.q.out 6c32ccf42d 
  ql/src/test/results/clientpositive/llap/vectorization_short_regress.q.out 
66764cf42b 
  ql/src/test/results/clientpositive/llap/vectorized_nested_mapjoin.q.out 
2a95065203 
  ql/src/test/results/clientpositive/perf/query23.q.out dde707d0ea 
  ql/src/test/results/clientpositive/spark/vectorization_10.q.out 1f953575e3 
  ql/src/test/results/clientpositive/spark/vectorization_12.q.out 4e26314499 
  ql/src/test/results/clientpositive/spark/vectorization_17.q.out b43c506c6f 
  ql/src/test/results/clientpositive/vectorization_10.q.out d2d9bf644b 
  ql/src/test/results/clientpositive/vectorization_12.q.out 63ea984a33 
  ql/src/test/results/clientpositive/vectorization_17.q.out c0186082df 
  ql/src/test/results/clientpositive/vectorization_7.q.out 218d307915 


Diff: https://reviews.apache.org/r/62309/diff/3/

Changes: https://reviews.apache.org/r/62309/diff/2-3/


Testing
---

* Added new tests
* Updated existing tests


Thanks,

Vineet Garg

[jira] [Created] (HIVE-17548) ThriftCliService reports inaccurate the number of current sessions in the log message

2017-09-17 Thread Xuefu Zhang (JIRA) 
Xuefu Zhang created HIVE-17548:
--

 Summary: ThriftCliService reports inaccurate the number of current 
sessions in the log message
 Key: HIVE-17548
 URL: https://issues.apache.org/jira/browse/HIVE-17548
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 1.1.0
Reporter: Xuefu Zhang


Currently ThriftCliService uses an atomic integer to keep track of the number 
of currently open sessions. It reports it through the following two log 
messages:
{code}
2017-09-18 04:14:31,722 INFO [HiveServer2-Handler-Pool: Thread-729979]: 
org.apache.hive.service.cli.thrift.ThriftCLIService: Opened a session: 
SessionHandle [99ec30d7-5c44-4a45-a8d6-0f0e7ecf4879], current sessions: 345
2017-09-18 04:14:41,926 INFO [HiveServer2-Handler-Pool: Thread-717542]: 
org.apache.hive.service.cli.thrift.ThriftCLIService: Closed session: 
SessionHandle [f38f7890-cba4-459c-872e-4c261b897e00], current sessions: 344
{code}
This assumes that all sessions are closed or opened thru Thrift API. This 
assumption isn't correct because sessions may be closed by the server such as 
in case of timeout. Therefore, such log messages tends to over-report the 
number of open sessions.

In order to accurately report the number of outstanding sessions, session 
manager should be consulted instead.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62360: HIVE-16898: Validation of source file after distcp in repl load

2017-09-17 Thread anishek 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62360/#review185534
---




ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
Lines 73 (patched)


Evaluation of doing a regularCopy or distCp can be done in the inner most 
function call, this will reduce passing in another variable from the top which 
can be evaluated later



ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
Lines 92 (patched)


I think eventually we have to move to a model of doing the checksum on 
sourceFS vs destinationFS as you have done here, though certain FS 
configurations change the value of checksum and unless we can guarantee that we 
calculate the checksum on the data by reading the data this might lead to more 
failures,

I thought the idea for now was that,

1>> we get the checksum of the file on sourceFS before copy
2>> we do the copy
3>> we get the checksum on the file on sourceFS again 
4>> we compare the checksum in 1 and 3 and if its not changed then during 
our copy the value wouldnt have either. 

until we can figure out the acutal solution to this, the fall back of doing 
the check on sourceFS might be the way to go.



ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
Lines 95 (patched)


the same problem here were our cleaner thread on CM is running sooner than 
expected then we will miss data, might be better to fail in case the file is 
not in CM



ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
Lines 116 (patched)


As a part of doing copy if the copy fails due to fileNotFoundException for 
a file location to actual location on hdfs then we should retry with the 
corresponding CMRoot Path for this file since it was moved while we were in the 
porcess of doing the copy.

Also if this happnes for a CM root file then there is an issue in our 
configuration such that the CM root FS is cleaned before the copy is done and 
we should log this as an error as the cleaner thread for CMroot is not 
configured for the right time. i did rather fail repl load, instead of just 
logging the error else we might not know how many such instances might happen 
before we realize that replication is broken.


- anishek


On Sept. 15, 2017, 6:10 p.m., Daniel Dai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62360/
> ---
> 
> (Updated Sept. 15, 2017, 6:10 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See HIVE-16898
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java 
> 88d6a7a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java 54746d3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java 28e7bcb 
> 
> 
> Diff: https://reviews.apache.org/r/62360/diff/1/
> 
> 
> Testing
> ---
> 
> Manually test it with debugger: setup a breakpoint right before copy, and 
> drop table in another session.
> 
> 
> Thanks,
> 
> Daniel Dai
> 
>

Review Request 62373: [HIVE-17483] HS2 kill command to kill queries using query id

2017-09-17 Thread Teddy Choi 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62373/
---

Review request for hive.


Bugs: HIVE-17483
https://issues.apache.org/jira/browse/HIVE-17483


Repository: hive-git


Description
---

For administrators, it is important to be able to kill queries if required. 
Currently, there is no clean way to do it.
It would help to have a "kill query " command that can be run using 
odbc/jdbc against a HiveServer2 instance, to kill a query with that queryid 
running in that instance.
Authorization will have to be done to ensure that the user that is invoking the 
API is allowed to perform this action.
In case of SQL std authorization, this would require admin role.


Diffs
-

  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
1108934df2 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
4a9af80fdc 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestServiceDiscoveryWithMiniHS2.java
 e8051e40f2 
  itests/src/test/resources/testconfiguration.properties d472bb3f9e 
  jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 1311d2d88c 
  jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java c6bd41feb7 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4e7c80f184 
  ql/src/java/org/apache/hadoop/hive/ql/QueryState.java fa7c32386b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 8aa2d90b76 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java 131c1e1bb5 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java da8c1e2305 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
251decac9b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g b5792ac485 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 429e0d995a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 003e09fd13 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
553dd64b5f 
  ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java 2b9e897a54 
  ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java e1f1f53c3c 
  ql/src/java/org/apache/hadoop/hive/ql/plan/KillQueryDesc.java PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/AuthorizationUtils.java
 04e5565506 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveOperationType.java
 3af97ea02f 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HivePrivilegeObject.java
 41983f1b4c 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/Operation2Privilege.java
 da99972e0c 
  ql/src/java/org/apache/hadoop/hive/ql/session/KillQuery.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/session/NullKillQuery.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 8b64407d53 
  
ql/src/test/org/apache/hadoop/hive/ql/parse/TestSQL11ReservedKeyWordsNegative.java
 1a89eb1263 
  ql/src/test/queries/clientnegative/authorization_kill_query.q PRE-CREATION 
  ql/src/test/queries/clientpositive/kill_query.q PRE-CREATION 
  ql/src/test/results/clientnegative/authorization_kill_query.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/kill_query.q.out PRE-CREATION 
  service-rpc/if/TCLIService.thrift 976ca9b6b3 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService.h 5fd423da6e 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService.cpp 3597d44f2d 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_constants.cpp 874a81bf6b 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_server.skeleton.cpp 5d7caf9783 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h cb62db00ea 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp ec3f066516 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIService.java
 6dba0512b4 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIServiceConstants.java
 955da0c987 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetQueryIdReq.java
 PRE-CREATION 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetQueryIdResp.java
 PRE-CREATION 
  service-rpc/src/gen/thrift/gen-php/TCLIService.php d283145b69 
  service-rpc/src/gen/thrift/gen-php/Types.php f34a0cf30a 
  service-rpc/src/gen/thrift/gen-py/TCLIService/TCLIService-remote 62b87a35cc 
  service-rpc/src/gen/thrift/gen-py/TCLIService/TCLIService.py 35f2ae47c7 
  service-rpc/src/gen/thrift/gen-py/TCLIService/constants.py 0427aa9380 
  service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py acc8c3a1b6 
  service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service.rb a50fe259f7 
  service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_constants.rb 2772170d2e 
  service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 6695aee298 
  service/src/java/org/apache/hive/service/cli/CLIService.java 68