[jira] [Created] (HIVE-12838) Add methods for getting and storing serialized ORC file tails

2016-01-11 Thread Owen O'Malley (JIRA) 
Owen O'Malley created HIVE-12838:


 Summary: Add methods for getting and storing serialized ORC file 
tails
 Key: HIVE-12838
 URL: https://issues.apache.org/jira/browse/HIVE-12838
 Project: Hive
  Issue Type: Sub-task
Reporter: Owen O'Malley
Assignee: Owen O'Malley


Provide a pair of routines for getting and restoring from a serialized file 
footer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12841) Document existence of multiple hive-site.xml files

2016-01-11 Thread Lefty Leverenz (JIRA) 
Lefty Leverenz created HIVE-12841:
-

 Summary: Document existence of multiple hive-site.xml files
 Key: HIVE-12841
 URL: https://issues.apache.org/jira/browse/HIVE-12841
 Project: Hive
  Issue Type: Bug
  Components: Documentation
Reporter: Lefty Leverenz


The wiki's AdminManual Configuration doc discusses hive-site.xml without giving 
any filepaths or explaining which files override the others.  This needs to be 
clarified, as well as the relationship between hive-site.xml and HiveConf.java.

* [AdminManual -- Configuration -- Configuring Hive | 
https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration#AdminManualConfiguration-ConfiguringHive]
* [AdminManual -- Configuration -- hive-site.xml and hive-default.xml.template 
| 
https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration#AdminManualConfiguration-hive-site.xmlandhive-default.xml.template]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12840) stats optimize subqueries whenever possible in a union query

2016-01-11 Thread Chaoyu Tang (JIRA) 
Chaoyu Tang created HIVE-12840:
--

 Summary: stats optimize subqueries whenever possible in a union 
query
 Key: HIVE-12840
 URL: https://issues.apache.org/jira/browse/HIVE-12840
 Project: Hive
  Issue Type: Improvement
  Components: Logical Optimizer
Reporter: Chaoyu Tang


HIVE-12788 addressed a data incorrect issue in union query with aggregate 
function when stats optimization is enabled. It won't stats optimize a query if 
any of its subqueries can not be. 
[~pxiong] suggested an enhancement to leverage the stats optimizer whenever 
possible (even only for a branch of a union), and we need investigate the 
possible solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 42081: HIVE-12788: Setting hive.optimize.union.remove to TRUE will break UNION ALL with aggregate functions

2016-01-11 Thread Chaoyu Tang 


> On Jan. 11, 2016, 11:05 p.m., pengcheng xiong wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/StatsOptimizer.java, line 
> > 226
> > 
> >
> > The patch looks good to me. +1. Could you put the reason that we 
> > discussed before here so that people will have a better understanding? And 
> > also please list the future work to improve it. Thanks.

This patch is going to address the critical data incorrect issue in a union 
with aggregate function when stats optimization is enabled. It bails out the 
query stats optimization if any of its branch is not able to be. A followed up 
JIRA HIVE-12840 has been opened to investigate the solution as Pengcheng 
suggested to leverage stats optimizer whenever possible (even only for a branch 
of a union). See our detailed discussion in HIVE-12788.


- Chaoyu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42081/#review113865
---


On Jan. 10, 2016, 2:17 p.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42081/
> ---
> 
> (Updated Jan. 10, 2016, 2:17 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, pengcheng xiong, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-12788
> https://issues.apache.org/jira/browse/HIVE-12788
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> adds StatsOptimizator support to union with aggregate function. Otherwise, it 
> always returns one row.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/StatsOptimizer.java 03c1c3f 
>   ql/src/test/queries/clientpositive/union_remove_26.q PRE-CREATION 
>   ql/src/test/results/clientpositive/union_remove_26.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/42081/diff/
> 
> 
> Testing
> ---
> 
> 1. Manual tests for some partitcular cases
> 2. submitted to precommit-tests
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>

[jira] [Created] (HIVE-12839) Upgrade Hive to Calcite 1.6

2016-01-11 Thread Pengcheng Xiong (JIRA) 
Pengcheng Xiong created HIVE-12839:
--

 Summary: Upgrade Hive to Calcite 1.6
 Key: HIVE-12839
 URL: https://issues.apache.org/jira/browse/HIVE-12839
 Project: Hive
  Issue Type: Improvement
Reporter: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 41153: HIVE-12640 : Allow StatsOptimizer to optimize the query for Constant GroupBy keys

2016-01-11 Thread Hari Sankar Sivarama Subramaniyan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41153/
---

(Updated Jan. 11, 2016, 10:44 p.m.)


Review request for hive, Ashutosh Chauhan and John Pullokkaran.


Repository: hive-git


Description
---

HIVE-12640 : Allow StatsOptimizer to optimize the query for Constant GroupBy 
keys


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/StatsOptimizer.java 03c1c3f 
  ql/src/test/queries/clientpositive/metadata_only_queries.q bce121d 
  ql/src/test/results/clientpositive/metadata_only_queries.q.out 65a4dfa 

Diff: https://reviews.apache.org/r/41153/diff/


Testing
---


Thanks,

Hari Sankar Sivarama Subramaniyan

Re: Review Request 42081: HIVE-12788: Setting hive.optimize.union.remove to TRUE will break UNION ALL with aggregate functions

2016-01-11 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42081/#review113865
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/StatsOptimizer.java (line 226)


The patch looks good to me. +1. Could you put the reason that we discussed 
before here so that people will have a better understanding? And also please 
list the future work to improve it. Thanks.


- pengcheng xiong


On Jan. 10, 2016, 2:17 p.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42081/
> ---
> 
> (Updated Jan. 10, 2016, 2:17 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, pengcheng xiong, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-12788
> https://issues.apache.org/jira/browse/HIVE-12788
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> adds StatsOptimizator support to union with aggregate function. Otherwise, it 
> always returns one row.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/StatsOptimizer.java 03c1c3f 
>   ql/src/test/queries/clientpositive/union_remove_26.q PRE-CREATION 
>   ql/src/test/results/clientpositive/union_remove_26.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/42081/diff/
> 
> 
> Testing
> ---
> 
> 1. Manual tests for some partitcular cases
> 2. submitted to precommit-tests
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>

Re: Review Request 42116: HIVE-12824

2016-01-11 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42116/#review113718
---

Ship it!


Ship It!

- Jesús Camacho Rodríguez


On Jan. 10, 2016, 5:25 p.m., Ashutosh Chauhan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42116/
> ---
> 
> (Updated Jan. 10, 2016, 5:25 p.m.)
> 
> 
> Review request for hive and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-12824
> https://issues.apache.org/jira/browse/HIVE-12824
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> CBO doesnt get triggered when aggregate function is used within windowing 
> function
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/PlanModifierForASTConv.java
>  b77beb8 
>   ql/src/test/queries/clientpositive/windowing_gby.q b948f76 
>   ql/src/test/results/clientpositive/groupby_resolution.q.out ea40014 
>   ql/src/test/results/clientpositive/quotedid_basic.q.out 29736af 
>   ql/src/test/results/clientpositive/tez/windowing_gby.q.out e65533f 
> 
> Diff: https://reviews.apache.org/r/42116/diff/
> 
> 
> Testing
> ---
> 
> windowing_gby.q
> 
> 
> Thanks,
> 
> Ashutosh Chauhan
> 
>

Re: Review Request 41928: HIVE-12777: Add capability to restore session

2016-01-11 Thread Rajat Khandelwal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41928/
---

(Updated Jan. 11, 2016, 3:52 p.m.)


Review request for hive.


Bugs: HIVE-12777
https://issues.apache.org/jira/browse/HIVE-12777


Repository: hive-git


Description
---

Extensions using Hive session handles should be able to restore the hive 
session from the handle. 

Apache Lens depends on a fork of hive and that fork has such a capability. 

Relevant commit: 
https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be


Diffs (updated)
-

  
itests/hive-unit/src/test/java/org/apache/hive/service/cli/TestEmbeddedThriftBinaryCLIService.java
 de66d9efb1cace9d32174e3020920d5e4002dc85 
  service/if/TCLIService.thrift baf583fb3ecd4a0ab248540ffb5db57100028968 
  service/src/gen/thrift/gen-cpp/TCLIService.h 
2ea80c76e4e8618c5535f8270f20c70c8b35f765 
  service/src/gen/thrift/gen-cpp/TCLIService.cpp 
7e2c028dc5e038e40d4dfb5e06a80789b4eabe3b 
  service/src/gen/thrift/gen-cpp/TCLIService_server.skeleton.cpp 
988bb4c11ddb717f585e0ba2fb4773ec5fff77e6 
  service/src/gen/thrift/gen-cpp/TCLIService_types.h 
b078c99db2f436e1ac32514b8f873c150952900b 
  service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 
b85237994f8019bd58aa1c8fdbef3aabce9940e2 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCLIService.java
 ded848f2b81bf698a819732a2e9ea4bc67fd2a26 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementReq.java
 30cf243f541304a2ac05c775c912d542520ebe05 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTablesReq.java
 4f2273214ab43b1116008183ff42a520769c6710 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRestoreSessionReq.java
 PRE-CREATION 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRestoreSessionResp.java
 PRE-CREATION 
  service/src/gen/thrift/gen-php/TCLIService.php 
eba62f12de0899280b9961dd82330bb149d40758 
  service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote 
56f5c5dc36d44627a2cd571c2bc1fcd1a79e62f2 
  service/src/gen/thrift/gen-py/TCLIService/TCLIService.py 
ad2d71d720376c76632d9e3a147f227b3ff67c17 
  service/src/gen/thrift/gen-py/TCLIService/ttypes.py 
ef5f5f568ffec92c093bdd0904c154f8ee67606e 
  service/src/gen/thrift/gen-rb/t_c_l_i_service.rb 
7d7f7a78a17cab50b18fa77805022bf4ca0197d9 
  service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 
f004ec40afd29ab4ed452f65f306cb1af5ff5207 
  service/src/java/org/apache/hive/service/cli/CLIService.java 
adc98098437b2d0429dd2af40d8eaae323fbf3f0 
  service/src/java/org/apache/hive/service/cli/CLIServiceClient.java 
3155c238ff688bfea16b0aaeea950599bb659b5b 
  service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 
9cad5be198c063115a8e90c67b1c2fd910ca8bc6 
  service/src/java/org/apache/hive/service/cli/ICLIService.java 
c9cc1f4da56f1cd10f6348ea2b9e17e203b87664 
  service/src/java/org/apache/hive/service/cli/SessionHandle.java 
52e0ad4834d8b44d5038bb5c1a851eb702cd38ca 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
50c912ee606bce73f447fba69ce0f57d8c7fc6c9 
  service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
637cdca3bcea9d320f397de032fd78d59ce6afef 
  
service/src/java/org/apache/hive/service/cli/thrift/RetryingThriftCLIServiceClient.java
 529eaa45d29d22e0e80ea2c08a27b84f2909eda3 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
8434965e87366e86d27685c5ca70ee904446bae9 
  
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java 
1af45398b895cd7616c5627d318422e14b81e734 
  service/src/test/org/apache/hive/service/cli/CLIServiceTest.java 
e78181a15993d99f1cab5a061c08bb21823d2171 

Diff: https://reviews.apache.org/r/41928/diff/


Testing
---


Thanks,

Rajat Khandelwal

Re: Anyone knows which tool does hive used to generate shell utilities?

2016-01-11 Thread zuoca 
@Owen Got it! thank you.

在 2016年1月12日，上午12:08，Owen O'Malley  写道：

> Do you mean the commands like orcfiledump? The scripts are in bin/ext/*
> from the source directory.
> 
> .. Owen
> 
> On Mon, Jan 11, 2016 at 8:02 AM, zuoca  wrote:
> 
>> Hi,all
>> 
>> Anyone knows which tool does hive used to generate shell utilities?
>> 
>> I’ve check the code but still cannot found the answer.
>> 
>> Thank you
>>

[jira] [Created] (HIVE-12842) Need to verify existing schema is compatible for Schema Evolution (e.g. doesn't have column renames, reorders, incompatible type changes, etc)

2016-01-11 Thread Matt McCline (JIRA) 
Matt McCline created HIVE-12842:
---

 Summary: Need to verify existing schema is compatible for Schema 
Evolution (e.g. doesn't have column renames, reorders, incompatible type 
changes, etc)
 Key: HIVE-12842
 URL: https://issues.apache.org/jira/browse/HIVE-12842
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 41482: StatsSetupConst.COLUMN_STATS_ACCURATE is not used correctly

2016-01-11 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41482/
---

(Updated Jan. 12, 2016, 5:17 a.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-12661


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 2ff76ee 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
432f7d0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java f6fbe74 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java ac0ecd9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java edf69fe 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java dca1198 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java f151871 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/StatsOptimizer.java 03c1c3f 
  ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java d87022d 
  ql/src/test/queries/clientpositive/columnStatsUpdateForStatsOptimizer_1.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/columnStatsUpdateForStatsOptimizer_2.q 
PRE-CREATION 
  ql/src/test/results/clientnegative/stats_partialscan_autogether.q.out d03c249 
  ql/src/test/results/clientnegative/unset_table_property.q.out 158ed38 
  ql/src/test/results/clientpositive/alter_file_format.q.out c9e88f8 
  ql/src/test/results/clientpositive/alter_merge_stats_orc.q.out cefe069 
  
ql/src/test/results/clientpositive/alter_numbuckets_partitioned_table2_h23.q.out
 29a4c4b 
  
ql/src/test/results/clientpositive/alter_numbuckets_partitioned_table_h23.q.out 
4d0f841 
  ql/src/test/results/clientpositive/alter_partition_clusterby_sortby.q.out 
f5e8d1f 
  ql/src/test/results/clientpositive/alter_partition_coltype.q.out 08bd2b8 
  ql/src/test/results/clientpositive/alter_skewed_table.q.out 03904e6 
  ql/src/test/results/clientpositive/alter_table_not_sorted.q.out 40974e4 
  ql/src/test/results/clientpositive/alter_table_serde2.q.out 3b63e7d 
  ql/src/test/results/clientpositive/analyze_table_null_partition.q.out c7254d1 
  ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 91b89d7 
  ql/src/test/results/clientpositive/auto_sortmerge_join_1.q.out a48c148 
  ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out 206a619 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 69ae0bf 
  ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out fbc500b 
  ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out 9ae6861 
  ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out 357bc4c 
  ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out b763e52 
  ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out cef3bd3 
  ql/src/test/results/clientpositive/auto_sortmerge_join_8.q.out 567e3cf 
  ql/src/test/results/clientpositive/binary_output_format.q.out 30d0037 
  ql/src/test/results/clientpositive/bucket1.q.out 8009514 
  ql/src/test/results/clientpositive/bucket2.q.out 75de8a9 
  ql/src/test/results/clientpositive/bucket3.q.out c459870 
  ql/src/test/results/clientpositive/bucket4.q.out e0e5fa6 
  ql/src/test/results/clientpositive/bucket5.q.out c1dd90d 
  ql/src/test/results/clientpositive/bucket_many.q.out 04ec255 
  ql/src/test/results/clientpositive/bucket_map_join_1.q.out c7a8a20 
  ql/src/test/results/clientpositive/bucket_map_join_2.q.out 3c3793f 
  ql/src/test/results/clientpositive/bucket_map_join_spark1.q.out 8411a9b 
  ql/src/test/results/clientpositive/bucket_map_join_spark2.q.out 008d57a 
  ql/src/test/results/clientpositive/bucket_map_join_spark3.q.out d5ca1ad 
  ql/src/test/results/clientpositive/bucket_map_join_spark4.q.out 42abaa3 
  ql/src/test/results/clientpositive/bucketcontext_1.q.out d28e582 
  ql/src/test/results/clientpositive/bucketcontext_2.q.out 9f92030 
  ql/src/test/results/clientpositive/bucketcontext_3.q.out 5a847f0 
  ql/src/test/results/clientpositive/bucketcontext_4.q.out 19ed7f9 
  ql/src/test/results/clientpositive/bucketcontext_5.q.out e1a911b 
  ql/src/test/results/clientpositive/bucketcontext_6.q.out 0d3bf6e 
  ql/src/test/results/clientpositive/bucketcontext_7.q.out 15f1335 
  ql/src/test/results/clientpositive/bucketcontext_8.q.out 25d0c3c 
  ql/src/test/results/clientpositive/bucketmapjoin1.q.out 8999fe6 
  ql/src/test/results/clientpositive/bucketmapjoin10.q.out ec6069f 
  ql/src/test/results/clientpositive/bucketmapjoin11.q.out 15be0af 
  ql/src/test/results/clientpositive/bucketmapjoin12.q.out b87685c 
  ql/src/test/results/clientpositive/bucketmapjoin13.q.out cdb7637 
  ql/src/test/results/clientpositive/bucketmapjoin2.q.out fb0e601 
  ql/src/test/results/clientpositive/bucketmapjoin3.q.out e71c5e1 
  ql/src/test/results/clientpositive/bucketmapjoin4.q.out 7dd2202 
  ql/src/test/results/clientpositive/bucketmapjoin5.q.out 15969db

Re: Hive UDF accessing https request

2016-01-11 Thread Prabhu Joseph 
Sergey,

 Yes there may be some difference between Hive Task and the separate
program and so Hive Task could not able to access https request.

Gopal,

We have ca-certificates installed and there is no java subdirectory in
/etc/ssl/certs folder.

ls -ltr /etc/ssl/certs/java/

-
ls: cannot access /etc/ssl/certs/java/: No such file or directory

-

We don't have any dir "/etc/ssl/certs/java/" we have dir till
"/etc/ssl/certs". There is no Java dir here.

=> rpm -qa | grep cert

-
ca-certificates-2013.1.95-65.1.el6_5.noarch
ca-certificates-2014.1.98-65.1.el6.noarch

-
Can you guys help me to check whether sample https request is accessed from
Hive UDF to ensure whether any configuration issue from my end or a Bug in
Hive.



Thanks,
Prabhu Joseph



On Mon, Jan 11, 2016 at 11:53 PM, Sergey Shelukhin 
wrote:

> Hmm, I’ve no idea off the top of my head what this exception means.
> My guess is something is different about the environment in which the Hive
> task vs the separate program is running. Different machine, different user,
> different Java args, path, not sure. It probably cannot find some Java SSL
> thing, e.g. a truststore, from the Hive task, or doesn’t have access to it.
>
>
> From: Prabhu Joseph 
> Reply-To: "u...@hive.apache.org" 
> Date: Sunday, January 10, 2016 at 22:06
> To: "u...@hive.apache.org" 
> Cc: "dev@hive.apache.org" 
> Subject: Re: Hive UDF accessing https request
>
> Thanks Sergey for looking into this.
>
> Below is the Exception we are getting when we use from Hive UDF, but from
> separate java program it works fine
>
> javax.net.ssl.SSLHandshakeException:
> sun.security.validator.ValidatorException: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
> valid certification path to requested target
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1884)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:276)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:270)
> at
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1341)
> at
> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:153)
> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
> at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1016)
> at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
> at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
> at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
> at
> sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563)
> at
> sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1300)
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
> at com.network.logs.udf.ProfoundNew.evaluate(ProfoundNew.java:30)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:1219)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.evaluate(GenericUDFBridge.java:182)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:79)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:540)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
> at

[jira] [Created] (HIVE-12843) One path for ORC ACID reading (Vectorized) when file is original and there are no deltas appears to not do Schema Evolution

2016-01-11 Thread Matt McCline (JIRA) 
Matt McCline created HIVE-12843:
---

 Summary: One path for ORC ACID reading (Vectorized) when file is 
original and there are no deltas appears to not do Schema Evolution
 Key: HIVE-12843
 URL: https://issues.apache.org/jira/browse/HIVE-12843
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Matt McCline
Priority: Critical


>From code reading, it appears that VectorizedOrcInputFormat.getRecordReader 
>fetches the file meta data when an OrcSplit is used ...  It can come down this 
>path for ACID when the split is original and there are no deltas.  Is this ok?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12845) When you try to view file from HWI , you get error with "HTTP ERROR 500" and stacktrace for JSP file. It says that "Unable to complie JSP file"

2016-01-11 Thread Monani Mihir (JIRA) 
Monani Mihir created HIVE-12845:
---

 Summary: When you try to view file from HWI , you get error with 
"HTTP ERROR 500" and stacktrace for JSP file. It says that "Unable to complie 
JSP file"
 Key: HIVE-12845
 URL: https://issues.apache.org/jira/browse/HIVE-12845
 Project: Hive
  Issue Type: Bug
  Components: Web UI
Affects Versions: 1.2.1
 Environment: Software Problem
Reporter: Monani Mihir
Priority: Minor


Here are the details for stacktrace :-

Problem accessing /hwi/view_file.jsp. Reason:

Unable to compile class for JSP: 

An error occurred at line: 61 in the jsp file: /view_file.jsp
The type java.nio.CharBuffer cannot be resolved. It is indirectly referenced 
from required .class files
58:   char [] c = new char [bsize] ;
59:   int cread=-1;
60:   
61:   if( ( cread=br.read(c)) != -1 ){
62:out.println( c ); 
63:   }
64:   br.close(); 


Stacktrace:
Caused by:

org.apache.jasper.JasperException: Unable to compile class for JSP: 

An error occurred at line: 61 in the jsp file: /view_file.jsp
The type java.nio.CharBuffer cannot be resolved. It is indirectly referenced 
from required .class files
58:   char [] c = new char [bsize] ;
59:   int cread=-1;
60:   
61:   if( ( cread=br.read(c)) != -1 ){
62:out.println( c ); 
63:   }
64:   br.close(); 


Stacktrace:
at 
org.apache.jasper.compiler.DefaultErrorHandler.javacError(DefaultErrorHandler.java:85)
at 
org.apache.jasper.compiler.ErrorDispatcher.javacError(ErrorDispatcher.java:330)
at 
org.apache.jasper.compiler.JDTCompiler.generateClass(JDTCompiler.java:435)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:298)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:277)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:265)
at 
org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:564)
at 
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:299)
at 
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:315)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at 
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at 
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at 
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at 
org.mortbay.jetty.handler.RequestLogHandler.handle(RequestLogHandler.java:49)
at 
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at 
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at 
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at 
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at 
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582

I have used below configuration :-


hive.hwi.listen.host
localhost
This is the host address the Hive Web Interface will listen 
on
  
  
hive.hwi.listen.port

This is the port the Hive Web Interface will listen 
on
  
  
hive.hwi.war.file
/lib/hive-hwi-0.11.0.war
This sets the path to the HWI war file, relative to 
${HIVE_HOME}. 
  

Note :- I have usedhwi-0.11.0.war as 'war' file for HWI is not available in tar 
file(1.2.1 version) , also i can't complie code in local to generate it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12844) hive-1.2.1 doesn't return correct value when run select count query

2016-01-11 Thread david (JIRA) 
david created HIVE-12844:


 Summary: hive-1.2.1 doesn't return correct value when run select 
count query
 Key: HIVE-12844
 URL: https://issues.apache.org/jira/browse/HIVE-12844
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 1.2.1
Reporter: david
Priority: Critical


in hbase 1.0.2,I created a table 'test1',it has below rows and values:
hbase(main):027:0> scan 'test1'
ROW   COLUMN+CELL   

 
 a1   column=df1:a2, 
timestamp=1452505991743, value=ddd  

 a1   column=df1:a3, 
timestamp=1452506082723, value=eee  

 a1   column=df1:c2, 
timestamp=1452505705391, value=bbb  

 b1   column=df1:a2, 
timestamp=1452505838737, value=ccc  

 b1   column=df1:a3, 
timestamp=1452506149461, value=fff  

 r1   column=df1:a, 
timestamp=1452507261849, value=hhh  
 
 r1   column=df1:a1, 
timestamp=1452507100774, value=ggg  

 r1   column=df1:c1, 
timestamp=1451221711588, value=aaa

then I created hive-1.2.1 table:
create external table test3(
  key string,
  coll string,
  col2 string,
  col3 string,
  col4 string,
  col5 string,
  col6 string)
  STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
  WITH SERDEPROPERTIES
  ("hbase.columns.mapping" =
  ":key,df1:a,df1:1,df1:a2,df1:a3,df1:c1,df1:c2")
  TBLPROPERTIES("hbase.table.name" = "test1"); 

when I run query in hive:
hive> select * from test3;
OK
a1  NULLNULLddd eee NULLbbb
b1  NULLNULLccc fff NULLNULL
r1  hhh NULLNULLNULLaaa NULL
the result is correct,but when I run:
select count(1) from test3;
Total MapReduce CPU Time Spent: 6 seconds 770 msec
OK
1
it returns "1",I find that it doesn't count the rows where the first column is 
null,
Could you help to analyze this?
by the way the hadoop version is 2.6.0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 41924: HIVE - 11485 : Session close should not close async SQL operations

2016-01-11 Thread Amareshwari Sriramadasu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41924/#review113729
---



ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 


If drop is not happening here, when will it happen?


- Amareshwari Sriramadasu


On Jan. 5, 2016, 10:23 a.m., Deepak Barr wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41924/
> ---
> 
> (Updated Jan. 5, 2016, 10:23 a.m.)
> 
> 
> Review request for hive and Amareshwari Sriramadasu.
> 
> 
> Bugs: HIVE-11485
> https://issues.apache.org/jira/browse/HIVE-11485
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Removed 3 actions on closing of a hive session  :
> 1. Closing of operations running on the session.
> 2. Dropping of the session paths.
> 3. FileSystem.closeAllForUGI() for the session.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 5f15557 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 50c912e 
>   
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java
>  441db7c 
> 
> Diff: https://reviews.apache.org/r/41924/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Barr
> 
>

Review Request 42134: More information to user on GetOperationStatus in Hive Server2 when query is still executing

2016-01-11 Thread Akshay Goyal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42134/
---

Review request for hive and Amareshwari Sriramadasu.


Bugs: HIVE-4570
https://issues.apache.org/jira/browse/HIVE-4570


Repository: hive-git


Description
---

Driver maintains list of running and runnable tasks although that info is not 
exposed outside. It's kept locally in the driver's execute method. We can add 
Driver.getTaskStatuses() to return status on all tasks (both running and 
runnable). Similarly, start and completion times for operations.

Proposed changes are :

struct TGetOperationStatusResp {
  1: required TStatus status
  2: optional TOperationState operationState

  // If operationState is ERROR_STATE, then the following fields may be set
  // sqlState as defined in the ISO/IEF CLI specification
  3: optional string sqlState

  // Internal error code
  4: optional i32 errorCode

  // Error message
  5: optional string errorMessage

  // List of statuses of sub tasks
  6: optional string taskStatus

  // When was the operation started
  7: optional i64 operationStarted
  // When was the operation completed
  8: optional i64 operationCompleted

}


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 29e6315 
  ql/src/java/org/apache/hadoop/hive/ql/TaskStatus.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 0eab63e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java ab7fd93 
  service/if/TCLIService.thrift baf583f 
  service/src/gen/thrift/gen-cpp/TCLIService_types.h b078c99 
  service/src/gen/thrift/gen-cpp/TCLIService_types.cpp b852379 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetOperationStatusResp.java
 99c2409 
  service/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
  service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
  service/src/java/org/apache/hive/service/cli/OperationStatus.java e45b828 
  
service/src/java/org/apache/hive/service/cli/operation/GetCatalogsOperation.java
 8868ec1 
  
service/src/java/org/apache/hive/service/cli/operation/GetColumnsOperation.java 
8ecdc2e 
  
service/src/java/org/apache/hive/service/cli/operation/GetFunctionsOperation.java
 6df1e8a 
  
service/src/java/org/apache/hive/service/cli/operation/GetSchemasOperation.java 
e56686a 
  
service/src/java/org/apache/hive/service/cli/operation/GetTableTypesOperation.java
 a09b39a 
  
service/src/java/org/apache/hive/service/cli/operation/GetTablesOperation.java 
740b851 
  
service/src/java/org/apache/hive/service/cli/operation/GetTypeInfoOperation.java
 2a0fec2 
  
service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java
 04dc6e3 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java d2b3f9c 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
2eaab4a 
  
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java 
1af4539 
  service/src/test/org/apache/hive/service/cli/CLIServiceTest.java e78181a 

Diff: https://reviews.apache.org/r/42134/diff/


Testing
---


Thanks,

Akshay Goyal

Re: Review Request 41928: HIVE-12777: Add capability to restore session

2016-01-11 Thread Rajat Khandelwal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41928/
---

(Updated Jan. 11, 2016, 5:24 p.m.)


Review request for hive.


Bugs: HIVE-12777
https://issues.apache.org/jira/browse/HIVE-12777


Repository: hive-git


Description
---

Extensions using Hive session handles should be able to restore the hive 
session from the handle. 

Apache Lens depends on a fork of hive and that fork has such a capability. 

Relevant commit: 
https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be


Diffs (updated)
-

  
itests/hive-unit/src/test/java/org/apache/hive/service/cli/TestEmbeddedThriftBinaryCLIService.java
 de66d9efb1cace9d32174e3020920d5e4002dc85 
  service/if/TCLIService.thrift baf583fb3ecd4a0ab248540ffb5db57100028968 
  service/src/gen/thrift/gen-cpp/TCLIService.h 
2ea80c76e4e8618c5535f8270f20c70c8b35f765 
  service/src/gen/thrift/gen-cpp/TCLIService.cpp 
7e2c028dc5e038e40d4dfb5e06a80789b4eabe3b 
  service/src/gen/thrift/gen-cpp/TCLIService_server.skeleton.cpp 
988bb4c11ddb717f585e0ba2fb4773ec5fff77e6 
  service/src/gen/thrift/gen-cpp/TCLIService_types.h 
b078c99db2f436e1ac32514b8f873c150952900b 
  service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 
b85237994f8019bd58aa1c8fdbef3aabce9940e2 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCLIService.java
 ded848f2b81bf698a819732a2e9ea4bc67fd2a26 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementReq.java
 30cf243f541304a2ac05c775c912d542520ebe05 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTablesReq.java
 4f2273214ab43b1116008183ff42a520769c6710 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRestoreSessionReq.java
 PRE-CREATION 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRestoreSessionResp.java
 PRE-CREATION 
  service/src/gen/thrift/gen-php/TCLIService.php 
eba62f12de0899280b9961dd82330bb149d40758 
  service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote 
56f5c5dc36d44627a2cd571c2bc1fcd1a79e62f2 
  service/src/gen/thrift/gen-py/TCLIService/TCLIService.py 
ad2d71d720376c76632d9e3a147f227b3ff67c17 
  service/src/gen/thrift/gen-py/TCLIService/ttypes.py 
ef5f5f568ffec92c093bdd0904c154f8ee67606e 
  service/src/gen/thrift/gen-rb/t_c_l_i_service.rb 
7d7f7a78a17cab50b18fa77805022bf4ca0197d9 
  service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 
f004ec40afd29ab4ed452f65f306cb1af5ff5207 
  service/src/java/org/apache/hive/service/cli/CLIService.java 
adc98098437b2d0429dd2af40d8eaae323fbf3f0 
  service/src/java/org/apache/hive/service/cli/CLIServiceClient.java 
3155c238ff688bfea16b0aaeea950599bb659b5b 
  service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 
9cad5be198c063115a8e90c67b1c2fd910ca8bc6 
  service/src/java/org/apache/hive/service/cli/ICLIService.java 
c9cc1f4da56f1cd10f6348ea2b9e17e203b87664 
  service/src/java/org/apache/hive/service/cli/SessionHandle.java 
52e0ad4834d8b44d5038bb5c1a851eb702cd38ca 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
50c912ee606bce73f447fba69ce0f57d8c7fc6c9 
  service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
637cdca3bcea9d320f397de032fd78d59ce6afef 
  
service/src/java/org/apache/hive/service/cli/thrift/RetryingThriftCLIServiceClient.java
 529eaa45d29d22e0e80ea2c08a27b84f2909eda3 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
8434965e87366e86d27685c5ca70ee904446bae9 
  
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java 
1af45398b895cd7616c5627d318422e14b81e734 
  service/src/test/org/apache/hive/service/cli/CLIServiceTest.java 
e78181a15993d99f1cab5a061c08bb21823d2171 

Diff: https://reviews.apache.org/r/41928/diff/


Testing
---


Thanks,

Rajat Khandelwal

Re: Review Request 41928: HIVE-12777: Add capability to restore session

2016-01-11 Thread Amareshwari Sriramadasu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41928/#review113730
---



service/src/test/org/apache/hive/service/cli/CLIServiceTest.java (line 94)


Can you do some operations on restored session?


- Amareshwari Sriramadasu


On Jan. 11, 2016, 10:22 a.m., Rajat Khandelwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41928/
> ---
> 
> (Updated Jan. 11, 2016, 10:22 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-12777
> https://issues.apache.org/jira/browse/HIVE-12777
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/TestEmbeddedThriftBinaryCLIService.java
>  de66d9efb1cace9d32174e3020920d5e4002dc85 
>   service/if/TCLIService.thrift baf583fb3ecd4a0ab248540ffb5db57100028968 
>   service/src/gen/thrift/gen-cpp/TCLIService.h 
> 2ea80c76e4e8618c5535f8270f20c70c8b35f765 
>   service/src/gen/thrift/gen-cpp/TCLIService.cpp 
> 7e2c028dc5e038e40d4dfb5e06a80789b4eabe3b 
>   service/src/gen/thrift/gen-cpp/TCLIService_server.skeleton.cpp 
> 988bb4c11ddb717f585e0ba2fb4773ec5fff77e6 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.h 
> b078c99db2f436e1ac32514b8f873c150952900b 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 
> b85237994f8019bd58aa1c8fdbef3aabce9940e2 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCLIService.java
>  ded848f2b81bf698a819732a2e9ea4bc67fd2a26 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementReq.java
>  30cf243f541304a2ac05c775c912d542520ebe05 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTablesReq.java
>  4f2273214ab43b1116008183ff42a520769c6710 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRestoreSessionReq.java
>  PRE-CREATION 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRestoreSessionResp.java
>  PRE-CREATION 
>   service/src/gen/thrift/gen-php/TCLIService.php 
> eba62f12de0899280b9961dd82330bb149d40758 
>   service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote 
> 56f5c5dc36d44627a2cd571c2bc1fcd1a79e62f2 
>   service/src/gen/thrift/gen-py/TCLIService/TCLIService.py 
> ad2d71d720376c76632d9e3a147f227b3ff67c17 
>   service/src/gen/thrift/gen-py/TCLIService/ttypes.py 
> ef5f5f568ffec92c093bdd0904c154f8ee67606e 
>   service/src/gen/thrift/gen-rb/t_c_l_i_service.rb 
> 7d7f7a78a17cab50b18fa77805022bf4ca0197d9 
>   service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 
> f004ec40afd29ab4ed452f65f306cb1af5ff5207 
>   service/src/java/org/apache/hive/service/cli/CLIService.java 
> adc98098437b2d0429dd2af40d8eaae323fbf3f0 
>   service/src/java/org/apache/hive/service/cli/CLIServiceClient.java 
> 3155c238ff688bfea16b0aaeea950599bb659b5b 
>   service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 
> 9cad5be198c063115a8e90c67b1c2fd910ca8bc6 
>   service/src/java/org/apache/hive/service/cli/ICLIService.java 
> c9cc1f4da56f1cd10f6348ea2b9e17e203b87664 
>   service/src/java/org/apache/hive/service/cli/SessionHandle.java 
> 52e0ad4834d8b44d5038bb5c1a851eb702cd38ca 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 50c912ee606bce73f447fba69ce0f57d8c7fc6c9 
>   service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
> 637cdca3bcea9d320f397de032fd78d59ce6afef 
>   
> service/src/java/org/apache/hive/service/cli/thrift/RetryingThriftCLIServiceClient.java
>  529eaa45d29d22e0e80ea2c08a27b84f2909eda3 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 8434965e87366e86d27685c5ca70ee904446bae9 
>   
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java
>  1af45398b895cd7616c5627d318422e14b81e734 
>   service/src/test/org/apache/hive/service/cli/CLIServiceTest.java 
> e78181a15993d99f1cab5a061c08bb21823d2171 
> 
> Diff: https://reviews.apache.org/r/41928/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Rajat Khandelwal
> 
>

[jira] [Created] (HIVE-12835) BC dates not properly documented / handled

2016-01-11 Thread Carter Shanklin (JIRA) 
Carter Shanklin created HIVE-12835:
--

 Summary: BC dates not properly documented / handled
 Key: HIVE-12835
 URL: https://issues.apache.org/jira/browse/HIVE-12835
 Project: Hive
  Issue Type: Bug
Reporter: Carter Shanklin
Priority: Minor


Minor issue IMO, but since I saw it I'm filing it to document it.

There's something screwy with date types. Documentation says they run from year 
0 to year  ("depending on Java support"). Tests I have done are consistent 
with supporting BC dates, but not identifying the dates as BC dates. Also there 
is some incorrect boundary handling. This is Hive 1.2.1 and Java 1.7.0_91.

To reproduce, create some table with 1 record and run these.

{code}
hive> select cast('2000-01-01' as date) - interval '2000' year from date_test;
OK
0001-01-01
{code}

So far so good

{code}
Time taken: 0.067 seconds, Fetched: 1 row(s)
hive> select cast('2000-01-01' as date) - interval '2100' year from date_test;
OK
0101-01-01
Time taken: 0.055 seconds, Fetched: 1 row(s)
hive> select cast('2000-01-01' as date) - interval '2200' year from date_test;
OK
0201-01-01
Time taken: 0.153 seconds, Fetched: 1 row(s)
{code}

Consistent with BC dates, but not identified as such.

{code}
hive> select cast('2000-01-01' as date) - interval '' year from date_test;
OK
8000-01-01
Time taken: 0.062 seconds, Fetched: 1 row(s)
hive> select cast('2000-01-01' as date) - interval '1' year from date_test;
OK
8001-01-01
Time taken: 0.056 seconds, Fetched: 1 row(s)
hive> select cast('2000-01-01' as date) - interval '12000' year from date_test;
OK
001-01-01
Time taken: 0.054 seconds, Fetched: 1 row(s)
hive> select cast('2000-01-01' as date) - interval '14000' year from date_test;
OK
001-01-01
Time taken: 0.057 seconds, Fetched: 1 row(s)
hive> select cast('2000-01-01' as date) - interval '5' year from date_test;
OK
001-01-01
Time taken: 0.06 seconds, Fetched: 1 row(s)
{code}

Everything from here out collapses down to 001-01-01.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Anyone knows which tool does hive used to generate shell utilities?

2016-01-11 Thread zuoca 
Hi,all

 Anyone knows which tool does hive used to generate shell utilities?

 I’ve check the code but still cannot found the answer.

 Thank you

Re: Anyone knows which tool does hive used to generate shell utilities?

2016-01-11 Thread Owen O'Malley 
Do you mean the commands like orcfiledump? The scripts are in bin/ext/*
from the source directory.

.. Owen

On Mon, Jan 11, 2016 at 8:02 AM, zuoca  wrote:

> Hi,all
>
>  Anyone knows which tool does hive used to generate shell utilities?
>
>  I’ve check the code but still cannot found the answer.
>
>  Thank you
>

Re: Hive UDF accessing https request

2016-01-11 Thread Sergey Shelukhin 
Hmm, I’ve no idea off the top of my head what this exception means.
My guess is something is different about the environment in which the Hive task 
vs the separate program is running. Different machine, different user, 
different Java args, path, not sure. It probably cannot find some Java SSL 
thing, e.g. a truststore, from the Hive task, or doesn’t have access to it.


From: Prabhu Joseph 
>
Reply-To: "u...@hive.apache.org" 
>
Date: Sunday, January 10, 2016 at 22:06
To: "u...@hive.apache.org" 
>
Cc: "dev@hive.apache.org" 
>
Subject: Re: Hive UDF accessing https request

Thanks Sergey for looking into this.

Below is the Exception we are getting when we use from Hive UDF, but from 
separate java program it works fine

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: 
PKIX path building failed: 
sun.security.provider.certpath.SunCertPathBuilderException: unable to find 
valid certification path to requested target
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1884)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:276)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:270)
at 
sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1341)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:153)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1016)
at 
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563)
at 
sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at 
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1300)
at 
sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
at com.network.logs.udf.ProfoundNew.evaluate(ProfoundNew.java:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:1219)
at 
org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.evaluate(GenericUDFBridge.java:182)
at 
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
at 
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
at 
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
at 
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:79)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
at 
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:540)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1469)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: sun.security.validator.ValidatorException: PKIX path building 
failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to 
find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
at sun.security.validator.Validator.validate(Validator.java:260)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
at 
sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231)
at 
sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
at

[jira] [Created] (HIVE-12836) Install wget & curl packages on LXC containers for HMS upgrade tests

2016-01-11 Thread JIRA 
Sergio Peña created HIVE-12836:
--

 Summary: Install wget & curl packages on LXC containers for HMS 
upgrade tests
 Key: HIVE-12836
 URL: https://issues.apache.org/jira/browse/HIVE-12836
 Project: Hive
  Issue Type: Task
  Components: Testing Infrastructure
Affects Versions: 2.1.0
Reporter: Sergio Peña
Assignee: Sergio Peña


Install wget & curl packages on LXC containers for HMS upgrade tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Column data mismatch after join !!!

2016-01-11 Thread David Morel 
On 11 Jan 2016, at 8:54, @Sanjiv Singh wrote:

> Adding to that 
>
> Surprisingly it is giving correct result if i use derived tables rather
> than original tables;

Related to https://issues.apache.org/jira/browse/HIVE-9613 ?

David

[jira] [Created] (HIVE-12837) Better memory estimation/allocation for hybrid grace hash join during hash table loading

2016-01-11 Thread Wei Zheng (JIRA) 
Wei Zheng created HIVE-12837:


 Summary: Better memory estimation/allocation for hybrid grace hash 
join during hash table loading
 Key: HIVE-12837
 URL: https://issues.apache.org/jira/browse/HIVE-12837
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 2.1.0
Reporter: Wei Zheng
Assignee: Wei Zheng


This is to avoid an edge case when the memory available is very little (less 
than a single write buffer size), and we start loading the hash table. Since 
the write buffer is lazily allocated, we will easily run out of memory before 
even checking if we should spill any hash partition.

e.g.
Total memory available: 210 MB
Size of ref array of BytesBytesMultiHashMap for each hash partition: ~16 MB
Size of write buffer: 8 MB (lazy allocation)
# hash partitions: 16
# hash partitions created in memory: 13
# hash partitions created on disk: 3
Available memory left after HybridHashTableContainer initialization: 
210-16*13=2MB

Now let's say a row is to be loaded into a hash partition in memory, it will 
try to allocate an 8MB write buffer for it, but we only have 2MB, thus OOM.

Solution is to perform the check for possible spilling earlier so we can spill 
partitions if memory is about to be full, to avoid OOM.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)