Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-06-20 Thread Takanobu Asanuma 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review138763
---




ql/src/java/org/apache/hadoop/hive/ql/Driver.java (line 1992)


I forgot to add a log when useBypass is true. I will add it in the next 
patch.


- Takanobu Asanuma


On 6月 21, 2016, 2:10 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated 6月 21, 2016, 2:10 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1e0ffa4 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 
> 84644d1 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 0c313a2 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 
> 637e51a 
>   jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 8ac040e 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 92fdbca 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java a242501 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 2263192 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java dff1815 
>   service-rpc/if/TCLIService.thrift 5a9a785 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h d23b3cd 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 0f53cb2 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
>  31472c8 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
>  7101fa5 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
>  1aa3f94 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionReq.java
>  c6279dc 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionResp.java
>  1eaeee6 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
>  14d50ed 
>   service-rpc/src/gen/thrift/gen-php/Types.php a6a257f 
>   service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py fcd330f 
>   service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 71148a0 
>   service/src/java/org/apache/hive/service/cli/CLIService.java ed52b4a 
>   service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java bfd7135 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> d48b92c 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> 2f18231 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 3bf40eb 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 78ff388 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 7341635 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 8bc3d94 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-06-20 Thread Takanobu Asanuma 


> On 6月 18, 2016, 12:21 a.m., Thejas Nair wrote:
> >

I modified some codes in the latest patch.


> On 6月 18, 2016, 12:21 a.m., Thejas Nair wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 57
> > 
> >
> > Use ReflectionUtils that exists within Hive ?
> > The hadoop one is not a public class.
> > I know that we use the hadoop one in other parts, but that is something 
> > we should move away from.

I fixed it to use Hive ReflectionUtils.


> On 6月 18, 2016, 12:21 a.m., Thejas Nair wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 536
> > 
> >
> > Wouldn't it be more performant to use LazyBinarySimpleSerde ?

Excuse me, what is LazyBinarySimpleSerde?


> On 6月 18, 2016, 12:21 a.m., Thejas Nair wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/Driver.java, line 1972
> > 
> >
> > We should log if the bypass is kicking in or not , and the reason for 
> > disabling it.
> > There are so many cases where it gets disabled, it would be hard to 
> > debug in a production environment what the reason is.

I fixed it to log the reasons. Since I am not a native speaker of English, 
please let me know if my English expressions are mistaken.


> On 6月 18, 2016, 12:21 a.m., Thejas Nair wrote:
> > service-rpc/if/TCLIService.thrift, line 277
> > 
> >
> > There is already typeDesc, why do we need typeName

It is to handle complex column types. typeDesc does not have enough infomation 
for complex column types. For example, "struct" is regarded as 
"struct". On the other hand, typename has detailed infomation as is.


- Takanobu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review138347
---


On 6月 21, 2016, 2:10 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated 6月 21, 2016, 2:10 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1e0ffa4 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 
> 84644d1 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 0c313a2 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 
> 637e51a 
>   jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 8ac040e 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 92fdbca 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java a242501 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 2263192 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java dff1815 
>   service-rpc/if/TCLIService.thrift 5a9a785 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h d23b3cd 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 0f53cb2 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
>  31472c8 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
>  7101fa5 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
>  1aa3f94 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionReq.java
>  c6279dc 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionResp.java
>  1eaeee6 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
>  14d50ed 
>   service-rpc/src/gen/thrift/gen-php/Types.php a6a257f 
>   service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py fcd330f 
>   service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 71148a0 
>   service/src/java/org/apache/hive/service/cli/CLIService.java ed52b4a 
>   service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java bfd7135 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> d48b92c 
>   
>

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-06-20 Thread Takanobu Asanuma 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/
---

(Updated 6月 21, 2016, 2:10 a.m.)


Review request for hive.


Changes
---

I updated the patch based on Thejas's advice.


Repository: hive-git


Description
---

This is a WIP patch for HIVE-11527

* I added a new configuration whose name is 
hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
true, clients use the bypass.

* I still have not considered security such as Kerberos and SSL at present.

* I have not implement Statement#setFetchSize for bypass yet.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1e0ffa4 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 
84644d1 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
0c313a2 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 
637e51a 
  jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 8ac040e 
  jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 92fdbca 
  jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java a242501 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 2263192 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java dff1815 
  service-rpc/if/TCLIService.thrift 5a9a785 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h d23b3cd 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 0f53cb2 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
 31472c8 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
 7101fa5 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
 1aa3f94 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionReq.java
 c6279dc 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionResp.java
 1eaeee6 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
 14d50ed 
  service-rpc/src/gen/thrift/gen-php/Types.php a6a257f 
  service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py fcd330f 
  service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 71148a0 
  service/src/java/org/apache/hive/service/cli/CLIService.java ed52b4a 
  service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java bfd7135 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java d48b92c 
  service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
2f18231 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
3bf40eb 
  service/src/java/org/apache/hive/service/cli/session/HiveSession.java 78ff388 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
7341635 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
8bc3d94 

Diff: https://reviews.apache.org/r/40867/diff/


Testing
---

I have tested few simple queries and they worked well. But I think there are 
some problems for some queries. I'm going to test more queries and fix bugs. 
I'm also going to add unit tests.


Thanks,

Takanobu Asanuma

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-06-17 Thread Thejas Nair 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review138347
---




jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 57)


Use ReflectionUtils that exists within Hive ?
The hadoop one is not a public class.
I know that we use the hadoop one in other parts, but that is something we 
should move away from.



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 535)


Wouldn't it be more performant to use LazyBinarySimpleSerde ?



ql/src/java/org/apache/hadoop/hive/ql/Driver.java (line 1972)


We should log if the bypass is kicking in or not , and the reason for 
disabling it.
There are so many cases where it gets disabled, it would be hard to debug 
in a production environment what the reason is.



service-rpc/if/TCLIService.thrift (line 277)


There is already typeDesc, why do we need typeName


- Thejas Nair


On June 15, 2016, 6:50 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated June 15, 2016, 6:50 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 761dbb2 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 
> 84644d1 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 0c313a2 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 
> 637e51a 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 92fdbca 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java a242501 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 2263192 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java dff1815 
>   service-rpc/if/TCLIService.thrift 5a9a785 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h d23b3cd 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 0f53cb2 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
>  31472c8 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
>  7101fa5 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
>  1aa3f94 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
>  14d50ed 
>   service-rpc/src/gen/thrift/gen-php/Types.php a6a257f 
>   service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py fcd330f 
>   service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 71148a0 
>   service/src/java/org/apache/hive/service/cli/CLIService.java ed52b4a 
>   service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java bfd7135 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> d48b92c 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> 2f18231 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 3bf40eb 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 78ff388 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 7341635 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 8bc3d94 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-06-15 Thread Takanobu Asanuma 


> On 6月 3, 2016, 5:50 p.m., Vaibhav Gumashta wrote:
> > itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java,
> >  line 157
> > 
> >
> > Can you add a test with a join query as well? The join query should 
> > write the results in a new intermediate file on hdfs and it will be good to 
> > test that.

I added it in the latest patch.


> On 6月 3, 2016, 5:50 p.m., Vaibhav Gumashta wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 552
> > 
> >
> > Nit: Usually an iterator implements 
> > https://docs.oracle.com/javase/7/docs/api/java/util/Iterator.html.

This class wrappers SequenceFile.Reader. And I think implementing an iterator 
is not suitable in this class. (Implementing hasNext() is not easy.) I just 
changed the class name to avoid confusion.


> On 6月 3, 2016, 5:50 p.m., Vaibhav Gumashta wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 573
> > 
> >
> > Nit: getXXX methods are usually used to return something other than 
> > void.

I think the name is not valid. I changed the name in the latest patch.


> On 6月 3, 2016, 5:50 p.m., Vaibhav Gumashta wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 590
> > 
> >
> > This should be a private method.

I fixed it.


> On 6月 3, 2016, 5:50 p.m., Vaibhav Gumashta wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/Driver.java, line 1978
> > 
> >
> > Shouldn't the read happen independent of the file format? For example, 
> > the default format used to be TextFile until very recently and the user can 
> > as well choose to configure it that way.

I thought handling other formats will make code complex. If this should handle 
TextFile or other formats, I want to create a follow-up jira.


> On 6月 3, 2016, 5:50 p.m., Vaibhav Gumashta wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 536
> > 
> >
> > This hardcodes the serde to LazySimpleSerde. There is work in 
> > https://issues.apache.org/jira/browse/HIVE-12049 where we write using a 
> > different serde in the final tasks. However, I'll create a follow-up jira 
> > for making this more generic.

I got it. Thank you.


> On 6月 3, 2016, 5:50 p.m., Vaibhav Gumashta wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/Driver.java, line 1976
> > 
> >
> > We can create a follow-up jira to handle this.

Shall I create the jira?


- Takanobu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review136073
---


On 6月 15, 2016, 6:50 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated 6月 15, 2016, 6:50 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 761dbb2 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 
> 84644d1 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 0c313a2 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 
> 637e51a 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 92fdbca 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java a242501 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 2263192 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java dff1815 
>   service-rpc/if/TCLIService.thrift 5a9a785 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h d23b3cd 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 0f53cb2 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
>  31472c8 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-06-15 Thread Takanobu Asanuma 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/
---

(Updated 6月 15, 2016, 6:50 a.m.)


Review request for hive.


Changes
---

I updated the patch based on Vaibhav's advice.


Repository: hive-git


Description
---

This is a WIP patch for HIVE-11527

* I added a new configuration whose name is 
hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
true, clients use the bypass.

* I still have not considered security such as Kerberos and SSL at present.

* I have not implement Statement#setFetchSize for bypass yet.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 761dbb2 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 
84644d1 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
0c313a2 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 
637e51a 
  jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 92fdbca 
  jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java a242501 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 2263192 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java dff1815 
  service-rpc/if/TCLIService.thrift 5a9a785 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h d23b3cd 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 0f53cb2 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
 31472c8 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
 7101fa5 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
 1aa3f94 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
 14d50ed 
  service-rpc/src/gen/thrift/gen-php/Types.php a6a257f 
  service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py fcd330f 
  service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 71148a0 
  service/src/java/org/apache/hive/service/cli/CLIService.java ed52b4a 
  service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java bfd7135 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java d48b92c 
  service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
2f18231 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
3bf40eb 
  service/src/java/org/apache/hive/service/cli/session/HiveSession.java 78ff388 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
7341635 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
8bc3d94 

Diff: https://reviews.apache.org/r/40867/diff/


Testing
---

I have tested few simple queries and they worked well. But I think there are 
some problems for some queries. I'm going to test more queries and fix bugs. 
I'm also going to add unit tests.


Thanks,

Takanobu Asanuma

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-06-03 Thread Vaibhav Gumashta 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review136073
---




itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 
(line 157)


Can you add a test with a join query as well? The join query should write 
the results in a new intermediate file on hdfs and it will be good to test that.



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 535)


This hardcodes the serde to LazySimpleSerde. There is work in 
https://issues.apache.org/jira/browse/HIVE-12049 where we write using a 
different serde in the final tasks. However, I'll create a follow-up jira for 
making this more generic.



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 551)


Nit: Usually an iterator implements 
https://docs.oracle.com/javase/7/docs/api/java/util/Iterator.html.



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 572)


Nit: getXXX methods are usually used to return something other than void.



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 589)


This should be a private method.



ql/src/java/org/apache/hadoop/hive/ql/Driver.java (line 1976)


We can create a follow-up jira to handle this.



ql/src/java/org/apache/hadoop/hive/ql/Driver.java (line 1978)


Shouldn't the read happen independent of the file format? For example, the 
default format used to be TextFile until very recently and the user can as well 
choose to configure it that way.


- Vaibhav Gumashta


On May 18, 2016, 9:54 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated May 18, 2016, 9:54 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cbb3a72 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 
> 84644d1 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 815ccfa 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 
> 637e51a 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 92fdbca 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 38ccc78 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 3fecc5c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java dff1815 
>   service-rpc/if/TCLIService.thrift 9879b1b 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 838bf17 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 5229230 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
>  31472c8 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
>  7101fa5 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
>  1aa3f94 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
>  14d50ed 
>   service-rpc/src/gen/thrift/gen-php/Types.php 9ed7403 
>   service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py 44e5462 
>   service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb b39ec1e 
>   service/src/java/org/apache/hive/service/cli/CLIService.java ed52b4a 
>   service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java bfd7135 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 0932884 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> 2f18231 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 67e0e52 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 78ff388 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> a0015eb 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 5464e58 
> 
> Diff:

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-05-18 Thread Takanobu Asanuma 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/
---

(Updated 5月 18, 2016, 9:54 a.m.)


Review request for hive.


Changes
---

I updated the patch based on Sergey's advice. Also I merged the latest master 
branch.


Repository: hive-git


Description
---

This is a WIP patch for HIVE-11527

* I added a new configuration whose name is 
hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
true, clients use the bypass.

* I still have not considered security such as Kerberos and SSL at present.

* I have not implement Statement#setFetchSize for bypass yet.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cbb3a72 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 
84644d1 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
815ccfa 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 
637e51a 
  jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 92fdbca 
  jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 38ccc78 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 3fecc5c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java dff1815 
  service-rpc/if/TCLIService.thrift 9879b1b 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 838bf17 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 5229230 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
 31472c8 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
 7101fa5 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
 1aa3f94 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
 14d50ed 
  service-rpc/src/gen/thrift/gen-php/Types.php 9ed7403 
  service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py 44e5462 
  service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb b39ec1e 
  service/src/java/org/apache/hive/service/cli/CLIService.java ed52b4a 
  service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java bfd7135 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java 0932884 
  service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
2f18231 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
67e0e52 
  service/src/java/org/apache/hive/service/cli/session/HiveSession.java 78ff388 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
a0015eb 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
5464e58 

Diff: https://reviews.apache.org/r/40867/diff/


Testing
---

I have tested few simple queries and they worked well. But I think there are 
some problems for some queries. I'm going to test more queries and fix bugs. 
I'm also going to add unit tests.


Thanks,

Takanobu Asanuma

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-05-18 Thread Takanobu Asanuma 


> On 4月 22, 2016, 1:33 a.m., Sergey Shelukhin wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 565
> > 
> >
> > nit: is it enough to just use Arrays.asList?

That's right. I will use just List here.


> On 4月 22, 2016, 1:33 a.m., Sergey Shelukhin wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 588
> > 
> >
> > why does this swallow the error? It should fail on error

This is my mistake. I think it should not use a try-catch statement.


- Takanobu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review130014
---


On 4月 15, 2016, 11:38 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated 4月 15, 2016, 11:38 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c7e5b33 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 
> 84644d1 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 857805a 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 
> 637e51a 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 0bbd0e3 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 92c2c76 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336 
>   service-rpc/if/TCLIService.thrift aa28b6e 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 7f1d9dd 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 3a27a60 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
>  31472c8 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
>  7101fa5 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
>  1aa3f94 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
>  14d50ed 
>   service-rpc/src/gen/thrift/gen-php/Types.php b7df50a 
>   service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py c691781 
>   service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 07ed97c 
>   service/src/java/org/apache/hive/service/cli/CLIService.java ab30ae2 
>   service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> b7d6549 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> 56a9c18 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 9ce6055 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 0f36cd6 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> be9833d 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-04-21 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review130014
---




jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 557)


nit: is it enough to just use Arrays.asList?



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 580)


why does this swallow the error? It should fail on error



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 581)


nit: null check, there could be an error before stream is created


- Sergey Shelukhin


On April 15, 2016, 11:38 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated April 15, 2016, 11:38 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c7e5b33 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 
> 84644d1 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 857805a 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 
> 637e51a 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 0bbd0e3 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 92c2c76 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336 
>   service-rpc/if/TCLIService.thrift aa28b6e 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 7f1d9dd 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 3a27a60 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
>  31472c8 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
>  7101fa5 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
>  1aa3f94 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
>  14d50ed 
>   service-rpc/src/gen/thrift/gen-php/Types.php b7df50a 
>   service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py c691781 
>   service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 07ed97c 
>   service/src/java/org/apache/hive/service/cli/CLIService.java ab30ae2 
>   service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> b7d6549 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> 56a9c18 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 9ce6055 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 0f36cd6 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> be9833d 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-04-15 Thread Takanobu Asanuma 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/
---

(Updated 4月 15, 2016, 11:38 a.m.)


Review request for hive.


Changes
---

I uploaded a new patch. I implemented some features for handling SequenceFile, 
handling multiple result files, handling ResultSet#beforeFirst. And I added 
some unit tests.
I think I have almost finished implementing the codes. So I will left a comment 
which is the summary of all my implementation.


Repository: hive-git


Description
---

This is a WIP patch for HIVE-11527

* I added a new configuration whose name is 
hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
true, clients use the bypass.

* I still have not considered security such as Kerberos and SSL at present.

* I have not implement Statement#setFetchSize for bypass yet.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c7e5b33 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 
84644d1 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
857805a 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 
637e51a 
  jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209 
  jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 0bbd0e3 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 92c2c76 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336 
  service-rpc/if/TCLIService.thrift aa28b6e 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 7f1d9dd 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 3a27a60 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
 31472c8 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
 7101fa5 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
 1aa3f94 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
 14d50ed 
  service-rpc/src/gen/thrift/gen-php/Types.php b7df50a 
  service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py c691781 
  service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 07ed97c 
  service/src/java/org/apache/hive/service/cli/CLIService.java ab30ae2 
  service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java b7d6549 
  service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
56a9c18 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
9ce6055 
  service/src/java/org/apache/hive/service/cli/session/HiveSession.java 4f4e92d 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
0f36cd6 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
be9833d 

Diff: https://reviews.apache.org/r/40867/diff/


Testing
---

I have tested few simple queries and they worked well. But I think there are 
some problems for some queries. I'm going to test more queries and fix bugs. 
I'm also going to add unit tests.


Thanks,

Takanobu Asanuma

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-03-19 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review124234
---




jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 210)


nit: whitespace



ql/src/java/org/apache/hadoop/hive/ql/Driver.java (line 1917)


why cannot the conf itself be passed in? why is the map necessary?
Or, if that doesn't work for some reason, what about creating a small 
struct class to hold the fields with logical names and types, instead of the 
map.


- Sergey Shelukhin


On March 16, 2016, 8:51 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated March 16, 2016, 8:51 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 98c6372 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java b4dba44 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 7327a42 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336 
>   service-rpc/if/TCLIService.thrift aa28b6e 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 7f1d9dd 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 3a27a60 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
>  31472c8 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
>  7101fa5 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
>  1aa3f94 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
>  14d50ed 
>   service-rpc/src/gen/thrift/gen-php/Types.php b7df50a 
>   service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py c691781 
>   service-rpc/src/gen/thrift/gen-py/__init__.py PRE-CREATION 
>   service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 07ed97c 
>   service/src/java/org/apache/hive/service/cli/CLIService.java ab30ae2 
>   service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> d9a273b 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> 56a9c18 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 04d816a 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 8baecdf 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 62fcde5 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-03-18 Thread Takanobu Asanuma 


> On 3月 18, 2016, 5:57 p.m., Sergey Shelukhin wrote:
> >

Thanks for the reviewing.


> On 3月 18, 2016, 5:57 p.m., Sergey Shelukhin wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 210
> > 
> >
> > nit: whitespace

Sorry, I'll fix it.


> On 3月 18, 2016, 5:57 p.m., Sergey Shelukhin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/Driver.java, line 1919
> > 
> >
> > why cannot the conf itself be passed in? why is the map necessary?
> > Or, if that doesn't work for some reason, what about creating a small 
> > struct class to hold the fields with logical names and types, instead of 
> > the map.

> why cannot the conf itself be passed in?

In this patch, HiveServer2 gives JDBC clients the configurations which is the 
necessary and sufficient condition for resolving namespace of HA.
In present implementation of Hive, the configurations on the JDBC clients side 
are different from ones on the cluster side. So if all the configurations are 
passed in here, it would destroy compatibility with existing JDBC applications. 
Even if it is better, I think it should be done in another jira.

> what about creating a small struct class to hold the fields with logical 
> names and types, instead of the map.

We need a simple key-value structure in here. Map is a simple way to do this 
and we don't need to define the new structure in the thrift API. So I used the 
map. What the advantages of creating a new structure class?


- Takanobu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review124234
---


On 3月 16, 2016, 8:51 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated 3月 16, 2016, 8:51 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 98c6372 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java b4dba44 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 7327a42 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336 
>   service-rpc/if/TCLIService.thrift aa28b6e 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 7f1d9dd 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 3a27a60 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
>  31472c8 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
>  7101fa5 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
>  1aa3f94 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
>  14d50ed 
>   service-rpc/src/gen/thrift/gen-php/Types.php b7df50a 
>   service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py c691781 
>   service-rpc/src/gen/thrift/gen-py/__init__.py PRE-CREATION 
>   service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 07ed97c 
>   service/src/java/org/apache/hive/service/cli/CLIService.java ab30ae2 
>   service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> d9a273b 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> 56a9c18 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 04d816a 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 8baecdf 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 62fcde5 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-03-16 Thread Takanobu Asanuma 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/
---

(Updated 3月 16, 2016, 8:51 a.m.)


Review request for hive.


Changes
---

Implemented features for handling HA. HiveServer2 puts HA configurations to a 
Map in Driver#getFinalDirName() and send it to jdbc clients.


Repository: hive-git


Description
---

This is a WIP patch for HIVE-11527

* I added a new configuration whose name is 
hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
true, clients use the bypass.

* I still have not considered security such as Kerberos and SSL at present.

* I have not implement Statement#setFetchSize for bypass yet.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 98c6372 
  jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209 
  jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java b4dba44 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 7327a42 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336 
  service-rpc/if/TCLIService.thrift aa28b6e 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 7f1d9dd 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 3a27a60 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
 31472c8 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
 7101fa5 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
 1aa3f94 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
 14d50ed 
  service-rpc/src/gen/thrift/gen-php/Types.php b7df50a 
  service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py c691781 
  service-rpc/src/gen/thrift/gen-py/__init__.py PRE-CREATION 
  service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 07ed97c 
  service/src/java/org/apache/hive/service/cli/CLIService.java ab30ae2 
  service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java d9a273b 
  service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
56a9c18 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
04d816a 
  service/src/java/org/apache/hive/service/cli/session/HiveSession.java 4f4e92d 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
8baecdf 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
62fcde5 

Diff: https://reviews.apache.org/r/40867/diff/


Testing
---

I have tested few simple queries and they worked well. But I think there are 
some problems for some queries. I'm going to test more queries and fix bugs. 
I'm also going to add unit tests.


Thanks,

Takanobu Asanuma

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-02-04 Thread Takanobu Asanuma 


> On Feb. 3, 2016, 10:41 p.m., Sergey Shelukhin wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 530
> > 
> >
> > actually, do we expect more than one file here, should we handle this 
> > case?

I have some questions.
1. Is there any SELECT statement which creates multiple result files in the 
final directory? I tested some queries, but I wasn't able to find that query.
2. If there are multiple result files, can we assume that all those files' 
names are numbers and each file's content accords with the series?


- Takanobu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review117697
---


On Feb. 3, 2016, 9:32 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated Feb. 3, 2016, 9:32 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6678de6 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java b4dba44 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4c89812 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336 
>   service-rpc/if/TCLIService.thrift 0aa9d13 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 5364293 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp e62e8b7 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
>  31472c8 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
>  7101fa5 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
>  14d50ed 
>   service-rpc/src/gen/thrift/gen-php/Types.php 76805df 
>   service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
>   service-rpc/src/gen/thrift/gen-py/__init__.py PRE-CREATION 
>   service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
>   service/src/gen/thrift/gen-cpp/ThriftHive.h PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/ThriftHive.cpp PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/ThriftHive_server.skeleton.cpp PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/hive_service_constants.h PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/hive_service_constants.cpp PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/hive_service_types.h PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/hive_service_types.cpp PRE-CREATION 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveClusterStatus.java
>  PRE-CREATION 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveServerException.java
>  PRE-CREATION 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/JobTrackerState.java
>  PRE-CREATION 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/ThriftHive.java
>  PRE-CREATION 
>   service/src/gen/thrift/gen-php/ThriftHive.php PRE-CREATION 
>   service/src/gen/thrift/gen-php/Types.php PRE-CREATION 
>   service/src/gen/thrift/gen-py/__init__.py PRE-CREATION 
>   service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote PRE-CREATION 
>   service/src/gen/thrift/gen-py/hive_service/ThriftHive.py PRE-CREATION 
>   service/src/gen/thrift/gen-py/hive_service/__init__.py PRE-CREATION 
>   service/src/gen/thrift/gen-py/hive_service/constants.py PRE-CREATION 
>   service/src/gen/thrift/gen-py/hive_service/ttypes.py PRE-CREATION 
>   service/src/gen/thrift/gen-rb/hive_service_constants.rb PRE-CREATION 
>   service/src/gen/thrift/gen-rb/hive_service_types.rb PRE-CREATION 
>   service/src/gen/thrift/gen-rb/thrift_hive.rb PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/CLIService.java cc6e138 
>   service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 0c263cf 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> f1ce6f6 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 01b1d3d 
>

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-02-03 Thread Takanobu Asanuma 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/
---

(Updated 2月 3, 2016, 9:32 a.m.)


Review request for hive.


Changes
---

Sorry, I forgot to regenerate thrift files in the before patch. Please ignore 
that.


About this patch:
I changed typeName from required to optional in thrift API. If this is null, 
when new client sends the request to the old server, clients use old path 
without bypass.
I regenerated thrift files.
I merged latest master branch.


Repository: hive-git


Description
---

This is a WIP patch for HIVE-11527

* I added a new configuration whose name is 
hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
true, clients use the bypass.

* I still have not considered security such as Kerberos and SSL at present.

* I have not implement Statement#setFetchSize for bypass yet.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6678de6 
  jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209 
  jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java b4dba44 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4c89812 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336 
  service-rpc/if/TCLIService.thrift 0aa9d13 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 5364293 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp e62e8b7 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
 31472c8 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
 7101fa5 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
 14d50ed 
  service-rpc/src/gen/thrift/gen-php/Types.php 76805df 
  service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
  service-rpc/src/gen/thrift/gen-py/__init__.py PRE-CREATION 
  service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
  service/src/gen/thrift/gen-cpp/ThriftHive.h PRE-CREATION 
  service/src/gen/thrift/gen-cpp/ThriftHive.cpp PRE-CREATION 
  service/src/gen/thrift/gen-cpp/ThriftHive_server.skeleton.cpp PRE-CREATION 
  service/src/gen/thrift/gen-cpp/hive_service_constants.h PRE-CREATION 
  service/src/gen/thrift/gen-cpp/hive_service_constants.cpp PRE-CREATION 
  service/src/gen/thrift/gen-cpp/hive_service_types.h PRE-CREATION 
  service/src/gen/thrift/gen-cpp/hive_service_types.cpp PRE-CREATION 
  
service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveClusterStatus.java
 PRE-CREATION 
  
service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveServerException.java
 PRE-CREATION 
  
service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/JobTrackerState.java
 PRE-CREATION 
  
service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/ThriftHive.java
 PRE-CREATION 
  service/src/gen/thrift/gen-php/ThriftHive.php PRE-CREATION 
  service/src/gen/thrift/gen-php/Types.php PRE-CREATION 
  service/src/gen/thrift/gen-py/__init__.py PRE-CREATION 
  service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote PRE-CREATION 
  service/src/gen/thrift/gen-py/hive_service/ThriftHive.py PRE-CREATION 
  service/src/gen/thrift/gen-py/hive_service/__init__.py PRE-CREATION 
  service/src/gen/thrift/gen-py/hive_service/constants.py PRE-CREATION 
  service/src/gen/thrift/gen-py/hive_service/ttypes.py PRE-CREATION 
  service/src/gen/thrift/gen-rb/hive_service_constants.rb PRE-CREATION 
  service/src/gen/thrift/gen-rb/hive_service_types.rb PRE-CREATION 
  service/src/gen/thrift/gen-rb/thrift_hive.rb PRE-CREATION 
  service/src/java/org/apache/hive/service/cli/CLIService.java cc6e138 
  service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java 0c263cf 
  service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
f1ce6f6 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
01b1d3d 
  service/src/java/org/apache/hive/service/cli/session/HiveSession.java 4f4e92d 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
6aee80c 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
d9a9e3b 

Diff: https://reviews.apache.org/r/40867/diff/


Testing
---

I have tested few simple queries and they worked well. But I think there are 
some problems for some queries. I'm going to test more queries and fix bugs. 
I'm also going to add unit tests.


Thanks,

Takanobu Asanuma

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-02-03 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review117680
---




ql/src/java/org/apache/hadoop/hive/ql/Driver.java (line 1870)


port can be missing in a valid HDFS url, and it will be missing if HA is 
used. It may be better to use URI class to get the components of the URL.


- Sergey Shelukhin


On Feb. 3, 2016, 9:32 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated Feb. 3, 2016, 9:32 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6678de6 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java b4dba44 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4c89812 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336 
>   service-rpc/if/TCLIService.thrift 0aa9d13 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 5364293 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp e62e8b7 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
>  31472c8 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
>  7101fa5 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
>  14d50ed 
>   service-rpc/src/gen/thrift/gen-php/Types.php 76805df 
>   service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
>   service-rpc/src/gen/thrift/gen-py/__init__.py PRE-CREATION 
>   service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
>   service/src/gen/thrift/gen-cpp/ThriftHive.h PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/ThriftHive.cpp PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/ThriftHive_server.skeleton.cpp PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/hive_service_constants.h PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/hive_service_constants.cpp PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/hive_service_types.h PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/hive_service_types.cpp PRE-CREATION 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveClusterStatus.java
>  PRE-CREATION 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveServerException.java
>  PRE-CREATION 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/JobTrackerState.java
>  PRE-CREATION 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/ThriftHive.java
>  PRE-CREATION 
>   service/src/gen/thrift/gen-php/ThriftHive.php PRE-CREATION 
>   service/src/gen/thrift/gen-php/Types.php PRE-CREATION 
>   service/src/gen/thrift/gen-py/__init__.py PRE-CREATION 
>   service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote PRE-CREATION 
>   service/src/gen/thrift/gen-py/hive_service/ThriftHive.py PRE-CREATION 
>   service/src/gen/thrift/gen-py/hive_service/__init__.py PRE-CREATION 
>   service/src/gen/thrift/gen-py/hive_service/constants.py PRE-CREATION 
>   service/src/gen/thrift/gen-py/hive_service/ttypes.py PRE-CREATION 
>   service/src/gen/thrift/gen-rb/hive_service_constants.rb PRE-CREATION 
>   service/src/gen/thrift/gen-rb/hive_service_types.rb PRE-CREATION 
>   service/src/gen/thrift/gen-rb/thrift_hive.rb PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/CLIService.java cc6e138 
>   service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 0c263cf 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> f1ce6f6 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 01b1d3d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 6aee80c 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> d9a9e3b 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-02-03 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review117697
---




jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 529)


actually, do we expect more than one file here, should we handle this case?


- Sergey Shelukhin


On Feb. 3, 2016, 9:32 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated Feb. 3, 2016, 9:32 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6678de6 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java b4dba44 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4c89812 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336 
>   service-rpc/if/TCLIService.thrift 0aa9d13 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 5364293 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp e62e8b7 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
>  31472c8 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
>  7101fa5 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
>  14d50ed 
>   service-rpc/src/gen/thrift/gen-php/Types.php 76805df 
>   service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
>   service-rpc/src/gen/thrift/gen-py/__init__.py PRE-CREATION 
>   service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
>   service/src/gen/thrift/gen-cpp/ThriftHive.h PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/ThriftHive.cpp PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/ThriftHive_server.skeleton.cpp PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/hive_service_constants.h PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/hive_service_constants.cpp PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/hive_service_types.h PRE-CREATION 
>   service/src/gen/thrift/gen-cpp/hive_service_types.cpp PRE-CREATION 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveClusterStatus.java
>  PRE-CREATION 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveServerException.java
>  PRE-CREATION 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/JobTrackerState.java
>  PRE-CREATION 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/ThriftHive.java
>  PRE-CREATION 
>   service/src/gen/thrift/gen-php/ThriftHive.php PRE-CREATION 
>   service/src/gen/thrift/gen-php/Types.php PRE-CREATION 
>   service/src/gen/thrift/gen-py/__init__.py PRE-CREATION 
>   service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote PRE-CREATION 
>   service/src/gen/thrift/gen-py/hive_service/ThriftHive.py PRE-CREATION 
>   service/src/gen/thrift/gen-py/hive_service/__init__.py PRE-CREATION 
>   service/src/gen/thrift/gen-py/hive_service/constants.py PRE-CREATION 
>   service/src/gen/thrift/gen-py/hive_service/ttypes.py PRE-CREATION 
>   service/src/gen/thrift/gen-rb/hive_service_constants.rb PRE-CREATION 
>   service/src/gen/thrift/gen-rb/hive_service_types.rb PRE-CREATION 
>   service/src/gen/thrift/gen-rb/thrift_hive.rb PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/CLIService.java cc6e138 
>   service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 0c263cf 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> f1ce6f6 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 01b1d3d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 6aee80c 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> d9a9e3b 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-01-28 Thread Takanobu Asanuma 


> On 1月 14, 2016, 5:36 p.m., Prasad Mujumdar wrote:
> > service/if/TCLIService.thrift, line 272
> > 
> >
> > Does it need to be a required parameter ? Adding a new required 
> > parameter in the middle will make it break the backward compatibility for 
> > existing clients.
> 
> Takanobu Asanuma wrote:
> Yes, it is necessary to be a required parameter since this patch had 
> TypeDescriptor created by the typename in ColumnDescriptor. (Please see last 
> Sergey's review.) Next patch fixes the order of the parameters.
> 
> Sergey Shelukhin wrote:
> if the new client sends the request to the old server, that is not going 
> to work... it needs to be optional and the code that gets it needs to work 
> around if it's absent - for example, it can just use the old path w/o bypass.

Sergey
I understand. Thanks for the notice!


- Takanobu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review114504
---


On 1月 25, 2016, 9:25 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated 1月 25, 2016, 9:25 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a44e75a 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java b4dba44 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 75187cf 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336 
>   service-rpc/if/TCLIService.thrift 0aa9d13 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 5364293 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp e62e8b7 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
>  31472c8 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
>  7101fa5 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
>  14d50ed 
>   service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
>   service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
>   service/src/java/org/apache/hive/service/cli/CLIService.java 0d5ae1e 
>   service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 113eddf 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> 92135cd 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> c8a69b9 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 3c5700b 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 78b4b31 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-01-28 Thread Sergey Shelukhin 


> On Jan. 14, 2016, 5:36 p.m., Prasad Mujumdar wrote:
> > service/if/TCLIService.thrift, line 272
> > 
> >
> > Does it need to be a required parameter ? Adding a new required 
> > parameter in the middle will make it break the backward compatibility for 
> > existing clients.
> 
> Takanobu Asanuma wrote:
> Yes, it is necessary to be a required parameter since this patch had 
> TypeDescriptor created by the typename in ColumnDescriptor. (Please see last 
> Sergey's review.) Next patch fixes the order of the parameters.

if the new client sends the request to the old server, that is not going to 
work... it needs to be optional and the code that gets it needs to work around 
if it's absent - for example, it can just use the old path w/o bypass.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review114504
---


On Jan. 25, 2016, 9:25 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated Jan. 25, 2016, 9:25 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a44e75a 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java b4dba44 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 75187cf 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336 
>   service-rpc/if/TCLIService.thrift 0aa9d13 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 5364293 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp e62e8b7 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
>  31472c8 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
>  7101fa5 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
>  14d50ed 
>   service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
>   service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
>   service/src/java/org/apache/hive/service/cli/CLIService.java 0d5ae1e 
>   service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 113eddf 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> 92135cd 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> c8a69b9 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 3c5700b 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 78b4b31 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-01-25 Thread Takanobu Asanuma 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/
---

(Updated 1月 25, 2016, 9:25 a.m.)


Review request for hive.


Changes
---

Fixed the parameter order of TCLIService.thrift


Repository: hive-git


Description
---

This is a WIP patch for HIVE-11527

* I added a new configuration whose name is 
hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
true, clients use the bypass.

* I still have not considered security such as Kerberos and SSL at present.

* I have not implement Statement#setFetchSize for bypass yet.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a44e75a 
  jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209 
  jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java b4dba44 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 75187cf 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336 
  service-rpc/if/TCLIService.thrift 0aa9d13 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 5364293 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp e62e8b7 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
 31472c8 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
 7101fa5 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
 14d50ed 
  service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
  service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
  service/src/java/org/apache/hive/service/cli/CLIService.java 0d5ae1e 
  service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java 113eddf 
  service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
92135cd 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
c8a69b9 
  service/src/java/org/apache/hive/service/cli/session/HiveSession.java 4f4e92d 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
3c5700b 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
78b4b31 

Diff: https://reviews.apache.org/r/40867/diff/


Testing
---

I have tested few simple queries and they worked well. But I think there are 
some problems for some queries. I'm going to test more queries and fix bugs. 
I'm also going to add unit tests.


Thanks,

Takanobu Asanuma

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-01-14 Thread Prasad Mujumdar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review114504
---


My apologies for jumping in late in the review

I am curious about why you chose HiveServer2 interface to implement this. I 
guess one can work it around by 'INSERT OVERWRITE DIRECTORY .. SELECT ..' and 
so one of the options could be to solve this in the query processing (eg 
rewrite the query during parsing etc). The problem for HS2 solution is that 
it's not going to be accessible to high level clients like JDBC/ODBC/Beeline. 
One has to write a custom HS2 client driver in order to pass the new parameters.

Regarding this patch, one comment below.


service/if/TCLIService.thrift (line 272)


Does it need to be a required parameter ? Adding a new required parameter 
in the middle will make it break the backward compatibility for existing 
clients.


- Prasad Mujumdar


On Jan. 8, 2016, 7:18 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated Jan. 8, 2016, 7:18 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 479fa46 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 245c6a3 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 3f9024a 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 29e6315 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1634143 
>   service/if/TCLIService.thrift baf583f 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.h b078c99 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.cpp b852379 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TColumnDesc.java
>  b01fadb 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java
>  0b9aa0f 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java
>  c936ada 
>   service/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
>   service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
>   service/src/java/org/apache/hive/service/cli/CLIService.java adc9809 
>   service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java f0bbf14 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> d2b3f9c 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> 92135cd 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 2eaab4a 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 50c912e 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 8434965 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-01-14 Thread Takanobu Asanuma 


> On 1月 13, 2016, 11:54 p.m., Sergey Shelukhin wrote:
> > service/if/TCLIService.thrift, line 271
> > 
> >
> > is it possible to avoid changing field IDs? I think backward compat 
> > might depend on them. Field should be added at the end.

Thank you for your note. The next patch fixes it.


> On 1月 13, 2016, 11:54 p.m., Sergey Shelukhin wrote:
> > service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java, line 46
> > 
> >
> > actually, why is this change necessary?

It is to handle complex column types. TTypeDesc, which is a return value of 
tColumnDesc.getTypeDesc(), does not have enough infomation for complex column 
types. For example, "struct" is regarded as "struct". On the other 
hand, typename has detail infomation for complex column types as is.


- Takanobu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review114334
---


On 1月 8, 2016, 7:18 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated 1月 8, 2016, 7:18 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 479fa46 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 245c6a3 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 3f9024a 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 29e6315 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1634143 
>   service/if/TCLIService.thrift baf583f 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.h b078c99 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.cpp b852379 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TColumnDesc.java
>  b01fadb 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java
>  0b9aa0f 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java
>  c936ada 
>   service/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
>   service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
>   service/src/java/org/apache/hive/service/cli/CLIService.java adc9809 
>   service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java f0bbf14 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> d2b3f9c 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> 92135cd 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 2eaab4a 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 50c912e 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 8434965 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-01-14 Thread Takanobu Asanuma 


> On 1月 14, 2016, 5:36 p.m., Prasad Mujumdar wrote:
> > My apologies for jumping in late in the review
> > 
> > I am curious about why you chose HiveServer2 interface to implement this. I 
> > guess one can work it around by 'INSERT OVERWRITE DIRECTORY .. SELECT ..' 
> > and so one of the options could be to solve this in the query processing 
> > (eg rewrite the query during parsing etc). The problem for HS2 solution is 
> > that it's not going to be accessible to high level clients like 
> > JDBC/ODBC/Beeline. One has to write a custom HS2 client driver in order to 
> > pass the new parameters.
> > 
> > Regarding this patch, one comment below.

Thank you for your review and comments, Prasad!

> I am curious about why you chose HiveServer2 interface to implement this.

When some JDBC clients download large data from HiveServer2, HiveServer2 will 
be a bottle neck. This bypass is for between JDBC clients and HDFS so that JDBC 
clients can download data from HDFS directly.

> I guess one can work it around by 'INSERT OVERWRITE DIRECTORY .. SELECT ..' 
> and so one of the options could be to solve this in the query processing (eg 
> rewrite the query during parsing etc).
> The problem for HS2 solution is that it's not going to be accessible to high 
> level clients like JDBC/ODBC/Beeline.

What do you mean by that?  I rather think this bypass would not work 'INSERT 
OVERWRITE DIRECTORY .. SELECT ..'. Because it does not return results to JDBC 
clients.


> On 1月 14, 2016, 5:36 p.m., Prasad Mujumdar wrote:
> > service/if/TCLIService.thrift, line 272
> > 
> >
> > Does it need to be a required parameter ? Adding a new required 
> > parameter in the middle will make it break the backward compatibility for 
> > existing clients.

Yes, it is necessary to be a required parameter since this patch had 
TypeDescriptor created by the typename in ColumnDescriptor. (Please see last 
Sergey's review.) Next patch fixes the order of the parameters.


- Takanobu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review114504
---


On 1月 8, 2016, 7:18 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated 1月 8, 2016, 7:18 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 479fa46 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 245c6a3 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 3f9024a 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 29e6315 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1634143 
>   service/if/TCLIService.thrift baf583f 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.h b078c99 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.cpp b852379 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TColumnDesc.java
>  b01fadb 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java
>  0b9aa0f 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java
>  c936ada 
>   service/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
>   service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
>   service/src/java/org/apache/hive/service/cli/CLIService.java adc9809 
>   service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java f0bbf14 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> d2b3f9c 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> 92135cd 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 2eaab4a 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 50c912e 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 8434965 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-01-13 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review114334
---



service/if/TCLIService.thrift (line 271)


is it possible to avoid changing field IDs? I think backward compat might 
depend on them. Field should be added at the end.



service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java (line 46)


actually, why is this change necessary?


- Sergey Shelukhin


On Jan. 8, 2016, 7:18 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated Jan. 8, 2016, 7:18 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 479fa46 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 245c6a3 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 3f9024a 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 29e6315 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1634143 
>   service/if/TCLIService.thrift baf583f 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.h b078c99 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.cpp b852379 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TColumnDesc.java
>  b01fadb 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java
>  0b9aa0f 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java
>  c936ada 
>   service/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
>   service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
>   service/src/java/org/apache/hive/service/cli/CLIService.java adc9809 
>   service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java f0bbf14 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> d2b3f9c 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> 92135cd 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 2eaab4a 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 50c912e 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 8434965 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-01-07 Thread Takanobu Asanuma 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/
---

(Updated 1月 8, 2016, 7:18 a.m.)


Review request for hive.


Changes
---

two addtinal updates for bypass
1. handle complex type such as array, struct and map.
2. handle Statement#setFetchSize.


Repository: hive-git


Description
---

This is a WIP patch for HIVE-11527

* I added a new configuration whose name is 
hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
true, clients use the bypass.

* I still have not considered security such as Kerberos and SSL at present.

* I have not implement Statement#setFetchSize for bypass yet.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 479fa46 
  jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 245c6a3 
  jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 3f9024a 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 29e6315 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1634143 
  service/if/TCLIService.thrift baf583f 
  service/src/gen/thrift/gen-cpp/TCLIService_types.h b078c99 
  service/src/gen/thrift/gen-cpp/TCLIService_types.cpp b852379 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TColumnDesc.java
 b01fadb 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java
 0b9aa0f 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java
 c936ada 
  service/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
  service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
  service/src/java/org/apache/hive/service/cli/CLIService.java adc9809 
  service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java f0bbf14 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java d2b3f9c 
  service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
92135cd 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
2eaab4a 
  service/src/java/org/apache/hive/service/cli/session/HiveSession.java 4f4e92d 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
50c912e 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
8434965 

Diff: https://reviews.apache.org/r/40867/diff/


Testing
---

I have tested few simple queries and they worked well. But I think there are 
some problems for some queries. I'm going to test more queries and fix bugs. 
I'm also going to add unit tests.


Thanks,

Takanobu Asanuma

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-01-05 Thread Takanobu Asanuma 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/
---

(Updated 1月 5, 2016, 8:15 a.m.)


Review request for hive.


Changes
---

I updated the patch. It fixes the codes to use webhdfs and improves logging in 
HiveQueryResultSet.


Repository: hive-git


Description
---

This is a WIP patch for HIVE-11527

* I added a new configuration whose name is 
hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
true, clients use the bypass.

* I still have not considered security such as Kerberos and SSL at present.

* I have not implement Statement#setFetchSize for bypass yet.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ffe0d9a 
  jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 245c6a3 
  jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 3f9024a 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 29e6315 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1634143 
  service/if/TCLIService.thrift baf583f 
  service/src/gen/thrift/gen-cpp/TCLIService_types.h b078c99 
  service/src/gen/thrift/gen-cpp/TCLIService_types.cpp b852379 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java
 0b9aa0f 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java
 c936ada 
  service/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
  service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
  service/src/java/org/apache/hive/service/cli/CLIService.java adc9809 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java d2b3f9c 
  service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
92135cd 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
2eaab4a 
  service/src/java/org/apache/hive/service/cli/session/HiveSession.java 4f4e92d 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
50c912e 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
8434965 

Diff: https://reviews.apache.org/r/40867/diff/


Testing
---

I have tested few simple queries and they worked well. But I think there are 
some problems for some queries. I'm going to test more queries and fix bugs. 
I'm also going to add unit tests.


Thanks,

Takanobu Asanuma

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2015-12-18 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review111316
---



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 314)


Nit: LOG.error has an overload to log an exception, LOG.error(, ex);


- Sergey Shelukhin


On Dec. 18, 2015, 7:13 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated Dec. 18, 2015, 7:13 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b5aee00 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 245c6a3 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 3f9024a 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 3d5f3b5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1634143 
>   service/if/TCLIService.thrift baf583f 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.h b078c99 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.cpp b852379 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java
>  0b9aa0f 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java
>  c936ada 
>   service/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
>   service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote e167d5b 
>   service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
>   service/src/java/org/apache/hive/service/cli/CLIService.java adc9809 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> d2b3f9c 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> 92135cd 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> d90dd0d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 50c912e 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 8434965 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2015-12-18 Thread Sergey Shelukhin 


> On Dec. 3, 2015, 9:40 p.m., Sergey Shelukhin wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 390
> > 
> >
> > some existing configuration is probably needed (and better)
> 
> Takanobu Asanuma wrote:
> IIUC, hive configurations are in the server side and HiveQueryResultSet 
> is in the client side. So I don't understand how to load hive configurations 
> in HiveQueryResultSet. Is there any idea?

configuration does exist on client side, although it looks like noone on JDBC 
currently creates and object. So these changes make sense


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review108889
---


On Dec. 18, 2015, 7:13 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated Dec. 18, 2015, 7:13 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b5aee00 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 245c6a3 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 3f9024a 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 3d5f3b5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1634143 
>   service/if/TCLIService.thrift baf583f 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.h b078c99 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.cpp b852379 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java
>  0b9aa0f 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java
>  c936ada 
>   service/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
>   service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote e167d5b 
>   service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
>   service/src/java/org/apache/hive/service/cli/CLIService.java adc9809 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> d2b3f9c 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> 92135cd 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> d90dd0d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 50c912e 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 8434965 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2015-12-17 Thread Takanobu Asanuma 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/
---

(Updated 12月 18, 2015, 7:13 a.m.)


Review request for hive.


Changes
---

I updated the patch based on Sergey's advice. Main points of modification is 
adding rows to rowset immediately after reading the final data.


Repository: hive-git


Description
---

This is a WIP patch for HIVE-11527

* I added a new configuration whose name is 
hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
true, clients use the bypass.

* I still have not considered security such as Kerberos and SSL at present.

* I have not implement Statement#setFetchSize for bypass yet.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b5aee00 
  jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 245c6a3 
  jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 3f9024a 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 3d5f3b5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1634143 
  service/if/TCLIService.thrift baf583f 
  service/src/gen/thrift/gen-cpp/TCLIService_types.h b078c99 
  service/src/gen/thrift/gen-cpp/TCLIService_types.cpp b852379 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java
 0b9aa0f 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java
 c936ada 
  service/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
  service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote e167d5b 
  service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
  service/src/java/org/apache/hive/service/cli/CLIService.java adc9809 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java d2b3f9c 
  service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
92135cd 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
d90dd0d 
  service/src/java/org/apache/hive/service/cli/session/HiveSession.java 4f4e92d 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
50c912e 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
8434965 

Diff: https://reviews.apache.org/r/40867/diff/


Testing
---

I have tested few simple queries and they worked well. But I think there are 
some problems for some queries. I'm going to test more queries and fix bugs. 
I'm also going to add unit tests.


Thanks,

Takanobu Asanuma

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2015-12-17 Thread Sergey Shelukhin 


> On Dec. 3, 2015, 9:48 p.m., Sergey Shelukhin wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 509
> > 
> >
> > please log instead
> 
> Takanobu Asanuma wrote:
> Like this? 
> ```java
> LOG.error(ex.toString());
> ```
> printStackTrace is used at other lines(312, 438). Should we log there too?

Yes, that works.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review108891
---


On Dec. 2, 2015, 12:52 p.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated Dec. 2, 2015, 12:52 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java db942b0 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 245c6a3 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 180f99e8 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 8fafd61 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1634143 
>   service/if/TCLIService.thrift baf583f 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.h b078c99 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.cpp b852379 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java
>  0b9aa0f 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java
>  c936ada 
>   service/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
>   service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote e167d5b 
>   service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
>   service/src/java/org/apache/hive/service/cli/CLIService.java adc9809 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 25cefc2 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> b0bd351 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 1331a99 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> a14908b 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 8434965 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2015-12-09 Thread Takanobu Asanuma 


> On 12月 3, 2015, 9:48 p.m., Sergey Shelukhin wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 481
> > 
> >
> > this seems similar with the code in SQLOperation, perhaps they can be 
> > refactored into an utility method used by both

Yes, I refereed to the codes of SQLOperation. I will create an utility method 
later.


> On 12月 3, 2015, 9:48 p.m., Sergey Shelukhin wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 509
> > 
> >
> > please log instead

Like this? 
```java
LOG.error(ex.toString());
```
printStackTrace is used at other lines(312, 438). Should we log there too?


> On 12月 3, 2015, 9:48 p.m., Sergey Shelukhin wrote:
> > service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java, 
> > line 511
> > 
> >
> > nit: IIRC setting it to null is also harmless

I agree with it. I will fix this.


- Takanobu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review108891
---


On 12月 2, 2015, 12:52 p.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated 12月 2, 2015, 12:52 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java db942b0 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 245c6a3 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 180f99e8 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 8fafd61 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1634143 
>   service/if/TCLIService.thrift baf583f 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.h b078c99 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.cpp b852379 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java
>  0b9aa0f 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java
>  c936ada 
>   service/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
>   service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote e167d5b 
>   service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
>   service/src/java/org/apache/hive/service/cli/CLIService.java adc9809 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 25cefc2 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> b0bd351 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 1331a99 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> a14908b 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 8434965 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2015-12-09 Thread Takanobu Asanuma 


> On 12月 3, 2015, 9:40 p.m., Sergey Shelukhin wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 390
> > 
> >
> > some existing configuration is probably needed (and better)

IIUC, hive configurations are in the server side and HiveQueryResultSet is in 
the client side. So I don't understand how to load hive configurations in 
HiveQueryResultSet. Is there any idea?


> On 12月 3, 2015, 9:40 p.m., Sergey Shelukhin wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 402
> > 
> >
> > also, I wonder if it makes sense to add rows to rowset immediately 
> > after reading. Storing all rows and then adding them all to rowset stores 
> > all rows twice in memory.

Thank you for the advice. I'll think it over.


> On 12月 3, 2015, 9:40 p.m., Sergey Shelukhin wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 456
> > 
> >
> > nit: why object?

I will change to string.


> On 12月 3, 2015, 9:40 p.m., Sergey Shelukhin wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 466
> > 
> >
> > I wonder why we need to have thrift serialization? One of the goals is 
> > to avoid it. Perhaps it can be done in a follow-up JIRA

Sorry to confuse you, what I want to do here is to convert a object to a 
standard Java object. I think we can just use 
ObjectInspectorUtils#copyToStandardObject here.


> On 12月 3, 2015, 9:40 p.m., Sergey Shelukhin wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 486
> > 
> >
> > nit: useless check, schema is already used above to get descriptors

I will fix it.


> On 12月 3, 2015, 9:40 p.m., Sergey Shelukhin wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 492
> > 
> >
> > nit: get(pos) can be done once

I will fix it.


- Takanobu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review108889
---


On 12月 2, 2015, 12:52 p.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated 12月 2, 2015, 12:52 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java db942b0 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 245c6a3 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 180f99e8 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 8fafd61 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1634143 
>   service/if/TCLIService.thrift baf583f 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.h b078c99 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.cpp b852379 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java
>  0b9aa0f 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java
>  c936ada 
>   service/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
>   service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote e167d5b 
>   service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
>   service/src/java/org/apache/hive/service/cli/CLIService.java adc9809 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 25cefc2 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> b0bd351 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 1331a99 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> a14908b 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 8434965 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2015-12-03 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review108891
---



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 480)


this seems similar with the code in SQLOperation, perhaps they can be 
refactored into an utility method used by both



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 508)


please log instead



service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java (line 
511)


nit: IIRC setting it to null is also harmless


- Sergey Shelukhin


On Dec. 2, 2015, 12:52 p.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated Dec. 2, 2015, 12:52 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java db942b0 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 245c6a3 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 180f99e8 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 8fafd61 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1634143 
>   service/if/TCLIService.thrift baf583f 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.h b078c99 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.cpp b852379 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java
>  0b9aa0f 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java
>  c936ada 
>   service/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
>   service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote e167d5b 
>   service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
>   service/src/java/org/apache/hive/service/cli/CLIService.java adc9809 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 25cefc2 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> b0bd351 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 1331a99 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> a14908b 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 8434965 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2015-12-03 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review108889
---



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 390)


some existing configuration is probably needed (and better)



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 401)


I am not sure how the original rowset works; this may not be good for large 
result sets. Should be ok as long as we have parity with the original, but 
original has fetchSize parameter which I assume could be doing some batching. 
That might be useful to have



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 402)


also, I wonder if it makes sense to add rows to rowset immediately after 
reading. Storing all rows and then adding them all to rowset stores all rows 
twice in memory.



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 455)


nit: why object?



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 465)


I wonder why we need to have thrift serialization? One of the goals is to 
avoid it. Perhaps it can be done in a follow-up JIRA



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 485)


nit: useless check, schema is already used above to get descriptors



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 491)


nit: get(pos) can be done once



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 506)


again please use existing conf object


- Sergey Shelukhin


On Dec. 2, 2015, 12:52 p.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated Dec. 2, 2015, 12:52 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java db942b0 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 245c6a3 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 180f99e8 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 8fafd61 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1634143 
>   service/if/TCLIService.thrift baf583f 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.h b078c99 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.cpp b852379 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java
>  0b9aa0f 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java
>  c936ada 
>   service/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
>   service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote e167d5b 
>   service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
>   service/src/java/org/apache/hive/service/cli/CLIService.java adc9809 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 25cefc2 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> b0bd351 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 1331a99 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> a14908b 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 8434965 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2015-12-02 Thread Takanobu Asanuma 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/
---

Review request for hive.


Repository: hive-git


Description
---

This is a WIP patch for HIVE-11527

* I added a new configuration whose name is 
hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
true, clients use the bypass.

* I still have not considered security such as Kerberos and SSL at present.

* I have not implement Statement#setFetchSize for bypass yet.


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java db942b0 
  jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 245c6a3 
  jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 180f99e8 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 8fafd61 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1634143 
  service/if/TCLIService.thrift baf583f 
  service/src/gen/thrift/gen-cpp/TCLIService_types.h b078c99 
  service/src/gen/thrift/gen-cpp/TCLIService_types.cpp b852379 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java
 0b9aa0f 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java
 c936ada 
  service/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
  service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote e167d5b 
  service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
  service/src/java/org/apache/hive/service/cli/CLIService.java adc9809 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java 25cefc2 
  service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
b0bd351 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
1331a99 
  service/src/java/org/apache/hive/service/cli/session/HiveSession.java 4f4e92d 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
a14908b 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
8434965 

Diff: https://reviews.apache.org/r/40867/diff/


Testing
---

I have tested few simple queries and they worked well. But I think there are 
some problems for some queries. I'm going to test more queries and fix bugs. 
I'm also going to add unit tests.


Thanks,

Takanobu Asanuma