Ok, the complete log will be very long because it creates around 100
connections and they all looks same as shown in the first email.
On Mon, Aug 8, 2016 at 7:28 PM, Puneet Gupta <puneet.gu...@inmobi.com>
wrote:
> Hi Tao,
>
> Please attach the complete log if it happens again. We will open a JIRA
> for this .
>
> I checked the production setup, I am not seeing too many connections.
>
> Thanks,
> Puneet Gupta
>
> On Tue, Aug 9, 2016 at 1:49 AM, Tao Yan <t...@linkedin.com> wrote:
>
>> Hi Rajat,
>>
>> The number of connections created is usually small (less than 10?
>> probably), I run the same query multiple times, and it only happen twice
>> when Lens server created too many connections. I wonder if it is something
>> unexpected.
>>
>> Thanks,
>>
>> On Fri, Aug 5, 2016 at 9:30 PM, Puneet Gupta <puneet.gu...@inmobi.com>
>> wrote:
>>
>>> Hi Tao
>>>
>>> these connections can be created various operations that interact with
>>> hive server like launching a query, pulling its status, etc (There is a
>>> Status Poller thread which polls the status of hive query at regular
>>> intervals.)
>>> The hive connections are cached per user with an configurable expiry
>>> time (which is 10 mins by default). You can configure it differently in
>>> hive driver config (lens.driver.hive.hs2.connection.expiry.delay)if
>>> needed.
>>>
>>> I see from the logs attached by you, the connections are created for two
>>> different users (test_user and anonymous)
>>>
>>> Do let me know if you need more details .
>>>
>>> Thanks,
>>> Puneet Gupta
>>>
>>> On Fri, Aug 5, 2016 at 5:24 AM, Tao Yan <t...@linkedin.com> wrote:
>>>
>>>> Hi Lens Developers,
>>>>
>>>> I am running Lens queries on example data. And I realized sometimes
>>>> Lens server will keep trying to connect to HiveServer2 during the query
>>>> execution, and Lens log shows:
>>>>
>>>> *...*
>>>> 04 Aug 2016 23:39:42 [1b0ba53f-e063-4dd0-a2f1-1519ce940839]
>>>> [grizzly-http-server-28] INFO org.apache.lens.driver.hive.HiveDriver
>>>> - New thrift connection class
>>>> org.apache.lens.driver.hive.RemoteThriftConnection
>>>> for thread: 112 for user: test_user *connection ID=89* on
>>>> driver:hive/hive1
>>>> 04 Aug 2016 23:39:42 [1b0ba53f-e063-4dd0-a2f1-1519ce940839]
>>>> [grizzly-http-server-28] INFO
>>>> org.apache.lens.driver.hive.RemoteThriftConnection
>>>> - HiveDriver connecting to HiveServer @ *******.com:10000
>>>> 04 Aug 2016 23:39:42 [1b0ba53f-e063-4dd0-a2f1-1519ce940839]
>>>> [grizzly-http-server-28] INFO org.apache.hive.service.cli.t
>>>> hrift.RetryingThriftCLIServiceClient - Connecting to *******.com:10000
>>>> 04 Aug 2016 23:39:42 [1b0ba53f-e063-4dd0-a2f1-1519ce940839]
>>>> [grizzly-http-server-28] INFO org.apache.hive.service.cli.t
>>>> hrift.RetryingThriftCLIServiceClient - Connected!
>>>> 04 Aug 2016 23:39:42 [1b0ba53f-e063-4dd0-a2f1-1519ce940839]
>>>> [grizzly-http-server-28] INFO
>>>> org.apache.lens.driver.hive.RemoteThriftConnection
>>>> - HiveDriver connected to HiveServer @ *******.com:10000
>>>> 04 Aug 2016 23:39:47 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2]
>>>> [HiveDriver-ConnectionExpiryThread] INFO
>>>> org.apache.lens.driver.hive.HiveDriver
>>>> - Closed connection: 30
>>>> 04 Aug 2016 23:39:47 [1b0ba53f-e063-4dd0-a2f1-1519ce940839]
>>>> [grizzly-http-server-28] INFO
>>>> org.apache.lens.server.api.query.QueryContext
>>>> - Updating status of 44c647ae-83dc-4ac7-a61f-040d2be07573 from Status
>>>> : RUNNING
>>>> Message : Query is running in HiveServer!
>>>> Progress : 0.0
>>>> Progress Message : [{"taskId":"Stage-1","type":"M
>>>> APRED","externalHandle":"job_1469684228847_467645","taskStat
>>>> e":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","exter
>>>> nalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
>>>> to Status : RUNNING
>>>> Message : Query is running in HiveServer!
>>>> Progress : 0.0
>>>> Progress Message : [{"taskId":"Stage-1","type":"M
>>>> APRED","externalHandle":"job_1469684228847_467645","taskStat
>>>> e":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","exter
>>>> nalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
>>>>
>>>> *...*
>>>>
>>>> 04 Aug 2016 23:42:23 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2]
>>>> [QueryPurger] INFO org.apache.lens.driver.hive.HiveDriver - New
>>>> thrift connection class org.apache.lens.driver.hive.RemoteThriftConnection
>>>> for thread: 25 for user: anonymous *connection ID=106* on
>>>> driver:hive/hive1
>>>> 04 Aug 2016 23:42:23 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2]
>>>> [QueryPurger] INFO org.apache.lens.driver.hive.RemoteThriftConnection
>>>> - HiveDriver connecting to HiveServer @ *******.com:10000
>>>> 04 Aug 2016 23:42:23 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2]
>>>> [QueryPurger] INFO org.apache.hive.service.cli.t
>>>> hrift.RetryingThriftCLIServiceClient - Connecting to *******.com:10000
>>>> 04 Aug 2016 23:42:23 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2]
>>>> [QueryPurger] INFO org.apache.hive.service.cli.t
>>>> hrift.RetryingThriftCLIServiceClient - Connected!
>>>> 04 Aug 2016 23:42:23 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2]
>>>> [QueryPurger] INFO org.apache.lens.driver.hive.RemoteThriftConnection
>>>> - HiveDriver connected to HiveServer @ *******.com:10000
>>>>
>>>>
>>>> It created 106 connections in total for the following example query:
>>>> lens-shell>query execute cube select product_id, store_sales from sales
>>>> where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01')
>>>>
>>>> And it even created 2 connection when the query was successful. Then,
>>>> it started to close those connections:
>>>>
>>>> *....*
>>>> 04 Aug 2016 23:49:22 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2]
>>>> [HiveDriver-ConnectionExpiryThread] INFO
>>>> org.apache.lens.driver.hive.HiveDriver
>>>> - Closed connection: 87
>>>> 04 Aug 2016 23:49:32 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2]
>>>> [HiveDriver-ConnectionExpiryThread] INFO
>>>> org.apache.lens.driver.hive.HiveDriver
>>>> - Closed connection: 88
>>>> 04 Aug 2016 23:49:42 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2]
>>>> [HiveDriver-ConnectionExpiryThread] INFO
>>>> org.apache.lens.driver.hive.HiveDriver
>>>> - Closed connection: 89
>>>> 04 Aug 2016 23:49:52 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2]
>>>> [HiveDriver-ConnectionExpiryThread] INFO
>>>> org.apache.lens.driver.hive.HiveDriver
>>>> - Closed connection: 90
>>>> *...*
>>>>
>>>> Do we know why did Lens created so many connections for such a simple
>>>> query? And it really takes time to create those queries.
>>>>
>>>> Thanks,
>>>> --
>>>>
>>>> *Tao Yan*
>>>> Software Engineer
>>>> Data Analytics Infrastructure Tools and Services
>>>>
>>>>
>>>>
>>>> 206.250.5345
>>>> t...@linkedin.com
>>>> https://www.linkedin.com/in/taousc
>>>>
>>>
>>>
>>> _____________________________________________________________
>>> The information contained in this communication is intended solely for
>>> the use of the individual or entity to whom it is addressed and others
>>> authorized to receive it. It may contain confidential or legally privileged
>>> information. If you are not the intended recipient you are hereby notified
>>> that any disclosure, copying, distribution or taking any action in reliance
>>> on the contents of this information is strictly prohibited and may be
>>> unlawful. If you have received this communication in error, please notify
>>> us immediately by responding to this email and then delete it from your
>>> system. The firm is neither liable for the proper and complete transmission
>>> of the information contained in this communication nor for any delay in its
>>> receipt.
>>
>>
>>
>>
>> --
>>
>> *Tao Yan*
>> Software Engineer
>> Data Analytics Infrastructure Tools and Services
>>
>>
>>
>> 206.250.5345
>> t...@linkedin.com
>> https://www.linkedin.com/in/taousc
>>
>
>
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
>
--
*Tao Yan*
Software Engineer
Data Analytics Infrastructure Tools and Services
206.250.5345
t...@linkedin.com
https://www.linkedin.com/in/taousc
