Hi Lens Developers,
I am running Lens queries on example data. And I realized sometimes Lens
server will keep trying to connect to HiveServer2 during the query
execution, and Lens log shows:
*...*
04 Aug 2016 23:39:42 [1b0ba53f-e063-4dd0-a2f1-1519ce940839]
[grizzly-http-server-28] INFO org.apache.lens.driver.hive.HiveDriver - New
thrift connection class org.apache.lens.driver.hive.RemoteThriftConnection
for thread: 112 for user: test_user *connection ID=89* on driver:hive/hive1
04 Aug 2016 23:39:42 [1b0ba53f-e063-4dd0-a2f1-1519ce940839]
[grizzly-http-server-28] INFO
org.apache.lens.driver.hive.RemoteThriftConnection - HiveDriver connecting
to HiveServer @ *******.com:10000
04 Aug 2016 23:39:42 [1b0ba53f-e063-4dd0-a2f1-1519ce940839]
[grizzly-http-server-28] INFO
org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient -
Connecting to *******.com:10000
04 Aug 2016 23:39:42 [1b0ba53f-e063-4dd0-a2f1-1519ce940839]
[grizzly-http-server-28] INFO
org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient -
Connected!
04 Aug 2016 23:39:42 [1b0ba53f-e063-4dd0-a2f1-1519ce940839]
[grizzly-http-server-28] INFO
org.apache.lens.driver.hive.RemoteThriftConnection - HiveDriver connected
to HiveServer @ *******.com:10000
04 Aug 2016 23:39:47 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2]
[HiveDriver-ConnectionExpiryThread] INFO
org.apache.lens.driver.hive.HiveDriver - Closed connection: 30
04 Aug 2016 23:39:47 [1b0ba53f-e063-4dd0-a2f1-1519ce940839]
[grizzly-http-server-28] INFO
org.apache.lens.server.api.query.QueryContext - Updating status of
44c647ae-83dc-4ac7-a61f-040d2be07573 from Status : RUNNING
Message : Query is running in HiveServer!
Progress : 0.0
Progress Message :
[{"taskId":"Stage-1","type":"MAPRED","externalHandle":"job_1469684228847_467645","taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
to Status : RUNNING
Message : Query is running in HiveServer!
Progress : 0.0
Progress Message :
[{"taskId":"Stage-1","type":"MAPRED","externalHandle":"job_1469684228847_467645","taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
*...*
04 Aug 2016 23:42:23 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] [QueryPurger]
INFO org.apache.lens.driver.hive.HiveDriver - New thrift connection class
org.apache.lens.driver.hive.RemoteThriftConnection for thread: 25 for user:
anonymous *connection ID=106* on driver:hive/hive1
04 Aug 2016 23:42:23 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] [QueryPurger]
INFO org.apache.lens.driver.hive.RemoteThriftConnection - HiveDriver
connecting to HiveServer @ *******.com:10000
04 Aug 2016 23:42:23 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] [QueryPurger]
INFO org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient -
Connecting to *******.com:10000
04 Aug 2016 23:42:23 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] [QueryPurger]
INFO org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient -
Connected!
04 Aug 2016 23:42:23 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] [QueryPurger]
INFO org.apache.lens.driver.hive.RemoteThriftConnection - HiveDriver
connected to HiveServer @ *******.com:10000
It created 106 connections in total for the following example query:
lens-shell>query execute cube select product_id, store_sales from sales
where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01')
And it even created 2 connection when the query was successful. Then, it
started to close those connections:
*....*
04 Aug 2016 23:49:22 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2]
[HiveDriver-ConnectionExpiryThread] INFO
org.apache.lens.driver.hive.HiveDriver - Closed connection: 87
04 Aug 2016 23:49:32 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2]
[HiveDriver-ConnectionExpiryThread] INFO
org.apache.lens.driver.hive.HiveDriver - Closed connection: 88
04 Aug 2016 23:49:42 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2]
[HiveDriver-ConnectionExpiryThread] INFO
org.apache.lens.driver.hive.HiveDriver - Closed connection: 89
04 Aug 2016 23:49:52 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2]
[HiveDriver-ConnectionExpiryThread] INFO
org.apache.lens.driver.hive.HiveDriver - Closed connection: 90
*...*
Do we know why did Lens created so many connections for such a simple
query? And it really takes time to create those queries.
Thanks,
--
*Tao Yan*
Software Engineer
Data Analytics Infrastructure Tools and Services
206.250.5345
t...@linkedin.com
https://www.linkedin.com/in/taousc
