[
https://issues.apache.org/jira/browse/LENS-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617850#comment-14617850
]
Raju Bairishetti commented on LENS-638:
---------------------------------------
SparkSql supports accessing of hive tables. As it understands HQL, it can
directly executes the rewritten query(HQL).
We can run hive
server([hiveserverDoc|https://spark.apache.org/docs/latest/sql-programming-guide.html#running-the-thrift-jdbcodbc-server])
in the spark cluster. Spark sql can acts as a distribution query execution
using this hive thrift server.
Tried it in local by starting spark hive thrift server, hive metastore and lens
server.
SetUp:
---------
1) Configure a remote hive metastore and started the metastore
(thrift://localhost:9083)
2) Started HiveThriftServer2, This server talks to metastore on 9083 port
3) Started Lens server, which connects to HiveThriftServer2 in the spark
cluster and hive metastore
Ran a cube query from the lens cli. *Query was executed successfully but seeing
issues with the results. Returning always zero rows in the result.* Currently,
looking into this issue.
{noformat}
lens-shell>debug
lens-shell>query execute cube select product_details.category, store_sales from
sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01')
07 Jul 2015 07:20:05,487 [Spring Shell] INFO
org.apache.lens.client.LensMetadataClient - URI for cube schema:
jar:file:/Users/raju.bairishetti/Desktop/git/apache-open-source-porjects/incubator-lens/lens-dist/target/apache-lens-2.3.0-beta-incubating-SNAPSHOT-bin/apache-lens-2.3.0-beta-incubating-SNAPSHOT-bin/client/lib/lens-api-2.3.0-beta-incubating-SNAPSHOT.jar!/cube-0.1.xsd
07 Jul 2015 07:20:06,628 [Spring Shell] INFO cliLogger - Query handle:
1fa4d870-0b39-4c7f-bb62-b72d37bbc8f1
07 Jul 2015 07:20:14,194 [Spring Shell] INFO cliLogger - User query: 'cube
select product_details.category, store_sales from sales where
time_range_in(order_time, '2015-04-11-00', '2015-04-13-01')' was submitted to
org.apache.lens.driver.hive.HiveDriver
07 Jul 2015 07:20:14,194 [Spring Shell] INFO cliLogger - Driver query:
'SELECT ( product_details . category ), sum(( sales . store_sales )) FROM
local_sales_aggr_fact1 sales join local_product_table product_details on
sales.product_id = product_details.id and (product_details.dt = 'latest') WHERE
(((((( sales . ot ) = '2015-04-11' ) or (( sales . ot ) = '2015-04-12' )) or
(( sales . ot ) = '2015-04-13-00' )))) GROUP BY ( product_details . category
)' and Driver handle: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=c6a44191-cea4-432f-9ff3-8dbf1bc6f011]
category _c1
0 rows process in (7) seconds.
{noformat}
*Query status: successful*
{noformat}
lens-shell>query status --query_handle 1fa4d870-0b39-4c7f-bb62-b72d37bbc8f1
Status : SUCCESSFUL
Message : Query is successful!
Progress : 1.0
{noformat}
> Add Spark Driver
> ----------------
>
> Key: LENS-638
> URL: https://issues.apache.org/jira/browse/LENS-638
> Project: Apache Lens
> Issue Type: Bug
> Reporter: Raju Bairishetti
> Assignee: Raju Bairishetti
>
> I would like to add a spark driver in lens. We can integrate with sparkSql
> support.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
