Hi,
We finally get the hive storage engine up (the previous problem is mainly due
to some environmental variable settings)
However, it has problem when we try to query a specific hive table.
For example, we have the following table in the INFROMATION_SCHEMA.TABLES:
| DRILL | hivestg | store_sales_fact_denorm | TABLE |
When we query:
> select * from hivestg.`store_sales_fact_denorm` limit 1;
It shows the following exception:
Query failed: org.apache.drill.exec.rpc.RpcException: Remote failure while
running query.[error_id: "8bcf902e-a0c6-461c-94db-916d2188d1b2"
endpoint {
address: <my host name>
user_port: 31016
control_port: 31017
data_port: 31018
}
error_type: 0
message: "Failure while parsing sql. < OptimizerException:[ Failure while
attempting to retrieve storage engine. ] < IOException:[
org.apache.drill.common.exceptions.ExecutionSetupException:
org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate
with client version 4 ] < ExecutionSetupException:[
org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate
with client version 4 ] < RemoteException:[ Server IPC version 9 cannot
communicate with client version 4 ]"
]
It seems a version problem. Any suggestion on this?
BTW, we use the following storage-engines settings:
{
...
hivestg : {
type:"hive",
config :
{
"hive.metastore.uris" : "thrift://<hostname>:9083",
"hive.metastore.sasl.enabled" : "false"
}
}
}
}
Thanks,
Xiao
-----Original Message-----
From: George Chow [mailto:[email protected]]
Sent: March-27-14 5:12 PM
To: [email protected]
Subject: Re: Setting up Hive...
Hi Venki,
Thanks for the tip. I think I may be confused about my setup.
I have a remote metastore running alongside my HiveServer1. Thus, my
HiveServer1's hive-site.xml has the line
ConnectionURL=jdbc:mysql://localhost:3306/metastore.
I have tried configuring my Drillbit by pointing hive.metastore.uris to this
same URL (substituting the IP for localhost). But I'm not seeing anything.
Looking thru the various logs doesn't turn up anything.
I have already tried starting up the metastore in addition to the
HiveServer1 to no avail.
What am I doing wrong?
George
On Tue, Mar 25, 2014 at 7:07 PM, Venki Korukanti
<[email protected]>wrote:
> On Tue, Mar 25, 2014 at 6:56 PM, George Chow <[email protected]> wrote:
>
> > Hi Venki,
> >
> > I made a mistake initially and thought I was embedded. But I'm
> > actually
> in
> > scenario 1 (remote metastore).
>
>
> > I am setting hive.metastore.uris to my HiveServer's address and
> > port. Is that correct?
> >
> hive.metastore.uris should point to metastore server which is
> different from HiveServer1/2. Drill uses Metastore thrift interface to
> fetch the table/db info. From your first email, it looks like you are
> embedding the metastore within HiveServer1. You can copy the following
> settings from you current hiveserver's hive-site.xml to
> storage-engines.json to embed metastore in Drillbit or start the
> metastore as separate service using 'hive --service metastore' and point
> hive.metastore.uris to the address.
>
> "javax.jdo.option.ConnectionURL" :
> "jdbc:mysql://hostname/hivedrill?createDatabaseIfNotExist=true",
>
> "javax.jdo.option.ConnectionDriverName" : "com.mysql.jdbc.Driver",
>
>
> "javax.jdo.option.ConnectionUserName" : "user",
>
> "javax.jdo.option.ConnectionPassword" : "passwd"
>
>
> Thanks
>
> Venki
>
>
>
>
> > George
> >
> >
> > On Tue, Mar 25, 2014 at 6:48 PM, Venki Korukanti
> > <[email protected]>wrote:
> >
> > > Hi George,
> > >
> > > Are you embedding the metastore in drillbit
> > > (hive.metastore.uris="") or drillbit is connecting to remote
> > > metastore (hive.metastore.uris=valid address)?
> > >
> > > If you are embedding the metastore, make sure you have the
> > > appropriate driver (if MySql backend then mysql jdbc driver) in
> > > classpath or
> > drill/lib
> > > folder.
> > >
> > > Thanks
> > > Venki
> > >
> > >
> > > On Tue, Mar 25, 2014 at 6:30 PM, George Chow <[email protected]>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > Thanks Tim for the tip. It helps... so I copied the settings
> > > > from my hive-site.xml over to storage-engines.json.
> > > >
> > > > I restart my drillbit but I still don't see the contents from my
> > > > Hive metastore. The log file
> > > > (/var/log/drill/node11/drillbits.out) doesn't contain anything about
> > > > the inclusion of Hive.
> > > >
> > > > (I'm assuming that SELECT * FROM INFORMATION_SCHEMA.TABLES will
> return
> > > more
> > > > than the usual 5 tables (VIEW/COLUMNS/TABLES/CATALOGS/SCHEMATA.)
> > > >
> > > > George
> > > >
> > > >
> > > > On Tue, Mar 25, 2014 at 6:12 PM, Timothy Chen
> > > > <[email protected]>
> > wrote:
> > > >
> > > > > Hi George,
> > > > >
> > > > > I believe we have a wiki section on setting up hive + drill :
> > > > >
> > > > >
> > https://cwiki.apache.org/confluence/display/DRILL/Hive+Storage+Engin
> > e
> > > > >
> > > > > I haven't personally tried it yet but see if this helps.
> > > > >
> > > > > Tim
> > > > >
> > > > >
> > > > > On Tue, Mar 25, 2014 at 6:10 PM, George Chow
> > > > > <[email protected]>
> > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > So I'm trying to setup a drillbit to reach across to my Hive
> > > > metastore. I
> > > > > > dug into JIRA and see that DRILL-354/357 are both resolved.
> > > > > > I've
> > also
> > > > > > looked into my drillbit and found .../conf/storage-engines.json.
> > > > > >
> > > > > > Inside, I see an entry for Hive that's commented out.
> > > > > > However, it
> > is
> > > > for
> > > > > an
> > > > > > instance Hive with a Derby metastore. My instance is a
> HiveServer1
> > > > with a
> > > > > > MySQL metastore. So both javax.jdo.option.ConnectionURL and
> > > > > > hive.metastore.sasl.enabled are clear to me. But what about
> > > > > > the
> > > > > remaining:
> > > > > >
> > > > > > hive.metastore.uris
> > > > > > hive.metastore.warehouse.dir
> > > > > > fs.default.name
> > > > > >
> > > > > > George
> > > > > >
> > > > > > --
> > > > > > --
> > > > > > "Not everything that can be counted counts, and not
> > > > > > everything
> that
> > > > > counts
> > > > > > can be counted." Albert Einstein
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > --
> > > > "Not everything that can be counted counts, and not everything
> > > > that
> > > counts
> > > > can be counted." Albert Einstein
> > > >
> > >
> >
> >
> >
> > --
> > --
> > "Not everything that can be counted counts, and not everything that
> counts
> > can be counted." Albert Einstein
> >
>
--
--
"Not everything that can be counted counts, and not everything that counts can
be counted." Albert Einstein
