Re: Spark thrift server on yarn
I registered it in a new Spark SQL CLI. Yeah I thought so too about how the temp tables were accessible across different applications without using a job-server. I see that running* HiveThriftServer2.startWithContext(hiveContext) *within the spark app starts up a thrift server. On Tue, Aug 25, 2015 at 5:32 PM, Cheng, Hao wrote: > Did you register temp table via the beeline or in a new Spark SQL CLI? > > > > As I know, the temp table cannot cross the HiveContext. > > > > Hao > > > > *From:* Udit Mehta [mailto:ume...@groupon.com] > *Sent:* Wednesday, August 26, 2015 8:19 AM > *To:* user > *Subject:* Spark thrift server on yarn > > > > Hi, > > I am trying to start a spark thrift server using the following command on > Spark 1.3.1 running on yarn: > > * ./sbin/start-thriftserver.sh --master yarn://resourcemanager.snc1:8032 > --executor-memory 512m --hiveconf > hive.server2.thrift.bind.host=test-host.sn1 --hiveconf > hive.server2.thrift.port=10001 --queue public* > > It starts up fine and is able to connect to the hive metastore. > > I now need to view some temporary tables using this thrift server so I > start up SparkSql and register a temp table. > > But the problem is that I am unable to view the temp table using the > beeline client. I am pretty sure I am going wrong somewhere and the spark > documentation does not clearly say how to run the thrift server in yarn > mode or maybe I missed something. > Could someone tell me how this is to be done or point me to some > documentation? > > Thanks in advance, > > Udit >
RE: Spark thrift server on yarn
Did you register temp table via the beeline or in a new Spark SQL CLI? As I know, the temp table cannot cross the HiveContext. Hao From: Udit Mehta [mailto:ume...@groupon.com] Sent: Wednesday, August 26, 2015 8:19 AM To: user Subject: Spark thrift server on yarn Hi, I am trying to start a spark thrift server using the following command on Spark 1.3.1 running on yarn: ./sbin/start-thriftserver.sh --master yarn://resourcemanager.snc1:8032 --executor-memory 512m --hiveconf hive.server2.thrift.bind.host=test-host.sn1 --hiveconf hive.server2.thrift.port=10001 --queue public It starts up fine and is able to connect to the hive metastore. I now need to view some temporary tables using this thrift server so I start up SparkSql and register a temp table. But the problem is that I am unable to view the temp table using the beeline client. I am pretty sure I am going wrong somewhere and the spark documentation does not clearly say how to run the thrift server in yarn mode or maybe I missed something. Could someone tell me how this is to be done or point me to some documentation? Thanks in advance, Udit
Spark thrift server on yarn
Hi, I am trying to start a spark thrift server using the following command on Spark 1.3.1 running on yarn: * ./sbin/start-thriftserver.sh --master yarn://resourcemanager.snc1:8032 --executor-memory 512m --hiveconf hive.server2.thrift.bind.host=test-host.sn1 --hiveconf hive.server2.thrift.port=10001 --queue public* It starts up fine and is able to connect to the hive metastore. I now need to view some temporary tables using this thrift server so I start up SparkSql and register a temp table. But the problem is that I am unable to view the temp table using the beeline client. I am pretty sure I am going wrong somewhere and the spark documentation does not clearly say how to run the thrift server in yarn mode or maybe I missed something. Could someone tell me how this is to be done or point me to some documentation? Thanks in advance, Udit