Spark thrift server on yarn
Hi, I am trying to start a spark thrift server using the following command on Spark 1.3.1 running on yarn: * ./sbin/start-thriftserver.sh --master yarn://resourcemanager.snc1:8032 --executor-memory 512m --hiveconf hive.server2.thrift.bind.host=test-host.sn1 --hiveconf hive.server2.thrift.port=10001 --queue public* It starts up fine and is able to connect to the hive metastore. I now need to view some temporary tables using this thrift server so I start up SparkSql and register a temp table. But the problem is that I am unable to view the temp table using the beeline client. I am pretty sure I am going wrong somewhere and the spark documentation does not clearly say how to run the thrift server in yarn mode or maybe I missed something. Could someone tell me how this is to be done or point me to some documentation? Thanks in advance, Udit
RE: Spark thrift server on yarn
Did you register temp table via the beeline or in a new Spark SQL CLI? As I know, the temp table cannot cross the HiveContext. Hao From: Udit Mehta [mailto:ume...@groupon.com] Sent: Wednesday, August 26, 2015 8:19 AM To: user Subject: Spark thrift server on yarn Hi, I am trying to start a spark thrift server using the following command on Spark 1.3.1 running on yarn: ./sbin/start-thriftserver.sh --master yarn://resourcemanager.snc1:8032 --executor-memory 512m --hiveconf hive.server2.thrift.bind.host=test-host.sn1 --hiveconf hive.server2.thrift.port=10001 --queue public It starts up fine and is able to connect to the hive metastore. I now need to view some temporary tables using this thrift server so I start up SparkSql and register a temp table. But the problem is that I am unable to view the temp table using the beeline client. I am pretty sure I am going wrong somewhere and the spark documentation does not clearly say how to run the thrift server in yarn mode or maybe I missed something. Could someone tell me how this is to be done or point me to some documentation? Thanks in advance, Udit
Re: Spark thrift server on yarn
I registered it in a new Spark SQL CLI. Yeah I thought so too about how the temp tables were accessible across different applications without using a job-server. I see that running* HiveThriftServer2.startWithContext(hiveContext) *within the spark app starts up a thrift server. On Tue, Aug 25, 2015 at 5:32 PM, Cheng, Hao hao.ch...@intel.com wrote: Did you register temp table via the beeline or in a new Spark SQL CLI? As I know, the temp table cannot cross the HiveContext. Hao *From:* Udit Mehta [mailto:ume...@groupon.com] *Sent:* Wednesday, August 26, 2015 8:19 AM *To:* user *Subject:* Spark thrift server on yarn Hi, I am trying to start a spark thrift server using the following command on Spark 1.3.1 running on yarn: * ./sbin/start-thriftserver.sh --master yarn://resourcemanager.snc1:8032 --executor-memory 512m --hiveconf hive.server2.thrift.bind.host=test-host.sn1 --hiveconf hive.server2.thrift.port=10001 --queue public* It starts up fine and is able to connect to the hive metastore. I now need to view some temporary tables using this thrift server so I start up SparkSql and register a temp table. But the problem is that I am unable to view the temp table using the beeline client. I am pretty sure I am going wrong somewhere and the spark documentation does not clearly say how to run the thrift server in yarn mode or maybe I missed something. Could someone tell me how this is to be done or point me to some documentation? Thanks in advance, Udit
