Re: how to use cluster sparkSession like localSession

Hi, I think what you need is to have a long running Spark cluster to which you can submit jobs dynamically. For SQL, you can start Spark's HiveServer2: https://spark.apache.org/docs/latest/sql-programming-guide.html#distributed-sql-engine This will start a long

RE: how to use cluster sparkSession like localSession

Hello, I think you can try with below , the reason is only yarn-cllient mode is supported for your scenario. master("yarn-client") Thanks very much. Keith From: 张万新 Sent: Thursday, November 1, 2018 11:36 PM To: 崔苗(数据与人工智能产品开发部) <0049003...@znv.com> Cc: user Subject: Re: ho

Re: how to use cluster sparkSession like localSession

Agree. Spark is not designed for embedding in business applications(those traditional J2EE) for real-time interaction. Thanks, Gabriel On Fri, Nov 2, 2018 at 2:36 PM 张万新 wrote: > I think you should investigate apache zeppelin and livy > 崔苗(数据与人工智能产品开发部) <0049003...@znv.com>于2018年11月2日 周五11:01

Re: how to use cluster sparkSession like localSession

I think you should investigate apache zeppelin and livy 崔苗(数据与人工智能产品开发部) <0049003...@znv.com>于2018年11月2日 周五11:01写道： > > Hi， > we want to execute spark code with out submit application.jar,like this > code: > > public static void main(String args[]) throws Exception{ > SparkSession spark =

Re: how to use cluster sparkSession like localSession

we use spark in web server , no application.jar and submit on the cluster

Re: how to use cluster sparkSession like localSession

remove master configuration from code and then submit it to any cluster, it should work. On Fri, Nov 2, 2018 at 10:52 AM 崔苗(数据与人工智能产品开发部) <0049003...@znv.com> wrote: > > then how about spark sql and spark MLlib , we use them at most time > 0049003208 > 0049003...@znv.com > >

Re: how to use cluster sparkSession like localSession

then how about spark sql and spark MLlib , we use them at most time

Re: how to use cluster sparkSession like localSession

Please, read about Spark Streaming or Spark Structured Streaming. Your web application can easily communicate through some API and you won’t have the overhead of start a new spark job, which is pretty heavy. On Thu, Nov 1, 2018 at 23:01 崔苗(数据与人工智能产品开发部) <0049003...@znv.com> wrote: > > Hi， > we wa

how to use cluster sparkSession like localSession

Hi， we want to execute spark code with out submit application.jar,like this code:public static void main(String args[]) throws Exception{         SparkSession spark = SparkSession                 .builder()                 .master