Re: how to use cluster sparkSession like localSession

2018-11-04 Thread Sumedh Wale
using REST API from your application providing it the required parameters like a remote procedure call. Or you can try SnappyData that provides both of these (and much more) out of the box. Regards, Sumedh Wale SnappyData (http://www.snappydata.io

Re: Is there a way to run Spark SQL through REST?

2017-07-24 Thread Sumedh Wale
, Jul 22, 2017 at 6:19 AM, Sumedh Wale <sw...@snappydata.io> wrote: On Saturday 22 July 2017 01:31 PM, kant kodali wrote: Is there a way to run Spark SQL throug

Re: custom joins on dataframe

2017-07-22 Thread Sumedh Wale
fuzzy joins). Its possible to do that using internal Spark APIs but its not easy and you have to implement an efficient join structure first. Or perhaps some existing libraries out there could work for you (like https://github.com/soundcloud/cosine-lsh-join-spark?). -- Sumedh Wale SnappyData

Re: Is there a way to run Spark SQL through REST?

2017-07-22 Thread Sumedh Wale
On Saturday 22 July 2017 01:31 PM, kant kodali wrote: Is there a way to run Spark SQL through REST? There is spark-jobserver (https://github.com/spark-jobserver/spark-jobserver). It does more than just REST API (like long running SparkContext). regards -- Sumedh Wale SnappyData (http

Re: about aggregateByKey of pairrdd.

2017-07-19 Thread Sumedh Wale
? Thanks. regards -- Sumedh Wale SnappyData (http://www.snappydata.io) - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: reduceByKey as Action or Transformation

2016-04-25 Thread Sumedh Wale
Thanks and Regards, Weiping regards -- Sumedh Wale SnappyData (http://www.snappydata.io) - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org

Re: SSL support for Spark Thrift Server

2016-03-08 Thread Sumedh Wale
/confluence/display/Hive/Setting+Up+HiveServer2#SettingUpHiveServer2-SSLEncryption Regards, Sourav thanks -- Sumedh Wale SnappyData (http://www.snappydata.io) - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org

Re: Best way to merge files from streaming jobs

2016-03-08 Thread Sumedh Wale
ing coalesce it will likely not use the full processing power. thanks -- Sumedh Wale SnappyData (http://www.snappydata.io) - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional comma

Re: Spark sql query taking long time

2016-03-03 Thread Sumedh Wale
. Regards, Gourav Sengupta thanks -- Sumedh Wale SnappyData (http://www.snappydata.io) On Thu, Mar 3, 2016 at 12:33 PM, Sumedh Wale <sw...@snappydata.io>

Re: Spark sql query taking long time

2016-03-03 Thread Sumedh Wale
is taking so long time. So can you suggest me the any optimized way, so i can reduce the execution time. My cluster has 3 slaves and 1 master. Thanks.

Re: SFTP Compressed CSV into Dataframe

2016-03-02 Thread Sumedh Wale
fuse/sshfs) and mount the remote directory locally. Since this uses FUSE, so works fine with normal user privileges. Thanks, Ben Thanks -- Sumedh Wale SnappyData (http://www.snappydata.io) - To unsubscribe, e-mail: use

Re: Add Jars to Master/Worker classpath

2016-03-02 Thread Sumedh Wale
So you can set SPARK_DIST_CLASSPATH in conf/spark-env.sh to add the required jars (separated by platform's File.pathSeparator). thanks -- Sumedh Wale SnappyData (http://www.snappydata.io) Am 2. März 2016 um 17:05 schrieb Igor Berman