Re: graphx vs graphframes

2019-10-17 Thread Nicolas Paris
Hi Alastair Cypher support looks like promising and the dev list thread discussion is interesting. thanks for your feedback. On Thu, Oct 17, 2019 at 09:19:28AM +0100, Alastair Green wrote: > Hi Nicolas, > > I was following the current thread on the dev channel about Spark > Graph, including

Re: spark streaming exception

2019-10-17 Thread Amit Sharma
Please update me if any one knows about it. Thanks Amit On Thu, Oct 10, 2019 at 3:49 PM Amit Sharma wrote: > Hi , we have spark streaming job to which we send a request through our UI > using kafka. It process and returned the response. We are getting below > error and this stareming is not

Spark Cluster over yarn cluster monitoring

2019-10-17 Thread Chetan Khatri
Hi Users, I do submit *X* number of jobs with Airflow to Yarn as a part of workflow for *Y *customer. I could potentially run workflow for customer *Z *but I need to check that how much resources are available over the cluster so jobs for next customer should start. Could you please tell what is

Re: Control Sqoop job from Spark job

2019-10-17 Thread Chetan Khatri
Shyam, As mark said - if we boost the parallelism with spark we can reach to performance of sqoop or better than that. On Tue, Sep 3, 2019 at 6:35 PM Shyam P wrote: > J Franke, > Leave alone sqoop , I am just asking about spark in ETL of Oracle ...? > > Thanks, > Shyam > >>

Spark - configuration setting doesn't work

2019-10-17 Thread Chetan Khatri
Hi Users, I am setting spark configuration in below way; val spark = SparkSession.builder().appName(APP_NAME).getOrCreate() spark.conf.set("spark.speculation", "false") spark.conf.set("spark.broadcast.compress", "true") spark.conf.set("spark.sql.broadcastTimeout", "36000")

Re: graphx vs graphframes

2019-10-17 Thread Alastair Green
Hi Nicolas, I was following the current thread on the dev channel about Spark Graph, including Cypher support, http://apache-spark-developers-list.1001551.n3.nabble.com/Add-spark-dependency-on-on-org-opencypher-okapi-shade-okapi-td28118.html