Re: graphx vs graphframes
Hi Alastair Cypher support looks like promising and the dev list thread discussion is interesting. thanks for your feedback. On Thu, Oct 17, 2019 at 09:19:28AM +0100, Alastair Green wrote: > Hi Nicolas, > > I was following the current thread on the dev channel about Spark > Graph, including Cypher support, > > http://apache-spark-developers-list.1001551.n3.nabble.com/ > Add-spark-dependency-on-on-org-opencypher-okapi-shade-okapi-td28118.html > > and I remembered your post. > > Actually, GraphX and GraphFrames are both not being developed actively, so far > as I can tell. > > The only activity on GraphX in the last two years was a fix for Scala 2.13 > functionality: to quote the PR > > > ### Does this PR introduce any user-facing change? > > No behavior change at all. > > The only activity on GraphFrames since the addition of Pregel support in Scala > back in December 2018, has been build/test improvements and recent builds > against 2.4 and 3.0 snapshots. I’m not sure there was a lot of functional > change before that either. > > The efforts to provide graph processing in Spark with the more full-featured > Cypher query language that you can see in the proposed 3.0 changes discussed > in > the dev list, and the related openCypher/morpheus project (which among many > other things allows you to cast a Morpheus graph into a GraphX graph) and > extends the proposed 3.0 changes in a compatible way, are active. > > Yrs, > > Alastair > > > Alastair Green > > Query Languages Standards and Research > > > Neo4j UK Ltd > > Union House > 182-194 Union Street > London, SE1 0LH > > > +44 795 841 2107 > > > On Sun, Sep 22, 2019 at 21:17, Nicolas Paris wrote: > > hi all > > graphframes was intended to replace graphx. > > however the former looks not maintained anymore while the latter is > still active. > > any thought ? > -- > nicolas > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > -- nicolas - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Re: spark streaming exception
Please update me if any one knows about it. Thanks Amit On Thu, Oct 10, 2019 at 3:49 PM Amit Sharma wrote: > Hi , we have spark streaming job to which we send a request through our UI > using kafka. It process and returned the response. We are getting below > error and this stareming is not processing any request. > > Listener StreamingJobProgressListener threw an exception > java.util.NoSuchElementException: key not found: 1570689515000 ms > at scala.collection.MapLike$class.default(MapLike.scala:228) > at scala.collection.AbstractMap.default(Map.scala:59) > at scala.collection.mutable.HashMap.apply(HashMap.scala:65) > at > org.apache.spark.streaming.ui.StreamingJobProgressListener.onOutputOperationCompleted(StreamingJobProgressListener.scala:134) > at > org.apache.spark.streaming.scheduler.StreamingListenerBus.doPostEvent(StreamingListenerBus.scala:67) > at > org.apache.spark.streaming.scheduler.StreamingListenerBus.doPostEvent(StreamingListenerBus.scala:29). > > Please help me in find out the root cause of this issue. >
Spark Cluster over yarn cluster monitoring
Hi Users, I do submit *X* number of jobs with Airflow to Yarn as a part of workflow for *Y *customer. I could potentially run workflow for customer *Z *but I need to check that how much resources are available over the cluster so jobs for next customer should start. Could you please tell what is the best way to handle this. Currently, I am just checking availableMB > 100 then trigger next Airflow DAG over Yarn. GET http://rm-http-address:port/ws/v1/cluster/metrics Thanks.
Re: Control Sqoop job from Spark job
Shyam, As mark said - if we boost the parallelism with spark we can reach to performance of sqoop or better than that. On Tue, Sep 3, 2019 at 6:35 PM Shyam P wrote: > J Franke, > Leave alone sqoop , I am just asking about spark in ETL of Oracle ...? > > Thanks, > Shyam > >>
Spark - configuration setting doesn't work
Hi Users,
I am setting spark configuration in below way;
val spark = SparkSession.builder().appName(APP_NAME).getOrCreate()
spark.conf.set("spark.speculation", "false")
spark.conf.set("spark.broadcast.compress", "true")
spark.conf.set("spark.sql.broadcastTimeout", "36000")
spark.conf.set("spark.network.timeout", "2500s")
spark.conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
spark.conf.set("spark.driver.memory", "10g")
spark.conf.set("spark.executor.memory", "10g")
import spark.implicits._
and submitting spark job with spark - submit. but none of the above
configuration is
getting reflected to the job, I have checked at Spark-UI.
I know setting up like this while creation of spark object, it's working well.
val spark = SparkSession.builder().appName(APP_NAME)
.config("spark.network.timeout", "1500s")
.config("spark.broadcast.compress", "true")
.config("spark.sql.broadcastTimeout", "36000")
.getOrCreate()
import spark.implicits._
Can someone please throw light?
Re: graphx vs graphframes
Hi Nicolas, I was following the current thread on the dev channel about Spark Graph, including Cypher support, http://apache-spark-developers-list.1001551.n3.nabble.com/Add-spark-dependency-on-on-org-opencypher-okapi-shade-okapi-td28118.html [http://apache-spark-developers-list.1001551.n3.nabble.com/Add-spark-dependency-on-on-org-opencypher-okapi-shade-okapi-td28118.html] and I remembered your post. Actually, GraphX and GraphFrames are both not being developed actively, so far as I can tell. The only activity on GraphX in the last two years was a fix for Scala 2.13 functionality: to quote the PR ### Does this PR introduce any user-facing change?No behavior change at all. The only activity on GraphFrames since the addition of Pregel support in Scala back in December 2018, has been build/test improvements and recent builds against 2.4 and 3.0 snapshots. I’m not sure there was a lot of functional change before that either. The efforts to provide graph processing in Spark with the more full-featured Cypher query language that you can see in the proposed 3.0 changes discussed in the dev list, and the related openCypher/morpheus project (which among many other things allows you to cast a Morpheus graph into a GraphX graph) and extends the proposed 3.0 changes in a compatible way, are active. Yrs, Alastair Alastair Green Query Languages Standards and Research Neo4j UK Ltd Union House 182-194 Union Street London, SE1 0LH +44 795 841 2107 On Sun, Sep 22, 2019 at 21:17, Nicolas Paris wrote: hi all graphframes was intended to replace graphx. however the former looks not maintained anymore while the latter is still active. any thought ? -- nicolas - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
