[Looping Anjana and Isuru] @Anjana, Niranda: is it possible to plugin a custom member discovery scheme for Spark? In the current implementation how does a wso2das with embedded Spark instance discover the other members? I'm referring to Spark cluster mode deployment here.
Thanks. On Wed, Jul 6, 2016 at 6:49 AM, Nirmal Fernando <nir...@wso2.com> wrote: > [Looping Niranda] > > On Tue, Jul 5, 2016 at 10:22 PM, Akila Ravihansa Perera < > raviha...@wso2.com> wrote: > >> Hi, >> >> Deploying wso2das in Mesos is bit tricky due to lack of overlay network >> support in Mesos OOTB. This is mainly because Spark cluster (when running >> in Spark cluster mode) is unable to communicate through Docker IP addresses >> when they are assigned to different host machines in a multi-node Mesos >> deployment. I checked the config options in [1] and tried setting >> SPARK_PUBLIC_DNS parameter to Mesos host IP without any success. >> >> The main problem is there is no way to instruct Spark members to bind to >> the local Docker IP and advertise a different IP (Mesos slave host IP) to >> other members. >> >> @Niranda, Nirmal: is this something we can fix from our side? AFAIU, we >> are using Hazelcast to discover Spark/wso2das members and adding them to >> Spark context, right? >> >> On a side note, there is a Wiki page explaining how Spark should be used >> with Mesos in [2]. This is available after Spark 1.2.0. In this approach, >> we treat each Mesos slave as a Spark member and Spark/Mesos driver can >> directly schedule tasks on Mesos slaves instead of running Spark itself as >> a container. We should consider this approach in our C5 based efforts. We >> can leverage Kubernetes in the same way. >> >> IMO, we should recommend users to use wso2das in Spark client mode on >> Mesos due to these complexities. >> > > +1, if we can't solve above from within DAS. > > >> There is a DCOS Mesos framework for Spark [3] which can be used to deploy >> Spark on Mesos very easily. We can even leverage DCOS Spark framework in >> our deploy scripts. >> >> [1] http://spark.apache.org/docs/latest/spark-standalone.html >> [2] http://spark.apache.org/docs/latest/running-on-mesos.html >> [3] https://docs.mesosphere.com/1.7/usage/service-guides/spark/ >> >> Thanks. >> >> -- >> Akila Ravihansa Perera >> WSO2 Inc.; http://wso2.com/ >> >> Blog: http://ravihansa3000.blogspot.com >> > > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- Akila Ravihansa Perera WSO2 Inc.; http://wso2.com/ Blog: http://ravihansa3000.blogspot.com
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
