Hi Akila, IMO it would better to use Spark Mesos Framework on Mesos DC/OS for production deployments rather than trying to cluster the built-in Spark module on Mesos. Then we might not need to worry much about this problem.
None production environments such as Dev should be able to use a standalone DAS server with a volume mount with auto healing. WDYT? Thanks On Wed, Jul 6, 2016 at 9:38 AM, Akila Ravihansa Perera <raviha...@wso2.com> wrote: > [Looping Anjana and Isuru] > > @Anjana, Niranda: is it possible to plugin a custom member discovery > scheme for Spark? In the current implementation how does a wso2das with > embedded Spark instance discover the other members? > I'm referring to Spark cluster mode deployment here. > > Thanks. > > On Wed, Jul 6, 2016 at 6:49 AM, Nirmal Fernando <nir...@wso2.com> wrote: > >> [Looping Niranda] >> >> On Tue, Jul 5, 2016 at 10:22 PM, Akila Ravihansa Perera < >> raviha...@wso2.com> wrote: >> >>> Hi, >>> >>> Deploying wso2das in Mesos is bit tricky due to lack of overlay network >>> support in Mesos OOTB. This is mainly because Spark cluster (when running >>> in Spark cluster mode) is unable to communicate through Docker IP addresses >>> when they are assigned to different host machines in a multi-node Mesos >>> deployment. I checked the config options in [1] and tried setting >>> SPARK_PUBLIC_DNS parameter to Mesos host IP without any success. >>> >>> The main problem is there is no way to instruct Spark members to bind to >>> the local Docker IP and advertise a different IP (Mesos slave host IP) to >>> other members. >>> >>> @Niranda, Nirmal: is this something we can fix from our side? AFAIU, we >>> are using Hazelcast to discover Spark/wso2das members and adding them to >>> Spark context, right? >>> >>> On a side note, there is a Wiki page explaining how Spark should be used >>> with Mesos in [2]. This is available after Spark 1.2.0. In this approach, >>> we treat each Mesos slave as a Spark member and Spark/Mesos driver can >>> directly schedule tasks on Mesos slaves instead of running Spark itself as >>> a container. We should consider this approach in our C5 based efforts. We >>> can leverage Kubernetes in the same way. >>> >>> IMO, we should recommend users to use wso2das in Spark client mode on >>> Mesos due to these complexities. >>> >> >> +1, if we can't solve above from within DAS. >> >> >>> There is a DCOS Mesos framework for Spark [3] which can be used to >>> deploy Spark on Mesos very easily. We can even leverage DCOS Spark >>> framework in our deploy scripts. >>> >>> [1] http://spark.apache.org/docs/latest/spark-standalone.html >>> [2] http://spark.apache.org/docs/latest/running-on-mesos.html >>> [3] https://docs.mesosphere.com/1.7/usage/service-guides/spark/ >>> >>> Thanks. >>> >>> -- >>> Akila Ravihansa Perera >>> WSO2 Inc.; http://wso2.com/ >>> >>> Blog: http://ravihansa3000.blogspot.com >>> >> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Team Lead - WSO2 Machine Learner >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> > > > -- > Akila Ravihansa Perera > WSO2 Inc.; http://wso2.com/ > > Blog: http://ravihansa3000.blogspot.com > -- *Imesh Gunaratne* Software Architect WSO2 Inc: http://wso2.com T: +94 11 214 5345 M: +94 77 374 2057 W: https://medium.com/@imesh TW: @imesh
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
