1. I am using ansible to deploy zeppelin on all slaves and to launch zeppelin instance for one user. So if zeppelin binaries are already deployed, the launch is very quick through marathon (1 or 2 sec). ooking for velocity solution (based on jfrog) on Mesos to manage binaries and artifacts with versioning, rights... No use of docker for network performance constraints
2. Same answer as John. Still running. I will test dynamic resource for spark interpreter but zeppelin daemon will still be up and taking 4GB 3. I have a service discovery that authenticate the user and route him to his instance (and only his instance). It's based right now on a simple shell script pulling marathon through its API and updating an apache configuration file every 15s. The username is in the marathon task. We will update this with a fully industrialized solution (consul ? haproxy ?...) 3. 2016-04-12 2:37 GMT+02:00 Johnny W. <jzw.ser...@gmail.com>: > Thanks John for your insights. > > For 2., one solution we have experimented is spark dynamic resource > allocation. We could define a timer to scale down. Hope that helps. > > J. > > On Mon, Apr 11, 2016 at 4:24 PM, John Omernik <j...@omernik.com> wrote: > >> 1. Things launch pretty fast for me, however, it depends if the docker >> container I am running Zeppelin in is cached on the node mesos wants to run >> it on. If not, it pulls from a local docker registry, so worst case, up to >> a minute to get things running if the image isn't cached. >> >> 2. No, if the user logs out it stays running. Ideally I would want to >> setup some sort of timer that could scale down an instance if left unused. >> I have some ideas here, but haven't put them into practice yet. I wanted >> to play with Nginx to see if I could do something there (lack of activity >> causes Nginx to shutdown Zeppelin for example). With spark resources, one >> thing I wanted to play with using fine grain scaling with mesos, to only >> use resources if queries were actually running. Lots of tools to fit the >> bill here, just need to identify the right ones. >> >> 3. Dns resolution is handed for me with mesos-dns. Each instance has its >> own Id and the dns name auto updates in mesos dns based on mesos tasks so >> I always know where Zeppelin is running. >> >> On Monday, April 11, 2016, Johnny W. <jzw.ser...@gmail.com> wrote: >> >>> John & Vincent, I am interested in the per instance per user approach. I >>> have some questions about this approach: >>> -- >>> 1. how long will it take to launch a Zeppelin instance (and initialize >>> SparkContext) when user log in? >>> 2. will the instance be destroyed when user log out? if not, how do you >>> deal with the resource assigned to Zeppelin/SparkContext? >>> 3. for auto failover through marathon, how do you deal with the DNS >>> resolve for clients? >>> >>> Thanks! >>> J. >>> >>> On Fri, Apr 8, 2016 at 10:09 AM, John Omernik <j...@omernik.com> wrote: >>> >>>> So for us, we are doing something similar to Vincent, however, instead >>>> of Gluster, we are using MapR-FS and the NFS mount. Basically, this gives >>>> us a shared filesystem that is running on all nodes, with strong security >>>> (Filesystem ACEs for fine grained permissions) built in auditing, Posix >>>> compliance, true random read/write (as opposed to HDFS), snapshots, and >>>> cluster to cluster replication. There are also some neat things with >>>> Volumes and Volume placement we are doing . That provides our storage >>>> layer. Then we have docker for actually running Zeppelin, and since it's a >>>> instance per User, that helps organize who has access to what (Still >>>> hashing out the details on that). Marathon on Mesos is how we ensure that >>>> Zeppelin is actually available, and then when it comes to spark, we are >>>> just submitting to Mesos, which is right there. Since everything is on one >>>> cluster, the user has a home directory (on a volume) where I store all >>>> configs for each instance of Zeppelin, and they can also put adhoc data in >>>> their home directory. Spark and Apache Drill can both query anything in >>>> MapR FS, making it a pretty powerful combination. >>>> >>>> >>>> >>>> On Fri, Apr 8, 2016 at 6:33 AM, vincent gromakowski < >>>> vincent.gromakow...@gmail.com> wrote: >>>> >>>>> Using it for 3 months without any incident >>>>> Le 8 avr. 2016 9:09 AM, "ashish rawat" <dceash...@gmail.com> a écrit : >>>>> >>>>>> Sounds great. How long have you been using glusterfs in prod? and >>>>>> have you encountered any challenges. The only difficulty for me to use >>>>>> it, >>>>>> would be a lack of expertise to fix broken things, so hope it's stability >>>>>> isn't something to be concerned about. >>>>>> >>>>>> Regards, >>>>>> Ashish >>>>>> >>>>>> On Fri, Apr 8, 2016 at 12:20 PM, vincent gromakowski < >>>>>> vincent.gromakow...@gmail.com> wrote: >>>>>> >>>>>>> use fuse interface. Gluster volume is directly accessible as local >>>>>>> storage on all nodes but performance is only 200 Mb/s. More than enough >>>>>>> for >>>>>>> notebooks. For data prefer tachyon/alluxio on top of gluster... >>>>>>> Le 8 avr. 2016 6:35 AM, "ashish rawat" <dceash...@gmail.com> a >>>>>>> écrit : >>>>>>> >>>>>>>> Thanks Eran and Vincent. >>>>>>>> Eran, I would definitely like to try it out, since it won't add to >>>>>>>> the complexity of my deployment. Would see the S3 implementation, to >>>>>>>> figure >>>>>>>> out how complex it would be. >>>>>>>> >>>>>>>> Vincent, >>>>>>>> I haven't explored glusterfs at all. Would it also require to write >>>>>>>> an implementation of storage interface? Or zeppelin can work with it, >>>>>>>> out >>>>>>>> of the box? >>>>>>>> >>>>>>>> Regards, >>>>>>>> Ashish >>>>>>>> >>>>>>>> On Wed, Apr 6, 2016 at 12:53 PM, vincent gromakowski < >>>>>>>> vincent.gromakow...@gmail.com> wrote: >>>>>>>> >>>>>>>>> For 1 marathon on mesos restart zeppelin daemon In case of failure. >>>>>>>>> For 2 glusterfs fuse mount allows to share notebooks on all mesos >>>>>>>>> nodes. >>>>>>>>> For 3 not available right now In our design but a manual restart >>>>>>>>> In zeppelin config page is acceptable for US. >>>>>>>>> Le 6 avr. 2016 8:18 AM, "Eran Witkon" <eranwit...@gmail.com> a >>>>>>>>> écrit : >>>>>>>>> >>>>>>>>>> Yes this is correct. >>>>>>>>>> For HA disk, if you don't have HA storage and no access to S3 >>>>>>>>>> then AFAIK you don't have other option at the moment. >>>>>>>>>> If you like to save notebooks to elastic then I suggest you look >>>>>>>>>> at the storage interface and implementation for git and s3 and >>>>>>>>>> implement >>>>>>>>>> that yourself. It does sound like an interesting feature >>>>>>>>>> Best >>>>>>>>>> Eran >>>>>>>>>> On Wed, 6 Apr 2016 at 08:57 ashish rawat <dceash...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Thanks Eran. So 3, seems to be something external to Zeppelin, >>>>>>>>>>> and hopefully 1 only means running "zeppelin-daemon.sh start" on a >>>>>>>>>>> slave >>>>>>>>>>> machine, when master become inaccessible. Is that correct? >>>>>>>>>>> >>>>>>>>>>> My main concern still remains on the storage front. And I don't >>>>>>>>>>> really have high availability disks or even hdfs in my setup. I >>>>>>>>>>> have been >>>>>>>>>>> using elastic search cluster for data high availability, but was >>>>>>>>>>> hoping >>>>>>>>>>> that zeppelin can save notebooks to a Elastic Search (like kibana) >>>>>>>>>>> or maybe >>>>>>>>>>> a document store. >>>>>>>>>>> >>>>>>>>>>> Any idea if anything is planned in that direction. Don't want to >>>>>>>>>>> fallback to 'rsync' like options. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Ashish >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 5, 2016 at 11:17 PM, Eran Witkon < >>>>>>>>>>> eranwit...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> For 1 you need to have both zeppelin web HA and zeppelin deamon >>>>>>>>>>>> HA >>>>>>>>>>>> For 2 I guess you can use HDFS if you implement the storage >>>>>>>>>>>> interface for HDFS. But i am not sure. >>>>>>>>>>>> For 3 I mean that if you connect to an external cluster for >>>>>>>>>>>> example a spark cluster you need to make sure your spark cluster >>>>>>>>>>>> is HA. >>>>>>>>>>>> Otherwise you will have zeppelin running but your notebook will >>>>>>>>>>>> fail as no >>>>>>>>>>>> spark cluster available. >>>>>>>>>>>> HTH >>>>>>>>>>>> Eran >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, 5 Apr 2016 at 20:20 ashish rawat <dceash...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Thanks Eran for your reply. >>>>>>>>>>>>> For 1) I am assuming that it would similar to HA of any other >>>>>>>>>>>>> web application, i.e. running multiple instances and switching to >>>>>>>>>>>>> the >>>>>>>>>>>>> backup server when master is down, is it not the case? >>>>>>>>>>>>> For 2) is it also possible to save it on hdfs? >>>>>>>>>>>>> Can you please explain 3, are you referring to interpreter >>>>>>>>>>>>> config? If I am using Spark interpreter and submitting jobs to >>>>>>>>>>>>> it, and if >>>>>>>>>>>>> zeppelin master node goes down, then what could be the problem in >>>>>>>>>>>>> slave >>>>>>>>>>>>> node pointing to the same cluster and submitting jobs? >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Ashish >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Apr 5, 2016 at 10:08 PM, Eran Witkon < >>>>>>>>>>>>> eranwit...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I would say you need to account for these things >>>>>>>>>>>>>> 1) availability of the zeppelin deamon >>>>>>>>>>>>>> 2) availability of the notebookd files >>>>>>>>>>>>>> 3) availability of the interpreters used. >>>>>>>>>>>>>> >>>>>>>>>>>>>> For 1 i don't know of out-of-box solution >>>>>>>>>>>>>> For 2 any ha storage will do, s3 or any ha external mounted >>>>>>>>>>>>>> disk >>>>>>>>>>>>>> For 3 it is up to the interpreter and your big data ha >>>>>>>>>>>>>> solution >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, 5 Apr 2016 at 19:29 ashish rawat <dceash...@gmail.com> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Is there a suggested architecture to run Zeppelin in high >>>>>>>>>>>>>>> availability mode. The only option I could find was by saving >>>>>>>>>>>>>>> notebooks to >>>>>>>>>>>>>>> S3. Are there any options if one is not using AWS? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> Ashish >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>> >>>> >>> >> >> -- >> Sent from my iThing >> > >
