Is this a specific Docker decision or a Zeppelin on Docker decision. I am curious on the amount of network traffic Zeppelin actually generates. I could be around, but I made the assumption that most of the network traffic with Zeppelin is results from the various endpoints (Spark, JDBC, Elastic Search etc) and not heavy lifting type activities.
John On Apr 12, 2016 5:03 PM, "vincent gromakowski" < vincent.gromakow...@gmail.com> wrote: > We decided to not use docker for network performance In production flows > not dor deployment. virtualisation of the network brings 50% decrease In > perf. It may change with calico because it abstract network with routing > not virtualizing like flannel > Le 12 avr. 2016 2:22 PM, "John Omernik" <j...@omernik.com> a écrit : > >> On 2. I had some thoughts there. How "expensive" would it be fore >> Zeppelin to run a timer of sorts that can be accessed via a specific URL. >> Basically, this URL would return the idle time. This thing that knows most >> if Zeppelin has activity is Zeppelin. So, any actions within Zeppelin >> would reset this timer basically, changing notebooks, opening, closing, >> moving notes around, running notes, adding new notes, changing interpreter >> settings. Any requests that are handled by Zeppelin in the UI, would reset >> said timer. A request to the "timer" URL obviously would NOT reset the >> timer, but basically, if nothing that was user actionable (we'd have to >> separate user actionable items from automated API requests) was run, the >> timer would not get reset. This would allow us using Zeppelin in a >> multi-user/multi-tenant environment to monitor for idle instances and take >> action when the occur. (Ideally, we could through an authenticated API >> issue a "save" of all notebooks before taking said action... >> >> So, to summarize: >> >> API that provides seconds since last human action... >> >> Monitor that API, when seconds since last human actions exceed enterprise >> threshold, then API can issue the "Safe Save all" to Zeppelin, which will >> go ahead and do a save (addition point, the timer API could return seconds >> since last human use and a bool value of "all saved" or not... basically, >> if normal Zeppelin processes have saved all human interaction, the API >> could indicate that, then, when the timer check hits the API, it knows, >> "The seconds past the threshold, and Zeppelin reports all saved, we can >> issue a termination, or if it's not all safe, it can issue the "save all" >> command, and wait for it to be safe... if something is keeping Zeppelin >> from being in a safe condition for shutdown, the API would reflect this and >> prevent a shutdown). >> >> Then, API seconds exceed enterprise threshold, we can safely shutdown the >> instance of Zeppelin returning resources to the cluster. >> >> Would love discussion here... >> >> On Tue, Apr 12, 2016 at 1:57 AM, vincent gromakowski < >> vincent.gromakow...@gmail.com> wrote: >> >>> 1. I am using ansible to deploy zeppelin on all slaves and to launch >>> zeppelin instance for one user. So if zeppelin binaries are already >>> deployed, the launch is very quick through marathon (1 or 2 sec). ooking >>> for velocity solution (based on jfrog) on Mesos to manage binaries and >>> artifacts with versioning, rights... No use of docker for network >>> performance constraints >>> >>> 2. Same answer as John. Still running. I will test dynamic resource for >>> spark interpreter but zeppelin daemon will still be up and taking 4GB >>> >>> 3. I have a service discovery that authenticate the user and route him >>> to his instance (and only his instance). It's based right now on a simple >>> shell script pulling marathon through its API and updating an apache >>> configuration file every 15s. The username is in the marathon task. We will >>> update this with a fully industrialized solution (consul ? haproxy ?...) >>> >>> >>> 3. >>> >>> 2016-04-12 2:37 GMT+02:00 Johnny W. <jzw.ser...@gmail.com>: >>> >>>> Thanks John for your insights. >>>> >>>> For 2., one solution we have experimented is spark dynamic resource >>>> allocation. We could define a timer to scale down. Hope that helps. >>>> >>>> J. >>>> >>>> On Mon, Apr 11, 2016 at 4:24 PM, John Omernik <j...@omernik.com> wrote: >>>> >>>>> 1. Things launch pretty fast for me, however, it depends if the docker >>>>> container I am running Zeppelin in is cached on the node mesos wants to >>>>> run >>>>> it on. If not, it pulls from a local docker registry, so worst case, up to >>>>> a minute to get things running if the image isn't cached. >>>>> >>>>> 2. No, if the user logs out it stays running. Ideally I would want to >>>>> setup some sort of timer that could scale down an instance if left unused. >>>>> I have some ideas here, but haven't put them into practice yet. I wanted >>>>> to play with Nginx to see if I could do something there (lack of activity >>>>> causes Nginx to shutdown Zeppelin for example). With spark resources, one >>>>> thing I wanted to play with using fine grain scaling with mesos, to only >>>>> use resources if queries were actually running. Lots of tools to fit the >>>>> bill here, just need to identify the right ones. >>>>> >>>>> 3. Dns resolution is handed for me with mesos-dns. Each instance has >>>>> its own Id and the dns name auto updates in mesos dns based on mesos >>>>> tasks >>>>> so I always know where Zeppelin is running. >>>>> >>>>> On Monday, April 11, 2016, Johnny W. <jzw.ser...@gmail.com> wrote: >>>>> >>>>>> John & Vincent, I am interested in the per instance per user >>>>>> approach. I have some questions about this approach: >>>>>> -- >>>>>> 1. how long will it take to launch a Zeppelin instance (and >>>>>> initialize SparkContext) when user log in? >>>>>> 2. will the instance be destroyed when user log out? if not, how do >>>>>> you deal with the resource assigned to Zeppelin/SparkContext? >>>>>> 3. for auto failover through marathon, how do you deal with the DNS >>>>>> resolve for clients? >>>>>> >>>>>> Thanks! >>>>>> J. >>>>>> >>>>>> On Fri, Apr 8, 2016 at 10:09 AM, John Omernik <j...@omernik.com> >>>>>> wrote: >>>>>> >>>>>>> So for us, we are doing something similar to Vincent, however, >>>>>>> instead of Gluster, we are using MapR-FS and the NFS mount. Basically, >>>>>>> this >>>>>>> gives us a shared filesystem that is running on all nodes, with strong >>>>>>> security (Filesystem ACEs for fine grained permissions) built in >>>>>>> auditing, >>>>>>> Posix compliance, true random read/write (as opposed to HDFS), >>>>>>> snapshots, >>>>>>> and cluster to cluster replication. There are also some neat things with >>>>>>> Volumes and Volume placement we are doing . That provides our storage >>>>>>> layer. Then we have docker for actually running Zeppelin, and since >>>>>>> it's a >>>>>>> instance per User, that helps organize who has access to what (Still >>>>>>> hashing out the details on that). Marathon on Mesos is how we ensure >>>>>>> that >>>>>>> Zeppelin is actually available, and then when it comes to spark, we are >>>>>>> just submitting to Mesos, which is right there. Since everything is on >>>>>>> one >>>>>>> cluster, the user has a home directory (on a volume) where I store all >>>>>>> configs for each instance of Zeppelin, and they can also put adhoc data >>>>>>> in >>>>>>> their home directory. Spark and Apache Drill can both query anything in >>>>>>> MapR FS, making it a pretty powerful combination. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Apr 8, 2016 at 6:33 AM, vincent gromakowski < >>>>>>> vincent.gromakow...@gmail.com> wrote: >>>>>>> >>>>>>>> Using it for 3 months without any incident >>>>>>>> Le 8 avr. 2016 9:09 AM, "ashish rawat" <dceash...@gmail.com> a >>>>>>>> écrit : >>>>>>>> >>>>>>>>> Sounds great. How long have you been using glusterfs in prod? and >>>>>>>>> have you encountered any challenges. The only difficulty for me to >>>>>>>>> use it, >>>>>>>>> would be a lack of expertise to fix broken things, so hope it's >>>>>>>>> stability >>>>>>>>> isn't something to be concerned about. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Ashish >>>>>>>>> >>>>>>>>> On Fri, Apr 8, 2016 at 12:20 PM, vincent gromakowski < >>>>>>>>> vincent.gromakow...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> use fuse interface. Gluster volume is directly accessible as >>>>>>>>>> local storage on all nodes but performance is only 200 Mb/s. More >>>>>>>>>> than >>>>>>>>>> enough for notebooks. For data prefer tachyon/alluxio on top of >>>>>>>>>> gluster... >>>>>>>>>> Le 8 avr. 2016 6:35 AM, "ashish rawat" <dceash...@gmail.com> a >>>>>>>>>> écrit : >>>>>>>>>> >>>>>>>>>>> Thanks Eran and Vincent. >>>>>>>>>>> Eran, I would definitely like to try it out, since it won't add >>>>>>>>>>> to the complexity of my deployment. Would see the S3 >>>>>>>>>>> implementation, to >>>>>>>>>>> figure out how complex it would be. >>>>>>>>>>> >>>>>>>>>>> Vincent, >>>>>>>>>>> I haven't explored glusterfs at all. Would it also require to >>>>>>>>>>> write an implementation of storage interface? Or zeppelin can work >>>>>>>>>>> with it, >>>>>>>>>>> out of the box? >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Ashish >>>>>>>>>>> >>>>>>>>>>> On Wed, Apr 6, 2016 at 12:53 PM, vincent gromakowski < >>>>>>>>>>> vincent.gromakow...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> For 1 marathon on mesos restart zeppelin daemon In case of >>>>>>>>>>>> failure. >>>>>>>>>>>> For 2 glusterfs fuse mount allows to share notebooks on all >>>>>>>>>>>> mesos nodes. >>>>>>>>>>>> For 3 not available right now In our design but a manual >>>>>>>>>>>> restart In zeppelin config page is acceptable for US. >>>>>>>>>>>> Le 6 avr. 2016 8:18 AM, "Eran Witkon" <eranwit...@gmail.com> a >>>>>>>>>>>> écrit : >>>>>>>>>>>> >>>>>>>>>>>>> Yes this is correct. >>>>>>>>>>>>> For HA disk, if you don't have HA storage and no access to S3 >>>>>>>>>>>>> then AFAIK you don't have other option at the moment. >>>>>>>>>>>>> If you like to save notebooks to elastic then I suggest you >>>>>>>>>>>>> look at the storage interface and implementation for git and s3 >>>>>>>>>>>>> and >>>>>>>>>>>>> implement that yourself. It does sound like an interesting feature >>>>>>>>>>>>> Best >>>>>>>>>>>>> Eran >>>>>>>>>>>>> On Wed, 6 Apr 2016 at 08:57 ashish rawat <dceash...@gmail.com> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks Eran. So 3, seems to be something external to >>>>>>>>>>>>>> Zeppelin, and hopefully 1 only means running "zeppelin-daemon.sh >>>>>>>>>>>>>> start" on >>>>>>>>>>>>>> a slave machine, when master become inaccessible. Is that >>>>>>>>>>>>>> correct? >>>>>>>>>>>>>> >>>>>>>>>>>>>> My main concern still remains on the storage front. And I >>>>>>>>>>>>>> don't really have high availability disks or even hdfs in my >>>>>>>>>>>>>> setup. I have >>>>>>>>>>>>>> been using elastic search cluster for data high availability, >>>>>>>>>>>>>> but was >>>>>>>>>>>>>> hoping that zeppelin can save notebooks to a Elastic Search >>>>>>>>>>>>>> (like kibana) >>>>>>>>>>>>>> or maybe a document store. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Any idea if anything is planned in that direction. Don't want >>>>>>>>>>>>>> to fallback to 'rsync' like options. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Ashish >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Apr 5, 2016 at 11:17 PM, Eran Witkon < >>>>>>>>>>>>>> eranwit...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> For 1 you need to have both zeppelin web HA and zeppelin >>>>>>>>>>>>>>> deamon HA >>>>>>>>>>>>>>> For 2 I guess you can use HDFS if you implement the storage >>>>>>>>>>>>>>> interface for HDFS. But i am not sure. >>>>>>>>>>>>>>> For 3 I mean that if you connect to an external cluster for >>>>>>>>>>>>>>> example a spark cluster you need to make sure your spark >>>>>>>>>>>>>>> cluster is HA. >>>>>>>>>>>>>>> Otherwise you will have zeppelin running but your notebook will >>>>>>>>>>>>>>> fail as no >>>>>>>>>>>>>>> spark cluster available. >>>>>>>>>>>>>>> HTH >>>>>>>>>>>>>>> Eran >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, 5 Apr 2016 at 20:20 ashish rawat < >>>>>>>>>>>>>>> dceash...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks Eran for your reply. >>>>>>>>>>>>>>>> For 1) I am assuming that it would similar to HA of any >>>>>>>>>>>>>>>> other web application, i.e. running multiple instances and >>>>>>>>>>>>>>>> switching to the >>>>>>>>>>>>>>>> backup server when master is down, is it not the case? >>>>>>>>>>>>>>>> For 2) is it also possible to save it on hdfs? >>>>>>>>>>>>>>>> Can you please explain 3, are you referring to interpreter >>>>>>>>>>>>>>>> config? If I am using Spark interpreter and submitting jobs to >>>>>>>>>>>>>>>> it, and if >>>>>>>>>>>>>>>> zeppelin master node goes down, then what could be the problem >>>>>>>>>>>>>>>> in slave >>>>>>>>>>>>>>>> node pointing to the same cluster and submitting jobs? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>> Ashish >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Apr 5, 2016 at 10:08 PM, Eran Witkon < >>>>>>>>>>>>>>>> eranwit...@gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I would say you need to account for these things >>>>>>>>>>>>>>>>> 1) availability of the zeppelin deamon >>>>>>>>>>>>>>>>> 2) availability of the notebookd files >>>>>>>>>>>>>>>>> 3) availability of the interpreters used. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> For 1 i don't know of out-of-box solution >>>>>>>>>>>>>>>>> For 2 any ha storage will do, s3 or any ha external >>>>>>>>>>>>>>>>> mounted disk >>>>>>>>>>>>>>>>> For 3 it is up to the interpreter and your big data ha >>>>>>>>>>>>>>>>> solution >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, 5 Apr 2016 at 19:29 ashish rawat < >>>>>>>>>>>>>>>>> dceash...@gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Is there a suggested architecture to run Zeppelin in high >>>>>>>>>>>>>>>>>> availability mode. The only option I could find was by >>>>>>>>>>>>>>>>>> saving notebooks to >>>>>>>>>>>>>>>>>> S3. Are there any options if one is not using AWS? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>> Ashish >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> Sent from my iThing >>>>> >>>> >>>> >>> >>
