Hi all, To try to give a little more details about this "new concept" that we want to bring to the marvin toolbox, I did this simple architecture draw.
[image: marvin-architecture-views (5).png] The general idea here is try to transform the toolbox something disconnected with the "language", something more agnostic. Also in this new architecture we could use remote resource to process engines and make easy the support for new languages. This "new toolbox" will be the only thing that a Marvin user must to install and also we could start to support multiples O.S (once the REPL is a dummy application that only interprets and by pass commands). Regards, Taka Em qua, 24 de out de 2018 às 09:52, Rafael Novello < rafa.reis.nove...@gmail.com> escreveu: > Alan, > > Yes! We are using the Docker SDK and it's possible to use the API to > automate, but it's a little bit harder than automate CLI calls. > > Atenciosamente, > Rafael J. R. Novello > > Skype: rafael.novello > Blog: http://rafanovello.blogspot.com.br/ > > > Em qua, 24 de out de 2018 às 12:02, Alan Silva <ju...@apache.org> > escreveu: > > > Hi, > > > > One question here, I understand that we start to use with this PoC the > > Docker SDK API, right? > > > > Why not use the API to expose some endpoints to permit this kind of > > automation by devops? > > > > I think it is possible and it solves the CLI problem, right? > > > > > > On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello < > > rafa.reis.nove...@gmail.com> > > wrote: > > > > > Lucas, > > > > > > Sorry, I didn't understood your question bellow. > > > > > > "Would it make sense to use the same solution that we will use for > > having a > > > single-language REPL to have a single-language CLI?" > > > > > > For DevOps purposes, maybe this new toolbox concept is not ideal. I > think > > > we can keep the CLI inside the docker container but it will not easy to > > > automate > > > jobs by this way. > > > > > > How to deal with this issue? Voting? > > > Atenciosamente, > > > Rafael J. R. Novello > > > > > > Skype: rafael.novello > > > Blog: http://rafanovello.blogspot.com.br/ > > > > > > > > > Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto Miguel < > > > lucasb...@apache.org> escreveu: > > > > > > > Got it! Thanks for clarifying. > > > > > > > > Would it make sense to use the same solution that we will use for > > having > > > a > > > > single-language REPL to have a single-language CLI? My only concern > > with > > > > killing the CLI is that you remove an essential feature for DevOps. > > > > > > > > - Lucas > > > > > > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello < > > > > rafa.reis.nove...@gmail.com> > > > > wrote: > > > > > > > > > Lucas, > > > > > > > > > > The idea is that REPL will substitute the actual CLI. It's because > > with > > > > the > > > > > actual concept (using language specific CLI) we will need to > > > re-implement > > > > > the same features many times and probably each CLI will have a > > > different > > > > > behavior because some language specific restrictions and/or > > > limitations. > > > > > > > > > > With this new concept, all users will have the same experience > > > > interacting > > > > > with Marvin REPL and they will use the bests tools to develop their > > > > engines > > > > > (Jupyter Notebook and/or Lab with different languages support or > even > > > > > Apache Zeppelin). All the interact will occur through REPL and > docker > > > > > protocol. > > > > > > > > > > Alan, Yes! As Lucas said, the concept is the same but we can use > > docker > > > > to > > > > > do the same job. > > > > > > > > > > Best! > > > > > > > > > > Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto Miguel < > > > > > lucasb...@apache.org> escreveu: > > > > > > > > > > > Thanks for the clarifications Rafael, so from what I understood, > > > > Marvin's > > > > > > developers would not use the REPL to explore data and test > models, > > > i.e. > > > > > > develop the engine. Is the idea to build something more like an > > > > > interactive > > > > > > CLI? I think an interactive CLI would be useful for the developer > > > > > > experience, however, it's important to keep the unattended > > (current) > > > > CLI > > > > > in > > > > > > place for scenarios where users want to automate Marvin tasks. > > > > > > > > > > > > Alan, I believe the idea is still the same as you started, but > > using > > > > > > docker SDK now. > > > > > > > > > > > > - Lucas > > > > > > > > > > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello < > > > > > > rafa.reis.nove...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Hi Lucas! > > > > > > > > > > > > > > First of all +1 for REPL POCs to Apache! > > > > > > > > > > > > > > Let me help with some comments: > > > > > > > > > > > > > > 1 - We have tested NodeJS, Scala and Python and the easiest one > > was > > > > > > Python. > > > > > > > We have found a small project [1] that have all features we > > desired > > > > for > > > > > > > REPL: > > > > > > > - Autocomplete > > > > > > > - Python commands disabled (the user have only the commands we > > > > > provide). > > > > > > > The other languages REPL options don't have this feature. > > > > > > > - Easy to show output > > > > > > > - Etc > > > > > > > So, I think the language chosen will not be important here > > because > > > > the > > > > > > user > > > > > > > will only interact with the commands that we create. > > > > > > > > > > > > > > 2 - The "engine-generate" command will download a docker image > > that > > > > we > > > > > > > create for that language and start a container with the basic > > > project > > > > > > > structure for that language. > > > > > > > > > > > > > > 3 - The REPL client will use the "docker protocol" to run > > command, > > > > > > > start/stop services and etc inside the container and it will > > > receive > > > > > the > > > > > > > log stream to show. No, the REPL will no pass code snippets for > > > > docker > > > > > > > container (I think it will not be necessary) > > > > > > > > > > > > > > 4 - Yep! Like I said on the first item. > > > > > > > > > > > > > > [1] - https://github.com/italorossi/ishell > > > > > > > > > > > > > > Let me know if there is any other question! > > > > > > > Best! > > > > > > > Rafael J. R. Novello > > > > > > > > > > > > > > Skype: rafael.novello > > > > > > > Blog: http://rafanovello.blogspot.com.br/ > > > > > > > > > > > > > > > > > > > > > Em qui, 18 de out de 2018 às 17:00, Lucas Bonatto Miguel < > > > > > > > lucasb...@apache.org> escreveu: > > > > > > > > > > > > > > > + 1 for migrating the REPL repo to Apache > > > > > > > > > > > > > > > > I have a few questions about the previous explanation: > > > > > > > > 1) The REPL itself, it would be an application in which > > > language? > > > > > > > Remember > > > > > > > > that the main idea is to allow the user to program on his > > > preferred > > > > > > > > language in the REPL. > > > > > > > > 2) Should the engine-generate command also generate a docker > > > image > > > > > > with > > > > > > > > the user's application? > > > > > > > > 3) What type of communication would happen between the REPL > > and > > > > the > > > > > > > engine > > > > > > > > via Docker SDK? Would the REPL pass snippets of code to be > > > executed > > > > > by > > > > > > > the > > > > > > > > docker container? > > > > > > > > 4) Have you considered code completion in the REPL? > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei < > > > > yifei.z.l...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Ok guys, > > > > > > > > > > > > > > > > > > The basic idea is to provide only one Toolbox for multiple > > > > > languages. > > > > > > > > > We are looking for possibility to build a single Marvin > Repl, > > > > > > > > > instead of severals toolboxes with differentes interfaces > or > > > > > > commands. > > > > > > > > > > > > > > > > > > In this case, the engine-generate command will download and > > > > start a > > > > > > > > Docker > > > > > > > > > container with basic engine structure corresponding the > > choosed > > > > > > > language. > > > > > > > > > this means we don't need to build Toolboxes of differents > > > > > languages, > > > > > > we > > > > > > > > > just > > > > > > > > > build the engine template of all languages that we want to > > > > support > > > > > > and > > > > > > > > > provide it > > > > > > > > > as Docker containers > > > > > > > > > > > > > > > > > > We have started researches around our basic requirements > > like: > > > > > > > > > - Repl interface > > > > > > > > > - System communication > > > > > > > > > - Connection security > > > > > > > > > - Tool popularity > > > > > > > > > - Update complexity > > > > > > > > > - Languages support > > > > > > > > > - ...... > > > > > > > > > > > > > > > > > > And we did some POC with code here: > > > > > > > > > https://github.com/marvin-ai/marvin-repl > > > > > > > > > > > > > > > > > > There is POC testing gRPC using Scala and Python, > > > > > > > > > Repl inteface and Docker SDK with NodeJS, > > > > > > > > > Repl interface and Docker SDK with Python. > > > > > > > > > > > > > > > > > > At this moment we prefer the Repl interface + Docker SDK > way, > > > > > because > > > > > > > > good > > > > > > > > > part of the requirements > > > > > > > > > will be guaranteed by Docker. > > > > > > > > > > > > > > > > > > With this informations, what do you think? Should we submit > > all > > > > > this > > > > > > > POCs > > > > > > > > > to Apache Repo? > > > > > > > > > Please feel free to opine. > > > > > > > > > > > > > > > > > > > > > > > > > > > Thats all, thanks!!! > > > > > > > > > > > > > > > > > > > > > > > > > > > Em ter, 16 de out de 2018 às 18:55, Daniel Takabayashi < > > > > > > > > > daniel.takabaya...@gmail.com> escreveu: > > > > > > > > > > > > > > > > > > > Zhang, > > > > > > > > > > > > > > > > > > > > I think the best approach is give us a better explanation > > > about > > > > > > this > > > > > > > > new > > > > > > > > > > feature and how this can help us to archive the desired > > > feature > > > > > > > > (support > > > > > > > > > > multiple languages). Than if the majority agree than I > can > > do > > > > the > > > > > > > > merge, > > > > > > > > > > just like I did before, but in another branch. > > > > > > > > > > > > > > > > > > > > Let me know if makes sense, ok? > > > > > > > > > > > > > > > > > > > > Taka > > > > > > > > > > > > > > > > > > > > Em ter, 16 de out de 2018 às 14:00, Luciano Resende < > > > > > > > > > luckbr1...@gmail.com> > > > > > > > > > > escreveu: > > > > > > > > > > > > > > > > > > > > > So, what is the POC, is it a refactoring of the > existing > > > > repo? > > > > > Or > > > > > > > is > > > > > > > > > > > it a new rewrite of the repo? > > > > > > > > > > > > > > > > > > > > > > Just asking as it might make sense to make it a branch > > then > > > > > > > actually > > > > > > > > a > > > > > > > > > > > thing in parallel, as this will have an effect for > > > releases, > > > > > etc. > > > > > > > but > > > > > > > > > > > you guys know more here than I do. > > > > > > > > > > > > > > > > > > > > > > Also, it's probably good to have a write up of the main > > > > > direction > > > > > > > of > > > > > > > > > > > the design that can help people get familiar with the > new > > > > > > approach. > > > > > > > > > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei < > > > > > > > yifei.z.l...@gmail.com > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hey guys, > > > > > > > > > > > > > > > > > > > > > > > > We have reorganized the Poc repo, and want to merge > it > > to > > > > our > > > > > > > > Apache > > > > > > > > > > > repo. > > > > > > > > > > > > Just making sure here before we do the merge, because > > im > > > > not > > > > > > sure > > > > > > > > if > > > > > > > > > > > doing > > > > > > > > > > > > this will perserve the Git commit history. > > > > > > > > > > > > What we are planning to do is: > > > > > > > > > > > > > > > > > > > > > > > > git filter-branch --subdirectory-filter <git > > > > > origin_repository > > > > > > > > > > directory> > > > > > > > > > > > > -- --all > > > > > > > > > > > > mkdir POCs/ > > > > > > > > > > > > git mv * POCs/ > > > > > > > > > > > > git commit -m "colleted the data to move" > > > > > > > > > > > > > > > > > > > > > > > > git clone <git destination_repository_url> cd <git > > > > > > > > > > > > destination_repository_directory> git remote add > > > > > > > > > > origin_repository_branch > > > > > > > > > > > > <git origin_repository directory> git pull > > > > > > > origin_repository_branch > > > > > > > > > > > master > > > > > > > > > > > > --allow-unrelated-histories git remote rm > > > > > > > origin_repository_branch > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks in advance ! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------------------------------------------------------- > > > > > > > > > > > > Zhang Yifei > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Luciano Resende > > > > > > > > > > > http://twitter.com/lresende1975 > > > > > > > > > > > http://lresende.blogspot.com/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > -------------------------------------------------------------- > > > > > > > > > Zhang Yifei > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
