For sure, both suggestions are to make sure we're not killing the CLI and that we have a component (the SDK client) that is shared by CLI and REPL.
Thanks for clarifying where the system calls will live. - Lucas On Fri, Oct 26, 2018 at 2:39 PM Daniel Takabayashi < daniel.takabaya...@gmail.com> wrote: > Hi Lucas, could you please justified your two suggestions? > > The general ideia is to use the Docker DSK to execute local commands, to > consume outputs and also to execute Marvin "specific language" commands. > Today almost all CLIs commands in Marvin is a mixed of systems commands and > library method calls. The plan is to put all "specific language calls" > inside the marvin-common-libs and group the systems call on the toolbox > (using the docker SDK in this task). > > > > > Em qui, 25 de out de 2018 às 20:09, Lucas Bonatto Miguel < > lucasb...@apache.org> escreveu: > > > Great, thanks. > > > > So here some suggestions: > > 1. We could create one new layer between Toolbox and the Engines, this > > would be Docker client. > > 2. Split the Toolbox layer in two. Toolbox CLI and Toolbox REPL > > > > It's not clear to me if the commands listed on General Commands will need > > to be written for every language, from the image I believe not, but how > it > > would work, since starting a notebook in Python is different than > starting > > in Scala. > > > > - Lucas > > On Thu, Oct 25, 2018 at 4:51 PM Daniel Takabayashi < > > daniel.takabaya...@gmail.com> wrote: > > > > > > > > > > > https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY > > > > > > Em qui, 25 de out de 2018 às 07:38, Rafael Novello < > > > rafa.reis.nove...@gmail.com> escreveu: > > > > > > > Hi Taka, > > > > > > > > I can't see the image! Maybe the Apache email list don't support > > > > attachments. > > > > > > > > Best! > > > > Rafael J. R. Novello > > > > > > > > Skype: rafael.novello > > > > Blog: http://rafanovello.blogspot.com.br/ > > > > > > > > > > > > Em qua, 24 de out de 2018 às 21:14, Daniel Takabayashi < > > > > daniel.takabaya...@gmail.com> escreveu: > > > > > > > > > Hi all, > > > > > > > > > > To try to give a little more details about this "new concept" that > we > > > > want > > > > > to bring to the marvin toolbox, I did this simple architecture > draw. > > > > > > > > > > [image: marvin-architecture-views (5).png] > > > > > > > > > > The general idea here is try to transform the toolbox something > > > > > disconnected with the "language", something more agnostic. Also in > > this > > > > new > > > > > architecture we could use remote resource to process engines and > make > > > > easy > > > > > the support for new languages. > > > > > > > > > > This "new toolbox" will be the only thing that a Marvin user must > to > > > > > install and also we could start to support multiples O.S (once the > > REPL > > > > is > > > > > a dummy application that only interprets and by pass commands). > > > > > > > > > > Regards, > > > > > Taka > > > > > > > > > > Em qua, 24 de out de 2018 às 09:52, Rafael Novello < > > > > > rafa.reis.nove...@gmail.com> escreveu: > > > > > > > > > >> Alan, > > > > >> > > > > >> Yes! We are using the Docker SDK and it's possible to use the API > to > > > > >> automate, but it's a little bit harder than automate CLI calls. > > > > >> > > > > >> Atenciosamente, > > > > >> Rafael J. R. Novello > > > > >> > > > > >> Skype: rafael.novello > > > > >> Blog: http://rafanovello.blogspot.com.br/ > > > > >> > > > > >> > > > > >> Em qua, 24 de out de 2018 às 12:02, Alan Silva <ju...@apache.org> > > > > >> escreveu: > > > > >> > > > > >> > Hi, > > > > >> > > > > > >> > One question here, I understand that we start to use with this > PoC > > > the > > > > >> > Docker SDK API, right? > > > > >> > > > > > >> > Why not use the API to expose some endpoints to permit this kind > > of > > > > >> > automation by devops? > > > > >> > > > > > >> > I think it is possible and it solves the CLI problem, right? > > > > >> > > > > > >> > > > > > >> > On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello < > > > > >> > rafa.reis.nove...@gmail.com> > > > > >> > wrote: > > > > >> > > > > > >> > > Lucas, > > > > >> > > > > > > >> > > Sorry, I didn't understood your question bellow. > > > > >> > > > > > > >> > > "Would it make sense to use the same solution that we will use > > for > > > > >> > having a > > > > >> > > single-language REPL to have a single-language CLI?" > > > > >> > > > > > > >> > > For DevOps purposes, maybe this new toolbox concept is not > > ideal. > > > I > > > > >> think > > > > >> > > we can keep the CLI inside the docker container but it will > not > > > easy > > > > >> to > > > > >> > > automate > > > > >> > > jobs by this way. > > > > >> > > > > > > >> > > How to deal with this issue? Voting? > > > > >> > > Atenciosamente, > > > > >> > > Rafael J. R. Novello > > > > >> > > > > > > >> > > Skype: rafael.novello > > > > >> > > Blog: http://rafanovello.blogspot.com.br/ > > > > >> > > > > > > >> > > > > > > >> > > Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto Miguel < > > > > >> > > lucasb...@apache.org> escreveu: > > > > >> > > > > > > >> > > > Got it! Thanks for clarifying. > > > > >> > > > > > > > >> > > > Would it make sense to use the same solution that we will > use > > > for > > > > >> > having > > > > >> > > a > > > > >> > > > single-language REPL to have a single-language CLI? My only > > > > concern > > > > >> > with > > > > >> > > > killing the CLI is that you remove an essential feature for > > > > DevOps. > > > > >> > > > > > > > >> > > > - Lucas > > > > >> > > > > > > > >> > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello < > > > > >> > > > rafa.reis.nove...@gmail.com> > > > > >> > > > wrote: > > > > >> > > > > > > > >> > > > > Lucas, > > > > >> > > > > > > > > >> > > > > The idea is that REPL will substitute the actual CLI. It's > > > > because > > > > >> > with > > > > >> > > > the > > > > >> > > > > actual concept (using language specific CLI) we will need > to > > > > >> > > re-implement > > > > >> > > > > the same features many times and probably each CLI will > > have a > > > > >> > > different > > > > >> > > > > behavior because some language specific restrictions > and/or > > > > >> > > limitations. > > > > >> > > > > > > > > >> > > > > With this new concept, all users will have the same > > experience > > > > >> > > > interacting > > > > >> > > > > with Marvin REPL and they will use the bests tools to > > develop > > > > >> their > > > > >> > > > engines > > > > >> > > > > (Jupyter Notebook and/or Lab with different languages > > support > > > or > > > > >> even > > > > >> > > > > Apache Zeppelin). All the interact will occur through REPL > > and > > > > >> docker > > > > >> > > > > protocol. > > > > >> > > > > > > > > >> > > > > Alan, Yes! As Lucas said, the concept is the same but we > can > > > use > > > > >> > docker > > > > >> > > > to > > > > >> > > > > do the same job. > > > > >> > > > > > > > > >> > > > > Best! > > > > >> > > > > > > > > >> > > > > Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto Miguel < > > > > >> > > > > lucasb...@apache.org> escreveu: > > > > >> > > > > > > > > >> > > > > > Thanks for the clarifications Rafael, so from what I > > > > understood, > > > > >> > > > Marvin's > > > > >> > > > > > developers would not use the REPL to explore data and > test > > > > >> models, > > > > >> > > i.e. > > > > >> > > > > > develop the engine. Is the idea to build something more > > like > > > > an > > > > >> > > > > interactive > > > > >> > > > > > CLI? I think an interactive CLI would be useful for the > > > > >> developer > > > > >> > > > > > experience, however, it's important to keep the > unattended > > > > >> > (current) > > > > >> > > > CLI > > > > >> > > > > in > > > > >> > > > > > place for scenarios where users want to automate Marvin > > > tasks. > > > > >> > > > > > > > > > >> > > > > > Alan, I believe the idea is still the same as you > started, > > > but > > > > >> > using > > > > >> > > > > > docker SDK now. > > > > >> > > > > > > > > > >> > > > > > - Lucas > > > > >> > > > > > > > > > >> > > > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello < > > > > >> > > > > > rafa.reis.nove...@gmail.com> > > > > >> > > > > > wrote: > > > > >> > > > > > > > > > >> > > > > > > Hi Lucas! > > > > >> > > > > > > > > > > >> > > > > > > First of all +1 for REPL POCs to Apache! > > > > >> > > > > > > > > > > >> > > > > > > Let me help with some comments: > > > > >> > > > > > > > > > > >> > > > > > > 1 - We have tested NodeJS, Scala and Python and the > > > easiest > > > > >> one > > > > >> > was > > > > >> > > > > > Python. > > > > >> > > > > > > We have found a small project [1] that have all > features > > > we > > > > >> > desired > > > > >> > > > for > > > > >> > > > > > > REPL: > > > > >> > > > > > > - Autocomplete > > > > >> > > > > > > - Python commands disabled (the user have only the > > > commands > > > > we > > > > >> > > > > provide). > > > > >> > > > > > > The other languages REPL options don't have this > > feature. > > > > >> > > > > > > - Easy to show output > > > > >> > > > > > > - Etc > > > > >> > > > > > > So, I think the language chosen will not be important > > here > > > > >> > because > > > > >> > > > the > > > > >> > > > > > user > > > > >> > > > > > > will only interact with the commands that we create. > > > > >> > > > > > > > > > > >> > > > > > > 2 - The "engine-generate" command will download a > docker > > > > image > > > > >> > that > > > > >> > > > we > > > > >> > > > > > > create for that language and start a container with > the > > > > basic > > > > >> > > project > > > > >> > > > > > > structure for that language. > > > > >> > > > > > > > > > > >> > > > > > > 3 - The REPL client will use the "docker protocol" to > > run > > > > >> > command, > > > > >> > > > > > > start/stop services and etc inside the container and > it > > > will > > > > >> > > receive > > > > >> > > > > the > > > > >> > > > > > > log stream to show. No, the REPL will no pass code > > > snippets > > > > >> for > > > > >> > > > docker > > > > >> > > > > > > container (I think it will not be necessary) > > > > >> > > > > > > > > > > >> > > > > > > 4 - Yep! Like I said on the first item. > > > > >> > > > > > > > > > > >> > > > > > > [1] - https://github.com/italorossi/ishell > > > > >> > > > > > > > > > > >> > > > > > > Let me know if there is any other question! > > > > >> > > > > > > Best! > > > > >> > > > > > > Rafael J. R. Novello > > > > >> > > > > > > > > > > >> > > > > > > Skype: rafael.novello > > > > >> > > > > > > Blog: http://rafanovello.blogspot.com.br/ > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > Em qui, 18 de out de 2018 às 17:00, Lucas Bonatto > > Miguel < > > > > >> > > > > > > lucasb...@apache.org> escreveu: > > > > >> > > > > > > > > > > >> > > > > > > > + 1 for migrating the REPL repo to Apache > > > > >> > > > > > > > > > > > >> > > > > > > > I have a few questions about the previous > explanation: > > > > >> > > > > > > > 1) The REPL itself, it would be an application in > > which > > > > >> > > language? > > > > >> > > > > > > Remember > > > > >> > > > > > > > that the main idea is to allow the user to program > on > > > his > > > > >> > > preferred > > > > >> > > > > > > > language in the REPL. > > > > >> > > > > > > > 2) Should the engine-generate command also > generate a > > > > >> docker > > > > >> > > image > > > > >> > > > > > with > > > > >> > > > > > > > the user's application? > > > > >> > > > > > > > 3) What type of communication would happen between > > the > > > > REPL > > > > >> > and > > > > >> > > > the > > > > >> > > > > > > engine > > > > >> > > > > > > > via Docker SDK? Would the REPL pass snippets of code > > to > > > be > > > > >> > > executed > > > > >> > > > > by > > > > >> > > > > > > the > > > > >> > > > > > > > docker container? > > > > >> > > > > > > > 4) Have you considered code completion in the REPL? > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei < > > > > >> > > > yifei.z.l...@gmail.com> > > > > >> > > > > > > > wrote: > > > > >> > > > > > > > > > > > >> > > > > > > > > Ok guys, > > > > >> > > > > > > > > > > > > >> > > > > > > > > The basic idea is to provide only one Toolbox for > > > > multiple > > > > >> > > > > languages. > > > > >> > > > > > > > > We are looking for possibility to build a single > > > Marvin > > > > >> Repl, > > > > >> > > > > > > > > instead of severals toolboxes with differentes > > > > interfaces > > > > >> or > > > > >> > > > > > commands. > > > > >> > > > > > > > > > > > > >> > > > > > > > > In this case, the engine-generate command will > > > download > > > > >> and > > > > >> > > > start a > > > > >> > > > > > > > Docker > > > > >> > > > > > > > > container with basic engine structure > corresponding > > > the > > > > >> > choosed > > > > >> > > > > > > language. > > > > >> > > > > > > > > this means we don't need to build Toolboxes of > > > > differents > > > > >> > > > > languages, > > > > >> > > > > > we > > > > >> > > > > > > > > just > > > > >> > > > > > > > > build the engine template of all languages that we > > > want > > > > to > > > > >> > > > support > > > > >> > > > > > and > > > > >> > > > > > > > > provide it > > > > >> > > > > > > > > as Docker containers > > > > >> > > > > > > > > > > > > >> > > > > > > > > We have started researches around our basic > > > requirements > > > > >> > like: > > > > >> > > > > > > > > - Repl interface > > > > >> > > > > > > > > - System communication > > > > >> > > > > > > > > - Connection security > > > > >> > > > > > > > > - Tool popularity > > > > >> > > > > > > > > - Update complexity > > > > >> > > > > > > > > - Languages support > > > > >> > > > > > > > > - ...... > > > > >> > > > > > > > > > > > > >> > > > > > > > > And we did some POC with code here: > > > > >> > > > > > > > > https://github.com/marvin-ai/marvin-repl > > > > >> > > > > > > > > > > > > >> > > > > > > > > There is POC testing gRPC using Scala and Python, > > > > >> > > > > > > > > Repl inteface and Docker SDK with NodeJS, > > > > >> > > > > > > > > Repl interface and Docker SDK with Python. > > > > >> > > > > > > > > > > > > >> > > > > > > > > At this moment we prefer the Repl interface + > Docker > > > SDK > > > > >> way, > > > > >> > > > > because > > > > >> > > > > > > > good > > > > >> > > > > > > > > part of the requirements > > > > >> > > > > > > > > will be guaranteed by Docker. > > > > >> > > > > > > > > > > > > >> > > > > > > > > With this informations, what do you think? Should > we > > > > >> submit > > > > >> > all > > > > >> > > > > this > > > > >> > > > > > > POCs > > > > >> > > > > > > > > to Apache Repo? > > > > >> > > > > > > > > Please feel free to opine. > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > Thats all, thanks!!! > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > Em ter, 16 de out de 2018 às 18:55, Daniel > > > Takabayashi < > > > > >> > > > > > > > > daniel.takabaya...@gmail.com> escreveu: > > > > >> > > > > > > > > > > > > >> > > > > > > > > > Zhang, > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > I think the best approach is give us a better > > > > >> explanation > > > > >> > > about > > > > >> > > > > > this > > > > >> > > > > > > > new > > > > >> > > > > > > > > > feature and how this can help us to archive the > > > > desired > > > > >> > > feature > > > > >> > > > > > > > (support > > > > >> > > > > > > > > > multiple languages). Than if the majority agree > > > than I > > > > >> can > > > > >> > do > > > > >> > > > the > > > > >> > > > > > > > merge, > > > > >> > > > > > > > > > just like I did before, but in another branch. > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > Let me know if makes sense, ok? > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > Taka > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > Em ter, 16 de out de 2018 às 14:00, Luciano > > Resende > > > < > > > > >> > > > > > > > > luckbr1...@gmail.com> > > > > >> > > > > > > > > > escreveu: > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > So, what is the POC, is it a refactoring of > the > > > > >> existing > > > > >> > > > repo? > > > > >> > > > > Or > > > > >> > > > > > > is > > > > >> > > > > > > > > > > it a new rewrite of the repo? > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > Just asking as it might make sense to make it > a > > > > branch > > > > >> > then > > > > >> > > > > > > actually > > > > >> > > > > > > > a > > > > >> > > > > > > > > > > thing in parallel, as this will have an effect > > for > > > > >> > > releases, > > > > >> > > > > etc. > > > > >> > > > > > > but > > > > >> > > > > > > > > > > you guys know more here than I do. > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > Also, it's probably good to have a write up of > > the > > > > >> main > > > > >> > > > > direction > > > > >> > > > > > > of > > > > >> > > > > > > > > > > the design that can help people get familiar > > with > > > > the > > > > >> new > > > > >> > > > > > approach. > > > > >> > > > > > > > > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei < > > > > >> > > > > > > yifei.z.l...@gmail.com > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > wrote: > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > Hey guys, > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > We have reorganized the Poc repo, and want > to > > > > merge > > > > >> it > > > > >> > to > > > > >> > > > our > > > > >> > > > > > > > Apache > > > > >> > > > > > > > > > > repo. > > > > >> > > > > > > > > > > > Just making sure here before we do the > merge, > > > > >> because > > > > >> > im > > > > >> > > > not > > > > >> > > > > > sure > > > > >> > > > > > > > if > > > > >> > > > > > > > > > > doing > > > > >> > > > > > > > > > > > this will perserve the Git commit history. > > > > >> > > > > > > > > > > > What we are planning to do is: > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > git filter-branch --subdirectory-filter <git > > > > >> > > > > origin_repository > > > > >> > > > > > > > > > directory> > > > > >> > > > > > > > > > > > -- --all > > > > >> > > > > > > > > > > > mkdir POCs/ > > > > >> > > > > > > > > > > > git mv * POCs/ > > > > >> > > > > > > > > > > > git commit -m "colleted the data to move" > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > git clone <git destination_repository_url> > cd > > > <git > > > > >> > > > > > > > > > > > destination_repository_directory> git remote > > add > > > > >> > > > > > > > > > origin_repository_branch > > > > >> > > > > > > > > > > > <git origin_repository directory> git pull > > > > >> > > > > > > origin_repository_branch > > > > >> > > > > > > > > > > master > > > > >> > > > > > > > > > > > --allow-unrelated-histories git remote rm > > > > >> > > > > > > origin_repository_branch > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > Thanks in advance ! > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > -------------------------------------------------------------- > > > > >> > > > > > > > > > > > Zhang Yifei > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > -- > > > > >> > > > > > > > > > > Luciano Resende > > > > >> > > > > > > > > > > http://twitter.com/lresende1975 > > > > >> > > > > > > > > > > http://lresende.blogspot.com/ > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > -- > > > > >> > > > > > > > > > > > > >> > -------------------------------------------------------------- > > > > >> > > > > > > > > Zhang Yifei > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >
