Re: Marvin’s mission discussion

2020-07-28 Thread Wei Chen
I think deploying to K8S does expend our capabilities for inference scaling
and managing.
I am not familiar with Luigi, but it makes sense since we are going to
setup data pipelines.

Best Regards,
Wei

On Wed, Jul 29, 2020 at 5:32 AM Lucas Cardoso Silva <
cardosolucas61@gmail.com> wrote:

> Great Wei! I find the suggestions really interesting. I think we can work
> with the deployment on K8s. The idea of it in Marvin would be, after
> development, the user would give some parameters and a script would
> facilitate a deployment in a kubernetes cluster, right? Regarding data
> acquisition, I think it would be great if we were able to integrate some
> third party library like Luigi. Thanks!
>
>
>
> Em qua., 22 de jul. de 2020 às 14:27, Wei Chen 
> escreveu:
>
> > Hello Lucas,
> >
> > I have some ideas:
> >
> > 1. Should we consider to use K8S or similar tools for inference container
> > scaling and management?
> > Marvin's current container management is not as powerful as some
> container
> > focus projects.
> > K8S can also be deployed into most environments now.
> >
> > 2. Is our current data cleaning stage flexible enough for multiple data
> > sources with table join?
> > Or if we should cut the data preparation stage out for the user to make
> > their own data pipeline on their data storage.
> > I figured that preprocessing might be too complex to be generalized for
> > different ML projects.
> >
> > Best Regards
> > Wei
> >
> >
> >
> > On Thu, Jul 23, 2020 at 12:26 AM Lucas Cardoso Silva <
> > cardosolucas61@gmail.com> wrote:
> >
> > > Hi guys.
> > > I would like to know if anyone else has any ideas about this evaluation
> > > phase. Both the opinion of those who have been in the community for a
> > long
> > > time and those who are still getting to know Marvin is now important
> for
> > > this step, so your suggestion or validation of the initial text is
> always
> > > welcome!
> > >
> > > Best regards,
> > > Lucas Cardoso
> > >
> > > Em sex., 10 de jul. de 2020 às 13:48, Lucas Cardoso Silva <
> > > cardosolucas61@gmail.com> escreveu:
> > >
> > > > Hello guys. The time has come for us to take the first step in
> > > > architectural assessment: the definition of the mission. Basically we
> > > have
> > > > to decide here what is important in Marvin and what is outside the
> > scope
> > > of
> > > > the project. This is important because, during this analysis and the
> > > > development process as a whole, we will be able to segment what is
> > really
> > > > important and make things more simple and functional. Also, if it
> looks
> > > > cool, we can include that on the Marvin-AI homepage.
> > > >
> > > > As stated earlier, I will post an initial draft and would like to
> > receive
> > > > your feedback to complete a few points:
> > > >
> > > > The Apache Marvin-AI platform aims to offer:
> > > >
> > > >-
> > > >
> > > >a practical and standardized solution,
> > > >-
> > > >
> > > >for the development and deployment of machine learning
> applications.
> > > >
> > > >
> > > > Aiming to offer the user:
> > > >
> > > >-
> > > >
> > > >scalability,
> > > >-
> > > >
> > > >language agnosticism,
> > > >-
> > > >
> > > >standardized pipeline (DASFE),
> > > >-
> > > >
> > > >possibility of remote versioning of artifacts.
> > > >
> > > >
> > > > Does anyone have any suggestions for more important features,
> resources
> > > or
> > > > design decisions in Marvin?
> > > >
> > > > Thank you very much,
> > > >
> > > > Lucas Cardoso
> > > >
> > >
> >
>


Re: Marvin’s mission discussion

2020-07-28 Thread Lucas Cardoso Silva
Great Wei! I find the suggestions really interesting. I think we can work
with the deployment on K8s. The idea of it in Marvin would be, after
development, the user would give some parameters and a script would
facilitate a deployment in a kubernetes cluster, right? Regarding data
acquisition, I think it would be great if we were able to integrate some
third party library like Luigi. Thanks!



Em qua., 22 de jul. de 2020 às 14:27, Wei Chen 
escreveu:

> Hello Lucas,
>
> I have some ideas:
>
> 1. Should we consider to use K8S or similar tools for inference container
> scaling and management?
> Marvin's current container management is not as powerful as some container
> focus projects.
> K8S can also be deployed into most environments now.
>
> 2. Is our current data cleaning stage flexible enough for multiple data
> sources with table join?
> Or if we should cut the data preparation stage out for the user to make
> their own data pipeline on their data storage.
> I figured that preprocessing might be too complex to be generalized for
> different ML projects.
>
> Best Regards
> Wei
>
>
>
> On Thu, Jul 23, 2020 at 12:26 AM Lucas Cardoso Silva <
> cardosolucas61@gmail.com> wrote:
>
> > Hi guys.
> > I would like to know if anyone else has any ideas about this evaluation
> > phase. Both the opinion of those who have been in the community for a
> long
> > time and those who are still getting to know Marvin is now important for
> > this step, so your suggestion or validation of the initial text is always
> > welcome!
> >
> > Best regards,
> > Lucas Cardoso
> >
> > Em sex., 10 de jul. de 2020 às 13:48, Lucas Cardoso Silva <
> > cardosolucas61@gmail.com> escreveu:
> >
> > > Hello guys. The time has come for us to take the first step in
> > > architectural assessment: the definition of the mission. Basically we
> > have
> > > to decide here what is important in Marvin and what is outside the
> scope
> > of
> > > the project. This is important because, during this analysis and the
> > > development process as a whole, we will be able to segment what is
> really
> > > important and make things more simple and functional. Also, if it looks
> > > cool, we can include that on the Marvin-AI homepage.
> > >
> > > As stated earlier, I will post an initial draft and would like to
> receive
> > > your feedback to complete a few points:
> > >
> > > The Apache Marvin-AI platform aims to offer:
> > >
> > >-
> > >
> > >a practical and standardized solution,
> > >-
> > >
> > >for the development and deployment of machine learning applications.
> > >
> > >
> > > Aiming to offer the user:
> > >
> > >-
> > >
> > >scalability,
> > >-
> > >
> > >language agnosticism,
> > >-
> > >
> > >standardized pipeline (DASFE),
> > >-
> > >
> > >possibility of remote versioning of artifacts.
> > >
> > >
> > > Does anyone have any suggestions for more important features, resources
> > or
> > > design decisions in Marvin?
> > >
> > > Thank you very much,
> > >
> > > Lucas Cardoso
> > >
> >
>