I'd like this approach, can't wait to review your PR!
пт, 31 мая 2019 г. в 15:33, Алексей Платонов <aplaton...@gmail.com>:
> No, I won't change them.
> In context of this architecture we should pass learning environment to
> ComputeUtils and this concerns just DatasetBuilders and ComputeUtils APIs.
> APIs of Preprocessors and Trainers won't be changed. We just set
> DeploymentContext in fit method but it doens't require to change APIs of
> these classes.
>
> Sincerely
> Alexey Platonov
>
> пт, 31 мая 2019 г. в 14:51, Alexey Zinoviev <zaleslaw....@gmail.com>:
>
> > It sounds great, could you please explain what are you going to change in
> > the main Vectorizer/Preprocessor API to support it?
> >
> > пт, 31 мая 2019 г. в 14:20, Алексей Платонов <aplaton...@gmail.com>:
> >
> > > Hi, Igniters!
> > > Currently we don't have an ability to deploy automatically user-defined
> > > preprocessors and vectorizers. Client's code should be deployed
> manually
> > to
> > > Ignite server nodes.
> > >
> > > I have an idea how to fix it. If we pass user's classloader and one of
> > > user-defined classes from fit-level to
> > > ComputeUtils.affinityCallWithRetries() then we wiil be able to use
> > > GridPeerDeployAware interface to send informtation about this
> classloader
> > > to server nodes.
> > >
> > > To support this ability we can define interfaces like these:
> > >
> > > public interface DeployableObject {
> > > public List<Object> getDependencies();
> > > }
> > >
> > > and
> > >
> > > public interface DeployingContext {
> > > public Class<?> userClass();
> > > public ClassLoader clientClassLoader();
> > > }
> > >
> > > DeployableObject will be mark for our ignite-ml final classes like
> > trainers
> > > or concrete preprocessors and it can be able to return all dependencies
> > > that should be deployed to server nodes if it's needed. If these
> > > dependencies are DeployableObjects too then depenndencies will be
> > unfolded
> > > recursively. Classes that isn't defined as DeployableObject will be
> > > recognized as user-defined (NOTE: all leaf classes in our hierarchy
> will
> > be
> > > DeployableObject).
> > >
> > > This list of DeployableObjects will be user for define user class
> loader
> > > and one of these objects will be used for passing to
> GridPeerDeployAware.
> > >
> > > So, this logic allows to pass user-defined Preprocessors and
> Vectorizers
> > to
> > > training algorithms and pipelines.
> > >
> > > What do you think?
> > >
> > > Sincerely
> > > Alexey Platonov
> > >
> >
>
