I think the focus of this discussion should be how we proceed not what to
do. The what comes from the committers anyway.
There are several people who like to commit, including people from the
Streamline project. Having pull requests that are older than 6 Month is not
good for any project.
The main question is how can we develop the library further with high
standards but without creating a bottleneck that holds things back to much.
In my opinion it would be best if we find enough resources to keep things
inside Flink. However if we have to depend on people who are
already stretched for time, splitting it out might be the better option.
(path 1 from Theos original mail)
cheers Martin
On Fri, Feb 12, 2016 at 3:54 PM, Suneel Marthi wrote:
> On Fri, Feb 12, 2016 at 9:40 AM, Simone Robutti <
> simone.robu...@radicalbit.io> wrote:
>
> > @Suneel
> >
> > 1) Totally agree, as I wrote before.
> >
> > 2)I agree that support for PMML is premature but we shouldn't
> underestimate
> > the variety and complexity of the uses of ML models in the industry. The
> > adoption of Flink, hopefully, will grow and reach less innovative
> realities
> > where Random Forests and SVMs are still the main algorithms in use. In
> > these same realities there are legacies that justify the use of PMML to
> > port models. Still, FlinkML is still in an early stage so as you said, it
> > doesn't make sense to spend time right now on such a feature.
> >
>
> +1, as I mentioned earlier the PMML spec only supports classification and
> clustering (I last checked this in Aug 2015, pretty sure it would not have
> changed since then); hence 'Yes' it has some limited uses; 'No' - its too
> premature to even talk about it given the present state of FlinkML.
>
> >
> > 3)This would be really interesting. How do you imagine that the
> integration
> > with a distributed processing engine would work?
> >
>
> I am not sure yet, we r still exploring this on Mahout project to add to
> Mahout-Samsara - most of the statistics and probabilistic modeling would
> then be supported by Figaro (Bayesian, MCMC etc) and hence can be external
> to FlinkML.
>
> Figaro is Scala based. See https://github.com/p2t2/figaro
>
> I believe there are few other similar DSLs out there, need to dig up my old
> emails.
>
> (Not sure if its ASLv2 License, need verification here)
>
>
> >
> > 5) Agree on this one too. To my knowledge it would be the best option
> > together with SAMOA (for the streaming part).
> >
>
> There's already Flink - Samoa integration in place IIRC.
>
>
> >
> > 2016-02-12 15:25 GMT+01:00 Suneel Marthi :
> >
> > > My 2 cents as someone who's done ML over the years - having worked on
> > Oryx
> > > 2.0 and Mahout and having used Spark MlLib (read as "had no choice due
> to
> > > strict workplace enforcement") and understands well their limitations.
> > >
> > > 1. FlinkML in its present form seems like "do it like how Spark did
> it".
> > >
> > > 2. The recent discussion about PMML support in Flink to my mind is a
> > clear
> > > example of putting the cart before the horse. Why are we even talking
> > PMML
> > > when there ain't much ML algos in FlinkML?
> > >
> > > For a real good implementation of PMML and how its being used (with
> > jPMML),
> > > suggest look at the Oryx 2.0 project. The PMML implementation in Oryx
> 2.0
> > > predates Spark and is a clean example of separating PMML from the
> > > underlying framework (Spark or Flink).
> > >
> > > We have had PMML discussions on the Mahout project in the past, but the
> > > idea never gained any traction in large part due to PMML spec
> limitations
> > > (mostly for clustering and classification algorithms) and the lack of
> > > adoption within the community.
> > >
> > > See the discussion here and specifically Ted Dunning's comment on PMML
> -
> > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/mahout-dev/201503.mbox/%3CCAJwFCa1%3DAw%2B3G54FgkYdTH%3DoNQBRqfeU-SS19iCFKMWbAfWzOQ%40mail.gmail.com%3E
> > >
> > > Most of the ML in practice (deployed in production) today are
> > Recommenders
> > > and Deep Learning - both of which are not supported by the PMML spec.
> > >
> > > 3. Leveraging a probabilistic programming language like Figaro might
> be a
> > > good way to go (just my thought) - that way most of the ML groundwork
> > would
> > > be external to Flink.
> > >
> > > 4. Within the Mahout community, we had been talking (and are working)
> on
> > > redoing the Samsara Distributed linear algebra framework to support
> Flink
> > > (in large part we realized that Flink is a better platform than the
> more
> > > popular one out there that Slim wouldn't wanna talk about :) ).
> > >
> > > We should be having a release out in the next few weeks (depending on
> > > committers' availability). It would be great if FlinkML had something
> > like
> > > it.
> > >
> > > There was a good audience to Sebastian's talk on this subject at #FF15
> in
> > > October.
> > >
> > > 5. Its a good idea to add Flink su
