Hi Katherin, we are also working in a similar direction. We implemented a prototype to integrate with SystemML: https://github.com/apache/incubator-systemml/pull/119 SystemML provides many different matrix formats, operations, GPU support and a couple of DL algorithms. Unfortunately, we realized that the lack of a caching operator and a broadcast issue highly effects the performance (e.g. compared to Spark). At the moment I am trying to tackle the broadcast issue. But caching is still a problem for us.
Best regards, Felix 2017-02-07 16:22 GMT+01:00 Katherin Eri <katherinm...@gmail.com>: > Thank you, Till. > > 1) Regarding ND4J, I didn’t know about such a pity and critical > restriction of it -> lack of sparsity optimizations, and you are right: > this issue is still actual for them. I saw that Flink uses Breeze, but I > thought its usage caused by some historical reasons. > > 2) Regarding integration with DL4J, I have read the source code of > DL4J/Spark integration, that’s why I have declined my idea of reuse of > their word2vec implementation for now, for example. I can perform deeper > investigation of this topic, if it required. > > > > So I feel that we have the following picture: > > 1) DL integration investigation, could be part of Apache Bahir. I can > perform futher investigation of this topic, but I thik we need some > separated ticket for this to track this activity. > > 2) GPU support, required for DL is interesting, but requires ND4J for > example. > > 3) ND4J couldn’t be incorporated because it doesn’t support sparsity > <https://deeplearning4j.org/roadmap.html> [1]. > > Regarding ND4J is this the single blocker for incorporation of it or may be > some others known? > > > [1] https://deeplearning4j.org/roadmap.html > > вт, 7 февр. 2017 г. в 16:26, Till Rohrmann <trohrm...@apache.org>: > > Thanks for initiating this discussion Katherin. I think you're right that > in general it does not make sense to reinvent the wheel over and over > again. Especially if you only have limited resources at hand. So if we > could integrate Flink with some existing library that would be great. > > In the past, however, we couldn't find a good library which provided enough > freedom to integrate it with Flink. Especially if you want to have > distributed and somewhat high-performance implementations of ML algorithms > you would have to take Flink's execution model (capabilities as well as > limitations) into account. That is mainly the reason why we started > implementing some of the algorithms "natively" on Flink. > > If I remember correctly, then the problem with ND4J was and still is that > it does not support sparse matrices which was a requirement from our side. > As far as I know, it is quite common that you have sparse data structures > when dealing with large scale problems. That's why we built our own > abstraction which can have different implementations. Currently, the > default implementation uses Breeze. > > I think the support for GPU based operations and the actual resource > management are two orthogonal things. The implementation would have to work > with no GPUs available anyway. If the system detects that GPUs are > available, then ideally it would exploit them. Thus, we could add this > feature later and maybe integrate it with FLINK-5131 [1]. > > Concerning the integration with DL4J I think that Theo's proposal to do it > in a separate repository (maybe as part of Apache Bahir) is a good idea. > We're currently thinking about outsourcing some of Flink's libraries into > sub projects. This could also be an option for the DL4J integration then. > In general I think it should be feasible to run DL4J on Flink given that it > also runs on Spark. Have you already looked at it closer? > > [1] https://issues.apache.org/jira/browse/FLINK-5131 > > Cheers, > Till > > On Tue, Feb 7, 2017 at 11:47 AM, Katherin Eri <katherinm...@gmail.com> > wrote: > > > Thank you Theodore, for your reply. > > > > 1) Regarding GPU, your point is clear and I agree with it, ND4J looks > > appropriate. But, my current understanding is that, we also need to cover > > some resource management questions -> when we need to provide GPU support > > we also need to manage it like resource. For example, Mesos has already > > supported GPU like resource item: Initial support for GPU resources. > > <https://issues.apache.org/jira/browse/MESOS-4424?jql=text%20~%20GPU> > > Flink > > uses Mesos as cluster manager, and this means that this feature of Mesos > > could be reused. Also memory managing questions in Flink regarding GPU > > should be clarified. > > > > 2) Regarding integration with DL4J: what stops us to initialize ticket > > and start the discussion around this topic? We need some user story or > the > > community is not sure that DL is really helpful? Why the discussion with > > Adam > > Gibson just finished with no implementation of any idea? What concerns do > > we have? > > > > пн, 6 февр. 2017 г. в 15:01, Theodore Vasiloudis < > > theodoros.vasilou...@gmail.com>: > > > > > Hell all, > > > > > > This is point that has come up in the past: Given the multitude of ML > > > libraries out there, should we have native implementations in FlinkML > or > > > try to integrate other libraries instead? > > > > > > We haven't managed to reach a consensus on this before. My opinion is > > that > > > there is definitely value in having ML algorithms written natively in > > > Flink, both for performance optimization, > > > but more importantly for engineering simplicity, we don't want to force > > > users to use yet another piece of software to run their ML algos (at > > least > > > for a basic set of algorithms). > > > > > > We have in the past discussed integrations with DL4J (particularly > ND4J) > > > with Adam Gibson, the core developer of the library, but we never got > > > around to implementing anything. > > > > > > Whether it makes sense to have an integration with DL4J as part of the > > > Flink distribution would be up for discussion. I would suggest to make > it > > > an independent repo to allow for > > > faster dev/release cycles, and because it wouldn't be directly related > to > > > the core of Flink so it would add extra reviewing burden to an already > > > overloaded group of committers. > > > > > > Natively supporting GPU calculations in Flink would be much better > > achieved > > > through a library like ND4J, the engineering burden would be too much > > > otherwise. > > > > > > Regards, > > > Theodore > > > > > > On Mon, Feb 6, 2017 at 11:26 AM, Katherin Eri <katherinm...@gmail.com> > > > wrote: > > > > > > > Hello, guys. > > > > > > > > Theodore, last week I started the review of the PR: > > > > https://github.com/apache/flink/pull/2735 related to *word2Vec for > > > Flink*. > > > > > > > > > > > > > > > > During this review I have asked myself: why do we need to implement > > such > > > a > > > > very popular algorithm like *word2vec one more time*, when there is > > > already > > > > available implementation in java provided by deeplearning4j.org > > > > <https://deeplearning4j.org/word2vec> library (DL4J -> Apache 2 > > > licence). > > > > This library tries to promote itself, there is a hype around it in ML > > > > sphere, and it was integrated with Apache Spark, to provide scalable > > > > deeplearning calculations. > > > > > > > > > > > > *That's why I thought: could we integrate with this library or not > also > > > and > > > > Flink? * > > > > > > > > 1) Personally I think, providing support and deployment of > > > > *Deeplearning(DL) > > > > algorithms/models in Flink* is promising and attractive feature, > > because: > > > > > > > > a) during last two years DL proved its efficiency and these > > > algorithms > > > > used in many applications. For example *Spotify *uses DL based > > algorithms > > > > for music content extraction: Recommending music on Spotify with deep > > > > learning AUGUST 05, 2014 > > > > <http://benanne.github.io/2014/08/05/spotify-cnns.html> for their > > music > > > > recommendations. Developers need to scale up DL manually, that causes > a > > > lot > > > > of work, so that’s why such platforms like Flink should support these > > > > models deployment. > > > > > > > > b) Here is presented the scope of Deeplearning usage cases > > > > <https://deeplearning4j.org/use_cases>, so many of this scenarios > > > related > > > > to scenarios, that could be supported on Flink. > > > > > > > > > > > > 2) But DL uncover such questions like: > > > > > > > > a) scale up calculations over machines > > > > > > > > b) perform these calculations both over CPU and GPU. GPU is > > required > > > to > > > > train big DL models, otherwise learning process could have very slow > > > > convergence. > > > > > > > > > > > > 3) I have checked this DL4J library, which already have reach support > > of > > > > many attractive DL models like: Recurrent Networks and LSTMs, > > > Convolutional > > > > Networks (CNN), Restricted Boltzmann Machines (RBM) and others. So we > > > won’t > > > > need to implement them independently, but only provide the ability of > > > > execution of this models over Flink cluster, the quite similar way > like > > > it > > > > was integrated with Apache Spark. > > > > > > > > > > > > Because of all of this I propose: > > > > > > > > 1) To create new ticket in Flink’s JIRA for integration of Flink > > with > > > > DL4J and decide on which side this integration should be implemented. > > > > > > > > 2) Support natively GPU resources in Flink and allow calculations > > over > > > > them, like that is described in this publication > > > > https://www.oreilly.com/learning/accelerating-spark- > > workloads-using-gpus > > > > > > > > > > > > > > > > *Regarding original issue Implement Word2Vec > > > > <https://issues.apache.org/jira/browse/FLINK-2094>in Flink, *I have > > > > investigated its implementation in DL4J and that implementation of > > > > integration DL4J with Apache Spark, and got several points: > > > > > > > > It seems that idea of building of our own implementation of word2vec > in > > > > Flink not such a bad solution, because: This DL4J was forced to > > > reimplement > > > > its original word2Vec over Spark. I have checked the integration of > > DL4J > > > > with Spark, and found that it is too strongly coupled with Spark API, > > so > > > > that it is impossible just to take some DL4J API and reuse it, > instead > > we > > > > need to implement independent integration for Flink. > > > > > > > > *That’s why we simply finish implementation of current PR > > > > **independently **from > > > > integration to DL4J.* > > > > > > > > > > > > > > > > Could you please provide your opinion regarding my questions and > > points, > > > > what do you think about them? > > > > > > > > > > > > > > > > пн, 6 февр. 2017 г. в 12:51, Katherin Eri <katherinm...@gmail.com>: > > > > > > > > > Sorry, guys I need to finish this letter first. > > > > > Full version of it will come shortly. > > > > > > > > > > пн, 6 февр. 2017 г. в 12:49, Katherin Eri <katherinm...@gmail.com > >: > > > > > > > > > > Hello, guys. > > > > > Theodore, last week I started the review of the PR: > > > > > https://github.com/apache/flink/pull/2735 related to *word2Vec for > > > > Flink*. > > > > > > > > > > During this review I have asked myself: why do we need to implement > > > such > > > > a > > > > > very popular algorithm like *word2vec one more time*, when there is > > > > > already availabe implementation in java provided by > > deeplearning4j.org > > > > > <https://deeplearning4j.org/word2vec> library (DL4J -> Apache 2 > > > > licence). > > > > > This library tries to promote it self, there is a hype around it in > > ML > > > > > sphere, and it was integrated with Apache Spark, to provide > scalable > > > > > deeplearning calculations. > > > > > That's why I thought: could we integrate with this library or not > > also > > > > and > > > > > Flink? > > > > > 1) Personally I think, providing support and deployment of > > Deeplearning > > > > > algorithms/models in Flink is promising and attractive feature, > > > because: > > > > > a) during last two years deeplearning proved its efficiency and > > > this > > > > > algorithms used in many applications. For example *Spotify *uses DL > > > based > > > > > algorithms for music content extraction: Recommending music on > > Spotify > > > > > with deep learning AUGUST 05, 2014 > > > > > <http://benanne.github.io/2014/08/05/spotify-cnns.html> for their > > > music > > > > > recommendations. Doing this natively scalable is very attractive. > > > > > > > > > > > > > > > I have investigated that implementation of integration DL4J with > > Apache > > > > > Spark, and got several points: > > > > > > > > > > 1) It seems that idea of building of our own implementation of > > word2vec > > > > > not such a bad solution, because the integration of DL4J with Spark > > is > > > > too > > > > > strongly coupled with Saprk API and it will take time from the side > > of > > > > DL4J > > > > > to adopt this integration to Flink. Also I have expected that we > will > > > be > > > > > able to call just some API, it is not such thing. > > > > > 2) > > > > > > > > > > https://deeplearning4j.org/use_cases > > > > > https://www.analyticsvidhya.com/blog/2017/01/t-sne- > > > > implementation-r-python/ > > > > > > > > > > > > > > > чт, 19 янв. 2017 г. в 13:29, Till Rohrmann <trohrm...@apache.org>: > > > > > > > > > > Hi Katherin, > > > > > > > > > > welcome to the Flink community. Always great to see new people > > joining > > > > the > > > > > community :-) > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Tue, Jan 17, 2017 at 1:02 PM, Katherin Sotenko < > > > > katherinm...@gmail.com> > > > > > wrote: > > > > > > > > > > > ok, I've got it. > > > > > > I will take a look at https://github.com/apache/flink/pull/2735 > . > > > > > > > > > > > > вт, 17 янв. 2017 г. в 14:36, Theodore Vasiloudis < > > > > > > theodoros.vasilou...@gmail.com>: > > > > > > > > > > > > > Hello Katherin, > > > > > > > > > > > > > > Welcome to the Flink community! > > > > > > > > > > > > > > The ML component definitely needs a lot of work you are > correct, > > we > > > > are > > > > > > > facing similar problems to CEP, which we'll hopefully resolve > > with > > > > the > > > > > > > restructuring Stephan has mentioned in that thread. > > > > > > > > > > > > > > If you'd like to help out with PRs we have many open, one I > have > > > > > started > > > > > > > reviewing but got side-tracked is the Word2Vec one [1]. > > > > > > > > > > > > > > Best, > > > > > > > Theodore > > > > > > > > > > > > > > [1] https://github.com/apache/flink/pull/2735 > > > > > > > > > > > > > > On Tue, Jan 17, 2017 at 12:17 PM, Fabian Hueske < > > fhue...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Katherin, > > > > > > > > > > > > > > > > welcome to the Flink community! > > > > > > > > Help with reviewing PRs is always very welcome and a great > way > > to > > > > > > > > contribute. > > > > > > > > > > > > > > > > Best, Fabian > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-01-17 11:17 GMT+01:00 Katherin Sotenko < > > > > katherinm...@gmail.com > > > > > >: > > > > > > > > > > > > > > > > > Thank you, Timo. > > > > > > > > > I have started the analysis of the topic. > > > > > > > > > And if it necessary, I will try to perform the review of > > other > > > > > pulls) > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 17 янв. 2017 г. в 13:09, Timo Walther < > > twal...@apache.org > > > >: > > > > > > > > > > > > > > > > > > > Hi Katherin, > > > > > > > > > > > > > > > > > > > > great to hear that you would like to contribute! Welcome! > > > > > > > > > > > > > > > > > > > > I gave you contributor permissions. You can now assign > > issues > > > > to > > > > > > > > > > yourself. I assigned FLINK-1750 to you. > > > > > > > > > > Right now there are many open ML pull requests, you are > > very > > > > > > welcome > > > > > > > to > > > > > > > > > > review the code of others, too. > > > > > > > > > > > > > > > > > > > > Timo > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Am 17/01/17 um 10:39 schrieb Katherin Sotenko: > > > > > > > > > > > Hello, All! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm Kate Eri, I'm java developer with 6-year enterprise > > > > > > experience, > > > > > > > > > also > > > > > > > > > > I > > > > > > > > > > > have some expertise with scala (half of the year). > > > > > > > > > > > > > > > > > > > > > > Last 2 years I have participated in several BigData > > > projects > > > > > that > > > > > > > > were > > > > > > > > > > > related to Machine Learning (Time series analysis, > > > > Recommender > > > > > > > > systems, > > > > > > > > > > > Social networking) and ETL. I have experience with > > Hadoop, > > > > > Apache > > > > > > > > Spark > > > > > > > > > > and > > > > > > > > > > > Hive. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I’m fond of ML topic, and I see that Flink project > > requires > > > > > some > > > > > > > work > > > > > > > > > in > > > > > > > > > > > this area, so that’s why I would like to join Flink and > > ask > > > > me > > > > > to > > > > > > > > grant > > > > > > > > > > the > > > > > > > > > > > assignment of the ticket > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1750 > > > > > > > > > > > to me. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
