Folks, It seems to me we've forgotten to remove -Ptensorflow profile from the ~Build Apache Ignite~ suite [1] on the project build. I've removed it. Please, revert it back if I'm missing something during this discussion.
[1] https://ci.ignite.apache.org/admin/editBuildRunners.html?id=buildType:IgniteTests24Java8_BuildApacheIgnite On Wed, 25 Mar 2020 at 02:21, Denis Magda <dma...@apache.org> wrote: > > Alexey, thanks for sharing details and your reasoning behind the taken > actions. It makes sense. I've updated the machine learning pages on the new > website that will be released in several days. > > - > Denis > > > On Tue, Mar 24, 2020 at 11:07 AM Alexey Zinoviev <zaleslaw....@gmail.com> > wrote: > > > Hi, Denis! > > > > Be honest, the significant amount of the ML contirbutors left the community > > previous year in frustration with unfinished parts. > > In this situation, I reduced the unsed and broken parts according our > > previous discussions peer-to-peer (not on devlist, our mistake) to release > > the stable core of ML which could be supported with reduced power. > > > > The reasons for GA removal > > 1. It doesn't related to the ML topic > > 2. It has no intersection with the ML package (as you mentioned) > > 3. It doesn't support Ignite code and in many places Java codestyle > > 4. It was experimental package placed in ML in time of earliest experiments > > in 2017 > > 5. Nobody doesn't want to support this for the years > > > > Genetic Algorithms could be moved to Ignite-extension (if somebody > > interested in it) > > > > A lot of things are changed since release 2.7 > > > > Lessons are learnt, I will start discussion topics next time for the > > significant changes or removal in API, moreover, the next releases I hope > > to use new @IgniteExperimental (it was added too late) and another > > annotations for the release cycle. > > > > вт, 24 мар. 2020 г. в 20:00, Denis Magda <dma...@apache.org>: > > > > > Alexey, > > > > > > I missed this thread and only now realized that TensorFlow, genetic > > > algorithms and some other APIs were expelled from 2.8. I would encourage > > us > > > to start a dedicated discussion for any APIs removal or significant > > changes > > > to let other community members share their opinions or take appropriate > > > actions (like proper documentation redirects setup for pages that are > > gone > > > and updates on the website like [1] and [2]). For instance, I have no > > glue > > > that the topic of TensorFlow removal was briefly mentioned in this > > > discussion thread. > > > > > > I see the reasoning about TensorFlow but why have we removed generic > > > algorithms that had a dependency on the compute APIs only? > > > > > > [1] https://ignite.apache.org/features/tensorflow.html > > > [2] https://ignite.apache.org/features/machinelearning.html#ga-grid > > > > > > - > > > Denis > > > > > > > > > On Mon, Feb 17, 2020 at 5:39 AM Alexey Zinoviev <zaleslaw....@gmail.com> > > > wrote: > > > > > > > Ok, agree, that I should start discussion before making changes, but I > > > was > > > > limited by release 2.8 and trying don;'t be a delayed person for that. > > > > During release I was focused on fixing bugs and don't tests TF and > > Ignite > > > > together > > > > > > > > I thought that as a maintainer of ML module I could do perform these > > > > actions. > > > > > > > > Below I will share my statement why it should be removed and why it > > > should > > > > be removed immediately > > > > > > > > About TensorFlow module (reason for removal) > > > > > > > > 1. This module is only one module that uses IGFS and needs in > > > FileSystem > > > > on Ignte side due to TensorFlow API > > > > 2. This module a part of bridge between Ignite ML and Tensorflow and > > > its > > > > broken after changes in TensorFlow on TensorFlow side > > > > 3. TensorFlow released new version without Ignite bridge, no chance > > to > > > > run them together for new releases > > > > 4. This module wasn't complete and developer who did this, left the > > > > community > > > > 5. The development skills for this story require python/C++/java > > > > programming together > > > > 6. The module is a source of bugs which could be fixed for release > > 2.8 > > > > and possibly for future releases (nobody in community could this) > > > > 7. The release size reduced from 6 Gb to 4.5 due to removed > > > dependencies > > > > 8. TensorFlow now is not popular among Data Scientists, the PyTorch > > is > > > > the most popular tool for Deep Learning (like NetBeans and IDEA) > > > > 9. Nobody uses that in production because it was developed between > > 2.7 > > > > and 2.8 (2.7 has only proof-of-concept) > > > > > > > > Nikolay, sorry for that, hope to share more information about the ML > > and > > > > discuss here the main changes before actions. > > > > > > > > пн, 17 февр. 2020 г. в 16:18, Nikolay Izhikov <nizhi...@apache.org>: > > > > > > > > > Hello, Alexey. > > > > > > > > > > > The main reason, the modules are not work proper way, were > > > > experimental, > > > > > > never released as a production-ready, support old, outdated > > version, > > > > the > > > > > > external frameworks, like Tensorflow, move integration with ignite > > to > > > > the > > > > > > special repos, they are not finished, the code there is broken and > > > > > couldn't > > > > > > be fixed, because and I have no power/C++ skills/permission to > > commit > > > > > > something to them and time to support this broken modules. > > > > > > > > > > > > > > > Do we have some tickets or wider explanation for it? > > > > > It very uncommon for me that the decision to remove modules from the > > > > > master and release is not discussed widely in the community. > > > > > > > > > > > 17 февр. 2020 г., в 14:39, Ravil Galeyev <rgale...@gmail.com> > > > > > написал(а): > > > > > > > > > > > > Hi Team, > > > > > > > > > > > > First of all, let me introduce myself. I’m Ravil, I contribute to > > the > > > > ML > > > > > > module since 2018 and from time to time I make talks about it. > > (I..e > > > > data > > > > > > science summit in Warsaw [1]). > > > > > > > > > > > > So, Alexey made a huge effort to develop the ML module but he is > > not > > > > > alone. > > > > > > If you check the repo you will find other contributors. > > > > > > > > > > > > Therefore the ML module is alive and is able to run and has the > > > > roadmap. > > > > > > For me, it means that it’s not a raw project. > > > > > > > > > > > > Regarding documentation, it’d like to mention the code is the best > > > > > > documentation :) > > > > > > > > > > > > We have examples for most algorithms [2]. But if it needed I’m > > ready > > > to > > > > > > help the community with documentation in English German Polish or > > > > > Russain. > > > > > > > > > > > > > > > > > > [1] https://dssconf.pl/ > > > > > > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/ml > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Ravil > > > > > > > > > > > > > > > > > > On Mon, 17 Feb 2020 at 11:49, Alexey Zinoviev < > > > zaleslaw....@gmail.com> > > > > > > wrote: > > > > > > > > > > > >> Hello, Igniters, and you, Nikolay. > > > > > >> > > > > > >> First of all, if you have real interest to the ML module and its > > > > state, > > > > > I > > > > > >> could make call with you and explain this. > > > > > >> > > > > > >> > > > > > >> *As far as I know, for now, we have only 1 active contributor to > > > this > > > > > area > > > > > >> -Alexey Zinoviev.* > > > > > >> Currently, we have 2 active contributors, me and Ravil Galeeyev, a > > > few > > > > > >> newbies, another guys who started tensorflow and another modules > > and > > > > > >> submodules don't visit the community for many months. > > > > > >> > > > > > >> *Is ML module production ready?* > > > > > >> This release will be the first release, than ML is production > > ready > > > > and > > > > > it > > > > > >> totally my work. > > > > > >> > > > > > >> > > > > > >> *Can someone related to the ML, please, give some examples of the > > > CVE > > > > > >> orissues that can be fixed only with removing a bunch of modules?* > > > > > >> CVE is not the main reason to remove the "bunch of modules", but > > > part > > > > of > > > > > >> the story. > > > > > >> The main reason, the modules are not work proper way, were > > > > experimental, > > > > > >> never released as a production-ready, support old, outdated > > version, > > > > the > > > > > >> external frameworks, like Tensorflow, move integration with ignite > > > to > > > > > the > > > > > >> special repos, they are not finished, the code there is broken and > > > > > couldn't > > > > > >> be fixed, because and I have no power/C++ skills/permission to > > > commit > > > > > >> something to them and time to support this broken modules. > > > > > >> > > > > > >> Also broken TF module blocks the removal of IGFS. > > > > > >> > > > > > >> Found CVE were related to the dependencies related to > > > > hadoop/tf/parquet > > > > > and > > > > > >> so on. > > > > > >> > > > > > >> *Should we mark it with the @IgniteExperimental? * > > > > > >> I don't know, we have no this RAW annotation a few weeks ago and I > > > > don't > > > > > >> know how we should use it. > > > > > >> It could be, if you finish the discussion about this annotation > > and > > > > the > > > > > >> write docs about it and share it with me. > > > > > >> > > > > > >> > > > > > >> * As far as I know, the ML module has no documentation. Is it > > > correct? > > > > > Dowe > > > > > >> have plans to fix it?* > > > > > >> The ML docs are here, on our Ignite documentation > > > > > >> https://apacheignite.readme.io/docs/machine-learning > > > > > >> of course, something could be wrong, 1.5 year we are not released > > > > Ignite > > > > > >> Yes, I have plan to fix. Of course, after fixing all bugs in > > release > > > > > branch > > > > > >> > > > > > >> *Should we move it to the ignite-extensions?* > > > > > >> No we shouldn't, I don't want this, and have a lot of arguments > > and > > > > > >> currently is not the time for discussion about this (they are too > > > > young > > > > > and > > > > > >> have now real infrastructure and release-cycle) > > > > > >> > > > > > >> P.S. Community, I understand that the removal of module looks > > > strange, > > > > > but > > > > > >> we could understand that ML was a strange experiment without > > roadmap > > > > and > > > > > >> this situation is finished. > > > > > >> Now, I have roadmap (will be published later), newbie tickets, the > > > > > ability > > > > > >> to prepare correct docs, the understanding what could be used by > > > > clients > > > > > >> and first of all, production-ready ML (it could be run on > > > > > Ignite-cluster, > > > > > >> really, it works.) > > > > > >> > > > > > >> If you, Igniters, believe, that I could be a good maintainer for > > ML > > > > > module, > > > > > >> please support me here in this thread > > > > > >> If you think, that I do something wrong, OK, please write it too, > > > I'll > > > > > read > > > > > >> carefully. > > > > > >> > > > > > >> I spent a few months to fix bugs in components, which were > > abandoned > > > > by > > > > > >> their creators. > > > > > >> > > > > > >> My goal: Ignite should have light-weight, easy-integrated ML > > without > > > > > >> strange and unfinished experiments which could be not maintained. > > > > It's a > > > > > >> part of common movement in Ignite (removal of modules or moving > > them > > > > to > > > > > the > > > > > >> separate repos). > > > > > >> > > > > > >> > > > > > >> пн, 17 февр. 2020 г. в 12:10, Nikolay Izhikov < > > nizhi...@apache.org > > > >: > > > > > >> > > > > > >>> Hello, Igniters. > > > > > >>> > > > > > >>> Can someone bring some light on the state of the ML module in > > > Ignite? > > > > > >>> As far as I know, for now, we have only 1 active contributor to > > > this > > > > > >> area - > > > > > >>> Alexey Zinoviev. > > > > > >>> I see how whole modules come and go from the module - [1] > > > > > >>> > > > > > >>> Please, also note this quote: > > > > > >>> > > > > > >>>> Also as a result of good testing from both side (from me and > > > Stepan) > > > > > we > > > > > >>>> found a lot of bugs and CVEs in hadoop related components that > > > > should > > > > > >> be > > > > > >>>> removed in release branch too. > > > > > >>> > > > > > >>> 0. As far as I know, the ML module has no documentation. Is it > > > > correct? > > > > > >> Do > > > > > >>> we have plans to fix it? > > > > > >>> > > > > > >>> 1. Can someone related to the ML, please, give some examples of > > the > > > > CVE > > > > > >> or > > > > > >>> issues that can be fixed only with removing a bunch of modules? > > > > > >>> > > > > > >>> 3. Is ML module production ready? > > > > > >>> > > > > > >>> 4. Should we mark it with the @IgniteExperimental? > > > > > >>> > > > > > >>> 5. Should we move it to the ignite-extensions? > > > > > >>> > > > > > >>> [1] > > > > > >>> > > > > > >>> > > > > > >> > > > > > > > > > > > > > > https://github.com/apache/ignite/commit/53e886b8ed38a6842cef8b44ace6851855dfad29 > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > > > > >
