In order to make this roadmap more concrete, I created the following epics for the target release 1.0 with about 50 subtasks, and linked related existing issues. Given the discussion on a short release cycle, the bare minimum would be SYSTEMML-1299 (which includes all changes that affect the external behavior), and a subset of SYSTEMML-1308 (especially features that address proper cleanups and robustness against OOMs).
SYSTEMML-1299 Language feature updates SYSTEMML-1321 Compiler feature extensions SYSTEMML-1308 Runtime feature extensions SYSTEMML-1284 Code generation for operator fusion SYSTEMML-1328 Perftest extensions I did not touch GPUs, Deep Learning, DSLs, and algorithms yet. So please have a look, and update or create them if necessary. Regards, Matthias On Mon, Jan 16, 2017 at 8:14 PM, <dusenberr...@gmail.com> wrote: > Yeah using the target release would be good. Actually, with that in mind, > I believe that we have been marking closed issues since the 0.11 release as > targeting an upcoming "1.0" release, but it would probably be more correct > to update those to "0.12" since we decided to release 0.12. In addition, we > should set the target of the Spark 2.x support issue to "0.13". > > As for the roadmap, it would be good to update the website with a > high-level overview, with links to associated JIRA issues. > > -- > > Mike Dusenberry > GitHub: github.com/dusenberrymw > LinkedIn: linkedin.com/in/mikedusenberry > > Sent from my iPhone. > > > > On Jan 16, 2017, at 7:35 PM, Luciano Resende <luckbr1...@gmail.com> > wrote: > > > > Instead of Epic, we could use the target release ? Also, we have a > roadmap > > page on the site and we should keep that up to date, or get rid of that > and > > use roadmap on jira. > > > >> On Mon, Jan 16, 2017 at 6:20 PM <dusenberr...@gmail.com> wrote: > >> > >> Now that we've had some discussion here, it would be good to transfer > this > >> discussion into a JIRA epic, containing sub tasks. That way, we can > >> properly track our progress on these items and facilitate contributions > >> from the community. Note that some of the sub tasks may already exist > as > >> individual issues. > >> > >> > >> > >> Would anyone in the community like to volunteer for creating these > issues? > >> > >> > >> > >> - Mike > >> > >> > >> > >> -- > >> > >> > >> > >> Mike Dusenberry > >> > >> GitHub: github.com/dusenberrymw > >> > >> LinkedIn: linkedin.com/in/mikedusenberry > >> > >> > >> > >> Sent from my iPhone. > >> > >> > >> > >> > >> > >>>> On Jan 4, 2017, at 6:00 PM, dusenberr...@gmail.com wrote: > >>> > >>> > >> > >>> Overall, this is a good list of items that should be worked on, > >> particularly because it contains several user-facing items. However, to > >> echo what Luciano said, I'm also concerned about the timeline. At this > >> stage, I agree that we need to release more often, and with a more > >> user-oriented "product" focus as a guide for timelines. I.e. we should > >> orient our release timelines around items that focus on the "product" of > >> allowing the user to work on a wide range of ML problems in a simple and > >> easy manner on top of Spark. > >> > >>> > >> > >>> With that in mind, I agree that a focus on a subset of (1) and (2) > would > >> be good for an immediate release, with a particular focus on Spark 2.0 > >> support as a priority. > >> > >>> > >> > >>> How about we aim for a February 1st release date for the initial items? > >> > >>> > >> > >>> -Mike > >> > >>> > >> > >>> -- > >> > >>> > >> > >>> Mike Dusenberry > >> > >>> GitHub: github.com/dusenberrymw > >> > >>> LinkedIn: linkedin.com/in/mikedusenberry > >> > >>> > >> > >>> Sent from my iPhone. > >> > >>> > >> > >>> > >> > >>>> On Jan 3, 2017, at 4:17 PM, Niketan Pansare <npan...@us.ibm.com> > wrote: > >> > >>>> > >> > >>>> Hi Matthias, > >> > >>>> > >> > >>>> Thanks for the detailed roadmap. > >> > >>>> > >> > >>>> +1 for all the items with few modifications. > >> > >>>> > >> > >>>> 1) APIs and Language: > >> > >>>> * Cleanup new MLContext (matrix/frame data types, move tests, etc) > >> > >>>>>> Ensure Python and Scala MLContext have same API capability. > >> > >>>> > >> > >>>> * Remove old MLContext > >> > >>>> * Consolidate MLContext and JMLC > >> > >>>> * Full support for Scala/Python DSLs > >> > >>>>>> +1 for Python DSL except for push-down of loop structures and > >> functions. > >> > >>>> > >> > >>>> * Remove old file-based transform > >> > >>>> * Scala/Python wrappers for all existing algorithms > >> > >>>> * Data converters (additional formats: e.g., libsvm; performance) > >> > >>>> > >> > >>>> 2) Updated Dependencies: > >> > >>>> * Spark 2.0 support > >> > >>>> * Matrix block library (isolated jar) > >> > >>>> > >> > >>>> 3) Compiler/Runtime Features: > >> > >>>> * GPU support (full compiler and runtime support) > >> > >>>>>> Can we break this down into phases: > >> https://issues.apache.org/jira/browse/SYSTEMML-445 ? We can discuss the > >> timeline of the phases in the JIRA. > >> > >>>> > >> > >>>> * Compressed linear algebra v2 > >> > >>>> * Code generation (automatic operator fusion) > >> > >>>> * Extended parfor (full spark exploitation, micro-batch support) > >> > >>>> * Scale-up architecture (large dense blocks, numa)? > >> > >>>> > >> > >>>> 4) Tools > >> > >>>> * Extended stats (task locality, shuffle, etc) > >> > >>>> * Cloud resource advisor (extended resource optimizer)? > >> > >>>> > >> > >>>> 5) Algorithms > >> > >>>> * Graduate "staging" algorithms (robustness/performance) > >> > >>>> * Perftest: include all algorithms into automated performance tests > >> > >>>>>> via spark-submit + via Scala/Python wrappers > >> > >>>> > >> > >>>> * Simplify usage decision trees, random forest, mlogreg, msvm > >> > >>>> (preprocessing, label representation, etc) > >> > >>>>>> + command-line variable naming. For example: maxi, maxiter, etc. > >> > >>>> > >> > >>>> Thanks, > >> > >>>> > >> > >>>> Niketan Pansare > >> > >>>> IBM Almaden Research Center > >> > >>>> E-mail: npansar At us.ibm.com > >> > >>>> http://researcher.watson.ibm.com/researcher/view.php? > person=us-npansar > >> > >>>> > >> > >>>> Matthias Boehm ---01/03/2017 02:44:39 PM---Yes indeed, most of (3) and > >> (4) can be done incrementally. For (5), some of the changes might also > >> > >>>> > >> > >>>> From: Matthias Boehm <mboe...@googlemail.com> > >> > >>>> To: dev@systemml.incubator.apache.org > >> > >>>> Date: 01/03/2017 02:44 PM > >> > >>>> Subject: Re: [DISCUSS] Roadmap SystemML 1.0 > >> > >>>> > >> > >>>> > >> > >>>> > >> > >>>> > >> > >>>> Yes indeed, most of (3) and (4) can be done incrementally. For (5), > some > >> > >>>> of the changes might also modify the signature of algorithms (i.e., > >> > >>>> parameters and required input data) but it would help, for example > with > >> > >>>> decision trees, as users no longer need to dummy code their inputs. > >> > >>>> > >> > >>>> Generally, I'm fine with making (3), (4), and part of (5) optional and > >> > >>>> let the "must-have" features from (1) and (2) determine the timeline. > >> > >>>> > >> > >>>> Regards, > >> > >>>> Matthias > >> > >>>> > >> > >>>> On 1/3/2017 11:27 PM, Luciano Resende wrote: > >> > >>>>> On Tue, Jan 3, 2017 at 11:50 AM, Matthias Boehm < > >> mboe...@googlemail.com> > >> > >>>>> wrote: > >> > >>>>> > >> > >>>>>> I'd like to initiate the discussion of a concrete roadmap for our > >> next > >> > >>>>>> release. According, to previous discussions, I'd think it's fair to > >> say > >> > >>>>>> that we agree on calling it SystemML 1.0. We should carefully plan > >> this > >> > >>>>>> release as it's an opportunity to change APIs and remove some older > >> > >>>>>> deprecated features. I'd like to encourage not just developers but > >> also the > >> > >>>>>> broader community to participate in this discussion. > >> > >>>>>> > >> > >>>>>> Personally, I think a target date of Q2/2017 is realistic. Let's > >> start > >> > >>>>>> with collecting the major features and changes that potentially > >> affect > >> > >>>>>> users. Here is an initial list, but please feel free to add and up- > >> or > >> > >>>>>> down-vote the individual items. > >> > >>>>>> > >> > >>>>>> 1) APIs and Language: > >> > >>>>>> * Cleanup new MLContext (matrix/frame data types, move tests, etc) > >> > >>>>>> * Remove old MLContext > >> > >>>>>> * Consolidate MLContext and JMLC > >> > >>>>>> * Full support for Scala/Python DSLs > >> > >>>>>> * Remove old file-based transform > >> > >>>>>> * Scala/Python wrappers for all existing algorithms > >> > >>>>>> * Data converters (additional formats: e.g., libsvm; performance) > >> > >>>>>> > >> > >>>>>> 2) Updated Dependencies: > >> > >>>>>> * Spark 2.0 support > >> > >>>>>> * Matrix block library (isolated jar) > >> > >>>>>> > >> > >>>>>> 3) Compiler/Runtime Features: > >> > >>>>>> * GPU support (full compiler and runtime support) > >> > >>>>>> * Compressed linear algebra v2 > >> > >>>>>> * Code generation (automatic operator fusion) > >> > >>>>>> * Extended parfor (full spark exploitation, micro-batch support) > >> > >>>>>> * Scale-up architecture (large dense blocks, numa)? > >> > >>>>>> > >> > >>>>>> 4) Tools > >> > >>>>>> * Extended stats (task locality, shuffle, etc) > >> > >>>>>> * Cloud resource advisor (extended resource optimizer)? > >> > >>>>>> > >> > >>>>>> 5) Algorithms > >> > >>>>>> * Graduate "staging" algorithms (robustness/performance) > >> > >>>>>> * Perftest: include all algorithms into automated performance tests > >> > >>>>>> * Simplify usage decision trees, random forest, mlogreg, msvm > >> > >>>>>> (preprocessing, label representation, etc) > >> > >>>>>> > >> > >>>>>> Items marked with a ? can potentially be moved out to subsequent > >> releases. > >> > >>>>>> > >> > >>>>>> > >> > >>>>>> Regards, > >> > >>>>>> Matthias > >> > >>>>>> > >> > >>>>> > >> > >>>>> My understanding is that most of the items in 1 and 2 are going to > >> break > >> > >>>>> backward compatibility, while the others can be done incrementally. > >> Is this > >> > >>>>> assumption correct? If so, can we finish 1 and 2 and do a 1.0 > >> release. and > >> > >>>>> them, continue with 3, 4, 5, etc ? as I don't think we should wait > for > >> > >>>>> 2017/Q2 to do a 1.0 release. I believe in release early, release > >> often, > >> > >>>>> particularly to attract new users, that can help verifying and > >> contributing > >> > >>>>> to specific releases. > >> > >>>>> > >> > >>>>> Thoughts ? > >> > >>>>> > >> > >>>> > >> > >>>> > >> > >>>> > >> > >>>> > >> > >> -- > > Sent from my Mobile device >
