just a quick update: the ANTLR issue and most other things that came up during QA are fixed now, except for performance issues with stratified stats for which I need some more time (SYSTEMML-2181 tracks the open issues). So it's probably a good idea to postpone the RC1 for a few days to avoid unnecessary release efforts.
Regards, Matthias On Sun, Mar 11, 2018 at 5:43 PM, Matthias Boehm <[email protected]> wrote: > well, after trying to run our perftest suite with Spark 2.3 and Spark 2.2 > this seems to be more complicated. Although the version update from 4.5.3 > to 4.7.1 solved the problem with Spark 2.3 (which uses 4.7.1), SystemML > would no longer be backwards compatible with Spark 2.2 and 2.1 (which use > 4.5.3) because ANTLR checks bidirectional mismatches.Unfortunately, > removing ANTLR from our jar does not help because the above versions are > binary incompatible. > > We need to hold off the release until we decided whether (1) we directly > release for Spark 2.3 and drop 2.2 and 2.1, or (2) we release for Spark 2.2 > and 2.1 with a subsequent ANTLR change and release for 2.3. > > Regards, > Matthias > > On Thu, Mar 8, 2018 at 5:38 PM, Niketan Pansare <[email protected]> > wrote: > >> +1. That will preserve current behavior on older Spark versions too. >> >> > On Mar 8, 2018, at 5:24 PM, Ted Yu <[email protected]> wrote: >> > >> > +1 on upgrading >> > -------- Original message --------From: Matthias Boehm < >> [email protected]> Date: 3/8/18 5:19 PM (GMT-08:00) To: >> [email protected] Subject: Re: Release Planning >> > related to Spark 2.3, we might want to update our ANTLR version because >> > with Spark 2.3 every parsed DML script (i.e., multiple times with >> imported >> > DML files) produces the following warning: >> > >> > "ANTLR Tool version 4.5.3 used for code generation does not match the >> > current runtime version 4.7" >> > >> > Regards, >> > Matthias >> > >> >> On Thu, Mar 1, 2018 at 5:22 PM, Matthias Boehm <[email protected]> >> wrote: >> >> >> >> Hi all, >> >> >> >> I'm sure you've seen that Spark 2.3 just got released. This lines up >> >> beautifully with our own SystemML 1.1 release. Accordingly, I would >> >> recommend to use Spark 2.3 for our due diligence algorithm-level Q/A. >> How >> >> about we shoot for an RC1 by March 12? This should give enough time to >> run >> >> over reasonably large data and fix all related issues. >> >> >> >> Regards, >> >> Matthias >> >> >> >>> On Tue, Feb 6, 2018 at 12:51 PM, Matthias Boehm <[email protected]> >> wrote: >> >>> >> >>> yes, absolutely. Here is a list of new features and improvements - >> please >> >>> feel free to extend as needed: >> >>> >> >>> 1) Extended Caffe2DML and Keras2DML APIs >> >>> 2) Support for large-dense blocks >16GB in CP >> >>> 3) New builtin functions ifelse, xor, as well as and/or/not over >> matrices >> >>> 4) Single-precision support for native conv2d and mm operations. >> >>> 5) Performance features and correctness of ultra-sparse operations >> >>> 6) Codegen: new plan cache, nary cbind >> >>> 7) Parfor: result merge with accumulators +=, reduced initialization >> >>> overhead >> >>> 8) Compiler improvements: avoid unnecessary spark instructions, >> corrected >> >>> memory estimates >> >>> >> >>> Until the RC, I'd like to support all deep learning builtin functions >> in >> >>> codegen, add a couple of pending parfor improvements, and potentially >> do a >> >>> first cut of the compiler/runtime integration for parameter servers. >> >>> >> >>> Regards, >> >>> Matthias >> >>> >> >>> >> >>> >> >>> On Tue, Feb 6, 2018 at 12:36 PM, Niketan Pansare <[email protected]> >> >>> wrote: >> >>> >> >>>> +1. >> >>>> >> >>>> We should consider including single precision native BLAS in the >> release >> >>>> notes as well. If possible, we should add JNI wrappers for PowerPC, >> Windows >> >>>> and Mac too in this release. >> >>>> >> >>>>> On Feb 6, 2018, at 12:27 PM, Berthold Reinwald <[email protected] >> > >> >>>> wrote: >> >>>>> >> >>>>> sure. >> >>>>> >> >>>>> Makes sense. Codegen and Keras2DML made good progress, and many >> other >> >>>>> fixes/improvements. >> >>>>> >> >>>>> What else do we want time/track/highlight for it? >> >>>>> >> >>>>> Regards, >> >>>>> Berthold Reinwald >> >>>>> IBM Almaden Research Center >> >>>>> office: (408) 927 2208; T/L: 457 2208 >> >>>>> e-mail: [email protected] >> >>>>> >> >>>>> >> >>>>> >> >>>>> From: Matthias Boehm <[email protected]> >> >>>>> To: [email protected] >> >>>>> Date: 02/05/2018 11:05 PM >> >>>>> Subject: Release Planning >> >>>>> >> >>>>> >> >>>>> >> >>>>> Hi all, >> >>>>> >> >>>>> since our 1.0 release in Dec, we already got a number enhancements >> and >> >>>> new >> >>>>> features in, so I think it would be good to discuss the timeline for >> >>>> our >> >>>>> next SystemML 1.1 release. How about, we target mid March for a >> first >> >>>> RC? >> >>>>> Also, Berthold would you be willing to serve again as the release >> >>>> manager? >> >>>>> >> >>>>> Regards, >> >>>>> Matthias >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>> >> >>>> >> >>> >> >> >> >> >
