Re: Questions about the future of UDTs and Encoders

2017-11-19 Thread Michael Lopez
Thank you for your response, Grandjean. Frameless looks great, but it is not quite what I need. From what I can tell, Frameless provides a layer of type-safety on top of Spark facilities, like column expressions and encoders. There are also some great quality enhancments in Frameless, like Injecti

Re: Thoughts on extedning ML exporting in Spark?

2017-11-19 Thread Sean Owen
To paraphrase, you are mostly suggesting a new API for reading/writing models, not a new serialization? and the API should be more like the other DataFrame writer APIs, and more extensible? That's better than introducing any new format for sure, as there are already 1.5 supported formats -- the na

Re: Thoughts on extedning ML exporting in Spark?

2017-11-19 Thread Timur Shenkao
Hello guys, Have you considered PFA? http://dmg.org/pfa/docs/document_structure/ As Sean noticed, "there are already 1.5 supported formats " + PMML is quite rigid. There are, at least, 2 implementations of PFA. *Scala* Hadrian: https://github.com/opendatagroup/hadrian. *Python* Titus: https//gi

Re: [CANCEL] Spark 2.2.1 (RC1)

2017-11-19 Thread Felix Cheung
This vote is cancelled due to no vote. I’m going to test or track down a few issues (please see link below for those targeting this release) and roll RC2 in a few days if we could make progress. On Tue, Nov 14, 2017 at 10:25 PM Felix Cheung wrote: > Please vote on releasing the following candi

Re: Thoughts on extedning ML exporting in Spark?

2017-11-19 Thread Holden Karau
Right so mostly suggesting a new API so that people can add things like PFA. Initially in spark we’d have built in support for the current Spark custom format and support for PMML for the models who already have that in MLlib. But ideally this API would allow folks looking at other formats to imple