Re: [DISCUSS] R-Interface to SystemML
Hi Can we please have the discussion in other thread on R4ML integration with SystemML :) sincerely Alok From: Niketan Pansare Sent: Friday, September 22, 2017 10:51 AM To: dev@systemml.apache.org Subject: Re: [DISCUSS] R-Interface to SystemML Hi Brendan/Alok, Thanks for taking time to answer our questions and also for considering to help SystemML community. I have couple more questions: 1. In case there is inconsistency, do you (as R4ML developers) feel comfortable changing R4ML interface to be compatible with our other APIs ? May be you can go over the below two links and imagine adding a corresponding R tab: - MLContext Programming guide: http://apache.github.io/systemml/spark-mlcontext-programming-guide apache.github.io<http://apache.github.io/systemml/spark-mlcontext-programming-guide> apache.github.io Spark MLContext Programming Guide. Overview; Spark Shell Example. Start Spark Shell with SystemML; Create MLContext; Hello World; LeNet on MNIST Example; DataFrame ... - Algorithm wrappers: http://apache.github.io/systemml/algorithms-classification.html#multinomial-logistic-regression 2. Classification - GitHub Pages<http://apache.github.io/systemml/algorithms-classification.html#multinomial-logistic-regression> apache.github.io SystemML Algorithms Reference 2. Classification 2.1. Multinomial Logistic Regression Description. The MultiLogReg.dml script performs both binomial and multinomial ... 2. Other than providing R interface to SystemML as the above APIs, what additional features/code R4ML plans to add in SystemML ? Just like we want the R API to be functionally complete with our Python and Scala API, we want Python and Scala APIs to be functionally complete with the R API. So a discussion on supporting the additional features in Python and Scala APIs is required :) Thanks, Niketan Pansare IBM Almaden Research Center E-mail: npansar At us.ibm.com http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar [http://researcher.watson.ibm.com/researcher/photos/3531.jpg]<http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar> Niketan Pansare - IBM<http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar> researcher.watson.ibm.com Niketan Pansare is a Senior Software Engineer at IBM Research Almaden, where he works on advanced information management systems that include analytics, distributed ... [Inactive hide details for Deron Eriksson ---09/22/2017 08:59:58 AM---Hi Brendan, Thank you for the detailed description. At a h]Deron Eriksson ---09/22/2017 08:59:58 AM---Hi Brendan, Thank you for the detailed description. At a high level that sounds From: Deron Eriksson To: dev@systemml.apache.org Date: 09/22/2017 08:59 AM Subject: Re: [DISCUSS] R-Interface to SystemML Hi Brendan, Thank you for the detailed description. At a high level that sounds feasible. Also, offering to help maintain the R codebase is extremely helpful. Please let us know if you have any questions so that we can assist you and Alok in your efforts, since as I said I think an R interface to SystemML makes a lot of sense. Deron On Thu, Sep 21, 2017 at 4:36 PM, Brendan Dwyer wrote: > Sorry for not responding sooner. I had some issues with my email client. > > > > I will do my best to address as many as the points that have been raised > as I can. Hopefully Alok will be able to jump in as well once he resolves > his email issues. > > > > - I would be happy to help maintain R4ML in SystemML and I’m sure Alok > would too. > > - R4ML does allow arbitrary DML script to be executed via the > `sysml.execute()` function. > > - I think we would like to merge the entire R4ML github repository into > SystemML. We could do this the same way SparkR was merged into Spark ( > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_spark_tree_master_R&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=zZv3hzGXYPsASGxiAIkKxrx6RwULOnp_PhMvzsrnwGs&s=-AsxMGFLlXCro0OSEJTWLLca2vPsQwTwBxAY8Bzp6kw&e= [https://avatars3.githubusercontent.com/u/47359?v=4&s=400]<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_spark_tree_master_R&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=zZv3hzGXYPsASGxiAIkKxrx6RwULOnp_PhMvzsrnwGs&s=-AsxMGFLlXCro0OSEJTWLLca2vPsQwTwBxAY8Bzp6kw&e=> apache/spark<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_spark_tree_master_R&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=zZv3hzGXYPsASGxiAIkKxrx6RwULOnp_PhMvzsrnwGs&s=-AsxMGFLlXCro0OSEJTWLLca2vPsQwTwBxAY8Bzp6kw&e=> urldefense.proofpoint.com spark - Mirror of Apache Spark ) > > - Currently the code is
Re: [DISCUSS] R-Interface to SystemML
Can you please have discussion in other thread 😊 1) Also note that current codebase in not R interface as MLCtx api for python etc. but it does provides all those functionality in one way or we might have to add it From: Deron Eriksson Sent: Friday, September 22, 2017 8:59 AM To: dev@systemml.apache.org Subject: Re: [DISCUSS] R-Interface to SystemML Hi Brendan, Thank you for the detailed description. At a high level that sounds feasible. Also, offering to help maintain the R codebase is extremely helpful. Please let us know if you have any questions so that we can assist you and Alok in your efforts, since as I said I think an R interface to SystemML makes a lot of sense. Deron On Thu, Sep 21, 2017 at 4:36 PM, Brendan Dwyer wrote: > Sorry for not responding sooner. I had some issues with my email client. > > > > I will do my best to address as many as the points that have been raised > as I can. Hopefully Alok will be able to jump in as well once he resolves > his email issues. > > > > - I would be happy to help maintain R4ML in SystemML and I’m sure Alok > would too. > > - R4ML does allow arbitrary DML script to be executed via the > `sysml.execute()` function. > > - I think we would like to merge the entire R4ML github repository into > SystemML. We could do this the same way SparkR was merged into Spark ( > https://github.com/apache/spark/tree/master/R) https://avatars3.githubusercontent.com/u/47359?v=4&s=400 spark/R at master · apache/spark · GitHub github.com README.md R on Spark. SparkR is an R package that provides a light-weight frontend to use Spark from R. Installing sparkR. Libraries of sparkR need to be created in ... > > - Currently the code is not ready to be merged into SystemML because we > are still on the old ML context. We have a PR in the works that will update > to the newest ML context. Once that happens we won’t need to duplicate the > DML scripts. > > - Documentation is generated automatically with the R package “roxygen”. > We would need to discuss how to incorporate this into the SystemML > documentation. Perhaps we could look to Spark/SparkR for ideas. > > - Test are done using the R testthat package. I can work with Alan to get > that integrated into the systemml Jenkins server > > Sent from IBM Verse > > > Matthias Boehm --- Re: [DISCUSS] R-Interface to SystemML --- > From:"Matthias Boehm" To: > dev@systemml.apache.org, deron@apache.orgDate:Thu, Sep 21, 2017 4:13 > PMSubject:Re: [DISCUSS] R-Interface to SystemML > > I pretty much agree with Niketan and Deron. In general, it would be > usefulto provide an R API as well. However, I'm a bit concerned for two > reasons:* Looking over the github repo, apparently R4ML is not under > activedevelopment/maintenance anymore (last commit Jul 20). So who would > bewilling to maintain and extend it?* Providing wrappers for our algorithm > scripts would be just a startbecause it hides our core value proposition of > custom large-scale ML.Hence, we would also need an MLContext equivalent > that allows to executearbitrary DML scripts or R functions. Is there > already a tentative designof such an API and if not, who would like to take > it over?Regards,MatthiasOn Thu, Sep 21, 2017 at 3:43 PM, Deron Eriksson < > deroneriks...@gmail.com>wrote:> I agree with Niketan. An R interface > definitely makes sense for SystemML.> DML itself is based on R, so it's > surprising we have Java/Scala/Python> interfaces to SystemML but we don't > have an R interface.>> Perhaps R4ML committers could supply a little more > info? For instance:> 1) Would they like to merge R4ML code into the main > SystemML project> itself? (Currently we have no modules.)> 2) What would > they like to merge?> 3) If so, how do they propose to do so?> 4) Who will > do the majority of the work to add R4ML code to SystemML? Or> who would > like to volunteer to do this?> 5) Who will maintain the contributed code? > Or who would like to volunteer> to do this?> 6) Documentation is needed > (fit in SystemML documentation framework).> 7) Testing is needed (fit into > SystemML testing framework).> 8) How is this packaged?>> From a technology > standpoint, I think an R interface totally makes sense.> As for a minor > criticism (which I apply to other parts of SystemML too), I> see script > wrappers at https://urldefense.proofpoint.com/v2/url?u=https-3A__github. > com_SparkTC_r4ml_tree_master_R4ML_R&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r= > oU0Hd6PZBeEjeAVlZmb0utBefJN1XdJBEF8eiZhdECk&m=kVcfoxaRYrbaD_ > gb_hA_jn4bjiKe_gcUpc6mF1xbEd4&s=_qGcWSDggH-K3E_ > wTPXBBjOClp2Jub4KtvBgfeW1kbU&e= .> This tightly binds the existing DM
Re: [DISCUSS] R-Interface to SystemML
Hi Brendan/Alok, Thanks for taking time to answer our questions and also for considering to help SystemML community. I have couple more questions: 1. In case there is inconsistency, do you (as R4ML developers) feel comfortable changing R4ML interface to be compatible with our other APIs ? May be you can go over the below two links and imagine adding a corresponding R tab: - MLContext Programming guide: http://apache.github.io/systemml/spark-mlcontext-programming-guide - Algorithm wrappers: http://apache.github.io/systemml/algorithms-classification.html#multinomial-logistic-regression 2. Other than providing R interface to SystemML as the above APIs, what additional features/code R4ML plans to add in SystemML ? Just like we want the R API to be functionally complete with our Python and Scala API, we want Python and Scala APIs to be functionally complete with the R API. So a discussion on supporting the additional features in Python and Scala APIs is required :) Thanks, Niketan Pansare IBM Almaden Research Center E-mail: npansar At us.ibm.com http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar From: Deron Eriksson To: dev@systemml.apache.org Date: 09/22/2017 08:59 AM Subject:Re: [DISCUSS] R-Interface to SystemML Hi Brendan, Thank you for the detailed description. At a high level that sounds feasible. Also, offering to help maintain the R codebase is extremely helpful. Please let us know if you have any questions so that we can assist you and Alok in your efforts, since as I said I think an R interface to SystemML makes a lot of sense. Deron On Thu, Sep 21, 2017 at 4:36 PM, Brendan Dwyer wrote: > Sorry for not responding sooner. I had some issues with my email client. > > > > I will do my best to address as many as the points that have been raised > as I can. Hopefully Alok will be able to jump in as well once he resolves > his email issues. > > > > - I would be happy to help maintain R4ML in SystemML and I’m sure Alok > would too. > > - R4ML does allow arbitrary DML script to be executed via the > `sysml.execute()` function. > > - I think we would like to merge the entire R4ML github repository into > SystemML. We could do this the same way SparkR was merged into Spark ( > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_spark_tree_master_R&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=zZv3hzGXYPsASGxiAIkKxrx6RwULOnp_PhMvzsrnwGs&s=-AsxMGFLlXCro0OSEJTWLLca2vPsQwTwBxAY8Bzp6kw&e= ) > > - Currently the code is not ready to be merged into SystemML because we > are still on the old ML context. We have a PR in the works that will update > to the newest ML context. Once that happens we won’t need to duplicate the > DML scripts. > > - Documentation is generated automatically with the R package “roxygen”. > We would need to discuss how to incorporate this into the SystemML > documentation. Perhaps we could look to Spark/SparkR for ideas. > > - Test are done using the R testthat package. I can work with Alan to get > that integrated into the systemml Jenkins server > > Sent from IBM Verse > > >Matthias Boehm --- Re: [DISCUSS] R-Interface to SystemML --- > From:"Matthias Boehm" To: > dev@systemml.apache.org, deron@apache.orgDate:Thu, Sep 21, 2017 4:13 > PMSubject:Re: [DISCUSS] R-Interface to SystemML > > I pretty much agree with Niketan and Deron. In general, it would be > usefulto provide an R API as well. However, I'm a bit concerned for two > reasons:* Looking over the github repo, apparently R4ML is not under > activedevelopment/maintenance anymore (last commit Jul 20). So who would > bewilling to maintain and extend it?* Providing wrappers for our algorithm > scripts would be just a startbecause it hides our core value proposition of > custom large-scale ML.Hence, we would also need an MLContext equivalent > that allows to executearbitrary DML scripts or R functions. Is there > already a tentative designof such an API and if not, who would like to take > it over?Regards,MatthiasOn Thu, Sep 21, 2017 at 3:43 PM, Deron Eriksson < > deroneriks...@gmail.com>wrote:> I agree with Niketan. An R interface > definitely makes sense for SystemML.> DML itself is based on R, so it's > surprising we have Java/Scala/Python> interfaces to SystemML but we don't > have an R interface.>> Perhaps R4ML committers could supply a little more > info? For instance:> 1) Would they like to merge R4ML code into the main > SystemML project> itself? (Currently we have no modules.)> 2) What would > they like to merge?> 3) If so, how do they propose to do so?> 4) Who will > do the majority of the work to add R4ML code to SystemML? Or> who would > like to volunte
Re: [DISCUSS] R-Interface to SystemML
Hi Brendan, Thank you for the detailed description. At a high level that sounds feasible. Also, offering to help maintain the R codebase is extremely helpful. Please let us know if you have any questions so that we can assist you and Alok in your efforts, since as I said I think an R interface to SystemML makes a lot of sense. Deron On Thu, Sep 21, 2017 at 4:36 PM, Brendan Dwyer wrote: > Sorry for not responding sooner. I had some issues with my email client. > > > > I will do my best to address as many as the points that have been raised > as I can. Hopefully Alok will be able to jump in as well once he resolves > his email issues. > > > > - I would be happy to help maintain R4ML in SystemML and I’m sure Alok > would too. > > - R4ML does allow arbitrary DML script to be executed via the > `sysml.execute()` function. > > - I think we would like to merge the entire R4ML github repository into > SystemML. We could do this the same way SparkR was merged into Spark ( > https://github.com/apache/spark/tree/master/R) > > - Currently the code is not ready to be merged into SystemML because we > are still on the old ML context. We have a PR in the works that will update > to the newest ML context. Once that happens we won’t need to duplicate the > DML scripts. > > - Documentation is generated automatically with the R package “roxygen”. > We would need to discuss how to incorporate this into the SystemML > documentation. Perhaps we could look to Spark/SparkR for ideas. > > - Test are done using the R testthat package. I can work with Alan to get > that integrated into the systemml Jenkins server > > Sent from IBM Verse > > >Matthias Boehm --- Re: [DISCUSS] R-Interface to SystemML --- > From:"Matthias Boehm" To: > dev@systemml.apache.org, deron@apache.orgDate:Thu, Sep 21, 2017 4:13 > PMSubject:Re: [DISCUSS] R-Interface to SystemML > > I pretty much agree with Niketan and Deron. In general, it would be > usefulto provide an R API as well. However, I'm a bit concerned for two > reasons:* Looking over the github repo, apparently R4ML is not under > activedevelopment/maintenance anymore (last commit Jul 20). So who would > bewilling to maintain and extend it?* Providing wrappers for our algorithm > scripts would be just a startbecause it hides our core value proposition of > custom large-scale ML.Hence, we would also need an MLContext equivalent > that allows to executearbitrary DML scripts or R functions. Is there > already a tentative designof such an API and if not, who would like to take > it over?Regards,MatthiasOn Thu, Sep 21, 2017 at 3:43 PM, Deron Eriksson < > deroneriks...@gmail.com>wrote:> I agree with Niketan. An R interface > definitely makes sense for SystemML.> DML itself is based on R, so it's > surprising we have Java/Scala/Python> interfaces to SystemML but we don't > have an R interface.>> Perhaps R4ML committers could supply a little more > info? For instance:> 1) Would they like to merge R4ML code into the main > SystemML project> itself? (Currently we have no modules.)> 2) What would > they like to merge?> 3) If so, how do they propose to do so?> 4) Who will > do the majority of the work to add R4ML code to SystemML? Or> who would > like to volunteer to do this?> 5) Who will maintain the contributed code? > Or who would like to volunteer> to do this?> 6) Documentation is needed > (fit in SystemML documentation framework).> 7) Testing is needed (fit into > SystemML testing framework).> 8) How is this packaged?>> From a technology > standpoint, I think an R interface totally makes sense.> As for a minor > criticism (which I apply to other parts of SystemML too), I> see script > wrappers at https://urldefense.proofpoint.com/v2/url?u=https-3A__github. > com_SparkTC_r4ml_tree_master_R4ML_R&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r= > oU0Hd6PZBeEjeAVlZmb0utBefJN1XdJBEF8eiZhdECk&m=kVcfoxaRYrbaD_ > gb_hA_jn4bjiKe_gcUpc6mF1xbEd4&s=_qGcWSDggH-K3E_ > wTPXBBjOClp2Jub4KtvBgfeW1kbU&e= .> This tightly binds the existing DML > scripts to R, which means DML> input/output modifications could potentially > require modifications to R> code.>> Deron>>>> On Thu, Sep 21, 2017 at 11:00 > AM, Niketan Pansare > wrote:>> > Janardhan: I believe > this is the R4ML repo: https://urldefense.proofpoint. > com/v2/url?u=https-3A__github.com_SparkTC_&d=DwIBaQ&c=jf_ > iaSHvJObTbx-siA1ZOg&r=oU0Hd6PZBeEjeAVlZmb0utBefJN1Xd > JBEF8eiZhdECk&m=kVcfoxaRYrbaD_gb_hA_jn4bjiKe_gcUpc6mF1xbEd4& > s=vj-Ogu1F2fnli1JwDjU1-S-Pauc7SSmSNG0g5sjgwUk&e= > > r4ml . Arvind: > please correct me if I am wrong.> >> > Over
Re: [DISCUSS] R-Interface to SystemML
Sorry for not responding sooner. I had some issues with my email client. I will do my best to address as many as the points that have been raised as I can. Hopefully Alok will be able to jump in as well once he resolves his email issues. - I would be happy to help maintain R4ML in SystemML and I’m sure Alok would too. - R4ML does allow arbitrary DML script to be executed via the `sysml.execute()` function. - I think we would like to merge the entire R4ML github repository into SystemML. We could do this the same way SparkR was merged into Spark (https://github.com/apache/spark/tree/master/R) - Currently the code is not ready to be merged into SystemML because we are still on the old ML context. We have a PR in the works that will update to the newest ML context. Once that happens we won’t need to duplicate the DML scripts. - Documentation is generated automatically with the R package “roxygen”. We would need to discuss how to incorporate this into the SystemML documentation. Perhaps we could look to Spark/SparkR for ideas. - Test are done using the R testthat package. I can work with Alan to get that integrated into the systemml Jenkins server Sent from IBM Verse Matthias Boehm --- Re: [DISCUSS] R-Interface to SystemML --- From:"Matthias Boehm" To:dev@systemml.apache.org, deron@apache.orgDate:Thu, Sep 21, 2017 4:13 PMSubject:Re: [DISCUSS] R-Interface to SystemML I pretty much agree with Niketan and Deron. In general, it would be usefulto provide an R API as well. However, I'm a bit concerned for two reasons:* Looking over the github repo, apparently R4ML is not under activedevelopment/maintenance anymore (last commit Jul 20). So who would bewilling to maintain and extend it?* Providing wrappers for our algorithm scripts would be just a startbecause it hides our core value proposition of custom large-scale ML.Hence, we would also need an MLContext equivalent that allows to executearbitrary DML scripts or R functions. Is there already a tentative designof such an API and if not, who would like to take it over?Regards,MatthiasOn Thu, Sep 21, 2017 at 3:43 PM, Deron Eriksson wrote:> I agree with Niketan. An R interface definitely makes sense for SystemML.> DML itself is based on R, so it's surprising we have Java/Scala/Python> interfaces to SystemML but we don't have an R interface.>> Perhaps R4ML committers could supply a little more info? For instance:> 1) Would they like to merge R4ML code into the main SystemML project> itself? (Currently we have no modules.)> 2) What would they like to merge?> 3) If so, how do they propose to do so?> 4) Who will do the majority of the work to add R4ML code to SystemML? Or> who would like to volunteer to do this?> 5) Who will maintain the contributed code? Or who would like to volunteer> to do this?> 6) Documentation is needed (fit in SystemML documentation framework).> 7) Testing is needed (fit into SystemML testing framework).> 8) How is this packaged?>> From a technology standpoint, I think an R interface totally makes sense.> As for a minor criticism (which I apply to other parts of SystemML too), I> see script wrappers at https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_SparkTC_r4ml_tree_master_R4ML_R&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=oU0Hd6PZBeEjeAVlZmb0utBefJN1XdJBEF8eiZhdECk&m=kVcfoxaRYrbaD_gb_hA_jn4bjiKe_gcUpc6mF1xbEd4&s=_qGcWSDggH-K3E_wTPXBBjOClp2Jub4KtvBgfeW1kbU&e= .> This tightly binds the existing DML scripts to R, which means DML> input/output modifications could potentially require modifications to R> code.>> Deron>>>> On Thu, Sep 21, 2017 at 11:00 AM, Niketan Pansare > wrote:>> > Janardhan: I believe this is the R4ML repo: https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_SparkTC_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=oU0Hd6PZBeEjeAVlZmb0utBefJN1XdJBEF8eiZhdECk&m=kVcfoxaRYrbaD_gb_hA_jn4bjiKe_gcUpc6mF1xbEd4&s=vj-Ogu1F2fnli1JwDjU1-S-Pauc7SSmSNG0g5sjgwUk&e= > > r4ml . Arvind: please correct me if I am wrong.> >> > Overall, having a R interface for SystemML is an awesome idea. Since I am> > not an R4ML expert, may be R4ML committers can comment on how they> envision> > "two code streams to work together".> >> > Also, comparing the features of R4ML with that of our Python APIs will be> > useful as it might make a stronger case for R4ML.> >> > As an FYI, here are different ways Python users can use SystemML:> > - Using MLContext to invoke DML script (https://urldefense.proofpoint.com/v2/url?u=http-3A__apache.github.io_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=oU0Hd6PZBeEjeAVlZmb0utBefJN1XdJBEF8eiZhdECk&m=kVcfoxaRYrbaD_gb_hA_jn4bjiKe_gcUpc6mF1xbEd4&s=
Re: [DISCUSS] R-Interface to SystemML
I pretty much agree with Niketan and Deron. In general, it would be useful to provide an R API as well. However, I'm a bit concerned for two reasons: * Looking over the github repo, apparently R4ML is not under active development/maintenance anymore (last commit Jul 20). So who would be willing to maintain and extend it? * Providing wrappers for our algorithm scripts would be just a start because it hides our core value proposition of custom large-scale ML. Hence, we would also need an MLContext equivalent that allows to execute arbitrary DML scripts or R functions. Is there already a tentative design of such an API and if not, who would like to take it over? Regards, Matthias On Thu, Sep 21, 2017 at 3:43 PM, Deron Eriksson wrote: > I agree with Niketan. An R interface definitely makes sense for SystemML. > DML itself is based on R, so it's surprising we have Java/Scala/Python > interfaces to SystemML but we don't have an R interface. > > Perhaps R4ML committers could supply a little more info? For instance: > 1) Would they like to merge R4ML code into the main SystemML project > itself? (Currently we have no modules.) > 2) What would they like to merge? > 3) If so, how do they propose to do so? > 4) Who will do the majority of the work to add R4ML code to SystemML? Or > who would like to volunteer to do this? > 5) Who will maintain the contributed code? Or who would like to volunteer > to do this? > 6) Documentation is needed (fit in SystemML documentation framework). > 7) Testing is needed (fit into SystemML testing framework). > 8) How is this packaged? > > From a technology standpoint, I think an R interface totally makes sense. > As for a minor criticism (which I apply to other parts of SystemML too), I > see script wrappers at https://github.com/SparkTC/r4ml/tree/master/R4ML/R. > This tightly binds the existing DML scripts to R, which means DML > input/output modifications could potentially require modifications to R > code. > > Deron > > > > On Thu, Sep 21, 2017 at 11:00 AM, Niketan Pansare > wrote: > > > Janardhan: I believe this is the R4ML repo: https://github.com/SparkTC/ > > r4ml . Arvind: please correct me if I am wrong. > > > > Overall, having a R interface for SystemML is an awesome idea. Since I am > > not an R4ML expert, may be R4ML committers can comment on how they > envision > > "two code streams to work together". > > > > Also, comparing the features of R4ML with that of our Python APIs will be > > useful as it might make a stronger case for R4ML. > > > > As an FYI, here are different ways Python users can use SystemML: > > - Using MLContext to invoke DML script (http://apache.github.io/ > > systemml/beginners-guide-python#invoking-dmlpydml- > scripts-using-mlcontext > > and http://apache.github.io/systemml/spark-mlcontext- > > programming-guide.html) > > - Python algorithms wrappers (http://apache.github.io/ > > systemml/beginners-guide-python#invoke-systemmls-algorithms) > > - (not important for R4ML discussion): Python DSL ( > > http://apache.github.io/systemml/beginners-guide- > python#matrix-operations) > > > > Thanks, > > > > Niketan Pansare > > IBM Almaden Research Center > > E-mail: npansar At us.ibm.com > > http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar > > > > [image: Inactive hide details for Janardhan ---09/21/2017 04:44:02 > AM---Hi > > Arvind, This is a great idea. One question: the R4ML generat]Janardhan > > ---09/21/2017 04:44:02 AM---Hi Arvind, This is a great idea. One > question: > > the R4ML generates any plan like the SystemML with `D > > > > From: Janardhan > > To: Arvind Surve , "dev@systemml.apache.org" < > > dev@systemml.apache.org> > > Date: 09/21/2017 04:44 AM > > Subject: Re: [DISCUSS] R-Interface to SystemML > > -- > > > > > > > > Hi Arvind, > > > > This is a great idea. One question: the R4ML generates any plan like the > > SystemML with `DML` Or with providing some interface we leverage this > > feature ?. Community effort in the sense of collaborative algorithm > > implementation.(?) > > > > Is this the Spark-R repo ( https://urldefense.proofpoint. > > com/v2/url?u=https-3A__github.com_rstudio_sparklyr&d=DwIGaQ& > > c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_ > > 6a6SaOkZrfiSrYZVby0&m=uxG7P-4VuICwg6yatnAEX5JBdZ- > PSwyvQzq5gwX1GL0&s=6VRs_ > > J7zXj9jZEavEP8iNvVfISAjDJeM8wFL2sBnb0g&e= ) ? > > > > Thanks, > > Janardhan > > > > Sent with [ProtonMail](https://urldefense.proo
Re: [DISCUSS] R-Interface to SystemML
I agree with Niketan. An R interface definitely makes sense for SystemML. DML itself is based on R, so it's surprising we have Java/Scala/Python interfaces to SystemML but we don't have an R interface. Perhaps R4ML committers could supply a little more info? For instance: 1) Would they like to merge R4ML code into the main SystemML project itself? (Currently we have no modules.) 2) What would they like to merge? 3) If so, how do they propose to do so? 4) Who will do the majority of the work to add R4ML code to SystemML? Or who would like to volunteer to do this? 5) Who will maintain the contributed code? Or who would like to volunteer to do this? 6) Documentation is needed (fit in SystemML documentation framework). 7) Testing is needed (fit into SystemML testing framework). 8) How is this packaged? >From a technology standpoint, I think an R interface totally makes sense. As for a minor criticism (which I apply to other parts of SystemML too), I see script wrappers at https://github.com/SparkTC/r4ml/tree/master/R4ML/R. This tightly binds the existing DML scripts to R, which means DML input/output modifications could potentially require modifications to R code. Deron On Thu, Sep 21, 2017 at 11:00 AM, Niketan Pansare wrote: > Janardhan: I believe this is the R4ML repo: https://github.com/SparkTC/ > r4ml . Arvind: please correct me if I am wrong. > > Overall, having a R interface for SystemML is an awesome idea. Since I am > not an R4ML expert, may be R4ML committers can comment on how they envision > "two code streams to work together". > > Also, comparing the features of R4ML with that of our Python APIs will be > useful as it might make a stronger case for R4ML. > > As an FYI, here are different ways Python users can use SystemML: > - Using MLContext to invoke DML script (http://apache.github.io/ > systemml/beginners-guide-python#invoking-dmlpydml-scripts-using-mlcontext > and http://apache.github.io/systemml/spark-mlcontext- > programming-guide.html) > - Python algorithms wrappers (http://apache.github.io/ > systemml/beginners-guide-python#invoke-systemmls-algorithms) > - (not important for R4ML discussion): Python DSL ( > http://apache.github.io/systemml/beginners-guide-python#matrix-operations) > > Thanks, > > Niketan Pansare > IBM Almaden Research Center > E-mail: npansar At us.ibm.com > http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar > > [image: Inactive hide details for Janardhan ---09/21/2017 04:44:02 AM---Hi > Arvind, This is a great idea. One question: the R4ML generat]Janardhan > ---09/21/2017 04:44:02 AM---Hi Arvind, This is a great idea. One question: > the R4ML generates any plan like the SystemML with `D > > From: Janardhan > To: Arvind Surve , "dev@systemml.apache.org" < > dev@systemml.apache.org> > Date: 09/21/2017 04:44 AM > Subject: Re: [DISCUSS] R-Interface to SystemML > -- > > > > Hi Arvind, > > This is a great idea. One question: the R4ML generates any plan like the > SystemML with `DML` Or with providing some interface we leverage this > feature ?. Community effort in the sense of collaborative algorithm > implementation.(?) > > Is this the Spark-R repo ( https://urldefense.proofpoint. > com/v2/url?u=https-3A__github.com_rstudio_sparklyr&d=DwIGaQ& > c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_ > 6a6SaOkZrfiSrYZVby0&m=uxG7P-4VuICwg6yatnAEX5JBdZ-PSwyvQzq5gwX1GL0&s=6VRs_ > J7zXj9jZEavEP8iNvVfISAjDJeM8wFL2sBnb0g&e= ) ? > > Thanks, > Janardhan > > Sent with [ProtonMail](https://urldefense.proofpoint.com/v2/ > url?u=https-3A__protonmail.com&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m= > uxG7P-4VuICwg6yatnAEX5JBdZ-PSwyvQzq5gwX1GL0&s=khkGV3oXz1W5m_ > ueQRuKWlNMVOXXCVhV_ytNCINjJWY&e= ) Secure Email. > > > Original Message > > Subject: [DISCUSS] R-Interface to SystemML > > Local Time: September 20, 2017 12:50 PM > > UTC Time: September 20, 2017 4:50 PM > > From: ac...@yahoo.com.INVALID > > To: dev@systemml.apache.org > > > > Hi, > > R4ML is an open source project which provides a R interface to > SystemML.Its a bridge between SystemML and Spark-R. > > Lets discuss here if and how we can get two code streams work together > to benefit development/community effort. > > > > Arvind Surve | Spark Technology Center | https://urldefense.proofpoint. > com/v2/url?u=http-3A__www.spark.tc_&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r= > HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=uxG7P-4VuICwg6yatnAEX5JBdZ- > PSwyvQzq5gwX1GL0&s=Yj8qfo7sjGBxX547UMGfLHRZFUxCAjZDTHWe8B7jLxI&e= > > >
Re: [DISCUSS] R-Interface to SystemML
Janardhan: I believe this is the R4ML repo: https://github.com/SparkTC/r4ml . Arvind: please correct me if I am wrong. Overall, having a R interface for SystemML is an awesome idea. Since I am not an R4ML expert, may be R4ML committers can comment on how they envision "two code streams to work together". Also, comparing the features of R4ML with that of our Python APIs will be useful as it might make a stronger case for R4ML. As an FYI, here are different ways Python users can use SystemML: - Using MLContext to invoke DML script ( http://apache.github.io/systemml/beginners-guide-python#invoking-dmlpydml-scripts-using-mlcontext and http://apache.github.io/systemml/spark-mlcontext-programming-guide.html) - Python algorithms wrappers ( http://apache.github.io/systemml/beginners-guide-python#invoke-systemmls-algorithms ) - (not important for R4ML discussion): Python DSL ( http://apache.github.io/systemml/beginners-guide-python#matrix-operations) Thanks, Niketan Pansare IBM Almaden Research Center E-mail: npansar At us.ibm.com http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar From: Janardhan To: Arvind Surve , "dev@systemml.apache.org" Date: 09/21/2017 04:44 AM Subject: Re: [DISCUSS] R-Interface to SystemML Hi Arvind, This is a great idea. One question: the R4ML generates any plan like the SystemML with `DML` Or with providing some interface we leverage this feature ?. Community effort in the sense of collaborative algorithm implementation.(?) Is this the Spark-R repo ( https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_rstudio_sparklyr&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=uxG7P-4VuICwg6yatnAEX5JBdZ-PSwyvQzq5gwX1GL0&s=6VRs_J7zXj9jZEavEP8iNvVfISAjDJeM8wFL2sBnb0g&e= ) ? Thanks, Janardhan Sent with [ProtonMail]( https://urldefense.proofpoint.com/v2/url?u=https-3A__protonmail.com&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=uxG7P-4VuICwg6yatnAEX5JBdZ-PSwyvQzq5gwX1GL0&s=khkGV3oXz1W5m_ueQRuKWlNMVOXXCVhV_ytNCINjJWY&e= ) Secure Email. > Original Message > Subject: [DISCUSS] R-Interface to SystemML > Local Time: September 20, 2017 12:50 PM > UTC Time: September 20, 2017 4:50 PM > From: ac...@yahoo.com.INVALID > To: dev@systemml.apache.org > > Hi, > R4ML is an open source project which provides a R interface to SystemML.Its a bridge between SystemML and Spark-R. > Lets discuss here if and how we can get two code streams work together to benefit development/community effort. > > Arvind Surve | Spark Technology Center | https://urldefense.proofpoint.com/v2/url?u=http-3A__www.spark.tc_&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=uxG7P-4VuICwg6yatnAEX5JBdZ-PSwyvQzq5gwX1GL0&s=Yj8qfo7sjGBxX547UMGfLHRZFUxCAjZDTHWe8B7jLxI&e=
Re: [DISCUSS] R-Interface to SystemML
Hi Arvind, This is a great idea. One question: the R4ML generates any plan like the SystemML with `DML` Or with providing some interface we leverage this feature ?. Community effort in the sense of collaborative algorithm implementation.(?) Is this the Spark-R repo ( https://github.com/rstudio/sparklyr ) ? Thanks, Janardhan Sent with [ProtonMail](https://protonmail.com) Secure Email. > Original Message > Subject: [DISCUSS] R-Interface to SystemML > Local Time: September 20, 2017 12:50 PM > UTC Time: September 20, 2017 4:50 PM > From: ac...@yahoo.com.INVALID > To: dev@systemml.apache.org > > Hi, > R4ML is an open source project which provides a R interface to SystemML.Its a > bridge between SystemML and Spark-R. > Lets discuss here if and how we can get two code streams work together to > benefit development/community effort. > > Arvind Surve | Spark Technology Center | http://www.spark.tc/
[DISCUSS] R-Interface to SystemML
Hi, R4ML is an open source project which provides a R interface to SystemML.Its a bridge between SystemML and Spark-R. Lets discuss here if and how we can get two code streams work together to benefit development/community effort. Arvind Surve | Spark Technology Center | http://www.spark.tc/
