Hi Tatsuya, Thanks for your interest in the SystemML project. Just so that we are on same page, there are two interpretation of REPL with respect to SystemML: 1. Using SystemML in Spark REPL. - Scala Spark-shell: http://apache.github.io/incubator-systemml/mlcontext-programming-guide.html . This also involves the integration with MLPipeline ( https://github.com/apache/incubator-systemml/blob/master/src/main/java/com/ibm/bi/dml/api/ml/LogisticRegression.java ) APIs. - PySpark shell (using SystemML.py) - Jupyter (using SystemML.py)
2. Having a "DML kernel/interpreter". - A DML kernel for Jupyter (will require a command-line interpreter) - A command-line interpreter where data scientists can type DML commands. Like Matthias, I would recommend that implementing an interpreter for DML might not be a good task to explore SystemML. Still, I will try to give you an overview and please pardon me for hand-waving a bit in below paragraph for sake of simplicity. We do have a "mini-interpreter" in the form of a debugger and it might be a good place to explore before implementing an interpreter. Please note: the debugger works at level of instruction, not at the level of DML statement. Also, the debugger takes as input a DML script and hence SystemML's optimizer has full scope for optimization. However, to allow for tasks such as "step into", we disable certain optimizations and have a special level for the debugger (O5_DEBUG_MODE). For example: algrebraic simplification, interprocedural analysis, branch removal, dynamic recompilations, sum product rewrites, etc are disabled. Browsing through the debugger code should give you a fair idea about the machinery required in SystemML for implementing a full-fledged interpreter (such as runtime instructions, symbol table, instructions-to-DML mapping, piggybacking, etc). Then one must step back and look at the high-level design decisions as well, such as lazy evaluation. To get started with the debugger, please look at http://apache.github.io/incubator-systemml/debugger-guide.html and also in the com.ibm.bi.dml.debug package. Please note the URL might change soon when we change our package names to org.apache.sysml. Reference: https://github.com/apache/incubator-systemml/blob/master/src/main/java/com/ibm/bi/dml/api/MLContext.java https://github.com/apache/incubator-systemml/blob/master/src/main/java/com/ibm/bi/dml/api/python/SystemML.py https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages https://github.com/apache/incubator-systemml/tree/master/src/main/java/com/ibm/bi/dml/debug Thanks, Niketan Pansare IBM Almaden Research Center E-mail: npansar At us.ibm.com http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar From: "Tatsuya Nishiyama" <nishiyama.tats...@lab.ntt.co.jp> To: <dev@systemml.incubator.apache.org> Date: 12/02/2015 01:06 AM Subject: RE: How to contribute SystemML Hi Matthias, Thanks for your reply. > Such a feature would, however, require substantial discussions, especially with regard to optimization scope etc and hence might not be a good starting task. > How about we select some smaller tasks on our existing APIs and once you feel comfortable get started on a larger feature like that? OK, I understand it. I will try to find such smaller tasks, and tackle it. If you already know any started tasks , please tell me. Best regards, - Tatsuya -- From: Matthias Boehm [mailto:mbo...@us.ibm.com] Sent: Wednesday, December 02, 2015 4:47 PM To: dev@systemml.incubator.apache.org Subject: Re: How to contribute SystemML Hi Tatsuya, thanks for your interest, we'd love to help you get started. Although we do have various APIs, including MLContext that allows you to invoke DML scripts from Spark's interactive shell, we don't have an actual REPL interface yet. Niketan built an initial prototype of a related API. @Niketan: Do you want to comment on that? Such a feature would, however, require substantial discussions, especially with regard to optimization scope etc and hence might not be a good starting task. How about we select some smaller tasks on our existing APIs and once you feel comfortable get started on a larger feature like that? Regards, Matthias "Tatsuya Nishiyama" ---12/01/2015 10:50:41 PM---Hi, I'm interested in contributing SystemML. I've checked the documentation, and From: "Tatsuya Nishiyama" <nishiyama.tats...@lab.ntt.co.jp> To: <dev@systemml.incubator.apache.org> Date: 12/01/2015 10:50 PM Subject: How to contribute SystemML ________________________________________ Hi, I'm interested in contributing SystemML. I've checked the documentation, and it looks like SystemML doesn't have REPL for interactive execution. Is it correct? If SystemML doesn't have REPL, I would like to contribute it. BTW, JIRA looks to be unavailable for now: https://issues.apache.org/jira/browse/SYSTEMML How can I start contribution? Thanks, - Tatsuya
