[ https://issues.apache.org/jira/browse/SYSTEMML-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15959961#comment-15959961 ]
Deron Eriksson edited comment on SYSTEMML-1471 at 4/6/17 11:33 PM: ------------------------------------------------------------------- I think you might be able to do something like this (prediction scenario) with the existing MLContext API. You could create a ScoringScriptExecutor class that extends ScriptExecutor. On this class, create a prepare(Script) method that contains: {code} setup(script); parseScript(); liveVariableAnalysis(); validateScript(); constructHops(); rewriteHops(); rewritePersistentReadsAndWrites(); constructLops(); generateRuntimeProgram(); showExplanation(); globalDataFlowOptimization(); countCompiledMRJobsAndSparkInstructions(); initializeCachingAndScratchSpace(); cleanupRuntimeProgram(); {code} Then override ScriptExecutor's execute(Script) method and have it contain: {code} if(statistics) { Statistics.startRunTimer(); } createAndInitializeExecutionContext(); executeRuntimeProgram(); cleanupAfterExecution(); // add symbol table to MLResults MLResults mlResults = new MLResults(script); script.setResults(mlResults); if (statistics) { Statistics.stopRunTimer(); System.out.println(Statistics.display(statisticsMaxHeavyHitters)); } return mlResults; {code} In the calling code, have something like: {code} ScoringScriptExecutor sse = new ScoringScriptExecutor(); sse.prepare(script); // create the dml program while (....) { ... MLResults results = ml.execute(script, sse); // execute the dml program } {code} was (Author: deron): I think you might be able to do something like this with the existing API. You could create a ScoringScriptExecutor class that extends ScriptExecutor. On this class, create a prepare(Script) method that contains: {code} setup(script); parseScript(); liveVariableAnalysis(); validateScript(); constructHops(); rewriteHops(); rewritePersistentReadsAndWrites(); constructLops(); generateRuntimeProgram(); showExplanation(); globalDataFlowOptimization(); countCompiledMRJobsAndSparkInstructions(); initializeCachingAndScratchSpace(); cleanupRuntimeProgram(); {code} Then override ScriptExecutor's execute(Script) method and have it contain: {code} if(statistics) { Statistics.startRunTimer(); } createAndInitializeExecutionContext(); executeRuntimeProgram(); cleanupAfterExecution(); // add symbol table to MLResults MLResults mlResults = new MLResults(script); script.setResults(mlResults); if (statistics) { Statistics.stopRunTimer(); System.out.println(Statistics.display(statisticsMaxHeavyHitters)); } return mlResults; {code} In the calling code, have something like: {code} ScoringScriptExecutor sse = new ScoringScriptExecutor(); sse.prepare(script); // create the dml program while (....) { ... MLResults results = ml.execute(script, sse); // execute the dml program } {code} > Support PreparedScript for MLContext > ------------------------------------ > > Key: SYSTEMML-1471 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1471 > Project: SystemML > Issue Type: Improvement > Reporter: Niketan Pansare > > The intent of this JIRA is three-fold: > 1. Allow MLContext to be used in prediction scenario. > 2. Consolidate the code of JMLC and MLContext. > 3. Explore what extensions are needed in SystemML to support Spark streaming. > For prediction scenario, it is important to reduce the parsing/validation > overhead as much as possible and reusing the JMLC infrastructure might be a > good step in that direction. It is also important that MLContext continues to > support dynamic recompilation and other optimization as the input size could > be small (similar to JMLC), but could also be large (if window size is large, > making MLContext ideal for this scenario). > {code} > val streamingContext = new StreamingContext(sc, SLIDE_INTERVAL) > val windowDStream = .....window(WINDOW_LENGTH, SLIDE_INTERVAL) > val preparedScript = ....prepareScript(....) > windowDStream.foreachRDD(currentWindow => { > if (currentWindow.count() > 0) { > ml.execute(preparedScript.in("X", currentWindow.toDF())) > ... > } > }) > {code} > [~deron] [~mboehm7] [~reinwald] [~freiss] [~mwdus...@us.ibm.com] [~nakul02] > Is this something that interest anyone of you ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)