Added: systemml/site/docs/1.1.0/jmlc.html URL: http://svn.apache.org/viewvc/systemml/site/docs/1.1.0/jmlc.html?rev=1828046&view=auto ============================================================================== --- systemml/site/docs/1.1.0/jmlc.html (added) +++ systemml/site/docs/1.1.0/jmlc.html Fri Mar 30 04:31:05 2018 @@ -0,0 +1,319 @@ +<!DOCTYPE html> +<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]--> +<!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]--> +<!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]--> +<!--[if gt IE 8]><!--> <html class="no-js"> <!--<![endif]--> + <head> + <title>Java Machine Learning Connector (JMLC) - SystemML 1.1.0</title> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> + + <meta name="description" content="Java Machine Learning Connector (JMLC)"> + + <meta name="viewport" content="width=device-width"> + <link rel="stylesheet" href="css/bootstrap.min.css"> + <link rel="stylesheet" href="css/main.css"> + <link rel="stylesheet" href="css/pygments-default.css"> + <link rel="shortcut icon" href="img/favicon.png"> + </head> + <body> + <!--[if lt IE 7]> + <p class="chromeframe">You are using an outdated browser. <a href="http://browsehappy.com/";>Upgrade your browser today</a> or <a href="http://www.google.com/chromeframe/?redirect=true";>install Google Chrome Frame</a> to better experience this site.</p> + <![endif]--> + + <header class="navbar navbar-default navbar-fixed-top" id="topbar"> + <div class="container"> + <div class="navbar-header"> + <div class="navbar-brand brand projectlogo"> + <a href="http://systemml.apache.org/";><img class="logo" src="img/systemml-logo.png" alt="Apache SystemML" title="Apache SystemML"/></a> + </div> + <div class="navbar-brand brand projecttitle"> + <a href="http://systemml.apache.org/";>Apache SystemML<sup id="trademark">â¢</sup></a><br/> + <span class="version">1.1.0</span> + </div> + <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target=".navbar-collapse"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + </div> + <nav class="navbar-collapse collapse"> + <ul class="nav navbar-nav navbar-right"> + <li><a href="index.html">Overview</a></li> + <li><a href="https://github.com/apache/systemml";>GitHub</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation<b class="caret"></b></a> + <ul class="dropdown-menu" role="menu"> + <li><b>Running SystemML:</b></li> + <li><a href="https://github.com/apache/systemml";>SystemML GitHub README</a></li> + <li><a href="spark-mlcontext-programming-guide.html">Spark MLContext</a></li> + <li><a href="spark-batch-mode.html">Spark Batch Mode</a> + <li><a href="hadoop-batch-mode.html">Hadoop Batch Mode</a> + <li><a href="standalone-guide.html">Standalone Guide</a></li> + <li><a href="jmlc.html">Java Machine Learning Connector (JMLC)</a> + <li class="divider"></li> + <li><b>Language Guides:</b></li> + <li><a href="dml-language-reference.html">DML Language Reference</a></li> + <li><a href="beginners-guide-to-dml-and-pydml.html">Beginner's Guide to DML and PyDML</a></li> + <li><a href="beginners-guide-python.html">Beginner's Guide for Python Users</a></li> + <li><a href="python-reference.html">Reference Guide for Python Users</a></li> + <li class="divider"></li> + <li><b>ML Algorithms:</b></li> + <li><a href="algorithms-reference.html">Algorithms Reference</a></li> + <li class="divider"></li> + <li><b>Tools:</b></li> + <li><a href="debugger-guide.html">Debugger Guide</a></li> + <li><a href="developer-tools-systemml.html">IDE Guide</a></li> + <li class="divider"></li> + <li><b>Other:</b></li> + <li><a href="contributing-to-systemml.html">Contributing to SystemML</a></li> + <li><a href="engine-dev-guide.html">Engine Developer Guide</a></li> + <li><a href="troubleshooting-guide.html">Troubleshooting Guide</a></li> + <li><a href="release-process.html">Release Process</a></li> + </ul> + </li> + + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">API Docs<b class="caret"></b></a> + <ul class="dropdown-menu" role="menu"> + <li><a href="./api/java/index.html">Java</a></li> + <li><a href="./api/python/index.html">Python</a></li> + </ul> + </li> + + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Issues<b class="caret"></b></a> + <ul class="dropdown-menu" role="menu"> + <li><b>JIRA:</b></li> + <li><a href="https://issues.apache.org/jira/browse/SYSTEMML";>SystemML JIRA</a></li> + + </ul> + </li> + </ul> + </nav> + </div> + </header> + + <div class="container" id="content"> + + <h1 class="title">Java Machine Learning Connector (JMLC)</h1> + + + <!-- + +--> + +<h1 id="overview">Overview</h1> + +<p>The <code>Java Machine Learning Connector (JMLC)</code> API is a programmatic interface for interacting with SystemML +in an embedded fashion. To use JMLC, the small footprint “in-memory” SystemML jar file needs to be included on the +classpath of the Java application, since JMLC invokes SystemML in an existing Java Virtual Machine. Because +of this, JMLC allows access to SystemML’s optimizations and fast linear algebra, but the bulk performance +gain from running SystemML on a large Spark or Hadoop cluster is not available. However, this embeddable nature +allows SystemML to be part of a production pipeline for tasks such as scoring.</p> + +<p>The primary purpose of JMLC is as a scoring API, where your scoring function is expressed using +SystemML’s DML (Declarative Machine Learning) language. Scoring occurs on a single machine in a single +JVM on a relatively small amount of input data which produces a relatively small amount of output data. +For consistency, it is important to be able to express a scoring function in the same DML language used for +training a model, since different implementations of linear algebra (for instance MATLAB and R) can deliver +slightly different results.</p> + +<p>In addition to scoring, embedded SystemML can be used for tasks such as unsupervised learning (for +example, clustering) in the context of a larger application running on a single machine.</p> + +<p>Performance penalties include startup costs, so JMLC has facilities to perform some startup tasks once, +such as script precompilation. Due to startup costs, it tends to be best practice to do batch scoring, such +as scoring 1000 records at a time. For large amounts of data, it is recommended to run DML in one +of SystemML’s distributed modes, such as Spark batch mode or Hadoop batch mode, to take advantage of SystemML’s +distributed computing capabilities. JMLC offers embeddability at the cost of performance, so its use is +dependent on the nature of the business use case being addressed.</p> + +<hr /> + +<h1 id="examples">Examples</h1> + +<p>JMLC is patterned loosely after JDBC. To interact with SystemML via JMLC, we can begin by creating a <code>Connection</code> +object. We can then prepare (precompile) a DML script by calling the <code>Connection</code>’s <code>prepareScript</code> method, +which returns a <code>PreparedScript</code> object. We can then call the <code>executeScript</code> method on the <code>PreparedScript</code> +object to invoke this script.</p> + +<p>Here, we see a “hello world” example, which invokes SystemML via JMLC and prints “hello world” to the console.</p> + +<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">Connection</span> <span class="n">conn</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Connection</span><span class="o">();</span> +<span class="n">String</span> <span class="n">dml</span> <span class="o">=</span> <span class="s">"print('hello world');"</span><span class="o">;</span> +<span class="n">PreparedScript</span> <span class="n">script</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="na">prepareScript</span><span class="o">(</span><span class="n">dml</span><span class="o">,</span> <span class="k">new</span> <span class="n">String</span><span class="o">[</span><span class="mi">0</span><span class="o">],</span> <span class="k">new</span> <span class="n">String</span><span class="o">[</span><span class="mi">0</span><span class="o">],</span> <span class="kc">false</span><span class="o">);</span> +<span class="n">script</span><span class="o">.</span><span class="na">executeScript</span><span class="o">();</span></code></pre></div> + +<hr /> + +<p>Next, let’s consider a more practical example. Consider the following DML script. It takes input data matrix <code>X</code> +and input model matrix <code>W</code>. Scores are computed, and it returns a (n x 1) matrix <code>predicted_y</code> consisting of the +column indexes of the maximum values of each row in the <code>scores</code> matrix. Note that since values are being read +in and out programmatically, we can ignore the parameters in the <code>read</code> and <code>write</code> statements.</p> + +<h4 id="dml">DML</h4> + +<div class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">X</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">read</span><span class="p">(</span><span class="s2">"./tmp/X"</span><span class="p">,</span><span class="w"> </span><span class="n">rows</span><span class="o">=</span><span class="m">-1</span><span class="p">,</span><span class="w"> </span><span class="n">cols</span><span class="o">=</span><span class="m">-1</span><span class="p">);</span><span class="w"> +</span><span class="n">W</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">read</span><span class="p">(</span><span class="s2">"./tmp/W"</span><span class="p">,</span><span class="w"> </span><span class="n">rows</span><span class="o">=</span><span class="m">-1</span><span class="p">,</span><span class="w"> </span><span class="n">cols</span><span class="o">=</span><span class="m">-1</span><span class="p">);</span><span class="w"> + +</span><span class="n">numRows</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">nrow</span><span class="p">(</span><span class="n">X</span><span class="p">);</span><span class="w"> +</span><span class="n">numCols</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ncol</span><span class="p">(</span><span class="n">X</span><span class="p">);</span><span class="w"> +</span><span class="n">b</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">W</span><span class="p">[</span><span class="n">numCols</span><span class="m">+1</span><span class="p">,]</span><span class="w"> +</span><span class="n">scores</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">X</span><span class="w"> </span><span class="o">%*%</span><span class="w"> </span><span class="n">W</span><span class="p">[</span><span class="m">1</span><span class="o">:</span><span class="n">numCols</span><span class="p">,]</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">b</span><span class="p">;</span><span class="w"> +</span><span class="n">predicted_y</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">rowIndexMax</span><span class="p">(</span><span class="n">scores</span><span class="p">);</span><span class="w"> + +</span><span class="n">write</span><span class="p">(</span><span class="n">predicted_y</span><span class="p">,</span><span class="w"> </span><span class="s2">"./tmp"</span><span class="p">,</span><span class="w"> </span><span class="n">format</span><span class="o">=</span><span class="s2">"text"</span><span class="p">);</span></code></pre></div> + +<p>In the Java below, we initialize SystemML by obtaining a <code>Connection</code> object. Next, we read in the above DML script +(<code>"scoring-example.dml"</code>) as a <code>String</code>. We precompile this script by calling the <code>prepareScript</code> method on the +<code>Connection</code> object with the names of the inputs (<code>"W"</code> and <code>"X"</code>) and outputs (<code>"predicted_y"</code>) to register.</p> + +<p>Following this, we set matrix <code>"W"</code> and we set a matrix of input data <code>"X"</code>. We execute the script and read +the resulting <code>"predicted_y"</code> matrix. We repeat this process. When done, we close the SystemML <code>Connection</code>.</p> + +<h4 id="java">Java</h4> + +<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="kn">package</span> <span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">sysml</span><span class="o">.</span><span class="na">example</span><span class="o">;</span> + + <span class="kn">import</span> <span class="nn">java.util.Random</span><span class="o">;</span> + + <span class="kn">import</span> <span class="nn">org.apache.sysml.api.jmlc.Connection</span><span class="o">;</span> + <span class="kn">import</span> <span class="nn">org.apache.sysml.api.jmlc.PreparedScript</span><span class="o">;</span> + + <span class="kd">public</span> <span class="kd">class</span> <span class="nc">JMLCExample</span> <span class="o">{</span> + + <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span> + + <span class="c1">// obtain connection to SystemML</span> + <span class="n">Connection</span> <span class="n">conn</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Connection</span><span class="o">();</span> + + <span class="c1">// read in and precompile DML script, registering inputs and outputs</span> + <span class="n">String</span> <span class="n">dml</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="na">readScript</span><span class="o">(</span><span class="s">"scoring-example.dml"</span><span class="o">);</span> + <span class="n">PreparedScript</span> <span class="n">script</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="na">prepareScript</span><span class="o">(</span><span class="n">dml</span><span class="o">,</span> <span class="k">new</span> <span class="n">String</span><span class="o">[]</span> <span class="o">{</span> <span class="s">"W"</span><span class="o">,</span> <span class="s">"X"</span> <span class="o">},</span> <span class="k">new</span> <span class="n">String</span><span class="o">[]</span> <span class="o">{</span> <span class="s">"predicted_y"</span> <span class="o">},</span> <span class="kc">false</span><span class="o">);</span> + + <span class="kt">double</span><span class="o">[][]</span> <span class="n">mtx</span> <span class="o">=</span> <span class="n">matrix</span><span class="o">(</span><span class="mi">4</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="k">new</span> <span class="kt">double</span><span class="o">[]</span> <span class="o">{</span> <span class="mi">1</span><span class="o">,</span> <span class="mi">2</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="mi">4</span><span class="o">,</span> <span class="mi">5</span><span class="o">,</span> <span class="mi">6</span><span class="o">,</span> <span class="mi">7</span><span class="o">,</span> <span class="mi">8</span><span class="o">,</span> <span class="mi">9</span> <span class="o">});</span> + <span class="kt">double</span><span class="o">[][]</span> <span class="n">result</span> <span class="o">=</span> <span class="kc">null</span><span class="o">;</span> + + <span class="c1">// set inputs, execute script, and obtain output</span> + <span class="n">script</span><span class="o">.</span><span class="na">setMatrix</span><span class="o">(</span><span class="s">"W"</span><span class="o">,</span> <span class="n">mtx</span><span class="o">);</span> + <span class="n">script</span><span class="o">.</span><span class="na">setMatrix</span><span class="o">(</span><span class="s">"X"</span><span class="o">,</span> <span class="n">randomMatrix</span><span class="o">(</span><span class="mi">3</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="o">-</span><span class="mi">1</span><span class="o">,</span> <span class="mi">1</span><span class="o">,</span> <span class="mf">0.7</span><span class="o">));</span> + <span class="n">result</span> <span class="o">=</span> <span class="n">script</span><span class="o">.</span><span class="na">executeScript</span><span class="o">().</span><span class="na">getMatrix</span><span class="o">(</span><span class="s">"predicted_y"</span><span class="o">);</span> + <span class="n">displayMatrix</span><span class="o">(</span><span class="n">result</span><span class="o">);</span> + + <span class="n">script</span><span class="o">.</span><span class="na">setMatrix</span><span class="o">(</span><span class="s">"W"</span><span class="o">,</span> <span class="n">mtx</span><span class="o">);</span> + <span class="n">script</span><span class="o">.</span><span class="na">setMatrix</span><span class="o">(</span><span class="s">"X"</span><span class="o">,</span> <span class="n">randomMatrix</span><span class="o">(</span><span class="mi">3</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="o">-</span><span class="mi">1</span><span class="o">,</span> <span class="mi">1</span><span class="o">,</span> <span class="mf">0.7</span><span class="o">));</span> + <span class="n">result</span> <span class="o">=</span> <span class="n">script</span><span class="o">.</span><span class="na">executeScript</span><span class="o">().</span><span class="na">getMatrix</span><span class="o">(</span><span class="s">"predicted_y"</span><span class="o">);</span> + <span class="n">displayMatrix</span><span class="o">(</span><span class="n">result</span><span class="o">);</span> + + <span class="n">script</span><span class="o">.</span><span class="na">setMatrix</span><span class="o">(</span><span class="s">"W"</span><span class="o">,</span> <span class="n">mtx</span><span class="o">);</span> + <span class="n">script</span><span class="o">.</span><span class="na">setMatrix</span><span class="o">(</span><span class="s">"X"</span><span class="o">,</span> <span class="n">randomMatrix</span><span class="o">(</span><span class="mi">3</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="o">-</span><span class="mi">1</span><span class="o">,</span> <span class="mi">1</span><span class="o">,</span> <span class="mf">0.7</span><span class="o">));</span> + <span class="n">result</span> <span class="o">=</span> <span class="n">script</span><span class="o">.</span><span class="na">executeScript</span><span class="o">().</span><span class="na">getMatrix</span><span class="o">(</span><span class="s">"predicted_y"</span><span class="o">);</span> + <span class="n">displayMatrix</span><span class="o">(</span><span class="n">result</span><span class="o">);</span> + + <span class="c1">// close connection</span> + <span class="n">conn</span><span class="o">.</span><span class="na">close</span><span class="o">();</span> + <span class="o">}</span> + + <span class="kd">public</span> <span class="kd">static</span> <span class="kt">double</span><span class="o">[][]</span> <span class="nf">matrix</span><span class="o">(</span><span class="kt">int</span> <span class="n">rows</span><span class="o">,</span> <span class="kt">int</span> <span class="n">cols</span><span class="o">,</span> <span class="kt">double</span><span class="o">[]</span> <span class="n">vals</span><span class="o">)</span> <span class="o">{</span> + <span class="kt">double</span><span class="o">[][]</span> <span class="n">matrix</span> <span class="o">=</span> <span class="k">new</span> <span class="kt">double</span><span class="o">[</span><span class="n">rows</span><span class="o">][</span><span class="n">cols</span><span class="o">];</span> + <span class="k">if</span> <span class="o">((</span><span class="n">vals</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">||</span> <span class="o">(</span><span class="n">vals</span><span class="o">.</span><span class="na">length</span> <span class="o">==</span> <span class="mi">0</span><span class="o">))</span> <span class="o">{</span> + <span class="k">return</span> <span class="n">matrix</span><span class="o">;</span> + <span class="o">}</span> + <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">vals</span><span class="o">.</span><span class="na">length</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span> + <span class="n">matrix</span><span class="o">[</span><span class="n">i</span> <span class="o">/</span> <span class="n">cols</span><span class="o">][</span><span class="n">i</span> <span class="o">%</span> <span class="n">cols</span><span class="o">]</span> <span class="o">=</span> <span class="n">vals</span><span class="o">[</span><span class="n">i</span><span class="o">];</span> + <span class="o">}</span> + <span class="k">return</span> <span class="n">matrix</span><span class="o">;</span> + <span class="o">}</span> + + <span class="kd">public</span> <span class="kd">static</span> <span class="kt">double</span><span class="o">[][]</span> <span class="nf">randomMatrix</span><span class="o">(</span><span class="kt">int</span> <span class="n">rows</span><span class="o">,</span> <span class="kt">int</span> <span class="n">cols</span><span class="o">,</span> <span class="kt">double</span> <span class="n">min</span><span class="o">,</span> <span class="kt">double</span> <span class="n">max</span><span class="o">,</span> <span class="kt">double</span> <span class="n">sparsity</span><span class="o">)</span> <span class="o">{</span> + <span class="kt">double</span><span class="o">[][]</span> <span class="n">matrix</span> <span class="o">=</span> <span class="k">new</span> <span class="kt">double</span><span class="o">[</span><span class="n">rows</span><span class="o">][</span><span class="n">cols</span><span class="o">];</span> + <span class="n">Random</span> <span class="n">random</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Random</span><span class="o">(</span><span class="n">System</span><span class="o">.</span><span class="na">currentTimeMillis</span><span class="o">());</span> + <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">rows</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span> + <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">j</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">j</span> <span class="o"><</span> <span class="n">cols</span><span class="o">;</span> <span class="n">j</span><span class="o">++)</span> <span class="o">{</span> + <span class="k">if</span> <span class="o">(</span><span class="n">random</span><span class="o">.</span><span class="na">nextDouble</span><span class="o">()</span> <span class="o">></span> <span class="n">sparsity</span><span class="o">)</span> <span class="o">{</span> + <span class="k">continue</span><span class="o">;</span> + <span class="o">}</span> + <span class="n">matrix</span><span class="o">[</span><span class="n">i</span><span class="o">][</span><span class="n">j</span><span class="o">]</span> <span class="o">=</span> <span class="o">(</span><span class="n">random</span><span class="o">.</span><span class="na">nextDouble</span><span class="o">()</span> <span class="o">*</span> <span class="o">(</span><span class="n">max</span> <span class="o">-</span> <span class="n">min</span><span class="o">)</span> <span class="o">+</span> <span class="n">min</span><span class="o">);</span> + <span class="o">}</span> + <span class="o">}</span> + <span class="k">return</span> <span class="n">matrix</span><span class="o">;</span> + <span class="o">}</span> + + <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">displayMatrix</span><span class="o">(</span><span class="kt">double</span><span class="o">[][]</span> <span class="n">matrix</span><span class="o">)</span> <span class="o">{</span> + <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Matrix size:"</span> <span class="o">+</span> <span class="n">matrix</span><span class="o">.</span><span class="na">length</span> <span class="o">+</span> <span class="s">"x"</span> <span class="o">+</span> <span class="n">matrix</span><span class="o">[</span><span class="mi">0</span><span class="o">].</span><span class="na">length</span><span class="o">);</span> + <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">matrix</span><span class="o">.</span><span class="na">length</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span> + <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">j</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">j</span> <span class="o"><</span> <span class="n">matrix</span><span class="o">[</span><span class="mi">0</span><span class="o">].</span><span class="na">length</span><span class="o">;</span> <span class="n">j</span><span class="o">++)</span> <span class="o">{</span> + <span class="k">if</span> <span class="o">(</span><span class="n">j</span> <span class="o">></span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span> + <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">print</span><span class="o">(</span><span class="s">", "</span><span class="o">);</span> + <span class="o">}</span> + <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">print</span><span class="o">(</span><span class="s">"["</span> <span class="o">+</span> <span class="n">i</span> <span class="o">+</span> <span class="s">","</span> <span class="o">+</span> <span class="n">j</span> <span class="o">+</span> <span class="s">"]:"</span> <span class="o">+</span> <span class="n">matrix</span><span class="o">[</span><span class="n">i</span><span class="o">][</span><span class="n">j</span><span class="o">]);</span> + <span class="o">}</span> + <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">();</span> + <span class="o">}</span> + <span class="o">}</span> + + <span class="o">}</span></code></pre></div> + +<hr /> + +<p>For additional information regarding programmatic access to SystemML, please see the +<a href="spark-mlcontext-programming-guide.html">Spark MLContext Programming Guide</a>.</p> + + + </div> <!-- /container --> + + + + <script src="js/vendor/jquery-1.12.0.min.js"></script> + <script src="js/vendor/bootstrap.min.js"></script> + <script src="js/vendor/anchor.min.js"></script> + <script src="js/main.js"></script> + + + + + + <!-- Analytics --> + <script> + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + ga('create', 'UA-71553733-1', 'auto'); + ga('send', 'pageview'); + </script> + + + + <!-- MathJax Section --> + <script type="text/x-mathjax-config"> + MathJax.Hub.Config({ + TeX: { equationNumbers: { autoNumber: "AMS" } } + }); + </script> + <script> + // Note that we load MathJax this way to work with local file (file://), HTTP and HTTPS. + // We could use "//cdn.mathjax...", but that won't support "file://". + (function(d, script) { + script = d.createElement('script'); + script.type = 'text/javascript'; + script.async = true; + script.onload = function(){ + MathJax.Hub.Config({ + tex2jax: { + inlineMath: [ ["$", "$"], ["\\\\(","\\\\)"] ], + displayMath: [ ["$$","$$"], ["\\[", "\\]"] ], + processEscapes: true, + skipTags: ['script', 'noscript', 'style', 'textarea', 'pre'] + } + }); + }; + script.src = ('https:' == document.location.protocol ? 'https://' : 'http://') + + 'cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML'; + d.getElementsByTagName('head')[0].appendChild(script); + }(document)); + </script> + </body> +</html>
Added: systemml/site/docs/1.1.0/native-backend.html URL: http://svn.apache.org/viewvc/systemml/site/docs/1.1.0/native-backend.html?rev=1828046&view=auto ============================================================================== --- systemml/site/docs/1.1.0/native-backend.html (added) +++ systemml/site/docs/1.1.0/native-backend.html Fri Mar 30 04:31:05 2018 @@ -0,0 +1,456 @@ +<!DOCTYPE html> +<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]--> +<!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]--> +<!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]--> +<!--[if gt IE 8]><!--> <html class="no-js"> <!--<![endif]--> + <head> + <title>Using SystemML with Native BLAS support - SystemML 1.1.0</title> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> + + <meta name="description" content="Using SystemML with Native BLAS support"> + + <meta name="viewport" content="width=device-width"> + <link rel="stylesheet" href="css/bootstrap.min.css"> + <link rel="stylesheet" href="css/main.css"> + <link rel="stylesheet" href="css/pygments-default.css"> + <link rel="shortcut icon" href="img/favicon.png"> + </head> + <body> + <!--[if lt IE 7]> + <p class="chromeframe">You are using an outdated browser. <a href="http://browsehappy.com/";>Upgrade your browser today</a> or <a href="http://www.google.com/chromeframe/?redirect=true";>install Google Chrome Frame</a> to better experience this site.</p> + <![endif]--> + + <header class="navbar navbar-default navbar-fixed-top" id="topbar"> + <div class="container"> + <div class="navbar-header"> + <div class="navbar-brand brand projectlogo"> + <a href="http://systemml.apache.org/";><img class="logo" src="img/systemml-logo.png" alt="Apache SystemML" title="Apache SystemML"/></a> + </div> + <div class="navbar-brand brand projecttitle"> + <a href="http://systemml.apache.org/";>Apache SystemML<sup id="trademark">â¢</sup></a><br/> + <span class="version">1.1.0</span> + </div> + <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target=".navbar-collapse"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + </div> + <nav class="navbar-collapse collapse"> + <ul class="nav navbar-nav navbar-right"> + <li><a href="index.html">Overview</a></li> + <li><a href="https://github.com/apache/systemml";>GitHub</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation<b class="caret"></b></a> + <ul class="dropdown-menu" role="menu"> + <li><b>Running SystemML:</b></li> + <li><a href="https://github.com/apache/systemml";>SystemML GitHub README</a></li> + <li><a href="spark-mlcontext-programming-guide.html">Spark MLContext</a></li> + <li><a href="spark-batch-mode.html">Spark Batch Mode</a> + <li><a href="hadoop-batch-mode.html">Hadoop Batch Mode</a> + <li><a href="standalone-guide.html">Standalone Guide</a></li> + <li><a href="jmlc.html">Java Machine Learning Connector (JMLC)</a> + <li class="divider"></li> + <li><b>Language Guides:</b></li> + <li><a href="dml-language-reference.html">DML Language Reference</a></li> + <li><a href="beginners-guide-to-dml-and-pydml.html">Beginner's Guide to DML and PyDML</a></li> + <li><a href="beginners-guide-python.html">Beginner's Guide for Python Users</a></li> + <li><a href="python-reference.html">Reference Guide for Python Users</a></li> + <li class="divider"></li> + <li><b>ML Algorithms:</b></li> + <li><a href="algorithms-reference.html">Algorithms Reference</a></li> + <li class="divider"></li> + <li><b>Tools:</b></li> + <li><a href="debugger-guide.html">Debugger Guide</a></li> + <li><a href="developer-tools-systemml.html">IDE Guide</a></li> + <li class="divider"></li> + <li><b>Other:</b></li> + <li><a href="contributing-to-systemml.html">Contributing to SystemML</a></li> + <li><a href="engine-dev-guide.html">Engine Developer Guide</a></li> + <li><a href="troubleshooting-guide.html">Troubleshooting Guide</a></li> + <li><a href="release-process.html">Release Process</a></li> + </ul> + </li> + + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">API Docs<b class="caret"></b></a> + <ul class="dropdown-menu" role="menu"> + <li><a href="./api/java/index.html">Java</a></li> + <li><a href="./api/python/index.html">Python</a></li> + </ul> + </li> + + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Issues<b class="caret"></b></a> + <ul class="dropdown-menu" role="menu"> + <li><b>JIRA:</b></li> + <li><a href="https://issues.apache.org/jira/browse/SYSTEMML";>SystemML JIRA</a></li> + + </ul> + </li> + </ul> + </nav> + </div> + </header> + + <div class="container" id="content"> + + <h1 class="title">Using SystemML with Native BLAS support</h1> + + + <!-- + +--> + +<ul id="markdown-toc"> + <li><a href="#user-guide" id="markdown-toc-user-guide">User Guide</a> <ul> + <li><a href="#step-1-install-blas" id="markdown-toc-step-1-install-blas">Step 1: Install BLAS</a> <ul> + <li><a href="#option-1-install-intel-mkl" id="markdown-toc-option-1-install-intel-mkl">Option 1: Install Intel MKL</a></li> + <li><a href="#option-2-install-openblas" id="markdown-toc-option-2-install-openblas">Option 2: Install OpenBLAS</a></li> + </ul> + </li> + <li><a href="#step-2-provide-the-location-of-the-native-libraries" id="markdown-toc-step-2-provide-the-location-of-the-native-libraries">Step 2: Provide the location of the native libraries</a></li> + <li><a href="#step-3-set-configuration-property-to-enable-native-blas" id="markdown-toc-step-3-set-configuration-property-to-enable-native-blas">Step 3: Set configuration property to enable native BLAS</a></li> + <li><a href="#common-issues-on-linux" id="markdown-toc-common-issues-on-linux">Common issues on Linux</a></li> + </ul> + </li> + <li><a href="#developer-guide" id="markdown-toc-developer-guide">Developer Guide</a> <ul> + <li><a href="#intro-to-cmake" id="markdown-toc-intro-to-cmake">Intro to CMake</a></li> + </ul> + </li> +</ul> + +<p><br /></p> + +<h1 id="user-guide">User Guide</h1> + +<p>By default, SystemML implements all its matrix operations in Java. +This simplifies deployment especially in a distributed environment.</p> + +<p>In some cases (such as deep learning), the user might want to use native BLAS +rather than SystemML’s internal Java library for performing single-node +operations such matrix multiplication, convolution, etc.</p> + +<p>To allow SystemML to use native BLAS rather than internal Java library, +please set the configuration property <code>sysml.native.blas</code> to <code>auto</code>. +Other possible options are: <code>mkl</code>, <code>openblas</code> and <code>none</code>. +The first two options will only attempt to use the respective BLAS libraries.</p> + +<p>By default, SystemML will first attempt to use Intel MKL (if installed) +and then OpenBLAS (if installed). +If both Intel MKL and OpenBLAS are not available, SystemML +falls back to its internal Java library.</p> + +<p>The current version of SystemML only supports BLAS on <strong>Linux</strong> machines.</p> + +<h2 id="step-1-install-blas">Step 1: Install BLAS</h2> + +<p>If BLAS is already installed, please skip this step.</p> + +<h3 id="option-1-install-intel-mkl">Option 1: Install Intel MKL</h3> + +<p>Download and install the <a href="https://software.intel.com/sites/campaigns/nest/";>community version of Intel MKL</a>. +Intel requires you to first register your email address and then sends the download link to your email address +with license key. Since we use MKL DNN primitives, we depend on Intel MKL version 2017 or higher.</p> + +<ul> + <li>Linux users will have to extract the downloaded <code>.tgz</code> file, execute <code>install.sh</code> and follow the guided setup.</li> +</ul> + +<h3 id="option-2-install-openblas">Option 2: Install OpenBLAS</h3> + +<p>The default OpenBLAS (via yum/apt-get) uses its internal threading rather than OpenMP, +which can lead to performance degradation when using SystemML. So, instead we recommend that you +compile OpenBLAS from the source instead of installing it with <code>yum</code> or <code>apt-get</code>.</p> + +<p>The steps to install OpenBLAS v0.2.20:</p> + +<p><code>bash +wget https://github.com/xianyi/OpenBLAS/archive/v0.2.20.tar.gz +tar -xzf v0.2.20.tar.gz +cd OpenBLAS-0.2.20/ +make clean +make USE_OPENMP=1 +sudo make install +# After installation, you may also want to add `/opt/OpenBLAS/lib` to your LD_LIBRARY_PATH or `java.library.path`. +</code></p> + +<p>When using OpenBLAS, we also depend on GNU OpenMP (gomp) which will be installed by GCC. +To find the location of <code>gomp</code> on your system, please use the command <code>ldconfig -p | grep libgomp</code>. +If gomp is available as <code>/lib64/libgomp.so.1</code> instead of <code>/lib64/libgomp.so</code>, +please add a softlink to it:</p> + +<p><code>bash +# Centos/RedHat +sudo yum install gcc-c++ +# Ubuntu +sudo apt-get install g++ +sudo ln -s /lib64/libgomp.so.1 /lib64/libgomp.so +</code></p> + +<h2 id="step-2-provide-the-location-of-the-native-libraries">Step 2: Provide the location of the native libraries</h2> + +<ol> + <li> + <p>Pass the location of the native libraries using command-line options:</p> + + <ul> + <li><a href="http://spark.apache.org/docs/latest/configuration.html";>Spark</a>: <code>--conf spark.executorEnv.LD_LIBRARY_PATH=/path/to/blas-n-other-dependencies</code></li> + <li>Java: <code>-Djava.library.path=/path/to/blas-n-other-dependencies</code></li> + </ul> + </li> + <li> + <p>Alternatively, you can add the location of the native libraries (i.e. BLAS and other dependencies) +to the environment variable <code>LD_LIBRARY_PATH</code> (on Linux). +If you want to use SystemML with Spark, please add the following line to <code>spark-env.sh</code> +(or to the bash profile).</p> + + <p>export LD_LIBRARY_PATH=/path/to/blas-n-other-dependencies</p> + </li> +</ol> + +<p>In cloud environment where you may not be able to set <code>LD_LIBRARY_PATH</code> or <code>spark.executorEnv.LD_LIBRARY_PATH</code> +before starting spark, you can use set the configuration property <code>sysml.native.blas.directory</code>. For example:</p> + +<p><code>python +mlCtx.setConfigProperty("sysml.native.blas.directory", "/path/to/blas-n-other-dependencies") +</code></p> + +<h2 id="step-3-set-configuration-property-to-enable-native-blas">Step 3: Set configuration property to enable native BLAS</h2> + +<p>The configuration property <code>sysml.native.blas</code> can be either set in the file <code>SystemML-config.xml</code> +or using <code>setConfigProperty</code> method of <code>MLContext</code> or <code>mllearn</code> classes. For example:</p> + +<p><code>python +mlCtx.setConfigProperty("sysml.native.blas", "openblas") +</code></p> + +<h2 id="common-issues-on-linux">Common issues on Linux</h2> + +<ul> + <li>Unable to load <code>gomp</code>.</li> +</ul> + +<p>First make sure if gomp is available on your system.</p> + +<pre><code>ldconfig -p | grep libgomp +</code></pre> + +<p>If the above command returns no results, then you may have to install <code>gcc</code>. +On the other hand, if the above command only returns libgomp with major suffix (such as <code>so.1</code>), +then please execute the below command:</p> + +<pre><code>sudo ln -s /lib64/libgomp.so.1 /usr/lib64/libgomp.so +</code></pre> + +<ul> + <li>Unable to load <code>mkl_rt</code>.</li> +</ul> + +<p>By default, Intel MKL libraries will be installed in the location <code>/opt/intel/mkl/lib/intel64/</code>. +Make sure that this path is accessible to Java as per instructions provided in the above section.</p> + +<ul> + <li>Unable to load <code>openblas</code>.</li> +</ul> + +<p>By default, OpenBLAS libraries will be installed in the location <code>/opt/OpenBLAS/lib/</code>. +Make sure that this path is accessible to Java as per instructions provided in the above section.</p> + +<ul> + <li>Using OpenBLAS without OpenMP can lead to performance degradation when using SystemML.</li> +</ul> + +<p>You can check if the OpenBLAS on you system is compiled with OpenMP or not using following commands: +If you don’t see any output after the second command, then OpenBLAS installed on your system is using its internal threading. +In this case, we highly recommend that you reinstall OpenBLAS using the above commands.</p> + +<pre><code>$ ldconfig -p | grep libopenblas.so +libopenblas.so (libc6,x86-64) => /opt/OpenBLAS/lib/libopenblas.so +$ ldd /opt/OpenBLAS/lib/libopenblas.so | grep libgomp +libgomp.so.1 => /lib64/libgomp.so.1 +</code></pre> + +<ul> + <li>Using MKL can lead to slow performance for convolution instruction.</li> +</ul> + +<p>We noticed that double-precision MKL DNN primitives for convolution instruction +is considerably slower than than the corresponding single-precision MKL DNN primitives +as of MKL 2017 Update 1. We anticipate that this performance bug will be fixed in the future MKL versions. +Until then or until SystemML supports single-precision matrices, we recommend that you use OpenBLAS when using script with <code>conv2d</code>.</p> + +<p>Here are the end-to-end runtime performance in seconds of 10 <code>conv2d</code> operations +on randomly generated 64 images of size 256 X 256 with sparsity 0.9 +and 32 filter of size 5x5 with stride = [1,1] and pad=[1,1].</p> + +<table> + <thead> + <tr> + <th> </th> + <th>MKL</th> + <th>OpenBLAS</th> + </tr> + </thead> + <tbody> + <tr> + <td>Single-precision, channels=3</td> + <td>5.144</td> + <td>7.918</td> + </tr> + <tr> + <td>Double-precision, channels=3</td> + <td>12.599</td> + <td>8.688</td> + </tr> + <tr> + <td>Single-precision, channels=32</td> + <td>10.765</td> + <td>21.963</td> + </tr> + <tr> + <td>Double-precision, channels=32</td> + <td>71.118</td> + <td>34.881</td> + </tr> + </tbody> +</table> + +<p>Setup used in the above experiment: +1. Intel MKL 2017 Update 1, OpenBLAS compiled with GNU OpenMP from source using <code>g++</code>. +2. CPU: <code>Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz</code></p> + +<h1 id="developer-guide">Developer Guide</h1> + +<p>This section describes how to compile shared libraries in the folder <code>src/main/cpp/lib</code>. +This is required when the developer makes changes to cpp directory or while validating the source package during the release process.</p> + +<h2 id="intro-to-cmake">Intro to CMake</h2> +<p>If you are familiar with cmake, skip this section.</p> + +<p>In a regular project with a Makefile, the compiled object files are placed in the same directory as the source. +Sometimes we don’t want to pollute the source tree. We might also want to have different binaries for different configurations. For instance, if we want to link a binary with separate libraries. +CMake supports out of source tree builds. As an illustration, you can create a directory called “BUILD” and invoke cmake like so : <code>cmake <path/to/source></code>. The makefile and other config files are placed in this “BUILD” directory. You can now say <code>make</code> and the compiled objects and binary files are created in this directory. You can then create another “BUILD2” directory and repeat the process. +You can pass options to cmake as well. In this instance, it might be to specify whether to build with Intel MKL or OpenBLAS. This can be done from the command line with a “-D” appended to it, but more interestingly, it can also be done form a n-curses GUI which is invoked as <code>ccmake <path/to/source></code>. (You may need to install this separately). +Also, the C, C++ compilers and their flags are picked up by cmake when set in standard environment variables. These are respectively <code>CC</code>, <code>CXX</code>, <code>CFLAGS</code> & <code>CXFLAGS</code>. As an example, they may be specified as:</p> + +<pre><code>CXX=gcc-6 cmake .. +</code></pre> + +<p>For this project, I typically make a directory in the <code>cpp</code> folder (this folder) and name it the config I use. For instance, <code>INTEL</code> for Intel MKL and <code>OPENBLAS</code> for OpenBLAS.</p> + +<ul> + <li> + <p>Install <code>g++</code>, OpenBLAS and MKL using the above instructions</p> + </li> + <li> + <p>Set <code>JAVA_HOME</code> to JDK.</p> + </li> +</ul> + +<p><code>bash +export JAVA_HOME=<path to JDK 1.8> +</code></p> + +<ul> + <li>Install cmake</li> +</ul> + +<p><code>bash +# Centos/RedHat +sudo yum install cmake3 +# Ubuntu +sudo apt-get install cmake +</code></p> + +<ul> + <li>Compile the libs using the below script.</li> +</ul> + +<p><code>bash +mkdir INTEL && cd INTEL +cmake -DUSE_INTEL_MKL=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_CXX_FLAGS="-DUSE_GNU_THREADING -m64" .. +make install +cd .. +mkdir OPENBLAS && cd OPENBLAS +cmake -DUSE_OPEN_BLAS=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_CXX_FLAGS="-m64" .. +make install +cd .. +# The below script helps maintain this document as well as avoid accidental inclusion of non-standard dependencies. +./check-dependency-linux-x86_64.sh +</code></p> + +<p>The generated library files are placed in src/main/cpp/lib. This location can be changed from the CMakeLists.txt file.</p> + +<p>The above script also validates whether additional dependencies have been added while compiling and warns the developer.<br /> +The current set of dependencies other than MKL and OpenBLAS, are as follows:</p> + +<ul> + <li>GNU Standard C++ Library: <code>libstdc++.so.6</code></li> + <li>GCC version 4.8 shared support library: <code>libgcc_s.so.1</code></li> + <li>The GNU libc libraries: <code>libm.so.6, libdl.so.2, libc.so.6, libpthread.so.0</code></li> + <li>GCC OpenMP v3.0 shared support library: <code>libgomp.so.1</code></li> + <li>Additional OpenBLAS dependencies: Fortran runtime (<code>libgfortran.so.3</code>) and GCC <code>__float128</code> shared support library (<code>libquadmath.so.0</code>)</li> +</ul> + +<p>If CMake cannot detect your OpenBLAS installation, set the <code>OpenBLAS_HOME</code> environment variable to the OpenBLAS Home.</p> + + + + </div> <!-- /container --> + + + + <script src="js/vendor/jquery-1.12.0.min.js"></script> + <script src="js/vendor/bootstrap.min.js"></script> + <script src="js/vendor/anchor.min.js"></script> + <script src="js/main.js"></script> + + + + + + <!-- Analytics --> + <script> + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + ga('create', 'UA-71553733-1', 'auto'); + ga('send', 'pageview'); + </script> + + + + <!-- MathJax Section --> + <script type="text/x-mathjax-config"> + MathJax.Hub.Config({ + TeX: { equationNumbers: { autoNumber: "AMS" } } + }); + </script> + <script> + // Note that we load MathJax this way to work with local file (file://), HTTP and HTTPS. + // We could use "//cdn.mathjax...", but that won't support "file://". + (function(d, script) { + script = d.createElement('script'); + script.type = 'text/javascript'; + script.async = true; + script.onload = function(){ + MathJax.Hub.Config({ + tex2jax: { + inlineMath: [ ["$", "$"], ["\\\\(","\\\\)"] ], + displayMath: [ ["$$","$$"], ["\\[", "\\]"] ], + processEscapes: true, + skipTags: ['script', 'noscript', 'style', 'textarea', 'pre'] + } + }); + }; + script.src = ('https:' == document.location.protocol ? 'https://' : 'http://') + + 'cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML'; + d.getElementsByTagName('head')[0].appendChild(script); + }(document)); + </script> + </body> +</html> Added: systemml/site/docs/1.1.0/python-performance-test.html URL: http://svn.apache.org/viewvc/systemml/site/docs/1.1.0/python-performance-test.html?rev=1828046&view=auto ============================================================================== --- systemml/site/docs/1.1.0/python-performance-test.html (added) +++ systemml/site/docs/1.1.0/python-performance-test.html Fri Mar 30 04:31:05 2018 @@ -0,0 +1,453 @@ +<!DOCTYPE html> +<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]--> +<!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]--> +<!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]--> +<!--[if gt IE 8]><!--> <html class="no-js"> <!--<![endif]--> + <head> + <title>SystemML Performance Testing - SystemML 1.1.0</title> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> + + <meta name="description" content="Description of SystemML performance testing."> + + <meta name="viewport" content="width=device-width"> + <link rel="stylesheet" href="css/bootstrap.min.css"> + <link rel="stylesheet" href="css/main.css"> + <link rel="stylesheet" href="css/pygments-default.css"> + <link rel="shortcut icon" href="img/favicon.png"> + </head> + <body> + <!--[if lt IE 7]> + <p class="chromeframe">You are using an outdated browser. <a href="http://browsehappy.com/";>Upgrade your browser today</a> or <a href="http://www.google.com/chromeframe/?redirect=true";>install Google Chrome Frame</a> to better experience this site.</p> + <![endif]--> + + <header class="navbar navbar-default navbar-fixed-top" id="topbar"> + <div class="container"> + <div class="navbar-header"> + <div class="navbar-brand brand projectlogo"> + <a href="http://systemml.apache.org/";><img class="logo" src="img/systemml-logo.png" alt="Apache SystemML" title="Apache SystemML"/></a> + </div> + <div class="navbar-brand brand projecttitle"> + <a href="http://systemml.apache.org/";>Apache SystemML<sup id="trademark">â¢</sup></a><br/> + <span class="version">1.1.0</span> + </div> + <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target=".navbar-collapse"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + </div> + <nav class="navbar-collapse collapse"> + <ul class="nav navbar-nav navbar-right"> + <li><a href="index.html">Overview</a></li> + <li><a href="https://github.com/apache/systemml";>GitHub</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation<b class="caret"></b></a> + <ul class="dropdown-menu" role="menu"> + <li><b>Running SystemML:</b></li> + <li><a href="https://github.com/apache/systemml";>SystemML GitHub README</a></li> + <li><a href="spark-mlcontext-programming-guide.html">Spark MLContext</a></li> + <li><a href="spark-batch-mode.html">Spark Batch Mode</a> + <li><a href="hadoop-batch-mode.html">Hadoop Batch Mode</a> + <li><a href="standalone-guide.html">Standalone Guide</a></li> + <li><a href="jmlc.html">Java Machine Learning Connector (JMLC)</a> + <li class="divider"></li> + <li><b>Language Guides:</b></li> + <li><a href="dml-language-reference.html">DML Language Reference</a></li> + <li><a href="beginners-guide-to-dml-and-pydml.html">Beginner's Guide to DML and PyDML</a></li> + <li><a href="beginners-guide-python.html">Beginner's Guide for Python Users</a></li> + <li><a href="python-reference.html">Reference Guide for Python Users</a></li> + <li class="divider"></li> + <li><b>ML Algorithms:</b></li> + <li><a href="algorithms-reference.html">Algorithms Reference</a></li> + <li class="divider"></li> + <li><b>Tools:</b></li> + <li><a href="debugger-guide.html">Debugger Guide</a></li> + <li><a href="developer-tools-systemml.html">IDE Guide</a></li> + <li class="divider"></li> + <li><b>Other:</b></li> + <li><a href="contributing-to-systemml.html">Contributing to SystemML</a></li> + <li><a href="engine-dev-guide.html">Engine Developer Guide</a></li> + <li><a href="troubleshooting-guide.html">Troubleshooting Guide</a></li> + <li><a href="release-process.html">Release Process</a></li> + </ul> + </li> + + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">API Docs<b class="caret"></b></a> + <ul class="dropdown-menu" role="menu"> + <li><a href="./api/java/index.html">Java</a></li> + <li><a href="./api/python/index.html">Python</a></li> + </ul> + </li> + + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Issues<b class="caret"></b></a> + <ul class="dropdown-menu" role="menu"> + <li><b>JIRA:</b></li> + <li><a href="https://issues.apache.org/jira/browse/SYSTEMML";>SystemML JIRA</a></li> + + </ul> + </li> + </ul> + </nav> + </div> + </header> + + <div class="container" id="content"> + + <h1 class="title">SystemML Performance Testing</h1> + + + <!-- + +--> + +<ul id="markdown-toc"> + <li><a href="#performance-testing-algorithms-user-manual" id="markdown-toc-performance-testing-algorithms-user-manual">Performance Testing Algorithms User Manual</a> <ul> + <li><a href="#architecture" id="markdown-toc-architecture">Architecture</a></li> + <li><a href="#adding-new-algorithms" id="markdown-toc-adding-new-algorithms">Adding New Algorithms</a></li> + <li><a href="#current-default-settings" id="markdown-toc-current-default-settings">Current Default Settings</a></li> + <li><a href="#examples" id="markdown-toc-examples">Examples</a></li> + <li><a href="#google-sheets-api" id="markdown-toc-google-sheets-api">Google sheets API</a></li> + <li><a href="#result-consolidation-and-plotting" id="markdown-toc-result-consolidation-and-plotting">Result Consolidation and Plotting</a></li> + <li><a href="#operational-notes" id="markdown-toc-operational-notes">Operational Notes</a></li> + <li><a href="#troubleshooting" id="markdown-toc-troubleshooting">Troubleshooting</a></li> + </ul> + </li> +</ul> + +<h1 id="performance-testing-algorithms-user-manual">Performance Testing Algorithms User Manual</h1> + +<p>This user manual contains details on how to conduct automated performance tests. Work was mostly done in this <a href="https://github.com/apache/systemml/pull/537";>PR</a> and part of <a href="https://issues.apache.org/jira/browse/SYSTEMML-1451";>SYSTEMML-1451</a>. Our aim was to move from existing <code>bash</code> based performance tests to automatic <code>python</code> based automatic performance tests.</p> + +<h2 id="architecture">Architecture</h2> + +<p>Our performance tests suit contains <code>7</code> families namely <code>binomial</code>, <code>multinomial</code>, <code>stats1</code>, <code>stats2</code>, <code>regression1</code>, <code>regression2</code>, <code>clustering</code>. Within these families we have algorithms grouped under it. Typically a family is a set of algorithms that require the same data generation script.</p> + +<ul> + <li>Exceptions: <code>regression1</code>, <code>regression2</code> and <code>binomial</code>. We decide to include these algorithms in separate families to keep the architecture simple.</li> +</ul> + +<p><img src="img/performance-test/perf_test_arch.png" alt="System ML Architecture" /></p> + +<p>On a very high level use construct a string with arguments required to run each operation. Once this string is constructed we use the subprocess module to execute this string and extract time from the standard out.</p> + +<p>We also use <code>json</code> module write our configurations to a json file. This ensure that our operations are easy to debug.</p> + +<p>We have <code>7</code> files in performance test suit:</p> + +<ul> + <li>Entry File <code>run_perftest.py</code></li> + <li>Supporting Files <code>datagen.py</code>, <code>train.py</code>, <code>predict.py</code></li> + <li>Utility Files <code>utils_exec.py</code>, <code>utils_fs.py</code>, <code>utils_misc.py</code></li> +</ul> + +<p><code>datagen.py</code>, <code>train.py</code> and <code>predict.py</code> generate a dictionary. Our key is the name of algorithm being processed and values is a list with path(s) where all the data required is present. We define this dictionary as a configuration packet.</p> + +<p>We will describe each of them in detail the following sections below.</p> + +<p><code>run_perftest.py</code> at a high level creates <code>algos_to_run</code> list. This list is tuple with key as algorithm and value as family to be executed in our performance test.</p> + +<p>In <code>datagen.py</code> script we have all functions required to generate data. We return the required configuration packet as a result of this script, that contains key as the <code>data-gen</code> script to run and values with location to read data-gen json files from.</p> + +<p>In <code>train.py</code> script we have functions required to generate training output. We return the required configuration packet as a result of this script, that contains key as the algorithm to run and values with location to read training json files from.</p> + +<p>The file <code>predict.py</code> contains all functions for all algorithms in the performance test that contain predict script. We return the required configuration packet as a result of this script, that contains key as the algorithm to run and values with location to read predict json files from.</p> + +<p>In the file(s) <code>utils_*.py</code> we have all the helper functions required in our performance test. These functions do operations like write <code>json</code> files, extract time from std out etc.</p> + +<h2 id="adding-new-algorithms">Adding New Algorithms</h2> + +<p>While adding a new algorithm we need know if it has to be part of the any pre existing family. If this algorithm depends on a new data generation script we would need to create a new family. Steps below to take below to add a new algorithm.</p> + +<p>Following changes to <code>run_perftest.py</code>:</p> + +<ul> + <li>Add the algorithm to <code>ML_ALGO</code> dictionary with its respective family.</li> + <li>Add the name of the data generation script in <code>ML_GENDATA</code> dictionary if it does not exist already.</li> + <li>Add the name of the training script in <code>ML_TRAIN</code> dictionary.</li> + <li>Add the name of the prediction script in <code>ML_PREDICT</code> incase the prediction script exists.</li> +</ul> + +<p>Following changes to <code>datagen.py</code>:</p> + +<ul> + <li>Check if the data generation algorithm has the ability to generate dense and sparse data. If it had the ability to generate only dense data add the corresponding family to <code>FAMILY_NO_MATRIX_TYPE</code> list.</li> + <li>Create a function with <code>familyname + _ + datagen</code> with same input arguments namely <code>matrix_dim</code>, <code>matrix_type</code>, <code>datagen_dir</code>.</li> + <li>Constants and arguments for the data generation script should be defined in function.</li> + <li>Test the perf test with the algorithm with <code>mode</code> as <code>data-gen</code>.</li> + <li>Check output folders, json files, output log.</li> + <li>Check for possible errors if these folders/files do not exist. (See the troubleshooting section).</li> +</ul> + +<p>Following changes to <code>train.py</code>:</p> + +<ul> + <li>Create the function with <code>familyname + _ + algoname + _ + train</code>.</li> + <li>This function needs to have the following arguments <code>save_folder_name</code>, <code>datagen_dir</code>, <code>train_dir</code>.</li> + <li>Constants and arguments for the training script should be defined in function.</li> + <li>Make sure that the return type is a list.</li> + <li>Test the perf test with the algorithm with <code>mode</code> as <code>train</code>.</li> + <li>Check output folders, json files, output log.</li> + <li>Check for possible errors if these folders/files do not exist. (See the troubleshooting section).</li> +</ul> + +<p>Following changes to <code>predict.py</code>:</p> + +<ul> + <li>Create the function with <code>algoname + _ + predict</code>.</li> + <li>This function needs to have the following arguments <code>save_file_name</code>, <code>datagen_dir</code>, <code>train_dir</code>, <code>predict_dir</code>.</li> + <li>Constants and arguments for the training script should be defined in function.</li> + <li>Test the perf test with the algorithm with <code>mode</code> as <code>predict</code>.</li> + <li>Check output folders, json files, output log.</li> + <li>Check for possible errors if these folders/files do not exist. (Please see the troubleshooting section).</li> + <li>Note: <code>predict.py</code> will not be executed if the current algorithm being executed does not have predict script.</li> +</ul> + +<h2 id="current-default-settings">Current Default Settings</h2> + +<p>Default setting for our performance test below:</p> + +<ul> + <li>Matrix size to 10,000 rows and 100 columns.</li> + <li>Execution mode <code>singlenode</code>.</li> + <li>Operation modes <code>data-gen</code>, <code>train</code> and <code>predict</code> in sequence.</li> + <li>Matrix type set to <code>all</code>. Which will generate <code>dense</code>, <code>sparse</code> matrices for all relevant algorithms.</li> +</ul> + +<h2 id="examples">Examples</h2> + +<p>Some examples of SystemML performance test with arguments shown below:</p> + +<p><code>./scripts/perftest/python/run_perftest.py --family binomial clustering multinomial regression1 regression2 stats1 stats2 +</code> +Test all algorithms with default parameters.</p> + +<p><code>./scripts/perftest/python/run_perftest.py --exec-type hybrid_spark --family binomial clustering multinomial regression1 regression2 stats1 stats2 +</code> +Test all algorithms in hybrid spark execution mode.</p> + +<p><code>./scripts/perftest/python/run_perftest.py --exec-type hybrid_spark --family clustering --mat-shape 10k_5 10k_10 10k_50 +</code> +Test all algorithms in <code>clustering</code> family in hybrid spark execution mode, on different matrix size <code>10k_10</code> (10,000 rows and 5 columns), <code>10k_10</code> and <code>10k_50</code>.</p> + +<p><code>./scripts/perftest/python/run_perftest.py --algo Univar-Stats bivar-stats +</code> +Run performance test for following algorithms <code>Univar-Stats</code> and <code>bivar-stats</code>.</p> + +<p><code>./scripts/perftest/python/run_perftest.py --algo m-svm --family multinomial binomial --mode data-gen train +</code> +Run performance test for the algorithms <code>m-svm</code> with <code>multinomial</code> family. Run only data generation and training operations.</p> + +<p><code>./scripts/perftest/python/run_perftest.py --family regression2 --filename new_log +</code> +Run performance test for all algorithms under the family <code>regression2</code> and log with filename <code>new_log</code>.</p> + +<p><code>./scripts/perftest/python/run_perftest.py --family binomial clustering multinomial regression1 regression2 stats1 stats2 --config-dir /Users/krishna/open-source/systemml/scripts/perftest/temp3 --temp-dir hdfs://localhost:9000/temp3</code> +Run performance test for all algorithms using HDFS.</p> + +<h2 id="google-sheets-api">Google sheets API</h2> + +<p>Steps below to configure google client API:</p> + +<ul> + <li>Navigate to <a href="https://console.developers.google.com/apis/";>Google APIs Console</a>.</li> + <li>Create a new project.</li> + <li>Click Enable API. Search for and enable the Google Drive API.</li> + <li>Create credentials for a Web Server to access Application Data.</li> + <li>Name the service account and grant it a Project Role of Editor.</li> + <li>Download the JSON file.</li> + <li>Copy the JSON file to your code directory and rename it to client_secret.json</li> +</ul> + +<p>Steps below to configure google sheets:</p> + +<ul> + <li>Create a new spread sheet with google sheets.</li> + <li>Create seperate sheets for <code>singlenode</code> and <code>hybrid_spark</code>.</li> + <li>Find the client_email inside client_secret.json and save it.</li> + <li>Back in your spreadsheet, click the Share button in the top right, and paste the client email into the People field to give it edit rights for each sheet.</li> + <li>Click Send</li> +</ul> + +<h2 id="result-consolidation-and-plotting">Result Consolidation and Plotting</h2> +<p>We have two scripts, <code>stats.py</code> forpulling results from google docs and <code>update.py</code> to updating results to google docs or local file system.</p> + +<p>Example of <code>update.py</code> would be below +<code>./scripts/perftest/python/google_docs/update.py --file ../../temp/perf_test_singlenode.out --exec-type singlenode --tag 2 --append test.csv</code> +The arguments being <code>--file</code> path of the perf-test output, <code>--exec-type</code> execution mode used to generate the perf-test output, <code>--tag</code> being the realease version or a unique name, <code>--append</code> being an optional argument that would append the a local csv file. If instead of <code>--append</code> the <code>--auth</code> argument needs the location of the <code>google api key</code> file.</p> + +<p>Example of <code>stats.py</code> below +` ./stats.py –auth ../key/client_json.json –exec-type singlenode –plot stats1_data-gen_none_dense_10k_100<code> +</code>–plot` argument needs the name of the composite key that you would like to compare results over. If this argument is not specified the results would be grouped by keys.</p> + +<h2 id="operational-notes">Operational Notes</h2> + +<p>All performance test depend mainly on two scripts for execution <code>systemml-standalone.py</code> and <code>systemml-spark-submit.py</code>. Incase we need to change standalone or spark parameters we need to manually change these parameters in their respective scripts.</p> + +<p>Constants like <code>DATA_FORMAT</code> currently set to <code>csv</code> and <code>MATRIX_TYPE_DICT</code> with <code>density</code> set to <code>0.9</code> and <code>sparsity</code> set to <code>0.01</code> are hardcoded in the performance test scripts. They can be changed easily as they are defined at the top of their respective operational scripts.</p> + +<p>The logs contain the following information below comma separated.</p> + +<table> + <thead> + <tr> + <th>algorithm</th> + <th>run_type</th> + <th>intercept</th> + <th>matrix_type</th> + <th>data_shape</th> + <th>time_sec</th> + </tr> + </thead> + <tbody> + <tr> + <td>multinomial</td> + <td>data-gen</td> + <td>0</td> + <td>10k_100</td> + <td>dense</td> + <td>0.33</td> + </tr> + <tr> + <td>MultiLogReg</td> + <td>train</td> + <td>0</td> + <td>10k_100</td> + <td>dense</td> + <td>6.956</td> + </tr> + <tr> + <td>MultiLogReg</td> + <td>predict</td> + <td>0</td> + <td>10k_100</td> + <td>dense</td> + <td>4.780</td> + </tr> + </tbody> +</table> + +<p>These logs and config <code>json</code> files can be found in <code>temp</code> folder (<code>$SYSTEMML_HOME/scripts/perftest/temp</code>) in-case not overridden by <code>--config-dir</code>.</p> + +<p><code>--temp-dir</code> by default points to local file system. We can change this to point to a hdfs path by <code>--temp-dir hdfs://localhost:9000/temp</code> where all files generated during execution will be saved.</p> + +<p>Every time a script executes in <code>data-gen</code> mode successfully, we write a <code>_SUCCESS</code> file. If this file exists we ensures that re-runs of the same script is not possible. Support for configuration options like <code>-stats</code>, <code>-explain</code>, <code>--conf</code> have also been added.</p> + +<p>Results obtained by our performance tests can be automatically uploaded to google docs.</p> + +<p><code>./update.py --file ../temp/singlenode.out --exec-mode singlenode --auth client_json.json --tag 1.0</code></p> + +<p>In the example above <code>--tag</code> can be a major/minor systemml version and <code>--auth</code> points to the <code>json</code> key required by <code>google docs</code>.</p> + +<p>Currently we only support time difference between algorithms in different versions. This can be obtained by running the script below +<code>./stats.py --auth client_json.json --exec-mode singlenode --tags 1.0 2.0</code></p> + +<p>We pass different <code>matrix shapes</code> using <code>--mat-shape</code> argument.</p> + +<table> + <thead> + <tr> + <th>Matrix Shape</th> + <th>Approximate Data Size</th> + </tr> + </thead> + <tbody> + <tr> + <td>10k_1k</td> + <td>80MB</td> + </tr> + <tr> + <td>100k_1k</td> + <td>800MB</td> + </tr> + <tr> + <td>1M_1k</td> + <td>8GB</td> + </tr> + <tr> + <td>10M_1k</td> + <td>80GB</td> + </tr> + <tr> + <td>100M_1k</td> + <td>800GB</td> + </tr> + </tbody> +</table> + +<p>For example the command below runs performance test for all data sizes described above +<code>run_perftest.py --family binomial clustering multinomial regression1 regression2 stats1 stats2 --mat-shape 10k_1k 100k_1k 1M_1k 10M_1k 100M_1k --master yarn-client --temp-dir hdfs://localhost:9000/user/systemml</code></p> + +<p>By default data generated in <code>hybrid_spark</code> execution mode is in the current users <code>hdfs</code> home directory.</p> + +<p>Note: Please use this command <code>pip3 install -r requirements.txt</code> before using the perftest scripts.</p> + +<h2 id="troubleshooting">Troubleshooting</h2> + +<p>We can debug the performance test by making changes in the following locations based on</p> + +<ul> + <li>Please see <code>utils_exec.py</code> function <code>subprocess_exec</code>.</li> + <li>Please see <code>run_perftest.py</code>. Changing the verbosity level to <code>0</code> allows us to log more information while the script runs.</li> + <li>Eyeballing the json files generated and making sure the configuration arguments are correct.</li> +</ul> + + + </div> <!-- /container --> + + + + <script src="js/vendor/jquery-1.12.0.min.js"></script> + <script src="js/vendor/bootstrap.min.js"></script> + <script src="js/vendor/anchor.min.js"></script> + <script src="js/main.js"></script> + + + + + + <!-- Analytics --> + <script> + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + ga('create', 'UA-71553733-1', 'auto'); + ga('send', 'pageview'); + </script> + + + + <!-- MathJax Section --> + <script type="text/x-mathjax-config"> + MathJax.Hub.Config({ + TeX: { equationNumbers: { autoNumber: "AMS" } } + }); + </script> + <script> + // Note that we load MathJax this way to work with local file (file://), HTTP and HTTPS. + // We could use "//cdn.mathjax...", but that won't support "file://". + (function(d, script) { + script = d.createElement('script'); + script.type = 'text/javascript'; + script.async = true; + script.onload = function(){ + MathJax.Hub.Config({ + tex2jax: { + inlineMath: [ ["$", "$"], ["\\\\(","\\\\)"] ], + displayMath: [ ["$$","$$"], ["\\[", "\\]"] ], + processEscapes: true, + skipTags: ['script', 'noscript', 'style', 'textarea', 'pre'] + } + }); + }; + script.src = ('https:' == document.location.protocol ? 'https://' : 'http://') + + 'cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML'; + d.getElementsByTagName('head')[0].appendChild(script); + }(document)); + </script> + </body> +</html>
