Repository: incubator-systemml Updated Branches: refs/heads/master d9de19295 -> 168ff41ec
[SYSTEMML-1606] Update notebook samples with latest code Project: http://git-wip-us.apache.org/repos/asf/incubator-systemml/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-systemml/commit/168ff41e Tree: http://git-wip-us.apache.org/repos/asf/incubator-systemml/tree/168ff41e Diff: http://git-wip-us.apache.org/repos/asf/incubator-systemml/diff/168ff41e Branch: refs/heads/master Commit: 168ff41ec186d8ccb68f2879d945b5fd4fc8db8a Parents: d9de192 Author: Arvind Surve <ac...@yahoo.com> Authored: Tue May 16 00:25:02 2017 -0700 Committer: Arvind Surve <ac...@yahoo.com> Committed: Tue May 16 00:25:02 2017 -0700 ---------------------------------------------------------------------- samples/jupyter-notebooks/Autoencoder.ipynb | 51 ++++++++-- .../Deep_Learning_Image_Classification.ipynb | 45 ++++++-- .../Linear_Regression_Algorithms_Demo.ipynb | 102 ++++++++++++------- .../SystemML-PySpark-Recommendation-Demo.ipynb | 46 +++++++-- samples/jupyter-notebooks/tutorial1.ipynb | 28 ++--- 5 files changed, 193 insertions(+), 79 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/168ff41e/samples/jupyter-notebooks/Autoencoder.ipynb ---------------------------------------------------------------------- diff --git a/samples/jupyter-notebooks/Autoencoder.ipynb b/samples/jupyter-notebooks/Autoencoder.ipynb index 47e8e03..0290b12 100644 --- a/samples/jupyter-notebooks/Autoencoder.ipynb +++ b/samples/jupyter-notebooks/Autoencoder.ipynb @@ -12,23 +12,39 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Install SystemML Python package and jar file" + "# Install SystemML Python package and jar file\n", + "You can install SystemML Python tgz file either from \n", + " 1. Distribution location (https://dist.apache.org/repos/dist/release/incubator/systemml/) or \n", + " 2. Latest daily built code (https://sparktc.ibmcloud.com/repo/latest/) or\n", + " 3. From your local system if you have extracted recent code and built locally. \n", + " (e.g. ~/git/incubator-systemml/target/systemml-1.0.0-incubating-SNAPSHOT-python.tgz)\n", + " \n", + "### This notebook is supported with SystemML 0.14.0 and above." ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ - "#!pip install --user systemml>=0.14.0\n", - "!pip install ~/git/incubator-systemml/target/systemml-1.0.0-incubating-SNAPSHOT-python.tgz" + "# !pip install --user systemml>=0.14.0\n", + "\n", + "!pip install https://dist.apache.org/repos/dist/release/incubator/systemml/0.14.0-incubating/systemml-0.14.0-incubating-python.tgz\n";, + " \n", + "# !pip install https://sparktc.ibmcloud.com/repo/latest/systemml-1.0.0-incubating-SNAPSHOT-python.tgz\n";, + " \n", + "# !pip install ~/git/incubator-systemml/target/systemml-1.0.0-incubating-SNAPSHOT-python.tgz" ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "!pip show systemml" @@ -37,7 +53,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "import pandas as pd\n", @@ -77,7 +95,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "X_pd = pd.DataFrame(range(1, 2001,1),dtype=float).values.reshape(100,20)\n", @@ -91,7 +111,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "!ls -l /tmp/data/Input" @@ -100,7 +122,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "autoencoderURL = \"https://raw.githubusercontent.com/apache/incubator-systemml/master/scripts/staging/autoencoder-2layer.dml\"\n";, @@ -120,7 +144,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "!ls -l /tmp/data/Output" @@ -137,6 +163,7 @@ "cell_type": "code", "execution_count": null, "metadata": { + "collapsed": true, "scrolled": true }, "outputs": [], @@ -168,7 +195,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "!yes | pip uninstall systemml" http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/168ff41e/samples/jupyter-notebooks/Deep_Learning_Image_Classification.ipynb ---------------------------------------------------------------------- diff --git a/samples/jupyter-notebooks/Deep_Learning_Image_Classification.ipynb b/samples/jupyter-notebooks/Deep_Learning_Image_Classification.ipynb index 3e58887..4285acd 100644 --- a/samples/jupyter-notebooks/Deep_Learning_Image_Classification.ipynb +++ b/samples/jupyter-notebooks/Deep_Learning_Image_Classification.ipynb @@ -35,26 +35,36 @@ "<a id=\"load_systemml\"></a>\n", "## Install and load SystemML and other libraries\n", "\n", - "You can install SystemML Python tgz file either from distribution location (https://dist.apache.org/repos/dist/release/incubator/systemml/) or from your local system if you have extracted recent code and built locally. \n", - "### This notebook is supported with master branch as of 05/15/2017 and later code." + "You can install SystemML Python tgz file either from \n", + " 1. Distribution location (https://dist.apache.org/repos/dist/release/incubator/systemml/) or \n", + " 2. Latest daily built code (https://sparktc.ibmcloud.com/repo/latest/) or\n", + " 3. From your local system if you have extracted recent code and built locally. \n", + " (e.g. ~/git/incubator-systemml/target/systemml-1.0.0-incubating-SNAPSHOT-python.tgz)\n", + " \n", + "### This notebook is supported with master branch (SystemML 1.0.0) as of 05/15/2017 and later code." ] }, { "cell_type": "code", "execution_count": null, "metadata": { + "collapsed": true, "scrolled": false }, "outputs": [], "source": [ "# !pip install --user systemml>=1.0.0\n", - "!pip install ~/git/systemml_scala/incubator-systemml/target/systemml-1.0.0-incubating-SNAPSHOT-python.tgz" + "\n", + "!pip install https://sparktc.ibmcloud.com/repo/latest/systemml-1.0.0-incubating-SNAPSHOT-python.tgz\n";, + " \n", + "# !pip install ~/git/incubator-systemml/target/systemml-1.0.0-incubating-SNAPSHOT-python.tgz" ] }, { "cell_type": "code", "execution_count": null, "metadata": { + "collapsed": true, "scrolled": false }, "outputs": [], @@ -65,7 +75,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "from systemml import MLContext, dml\n", @@ -81,6 +93,7 @@ "cell_type": "code", "execution_count": null, "metadata": { + "collapsed": true, "scrolled": true }, "outputs": [], @@ -131,6 +144,7 @@ "cell_type": "code", "execution_count": null, "metadata": { + "collapsed": true, "scrolled": true }, "outputs": [], @@ -165,6 +179,7 @@ "cell_type": "code", "execution_count": null, "metadata": { + "collapsed": true, "scrolled": true }, "outputs": [], @@ -185,7 +200,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "trainData = np.genfromtxt('data/mnist/mnist_train.csv', delimiter=\",\")\n", @@ -198,7 +215,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "pd.set_option('display.max_columns', 200)\n", @@ -241,6 +260,7 @@ "cell_type": "code", "execution_count": null, "metadata": { + "collapsed": true, "scrolled": true }, "outputs": [], @@ -288,7 +308,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "scriptPredict = \"\"\"\n", @@ -360,6 +382,7 @@ "cell_type": "code", "execution_count": null, "metadata": { + "collapsed": true, "scrolled": true }, "outputs": [], @@ -376,7 +399,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "pd.set_option('display.max_columns', 28)\n", @@ -393,7 +418,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "!yes | pip uninstall systemml" http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/168ff41e/samples/jupyter-notebooks/Linear_Regression_Algorithms_Demo.ipynb ---------------------------------------------------------------------- diff --git a/samples/jupyter-notebooks/Linear_Regression_Algorithms_Demo.ipynb b/samples/jupyter-notebooks/Linear_Regression_Algorithms_Demo.ipynb index 90a8048..b39d271 100644 --- a/samples/jupyter-notebooks/Linear_Regression_Algorithms_Demo.ipynb +++ b/samples/jupyter-notebooks/Linear_Regression_Algorithms_Demo.ipynb @@ -25,24 +25,39 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Install SystemML Python package and jar file" + "# Install SystemML Python package and jar file\n", + "You can install SystemML Python tgz file either from \n", + " 1. Distribution location (https://dist.apache.org/repos/dist/release/incubator/systemml/) or \n", + " 2. Latest daily built code (https://sparktc.ibmcloud.com/repo/latest/) or\n", + " 3. From your local system if you have extracted recent code and built locally. \n", + " (e.g. ~/git/incubator-systemml/target/systemml-1.0.0-incubating-SNAPSHOT-python.tgz)\n", + " \n", + "### This notebook is supported with SystemML 0.14.0 and above." ] }, { "cell_type": "code", "execution_count": null, "metadata": { + "collapsed": true, "scrolled": false }, "outputs": [], "source": [ - "!pip install ~/git/incubator-systemml/target/systemml-1.0.0-incubating-SNAPSHOT-python.tgz" + "# !pip install --user systemml>=0.14.0\n", + "\n", + "!pip install https://dist.apache.org/repos/dist/release/incubator/systemml/0.14.0-incubating/systemml-0.14.0-incubating-python.tgz\n";, + " \n", + "# !pip install https://sparktc.ibmcloud.com/repo/latest/systemml-1.0.0-incubating-SNAPSHOT-python.tgz\n";, + " \n", + "# !pip install ~/git/incubator-systemml/target/systemml-1.0.0-incubating-SNAPSHOT-python.tgz" ] }, { "cell_type": "code", "execution_count": null, "metadata": { + "collapsed": true, "scrolled": true }, "outputs": [], @@ -60,7 +75,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "from systemml import MLContext, dml, dmlFromResource\n", @@ -75,7 +92,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "ml.execute(dml(\"\"\"s = 'Hello World!'\"\"\").output(\"s\")).get(\"s\")" @@ -108,19 +127,6 @@ ] }, { - "cell_type": "code", - "execution_count": null, - "metadata": { - "scrolled": false - }, - "outputs": [], - "source": [ - "import sys, os\n", - "SCRIPTS = os.sep.join([os.environ[\"HOME\"],'anaconda', 'lib', 'python' + sys.version[:3], 'site-packages', 'systemml', 'systemml-java', 'scripts'])\n", - "print SCRIPTS" - ] - }, - { "cell_type": "markdown", "metadata": {}, "source": [ @@ -155,7 +161,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "prog = dml(script).input('$nr', 1e5).output('s')\n", @@ -181,22 +189,16 @@ "source": [ "ml = MLContext(sc)\n", "ml = ml.setStatistics(True)\n", - "# re-execute ML program" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "printLastLogLines(22)" + "# re-execute ML program\n", + "# printLastLogLines(22)" ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "prog = dml(script).input('$nr', 1e6).output('s')\n", @@ -237,7 +239,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "diabetes = datasets.load_diabetes()\n", @@ -254,7 +258,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "diabetes.data.shape" @@ -324,6 +330,7 @@ "cell_type": "code", "execution_count": null, "metadata": { + "collapsed": true, "scrolled": true }, "outputs": [], @@ -406,6 +413,7 @@ "cell_type": "code", "execution_count": null, "metadata": { + "collapsed": true, "scrolled": false }, "outputs": [], @@ -489,6 +497,7 @@ "cell_type": "code", "execution_count": null, "metadata": { + "collapsed": true, "scrolled": false }, "outputs": [], @@ -514,16 +523,29 @@ }, "outputs": [], "source": [ - "prog = dml(\"/Users/user_name/git/incubator-systemml/scripts/algorithms/LinearRegDS.dml\").input(X=diabetes_X_train, y=diabetes_y_train).input('$icpt',1.0).output('beta_out')\n", + "import os\n", + "from subprocess import call\n", + "\n", + "dirName = os.path.dirname(os.path.realpath(\"~\")) + \"/scripts\"\n", + "call([\"mkdir\", \"-p\", dirName])\n", + "call([\"wget\", \"-N\", \"-q\", \"-P\", dirName, \"https://raw.githubusercontent.com/apache/incubator-systemml/master/scripts/algorithms/LinearRegDS.dml\";])\n", + "\n", + "scriptName = dirName + \"/LinearRegDS.dml\"\n", + "dml_script = dmlFromResource(scriptName)\n", + "\n", + "prog = dml_script.input(X=diabetes_X_train, y=diabetes_y_train).input('$icpt',1.0).output('beta_out')\n", "w = ml.execute(prog).get('beta_out')\n", "w = w.toNumPy()\n", - "bias=w[1]" + "bias=w[1]\n", + "print bias" ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "plt.scatter(diabetes_X_train, diabetes_y_train, color='black')\n", @@ -562,7 +584,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "regr = LinearRegression(sqlCtx)\n", @@ -584,7 +608,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "# Use the trained model to perform prediction\n", @@ -605,7 +631,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "!yes | pip uninstall systemml" http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/168ff41e/samples/jupyter-notebooks/SystemML-PySpark-Recommendation-Demo.ipynb ---------------------------------------------------------------------- diff --git a/samples/jupyter-notebooks/SystemML-PySpark-Recommendation-Demo.ipynb b/samples/jupyter-notebooks/SystemML-PySpark-Recommendation-Demo.ipynb index eceb6c7..e706329 100644 --- a/samples/jupyter-notebooks/SystemML-PySpark-Recommendation-Demo.ipynb +++ b/samples/jupyter-notebooks/SystemML-PySpark-Recommendation-Demo.ipynb @@ -18,23 +18,39 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Install and load SystemML library" + "# Install SystemML Python package and jar file\n", + "You can install SystemML Python tgz file either from \n", + " 1. Distribution location (https://dist.apache.org/repos/dist/release/incubator/systemml/) or \n", + " 2. Latest daily built code (https://sparktc.ibmcloud.com/repo/latest/) or\n", + " 3. From your local system if you have extracted recent code and built locally. \n", + " (e.g. ~/git/incubator-systemml/target/systemml-1.0.0-incubating-SNAPSHOT-python.tgz)\n", + " \n", + "### This notebook is supported with SystemML 0.14.0 and above." ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ - "#!pip install --user systemml>=0.14.0\n", - "!pip install ~/git/incubator-systemml/target/systemml-1.0.0-incubating-SNAPSHOT-python.tgz" + "# !pip install --user systemml>=0.14.0\n", + "\n", + "!pip install https://dist.apache.org/repos/dist/release/incubator/systemml/0.14.0-incubating/systemml-0.14.0-incubating-python.tgz\n";, + " \n", + "# !pip install https://sparktc.ibmcloud.com/repo/latest/systemml-1.0.0-incubating-SNAPSHOT-python.tgz\n";, + " \n", + "# !pip install ~/git/incubator-systemml/target/systemml-1.0.0-incubating-SNAPSHOT-python.tgz" ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "!pip show systemml" @@ -43,7 +59,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "%load_ext autoreload\n", @@ -66,7 +84,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "%%sh\n", @@ -78,7 +98,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "# Load data\n", @@ -173,7 +195,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "# Plot training loss over time\n", @@ -195,7 +219,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "!yes | pip uninstall systemml" http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/168ff41e/samples/jupyter-notebooks/tutorial1.ipynb ---------------------------------------------------------------------- diff --git a/samples/jupyter-notebooks/tutorial1.ipynb b/samples/jupyter-notebooks/tutorial1.ipynb index 6da2cee..ce47cf8 100644 --- a/samples/jupyter-notebooks/tutorial1.ipynb +++ b/samples/jupyter-notebooks/tutorial1.ipynb @@ -12,22 +12,24 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Install latest SystemML jar file." + "# Install SystemML Python package and jar file\n", + "You can configre SystemML jar file in Scala kernel by downloading either from \n", + " 1. Distribution location (https://dist.apache.org/repos/dist/release/incubator/systemml/) or \n", + " 2. Latest daily built code (https://sparktc.ibmcloud.com/repo/latest/) or\n", + " 3. From your local system if you have extracted recent code and built locally. \n", + " (e.g. ~/git/incubator-systemml/target/systemml-1.0.0-incubating-SNAPSHOT.jar)\n", + " \n", + "Please visit SystemML site to know \"How to configure Toree(Scala) Kernel\".\n", + " \n", + "### This notebook is supported with SystemML 0.14.0 and above." ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "%AddJar https://sparktc.ibmcloud.com/repo/latest/SystemML.jar"; - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "import org.apache.sysml.api.mlcontext.MLContext\n", @@ -79,7 +81,9 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "collapsed": true + }, "outputs": [], "source": [ "outMatrix.show"
