This is an automated email from the ASF dual-hosted git repository.
janniklinde pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new e556a6f1bf [SYSTEMDS-3331] Documentation Cleanup of Quickstart Guide
e556a6f1bf is described below
commit e556a6f1bf0cc704c510d6ae284e5115d5ce77a1
Author: yiseungmi87 <[email protected]>
AuthorDate: Mon Mar 2 11:20:31 2026 +0100
[SYSTEMDS-3331] Documentation Cleanup of Quickstart Guide
Closes #2388
---
docs/site/install.md | 140 ++++++++++++++---------
docs/site/quickstart_extended.md | 75 +++++++++++++
docs/site/release_install.md | 219 ++++++++++++++++++++++++++++++++++++
docs/site/run.md | 236 +++++++++++++++++++++++++++++++++------
4 files changed, 581 insertions(+), 89 deletions(-)
diff --git a/docs/site/install.md b/docs/site/install.md
index 140f2e5b10..50ffbd4a20 100644
--- a/docs/site/install.md
+++ b/docs/site/install.md
@@ -21,99 +21,126 @@ limitations under the License.
{% endcomment %}
-->
-This guide helps in the install and setup of SystemDS from source code.
-
-- [Install](#install)
- - [Windows](#windows)
- - [Ubuntu 22.04](#ubuntu-2204)
- - [Testing](#testing)
- - [MAC](#mac)
-- [2. Build the project](#2-build-the-project)
-- [3. Run A Component Test](#3-run-a-component-test)
+# Install SystemDS from Source
-Once the individual versions is set up skip to the common part of building the
system.
+This guide helps in the install and setup of SystemDS from source code.
---
-## Install
+- [1. Install on Windows](#1-install-on-windows)
+- [2. Install on Ubuntu](#2-install-on-ubuntu-2204--2404)
+- [3. Install on macOS](#3-install-on-macos)
+- [4. Build the Project](#4-build-the-project)
+- [5. Run a Component Test](#5-run-a-component-test)
+- [6. Next Steps](#6-next-steps)
+
+Once the individual environment is set up, you can continue with the common
build steps below.
---
-### Windows
+# 1. Install on Windows
-First setup java and maven to compile the system note the java version is 17,
we suggest using Java OpenJDK 17.
+First setup Java and maven to compile the system note the Java version is 17,
we suggest using Java OpenJDK 17.
- <https://openjdk.org/>
- <https://maven.apache.org/download.cgi?.>
-Setup your environment variables with JAVA_HOME and MAVEN_HOME. Using these
variables add the JAVA_HOME/bin and MAVEN_HOME/bin to the path environment
variable. An example of setting it for java can be found here:
<https://www.thewindowsclub.com/set-java_home-in-windows-10>
+Setup your environment variables with JAVA_HOME and MAVEN_HOME. Using these
variables add the JAVA_HOME/bin and MAVEN_HOME/bin to the path environment
variable. An example of setting it for Java can be found here:
<https://www.thewindowsclub.com/set-java_home-in-windows-10>
-To run the system we also have to setup some Hadoop and spark specific
libraries. These can be found in the SystemDS repository. To add this, simply
take out the files, or add 'src/test/config/hadoop_bin_windows/bin' to PATH.
Just like for JAVA_HOME set a HADOOP_HOME to the environment variable without
the bin part, and add the %HADOOP_HOME%/bin to path.
+To run the system we also have to setup some Hadoop and Spark specific
libraries. These can be found in the SystemDS repository. To add this, simply
take out the files, or add 'src/test/config/hadoop_bin_windows/bin' to PATH.
Just like for JAVA_HOME set a HADOOP_HOME to the environment variable without
the bin part, and add the `%HADOOP_HOME%/bin` to path.
-Finally if you want to run systemds from command line, add a SYSTEMDS_ROOT
that points to the repository root, and add the bin folder to the path.
+On Windows, cloning large repositories via GitHub Desktop may stall in some
environments. If this happens, cloning via the Git command line is a reliable
alternative.
+Example:
+```bash
+git clone https://github.com/apache/systemds.git
+```
-To make the build go faster set the IDE or environment variables for java:
'-Xmx16g -Xms16g -Xmn1600m'. Here set the memory to something close to max
memory of the device you are using.
+To make the build go faster set the IDE or environment variables for Java:
'-Xmx16g -Xms16g -Xmn1600m'. Here set the memory to something close to max
memory of the device you are using.
To start editing the files remember to import the code style formatting into
the IDE, to keep the changes of the files consistent.
A suggested starting point would be to run some of the component tests from
your IDE.
----
+# 2. Install on Ubuntu (22.04 / 24.04)
-### Ubuntu 22.04
+### 2.1 Install Java 17 and Maven
-First setup java and maven to compile the system note that the java version is
17.
+First setup Java, maven and git to compile the system note that the Java
version is 17.
```bash
-sudo apt install openjdk-17-jdk
-sudo apt install maven
+sudo apt update
+sudo apt install openjdk-17-jdk maven
+sudo apt install -y git
```
Verify the install with:
-
```bash
java -version
mvn -version
+git --version
```
This should return something like:
+```bash
+openjdk 17.x.x
+Apache Maven 3.x.x
+git version 2.x.x
+```
+
+### 2.2 Set JAVA_HOME for Javadocs
+Set `JAVA_HOME` (required for generating Javadocs during the Maven build):
```bash
-openjdk 17.0.11 2024-04-16
-OpenJDK Runtime Environment Temurin-17.0.11+9 (build 17.0.11+9)
-OpenJDK 64-Bit Server VM Temurin-17.0.11+9 (build 17.0.11+9, mixed mode,
sharing)
+export JAVA_HOME=$(dirname $(dirname $(readlink -f $(which javac))))
+export PATH="$JAVA_HOME/bin:$PATH"
+```
-Apache Maven 3.9.9 (8e8579a9e76f7d015ee5ec7bfcdc97d260186937)
-Maven home: /home/usr/Programs/maven
-Java version: 17.0.11, vendor: Eclipse Adoptium, runtime:
/home/usr/Programs/jdk-17.0.11+9
-Default locale: en_US, platform encoding: UTF-8
-OS name: "linux", version: "6.8.0-59-generic", arch: "amd64", family: "unix"
+### 2.3 Clone Source Code
+
+Clone the source code:
+```bash
+cd /opt
+git clone https://github.com/apache/systemds.git
+cd systemds
```
-#### Testing
+### 2.4 Testing
-R is required to be install to run the test suite, since many tests are
constructed to compare output with common R packages.
-One option to install this is to follow the guide on the following link:
<https://linuxize.com/post/how-to-install-r-on-ubuntu-20-04/>
+R should be installed to run the test suite, since many tests are constructed
to compare output with common R packages. One option to install this is to
follow the guide on the following link:
<https://linuxize.com/post/how-to-install-r-on-ubuntu-20-04/>
-At the time of writing the commands to install R 4.0.2 are:
+R can be installed using the CRAN repository.
+
+**Ubuntu 22.04**
```bash
sudo apt install dirmngr gnupg apt-transport-https ca-certificates
software-properties-common
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys
E298A3A825C0D65DFD57CBB651716619E084DAB9
sudo add-apt-repository 'deb https://cloud.r-project.org/bin/linux/ubuntu
focal-cran40/'
+sudo apt update
sudo apt install r-base
```
-Optionally, you need to install the R dependencies for integration tests, like
this:
-(use `sudo` mode if the script couldn't write to local R library)
+**Ubuntu 24.04**
```bash
-Rscript ./src/test/scripts/installDependencies.R
+sudo apt install dirmngr gnupg apt-transport-https ca-certificates
software-properties-common
+sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys
E298A3A825C0D65DFD57CBB651716619E084DAB9
+sudo add-apt-repository 'deb https://cloud.r-project.org/bin/linux/ubuntu
noble-cran40/'
+sudo apt update
+sudo apt install r-base
```
----
+Verify the installation:
+```bash
+R --version
+```
+
+**Install R Dependencies for Integration Tests (Optional)** If you want to run
integration tests that depend on additional R packages, install them via:
+```bash
+Rscript ./src/test/scripts/installDependencies.R
+```
-### MAC
+# 3. Install on MacOS
Prerequisite install homebrew on the device.
@@ -133,7 +160,7 @@ java --version
mvn --version
```
-This should print java version.
+This should print Java version.
Note that if you have multiple __java__ versions installed then you have to
change the used version to 17, on __both java and javadoc__. This is done by
setting the environment variable JAVA_HOME to the install path of open JDK 17 :
@@ -150,16 +177,14 @@ Optionally, you need to install the R dependencies for
integration tests, like t
Rscript ./src/test/scripts/installDependencies.R
```
----
-
-## 2. Build the project
-
-To compile the project use:
+# 4. Build the project
+To compile the project use in the directory of the source code:
```bash
mvn package -P distribution
```
+Example output:
```bash
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
@@ -169,10 +194,20 @@ mvn package -P distribution
[INFO] ------------------------------------------------------------------------
```
-The first time you package the system it will take longer since maven will
download the dependencies.
-But successive compiles should become faster.
+The first time you package the system it will take longer since maven will
download the dependencies. But successive compiles should become faster. The
runnable JAR files will appear in `target/`.
+
+### (Optional) Add SystemDS CLI to PATH
+
+After building SystemDS from source, you can add the `bin` directory to your
+`PATH` in order to run `systemds` directly from the command line:
+
+```bash
+export SYSTEMDS_ROOT=$(pwd)
+export PATH="$SYSTEMDS_ROOT/bin:$PATH"
+```
+This allows you to run `systemds` from the repository root. For running the
freshly built executable JAR (e.g., `target/SystemDS.jar`) on Spark, see the
Spark section in [Execute SystemDS](run).
-## 3. Run A Component Test
+# 5. Run A Component Test
As an example here is how to run the component matrix tests from command line
via maven.
@@ -180,9 +215,8 @@ As an example here is how to run the component matrix tests
from command line vi
mvn test -Dtest="**.component.matrix.**"
```
-To run other tests simply specify other packages by modifying the
-test argument part of the command.
+To run other tests simply specify other packages by modifying the test
argument part of the command.
-Now everything is setup and ready to go!
+# 6. Next Steps
-To execute dml scripts look at [Execute SystemDS](run), this step is not
needed to develop in systemds, but it helps setting up the command-line
execution of systemds.
+Now everything is setup and ready to go! For running scripts in Spark mode or
experimenting with federated workers, see the Execution Guide: [Execute
SystemDS](run)
diff --git a/docs/site/quickstart_extended.md b/docs/site/quickstart_extended.md
new file mode 100644
index 0000000000..ce7c9914f9
--- /dev/null
+++ b/docs/site/quickstart_extended.md
@@ -0,0 +1,75 @@
+---
+layout: site
+title: SystemDS Quickstart
+description: Quickstart guide for installing and running SystemDS on Windows,
Linux, and macOS
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+# Extended Quickstart Guide
+
+Welcome to the extended quickstart guide for Apache SystemDS. This quickstart
page provides a high-level overview of both installation and points you to the
detailed documentation for each path.
+
+SystemDS can be installed and used in two different ways:
+
+1. Using a **downloaded release**
+2. Using a **source build**
+
+If you are primarily a user of SystemDS, start with the Release installation.
If you plan to contribute or modify internals, follow the Source installation.
+
+Each method is demonstrated in:
+- Local mode
+- Spark mode
+- Federated mode (simple example)
+
+For detailed configuration topics (BLAS, GPU, federated setup, contributing),
see the links at the end.
+
+---
+
+# 1. Install from a Release
+
+If you simply want to *use* SystemDS without modifying the source code, the
recommended approach is to install SystemDS from an official Apache release.
+
+**Full Release Installation Guide:** [Install SystemDS from a
Release](release_install)
+
+# 2. Install from Source
+
+If you plan to contribute to SystemDS or need to modify its internals, you can
build SystemDS from source.
+
+**Full Source Build Guide:** [Install SystemDS from Source](install)
+
+# 3. After Installation
+
+Once either installation path is completed, you can start running scripts:
+
+- Local Mode - Run SystemDS locally
+- Spark Mode - Execute scripts on Spark through `spark-submit`
+- Federated Mode - Run operations on remote data using federated workers
+
+For detailed commands and examples: [Execute SystemDS](run_extended)
+
+# 4. More Configuration
+
+SystemDS provides advanced configuration options for performance tuning and
specialized execution environments.
+
+- GPU Support — [GPU Guide](https://apache.github.io/systemds/site/gpu)
+- BLAS / Native Acceleration — [Native Backend (BLAS) Guide](native-backend)
+- Federated Backend Deployment — [Federated Guide](federated-monitoring)
+- Contributing to SystemDS — [Contributing
Guide](https://github.com/apache/systemds/blob/main/CONTRIBUTING.md)
+
diff --git a/docs/site/release_install.md b/docs/site/release_install.md
new file mode 100644
index 0000000000..bb0a230e7a
--- /dev/null
+++ b/docs/site/release_install.md
@@ -0,0 +1,219 @@
+---
+layout: site
+title: Install SystemDS from a Release
+description: Installation guide for SystemDS using release archives
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+# Install SystemDS from a Release
+
+This guide explains how to install and set up SystemDS using the pre-built
release archives.
+
+---
+
+- [1. Download a Release](#1-download-a-release)
+- [2. Install on Windows](#2-install-on-windows)
+- [3. Install on Ubuntu 22.04](#3-install-on-ubuntu-2204)
+- [4. Install on macOS](#4-install-on-macos)
+
+---
+
+# 1. Download a Release
+
+Download the official release archive from the Apache SystemDS website:
+
+https://systemds.apache.org/download
+
+After downloading the file `systemds-<VERSION>.tar.gz`, place the file in any
directory you choose for installation.
+
+### Java Requirement ###
+For compatability with Spark execution and parser components, **Java 17** is
strongly recommended for SystemDS.
+
+Verify Java 17:
+
+```bash
+java -version
+```
+
+If missing, install a JDK 17 distribution.
+
+---
+
+# 2. Install on Windows
+
+### 2.1 Extract the Release Archive
+
+Use Windows built-in extractor.
+
+### 2.2 Verify the Installation by Checking the CLI
+
+On Windows, the `systemds`CLI wrapper may not be executable. This is expected
because the `bin/systemds`launcher is implemented as a shell script, which
Windows cannot execute natively. To verify the installation on Windows,
navigate to the bin directory and run the JAR directly. Note that running
`systemds -help` without JAR may result in a CommandNotFoundExeption:
+
+```bash
+java -jar systemds-<VERSION>.jar -help
+```
+
+You should see usage information as an output printed to the console.
+
+### 2.3 Create a Simple Script
+
+On Windows, especially when using PowerShell, creating text files via shell
redirection (e.g., echo...) may result in unexpected encoding or invisible
characters. This can lead to parsing errors when executing the script, even
though the file appears correct in an editor. Therefore, you may try creating
the file explicitly using PowerShell:
+```bash
+Set-Content -Path .\hello.dml -Value 'print("Hello World!")' -Encoding Ascii
+```
+
+This ensures the script is stored as plain text without additional encoding
metadata.
+Note: This behavior depends on the shell and environment configuration and may
not affect all Windows setups.
+
+Verify the file contents:
+```bash
+Get-Content .\hello.dml
+```
+
+Expected output:
+```bash
+print("Hello World!")
+```
+
+### 2.4 Run the Script
+
+Now run the script:
+```bash
+java -jar systemds-3.3.0.jar -f .\hello.dml
+```
+
+Expected output:
+```bash
+Hello World!
+SystemDS Statistics:
+Total execution time: 0.012 sec.
+```
+
+# 3. Install on Ubuntu
+
+### 3.1 Extract the Release
+
+```bash
+cd /path/to/install
+tar -xvf systemds-<VERSION>-bin.tgz
+cd systemds-<VERSION>-bin
+```
+
+### 3.2 Configure Environment Variables
+
+The SystemDS CLI requires an environment variable pointing to the SystemDS
JAR. Make sure to export both `SYSTEMDS_ROOT` and `SYSTEMDS_JAR_FILE`.
+
+```bash
+export SYSTEMDS_ROOT=$(pwd)
+export SYSTEMDS_JAR_FILE=$(find "$SYSTEMDS_ROOT" -maxdepth 1 -type f -name
"systemds-*.jar" | head -n 1)
+export PATH="$SYSTEMDS_ROOT/bin:$PATH"
+```
+
+Verify that the JAR was found correctly:
+```bash
+echo "Using SystemDS JAR: $SYSTEMDS_JAR_FILE"
+```
+
+(Optional but recommended) To make SystemDS available in new terminals, add
the following lines to your shell configuration (e.g., `~/.bashrc` or
`~/.profile`):
+```bash
+export SYSTEMDS_ROOT=/absolute/path/to/systemds-<VERSION>
+export
SYSTEMDS_JAR_FILE=/absolute/path/to/systemds-<VERSION>-bin/lib/systemds-<VERSION>.jar
+export PATH="$SYSTEMDS_ROOT/bin:$PATH"
+```
+
+### 3.3 Verify the Installation by Checking the CLI
+
+```bash
+systemds -help
+```
+
+You should see usage information printed to the console.
+
+### 3.4 Create a Simple Script
+
+```bash
+echo 'print("Hello World!")' > hello.dml
+```
+
+### 3.5 Run the Script
+
+With `SYSTEMDS_JAR_FILE` properly set, the script can be executed directly via
the CLI:
+```bash
+systemds -f hello.dml
+```
+
+Expected output:
+```bash
+Hello World!
+```
+
+# 4. Install on macOS
+
+### 4.1 Extract the Release
+
+```bash
+cd /path/to/install
+tar -xvf systemds-<VERSION>-bin.tgz
+cd systemds-<VERSION>-bin
+```
+### 4.2 Add SystemDS to PATH
+
+```bash
+export SYSTEMDS_ROOT=$(pwd)
+export PATH="$SYSTEMDS_ROOT/bin:$PATH"
+```
+
+(Optional but recommended)
+To make SystemDS available in new terminals, add the following lines
+to your shell configuration (e.g., ~/.bashrc or ~/.profile):
+```bash
+export SYSTEMDS_ROOT=/absolute/path/to/systemds-<VERSION>
+export PATH=$SYSTEMDS_ROOT/bin:$PATH
+```
+
+### 4.3 Verify the Installation by Checking the CLI
+
+```bash
+systemds -help
+```
+
+You should see usage information printed to the console.
+
+### 4.4 Create a Simple Script
+
+```bash
+echo 'print("Hello World!")' > hello.dml
+```
+
+### 4.5 Run the Script
+
+```bash
+systemds -f hello.dml
+```
+
+Expected output:
+
+```bash
+Hello World!
+```
+
+# Next Steps
+
+For running scripts in Spark mode or experimenting with federated workers, see
the Execution Guide: [Execute SystemDS](run)
diff --git a/docs/site/run.md b/docs/site/run.md
index 66c3a970b4..e90349af99 100644
--- a/docs/site/run.md
+++ b/docs/site/run.md
@@ -21,58 +21,70 @@ limitations under the License.
{% endcomment %}
-->
-If you want to execute from source code follow the [Install from
source](install) guide first.
+# Running SystemDS
-## Setting SYSTEMDS_ROOT environment variable
+This guide explains how to run SystemDS regardless of whether you installed it
from a Release or built it from Source. All execution modes -local, Spark, and
federated- are covered in this document.
-In order to run SystemDS it is highly recommended to setup SystemDS root on
path.
-This works both from your development directory containing source code and if
-you download a release of SystemDS.
+---
-The following example works if you open an terminal at the root of the
downloaded release,
-or a cloned repository. (You can also change the `$(pwd)` with the full path
to the folder.)
+- [1. Prerequisites](#1-prerequisites)
+- [2. Run a Simple Script Locally](#2-run-a-simple-script-locally)
+- [3. Run a Script on Spark](#3-run-a-script-on-spark)
+- [4. Run a Script in Federated Mode](#4-run-a-script-in-federated-mode)
-```bash
-export SYSTEMDS_ROOT=$(pwd)
-export PATH=$SYSTEMDS_ROOT/bin:$PATH
-```
+---
-It can be beneficial to enter these into your `~/.profile` or `~/.bashrc` for
linux,
-(but remember to change `$(pwd` to the full folder path)
-or your environment variables in windows to enable reuse between terminals and
restarts.
+# 1. Prerequisites
-```bash
-echo 'export SYSTEMDS_ROOT='$(pwd) >> ~/.bashrc
-echo 'export PATH=$SYSTEMDS_ROOT/bin:$PATH' >> ~/.bashrc
-```
+This guide assumes that SystemDS has already been installed successfully.
+
+Please make sure you have followed one of the installation guides:
+- [Install SystemDS from a Release](release_install)
+- [Install SystemDS from Source](source_install)
+
+In particular, ensure that:
+- Java 17 is installed
+- Spark 3.x is available if you plan to run SystemDS on Spark
+- SystemDS is installed following the Release or Source installation guide
+- Required environment variables (`SYSTEMDS_ROOT`, `PATH`, and if applicable
`SYSTEMDS_JAR_FILE`) are set
+---
-## Hello, World! example
+# 2. Run a Simple Script Locally
-To quickly verify that the system is setup correctly.
-You can run a simple hello world, using the launch script.
+This mode does not require Spark. It only needs Java 17.
-Open an terminal and go to an empty folder, then execute the following.
+### 2.1 Create and Run a Hello World
```bash
-# Create a hello World script
echo 'print("Hello, World!")' > hello.dml
-# Execute hello world Script
+```
+
+Run:
+```bash
systemds hello.dml
-# Remove the hello.dml
-rm hello.dml
```
+Expected output:
+```bash
+Hello, World!
+```
+
+### (Optional) MacOS Note: `realpath: illegal option -- -` Error
If you are running MacOS and encounter an error message similar to `realpath:
illegal option -- -` when executing `systemds hello.dml`. You may try to
replace the system-wide command `realpath` with the homebrew version
`grealpath` that comes with the `coreutils`. Alternatively, you may change all
occurrences within the script accordingly, i.e., by prepending a `g` to avoid
any side effects.
-## Running a real first example
+### (Optional) Ubuntu Note: `Invalid or corrupt jarfile hello.dml`
+On some Ubuntu setups (especially clean environments such as Docker images),
running `systemds -f hello.dml` may result in an error like `Invalid or corrupt
jarfile hello.dml`. If this happens, the SystemDS launcher may not
automatically locate the correct JAR. To fix this, export `SYSTEMDS_JAR_FILE`
to point to the JAR shipped with the release. Please refer to the Ubuntu
troubleshooting section in the installation guide for a detailed workaround:
[Release Installation – Ubuntu Note](rel [...]
+
+### (Optional) Windows Note: `systemds` Command Not Found
+On Windows (e.g., PowerShell), running `systemds -f hello.dml` may fail with
an error indicating that `systemds` is not recognized as a command. This is
expected, since the `systemds` launcher in `bin/` is implemented as a shell
script,
+which cannot be executed natively on Windows. In this case, SystemDS should be
invoked directly via the runnable JAR using `java -jar`. For a detailed
Windows-specific walkthrough, please refer to the installation guide: [Release
Installation – Windows Notes](release_install#2-install-on-windows)
+
-To see SystemDS in action a simple example using the `Univar-stats.dml`
-script can be executed.
-The relevant commands to run this example with SystemDS is described in the
DML Language reference guide at [link](dml-language-reference.html).
-See their documentation for further details.
+### 2.2 Create a Real Example
-### Example preparations
+This example demonstrates local execution of a real script `Univar-stats.dml`.
The relevant commands to run this example with SystemDS is described in the DML
Language reference guide at [DML Language Reference](dml-language-reference).
+Prepare the data (macOS: use `curl`instead of `wget`):
```bash
# download test data
wget -P data/
http://archive.ics.uci.edu/ml/machine-learning-databases/haberman/haberman.data
@@ -85,16 +97,168 @@ echo '1,1,1,2' > data/types.csv
echo '{"rows": 1, "cols": 4, "format": "csv"}' > data/types.csv.mtd
```
-### Executing the DML script
+Execute the DML Script:
+```bash
+systemds "$SYSTEMDS_ROOT/scripts/algorithms/Univar-Stats.dml" -nvargs \
+ X=data/haberman.data \
+ TYPES=data/types.csv \
+ STATS=data/univarOut.mtx \
+ CONSOLE_OUTPUT=TRUE
+```
+
+### (Optional) MacOS Note: `SparkException` Error
+If SystemDS tries to initialize Spark and you see `SparkException: A master
URL must be set in your configuration`, you can force single-node execution
without Spark/Hadoop initialization via:
+```bash
+systemds -exec singlenode scripts/algorithms/Univar-Stats.dml -nvargs \
+ X=data/haberman.data \
+ TYPES=data/types.csv \
+ STATS=data/univarOut.mtx \
+ CONSOLE_OUTPUT=TRUE
+```
+
+### 2.3 Run the Real Example
+
+The script computes basic statistics (min, max, variance, skewness, etc) for
each column of a dataset. Expected output (example):
+```bash
+-------------------------------------------------
+Feature [1]: Scale
+ (01) Minimum | 30.0
+ (02) Maximum | 83.0
+ (03) Range | 53.0
+ (04) Mean | 52.45751633986928
+ (05) Variance | 116.71458266366658
+ (06) Std deviation | 10.803452349303281
+ (07) Std err of mean | 0.6175922641866753
+ (08) Coeff of variation | 0.20594669940735139
+ (09) Skewness | 0.1450718616532357
+ (10) Kurtosis | -0.6150152487211726
+ (11) Std err of skewness | 0.13934809593495995
+ (12) Std err of kurtosis | 0.277810485320835
+ (13) Median | 52.0
+ (14) Interquartile mean | 52.16013071895425
+-------------------------------------------------
+Feature [2]: Scale
+ (01) Minimum | 58.0
+ (02) Maximum | 69.0
+ (03) Range | 11.0
+ (04) Mean | 62.85294117647059
+ (05) Variance | 10.558630665380907
+ (06) Std deviation | 3.2494046632238507
+ (07) Std err of mean | 0.18575610076612029
+ (08) Coeff of variation | 0.051698529971741194
+ (09) Skewness | 0.07798443581479181
+ (10) Kurtosis | -1.1324380182967442
+ (11) Std err of skewness | 0.13934809593495995
+ (12) Std err of kurtosis | 0.277810485320835
+ (13) Median | 63.0
+ (14) Interquartile mean | 62.80392156862745
+-------------------------------------------------
+Feature [3]: Scale
+ (01) Minimum | 0.0
+ (02) Maximum | 52.0
+ (03) Range | 52.0
+ (04) Mean | 4.026143790849673
+ (05) Variance | 51.691117539912135
+ (06) Std deviation | 7.189653506248555
+ (07) Std err of mean | 0.41100513466216837
+ (08) Coeff of variation | 1.7857418611299172
+ (09) Skewness | 2.954633471088322
+ (10) Kurtosis | 11.425776549251449
+ (11) Std err of skewness | 0.13934809593495995
+ (12) Std err of kurtosis | 0.277810485320835
+ (13) Median | 1.0
+ (14) Interquartile mean | 1.2483660130718954
+-------------------------------------------------
+Feature [4]: Categorical (Nominal)
+ (15) Num of categories | 2
+ (16) Mode | 1
+ (17) Num of modes | 1
+SystemDS Statistics:
+Total execution time: 0,470 sec.
+```
-```shell script
-bin/systemds Univar-Stats.dml -nvargs X=data/haberman.data
TYPES=data/types.csv STATS=data/univarOut.mtx CONSOLE_OUTPUT=TRUE
+To check the location of output file created:
+```bash
+ls -l data/univarOut.mtx
```
-## Using Intel MKL native instructions
+---
+# 3. Run a Script on Spark
+
+SystemDS can be executed on Spark using the main executable JAR. The location
of this JAR differs depending on whether you installed SystemDS from:
+
+- a **Release archive**, or
+- a **Source-build installation** (built with Maven)
+
+### 3.1 Running with a Release installation
+
+If you installed SystemDS from a release archive, locate the runnable JAR in
the release root directory. It is typically named like `systemds-<version>.jar`.
+
+Example:
+```bash
+ls -1 "$SYSTEMDS_ROOT"/*.jar
+```
+
+Run:
+```bash
+spark-submit "$SYSTEMDS_ROOT/systemds-<version>.jar" -f hello.dml
+```
+
+### 3.2 Running with a Source-build installation
+
+If you cloned the SystemDS repository and built it yourself, you must first
run Maven to generate the executable JAR.
+
+```bash
+mvn -P distribution package
+```
+This creates several JAR files in `target/`:
+
+Example output:
+
+```bash
+target/systemds-3.3.0-shaded.jar
+target/systemds-3.3.0.jar
+target/systemds-3.3.0-unshaded.jar
+target/systemds-3.3.0-extra.jar
+target/SystemDS.jar <-- main runnable JAR
+target/systemds-3.3.0-ropt.jar
+target/systemds-3.3.0-javadoc.jar
+```
+
+Run:
+```bash
+spark-submit target/SystemDS.jar -f hello.dml
+```
+
+---
+# 4. Run a Script in Federated Mode
+
+Federated mode allows SystemDS to execute operations on data located on remote
or distributed workers. Federated execution requires:
+
+1. One or more **federated workers**
+2. A **driver program** (DML or Python) that sends operations to those workers.
+
+Note: The SystemDS documentation provides federated execution examples
primarily via the Python API. This Quickstart demonstrates only how to start a
federated worker, and refers users to the official Federated Environment guide
for complete end-to-end examples.
+
+### 4.1 Start a federated worker
+
+Run in a separate terminal:
+
+```bash
+systemds WORKER 8001
+```
+
+This starts a worker on port `8001`.
+
+### 4.2 Next steps and full examples
+
+For complete, runnable examples of federated execution (including data files,
metadata, and Python code), see the official [Federated Environment
guide](https://systemds.apache.org/docs/3.3.0/api/python/guide/federated)
+
+### Using Intel MKL native instructions
To use the MKL acceleration download and install the latest supported MKL
library (<=2019.5) from [1],
set the environment variables with the MKL-provided script `.
/opt/intel/bin/compilervars.sh intel64` (note the dot and
the default install location) and set the option `sysds.native.blas` in
`SystemDS-config.xml` to mkl.
[1]: https://software.intel.com/mkl "Intel Math Kernel Library"
+
