[jira] [Commented] (SYSTEMML-52) User Documentation Enhancement
[ https://issues.apache.org/jira/browse/SYSTEMML-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15409871#comment-15409871 ] Deron Eriksson commented on SYSTEMML-52: The documentation has significantly been expanded since this issue was created. We now have a README, Quick Start Guide, Spark MLContext Programming Guide, Hadoop Batch Mode Guide, JMLC Guide, DML Language Reference, Beginner's Guide to DML and PyDML, Algorithms Reference, Debugger Guide, IDE Guide, Contributing to SystemML Guide, Troubleshooting Guide, and Release Process Guide. So, I will consider this issue resolved and closed. > User Documentation Enhancement > -- > > Key: SYSTEMML-52 > URL: https://issues.apache.org/jira/browse/SYSTEMML-52 > Project: SystemML > Issue Type: Epic >Reporter: Frederick Reiss >Assignee: Deron Eriksson > Fix For: SystemML 0.10 > > > Developer ramping up -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (SYSTEMML-707) diag not generate square matrix if given Nx1 matrix of zeroes
[
https://issues.apache.org/jira/browse/SYSTEMML-707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deron Eriksson closed SYSTEMML-707.
---
Verified that diag now generates a square matrix if given Nx1 matrix of zeroes
> diag not generate square matrix if given Nx1 matrix of zeroes
> -
>
> Key: SYSTEMML-707
> URL: https://issues.apache.org/jira/browse/SYSTEMML-707
> Project: SystemML
> Issue Type: Bug
> Components: APIs
>Reporter: Deron Eriksson
>Assignee: Matthias Boehm
> Fix For: SystemML 0.11
>
>
> Thank you Matthew Plourde for finding this!
> If an Nx1 matrix of 0's is given to the diag() function, an Nx1 matrix of 0's
> is returned. However, if an Nx1 matrix consists of any values that aren't
> 0's, an NxN diagonal matrix is returned. This is inconsistent and the Nx1
> matrix of 0's to diag() should probably return an NxN matrix.
> Example 1:
> {code}
> zeroes=matrix(0, 5, 1);
> print(toString(zeroes));
> print(toString(diag(zeroes)));
> {code}
> gives:
> {code}
> 0.000
> 0.000
> 0.000
> 0.000
> 0.000
> 0.000
> 0.000
> 0.000
> 0.000
> 0.000
> {code}
> Example 2:
> {code}
> ones=matrix(1, 5, 1);
> print(toString(ones));
> print(toString(diag(ones)));
> {code}
> gives:
> {code}
> 1.000
> 1.000
> 1.000
> 1.000
> 1.000
> 1.000 0.000 0.000 0.000 0.000
> 0.000 1.000 0.000 0.000 0.000
> 0.000 0.000 1.000 0.000 0.000
> 0.000 0.000 0.000 1.000 0.000
> 0.000 0.000 0.000 0.000 1.000
> {code}
> Example 3:
> {code}
> nums=matrix("0 1 2 3 4", 5, 1);
> print(toString(nums));
> print(toString(diag(nums)));
> {code}
> gives:
> {code}
> 0.000
> 1.000
> 2.000
> 3.000
> 4.000
> 0.000 0.000 0.000 0.000 0.000
> 0.000 1.000 0.000 0.000 0.000
> 0.000 0.000 2.000 0.000 0.000
> 0.000 0.000 0.000 3.000 0.000
> 0.000 0.000 0.000 0.000 4.000
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (SYSTEMML-708) Release checklist for 0.10.0-incubating-rc1
[
https://issues.apache.org/jira/browse/SYSTEMML-708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deron Eriksson resolved SYSTEMML-708.
-
Resolution: Fixed
Fix Version/s: SystemML 0.10
> Release checklist for 0.10.0-incubating-rc1
> ---
>
> Key: SYSTEMML-708
> URL: https://issues.apache.org/jira/browse/SYSTEMML-708
> Project: SystemML
> Issue Type: Task
>Reporter: Deron Eriksson
>Assignee: Deron Eriksson
> Fix For: SystemML 0.10
>
>
> || Task || Status || Notes ||
> | All Artifacts and Checksums Present | {panel:bgColor=#bfffba}Pass{panel} | |
> | Release Candidate Build - Windows | {panel:bgColor=#bfffba}Pass{panel} | |
> | Release Candidate Build - OS X | {panel:bgColor=#bfffba}Pass{panel} | |
> | Release Candidate Build - Linux | {panel:bgColor=#bfffba}Pass{panel} | |
> | Test Suite Passes - Windows |{panel:bgColor=#bfffba}Pass{panel} |
> SYSTEMML-712 opened for intermittent test failure |
> | Test Suite Passes - OS X| {panel:bgColor=#bfffba}Pass{panel} | |
> | Test Suite Passes - Linux | {panel:bgColor=#bfffba}Pass{panel} | |
> | All Binaries Execute| {panel:bgColor=#bfffba}Pass{panel} |
> Verified on OS X |
> | Check LICENSE and NOTICE Files | {panel:bgColor=#bfffba}Pass{panel} |
> non-blocker, SYSTEMML-711 filed |
> | Src Artifact Builds and Tests Pass | {panel:bgColor=#bfffba}Pass{panel} |
> 5037 of 5038 passed on OS X (RightIndexingMatrixTest failed) |
> | Single-Node Standalone - Windows| {panel:bgColor=#bfffba}Pass{panel} | |
> | Single-Node Standalone - OS X | {panel:bgColor=#bfffba}Pass{panel} | |
> | Single-Node Standalone - Linux | {panel:bgColor=#bfffba}Pass{panel} | |
> | Single-Node Spark | {panel:bgColor=#bfffba}Pass{panel} |
> Verified on OS X
> | Single-Node Hadoop | {panel:bgColor=#bfffba}Pass{panel} |
> Verified on OS X
> | Notebooks - Jupyter | {panel:bgColor=#bfffba}Pass{panel} |
> Verified on OS X |
> | Notebooks - Zeppelin| {panel:bgColor=#bfffba}Pass{panel} |
> Verified on OS X |
> | Performance Suite - Spark | {panel:bgColor=#bfffba}Pass{panel} |
> Run on Spark 1.6.1 for data sizes {80MB, 800MB, 8GB, 80GB} |
> | Performance Suite - Hadoop | | |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (SYSTEMML-708) Release checklist for 0.10.0-incubating-rc1
[
https://issues.apache.org/jira/browse/SYSTEMML-708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deron Eriksson closed SYSTEMML-708.
---
> Release checklist for 0.10.0-incubating-rc1
> ---
>
> Key: SYSTEMML-708
> URL: https://issues.apache.org/jira/browse/SYSTEMML-708
> Project: SystemML
> Issue Type: Task
>Reporter: Deron Eriksson
>Assignee: Deron Eriksson
> Fix For: SystemML 0.10
>
>
> || Task || Status || Notes ||
> | All Artifacts and Checksums Present | {panel:bgColor=#bfffba}Pass{panel} | |
> | Release Candidate Build - Windows | {panel:bgColor=#bfffba}Pass{panel} | |
> | Release Candidate Build - OS X | {panel:bgColor=#bfffba}Pass{panel} | |
> | Release Candidate Build - Linux | {panel:bgColor=#bfffba}Pass{panel} | |
> | Test Suite Passes - Windows |{panel:bgColor=#bfffba}Pass{panel} |
> SYSTEMML-712 opened for intermittent test failure |
> | Test Suite Passes - OS X| {panel:bgColor=#bfffba}Pass{panel} | |
> | Test Suite Passes - Linux | {panel:bgColor=#bfffba}Pass{panel} | |
> | All Binaries Execute| {panel:bgColor=#bfffba}Pass{panel} |
> Verified on OS X |
> | Check LICENSE and NOTICE Files | {panel:bgColor=#bfffba}Pass{panel} |
> non-blocker, SYSTEMML-711 filed |
> | Src Artifact Builds and Tests Pass | {panel:bgColor=#bfffba}Pass{panel} |
> 5037 of 5038 passed on OS X (RightIndexingMatrixTest failed) |
> | Single-Node Standalone - Windows| {panel:bgColor=#bfffba}Pass{panel} | |
> | Single-Node Standalone - OS X | {panel:bgColor=#bfffba}Pass{panel} | |
> | Single-Node Standalone - Linux | {panel:bgColor=#bfffba}Pass{panel} | |
> | Single-Node Spark | {panel:bgColor=#bfffba}Pass{panel} |
> Verified on OS X
> | Single-Node Hadoop | {panel:bgColor=#bfffba}Pass{panel} |
> Verified on OS X
> | Notebooks - Jupyter | {panel:bgColor=#bfffba}Pass{panel} |
> Verified on OS X |
> | Notebooks - Zeppelin| {panel:bgColor=#bfffba}Pass{panel} |
> Verified on OS X |
> | Performance Suite - Spark | {panel:bgColor=#bfffba}Pass{panel} |
> Run on Spark 1.6.1 for data sizes {80MB, 800MB, 8GB, 80GB} |
> | Performance Suite - Hadoop | | |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SYSTEMML-708) Release checklist for 0.10.0-incubating-rc1
[
https://issues.apache.org/jira/browse/SYSTEMML-708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15409894#comment-15409894
]
Deron Eriksson commented on SYSTEMML-708:
-
Note that RC1 was rejected due to unexpected binaries in the src release. This
was fixed in RC2.
> Release checklist for 0.10.0-incubating-rc1
> ---
>
> Key: SYSTEMML-708
> URL: https://issues.apache.org/jira/browse/SYSTEMML-708
> Project: SystemML
> Issue Type: Task
>Reporter: Deron Eriksson
>Assignee: Deron Eriksson
> Fix For: SystemML 0.10
>
>
> || Task || Status || Notes ||
> | All Artifacts and Checksums Present | {panel:bgColor=#bfffba}Pass{panel} | |
> | Release Candidate Build - Windows | {panel:bgColor=#bfffba}Pass{panel} | |
> | Release Candidate Build - OS X | {panel:bgColor=#bfffba}Pass{panel} | |
> | Release Candidate Build - Linux | {panel:bgColor=#bfffba}Pass{panel} | |
> | Test Suite Passes - Windows |{panel:bgColor=#bfffba}Pass{panel} |
> SYSTEMML-712 opened for intermittent test failure |
> | Test Suite Passes - OS X| {panel:bgColor=#bfffba}Pass{panel} | |
> | Test Suite Passes - Linux | {panel:bgColor=#bfffba}Pass{panel} | |
> | All Binaries Execute| {panel:bgColor=#bfffba}Pass{panel} |
> Verified on OS X |
> | Check LICENSE and NOTICE Files | {panel:bgColor=#bfffba}Pass{panel} |
> non-blocker, SYSTEMML-711 filed |
> | Src Artifact Builds and Tests Pass | {panel:bgColor=#bfffba}Pass{panel} |
> 5037 of 5038 passed on OS X (RightIndexingMatrixTest failed) |
> | Single-Node Standalone - Windows| {panel:bgColor=#bfffba}Pass{panel} | |
> | Single-Node Standalone - OS X | {panel:bgColor=#bfffba}Pass{panel} | |
> | Single-Node Standalone - Linux | {panel:bgColor=#bfffba}Pass{panel} | |
> | Single-Node Spark | {panel:bgColor=#bfffba}Pass{panel} |
> Verified on OS X
> | Single-Node Hadoop | {panel:bgColor=#bfffba}Pass{panel} |
> Verified on OS X
> | Notebooks - Jupyter | {panel:bgColor=#bfffba}Pass{panel} |
> Verified on OS X |
> | Notebooks - Zeppelin| {panel:bgColor=#bfffba}Pass{panel} |
> Verified on OS X |
> | Performance Suite - Spark | {panel:bgColor=#bfffba}Pass{panel} |
> Run on Spark 1.6.1 for data sizes {80MB, 800MB, 8GB, 80GB} |
> | Performance Suite - Hadoop | | |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (SYSTEMML-478) Upgrade maven-failsafe-plugin if fixed
[ https://issues.apache.org/jira/browse/SYSTEMML-478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson closed SYSTEMML-478. --- > Upgrade maven-failsafe-plugin if fixed > -- > > Key: SYSTEMML-478 > URL: https://issues.apache.org/jira/browse/SYSTEMML-478 > Project: SystemML > Issue Type: Improvement >Reporter: Deron Eriksson >Assignee: Deron Eriksson >Priority: Minor > Fix For: SystemML 0.11 > > > The maven-failsafe-plugin entry in pom.xml is set to version 2.17 due to: > "Failsafe 2.18 has a bug in computing # cores, so use 2.17" > The maven central repo has the following later versions of > maven-failsafe-plugin available: > 2.19.1 > 2.19 > 2.18.1 > We should determine if a later version of the plugin can be used and the > "HACK ALERT" message can be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SYSTEMML-478) Upgrade maven-failsafe-plugin if fixed
[ https://issues.apache.org/jira/browse/SYSTEMML-478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson resolved SYSTEMML-478. - Resolution: Won't Fix Assignee: Deron Eriksson Fix Version/s: SystemML 0.11 Since issue seems to currently exist with all versions later than 2.17, we shouldn't upgrade this plugin for now. > Upgrade maven-failsafe-plugin if fixed > -- > > Key: SYSTEMML-478 > URL: https://issues.apache.org/jira/browse/SYSTEMML-478 > Project: SystemML > Issue Type: Improvement >Reporter: Deron Eriksson >Assignee: Deron Eriksson >Priority: Minor > Fix For: SystemML 0.11 > > > The maven-failsafe-plugin entry in pom.xml is set to version 2.17 due to: > "Failsafe 2.18 has a bug in computing # cores, so use 2.17" > The maven central repo has the following later versions of > maven-failsafe-plugin available: > 2.19.1 > 2.19 > 2.18.1 > We should determine if a later version of the plugin can be used and the > "HACK ALERT" message can be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SYSTEMML-458) Create Spark batch mode guide
[ https://issues.apache.org/jira/browse/SYSTEMML-458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson updated SYSTEMML-458: Assignee: Mike Dusenberry > Create Spark batch mode guide > - > > Key: SYSTEMML-458 > URL: https://issues.apache.org/jira/browse/SYSTEMML-458 > Project: SystemML > Issue Type: Task > Components: Documentation >Reporter: Deron Eriksson >Assignee: Mike Dusenberry > > Spark batch mode is one of the primary execution modes of SystemML. Document > is needed similar to Hadoop Batch Mode document. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SYSTEMML-457) Create Standalone mode guide
[ https://issues.apache.org/jira/browse/SYSTEMML-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson updated SYSTEMML-457: Assignee: Mike Dusenberry > Create Standalone mode guide > > > Key: SYSTEMML-457 > URL: https://issues.apache.org/jira/browse/SYSTEMML-457 > Project: SystemML > Issue Type: Task > Components: Documentation >Reporter: Deron Eriksson >Assignee: Mike Dusenberry > > Create a document that focuses on Standalone mode execution. Documentation > has recently been reorganized to emphasize the various execution modes. > Quick Start Guide can be used as a starting point for this. Quick Start Guide > has a bit of a broader focus than only Standalone mode. > Topics to include: > Download release > Standalone on Linux, Mac, Windows > Hello World > Algorithm examples > Execution on single machine, Hadoop, and Spark. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (SYSTEMML-836) Create ScriptFactory convenience methods for resources on classpath
[
https://issues.apache.org/jira/browse/SYSTEMML-836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deron Eriksson reassigned SYSTEMML-836:
---
Assignee: Deron Eriksson
> Create ScriptFactory convenience methods for resources on classpath
> ---
>
> Key: SYSTEMML-836
> URL: https://issues.apache.org/jira/browse/SYSTEMML-836
> Project: SystemML
> Issue Type: Task
> Components: APIs
>Reporter: Deron Eriksson
>Assignee: Deron Eriksson
>Priority: Minor
>
> Currently a DML Script object can be created by ScriptFactory by getting an
> input stream to a resource on the classpath, such as this example:
> {code}
> val inputStream =
> getClass.getResourceAsStream("/scripts/algorithms/Univar-Stats.dml")
> val script = ScriptFactory.dmlFromInputStream(inputStream)
> {code}
> This can be further simplified by creating a ScriptFactory method like
> dmlFromResourcePath (or dmlFromClasspath) so a user could instead do:
> {code}
> val script =
> ScriptFactory.dmlFromResourcePath("/scripts/algorithms/Univar-Stats.dml")
> {code}
> An addition method should be created for PyDML, such as pydmlFromResourcePath
> (or pydmlFromClasspath).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (SYSTEMML-851) MLContext source from URLs
Deron Eriksson created SYSTEMML-851: --- Summary: MLContext source from URLs Key: SYSTEMML-851 URL: https://issues.apache.org/jira/browse/SYSTEMML-851 Project: SystemML Issue Type: Improvement Components: APIs Reporter: Deron Eriksson The new MLContext API allows scripts to be created from URLs. Additionally, it should allow scripts or groups of scripts to be imported/sourced from URLs. For example, there should be a mechanism to pull in the SystemML-NN package of DML scripts from a URL. Preferably, a user can do this with a single line of code (specify a URL to a script and that script and all sourced scripts get pulled in). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SYSTEMML-852) MLContext API javadocs online
Deron Eriksson created SYSTEMML-852: --- Summary: MLContext API javadocs online Key: SYSTEMML-852 URL: https://issues.apache.org/jira/browse/SYSTEMML-852 Project: SystemML Issue Type: Task Components: APIs, Documentation Reporter: Deron Eriksson Assignee: Deron Eriksson The new MLContext API has a significant amount of Javadoc documentation associated with it. This documentation needs to be available online for users. A determination needs to be made where this documentation should be placed. Possibilities include the main site and the documentation site on GitHub. In addition, these Javadocs should be included in the documentation for each version of SystemML that is released. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SYSTEMML-842) Remove useless javadoc comments and fix javadoc warnings
Deron Eriksson created SYSTEMML-842: --- Summary: Remove useless javadoc comments and fix javadoc warnings Key: SYSTEMML-842 URL: https://issues.apache.org/jira/browse/SYSTEMML-842 Project: SystemML Issue Type: Task Components: Documentation Reporter: Deron Eriksson The project contains an enormous number of useless empty javadoc comments. In addition, it has many other javadoc warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SYSTEMML-853) Create Python wrapper API for new MLContext API
Deron Eriksson created SYSTEMML-853: --- Summary: Create Python wrapper API for new MLContext API Key: SYSTEMML-853 URL: https://issues.apache.org/jira/browse/SYSTEMML-853 Project: SystemML Issue Type: Task Components: APIs Reporter: Deron Eriksson Assignee: Deron Eriksson The new MLContext API needs a Python wrapper API to access MLContext from Python. This will allow Jupyter notebooks to use the new API. The old Python API is located at src/main/java/org/apache/sysml/api/python/SystemML.py The new MLContext API is located in the org.apache.sysml.api.mlcontext package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SYSTEMML-853) Create Python wrapper API for new MLContext API
[ https://issues.apache.org/jira/browse/SYSTEMML-853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson updated SYSTEMML-853: Description: The new MLContext API needs a Python wrapper API to access MLContext from Python. This will allow Jupyter notebooks to use the new API. The old Python API is located at src/main/java/org/apache/sysml/api/python/SystemML.py The new MLContext API is located in the org.apache.sysml.api.mlcontext package. Examples of using the new MLContext API from Spark Shell can be found at http://apache.github.io/incubator-systemml/spark-mlcontext-programming-guide.html was: The new MLContext API needs a Python wrapper API to access MLContext from Python. This will allow Jupyter notebooks to use the new API. The old Python API is located at src/main/java/org/apache/sysml/api/python/SystemML.py The new MLContext API is located in the org.apache.sysml.api.mlcontext package. > Create Python wrapper API for new MLContext API > --- > > Key: SYSTEMML-853 > URL: https://issues.apache.org/jira/browse/SYSTEMML-853 > Project: SystemML > Issue Type: Task > Components: APIs >Reporter: Deron Eriksson >Assignee: Deron Eriksson > > The new MLContext API needs a Python wrapper API to access MLContext from > Python. This will allow Jupyter notebooks to use the new API. > The old Python API is located at > src/main/java/org/apache/sysml/api/python/SystemML.py > The new MLContext API is located in the org.apache.sysml.api.mlcontext > package. > Examples of using the new MLContext API from Spark Shell can be found at > http://apache.github.io/incubator-systemml/spark-mlcontext-programming-guide.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SYSTEMML-565) Documentation frame support
[ https://issues.apache.org/jira/browse/SYSTEMML-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15423314#comment-15423314 ] Deron Eriksson commented on SYSTEMML-565: - For now, let's add this information to the DML Language Reference in its own section. The DML Language Reference needs to be restructured/refactored, so adding Frames in its own section would be helpful for this documentation refactoring. > Documentation frame support > --- > > Key: SYSTEMML-565 > URL: https://issues.apache.org/jira/browse/SYSTEMML-565 > Project: SystemML > Issue Type: Documentation >Reporter: Matthias Boehm >Assignee: Arvind Surve > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SYSTEMML-744) Update LogisticRegression wrapper class example for newer Spark version
[ https://issues.apache.org/jira/browse/SYSTEMML-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson resolved SYSTEMML-744. - Resolution: Fixed Fix Version/s: SystemML 0.11 LogisticRegression.java is now deleted and MLBlock.java has getTimestamp, so this issue should be resolved now. > Update LogisticRegression wrapper class example for newer Spark version > --- > > Key: SYSTEMML-744 > URL: https://issues.apache.org/jira/browse/SYSTEMML-744 > Project: SystemML > Issue Type: Improvement > Components: APIs >Reporter: Glenn Weidner >Assignee: Deron Eriksson >Priority: Minor > Fix For: SystemML 0.11 > > > When building SystemML against newer Spark version (e.g., 1.6.1), two files > failed to compile: > src/main/java/org/apache/sysml/api/javaml/LogisticRegression.java > src/main/java/org/apache/sysml/api/MLBlock.java > Default version uses Spark 1.4.1 to maintain backwards compatibility and > prevent utilizing interfaces only available in newer Spark versions. It > would be helpful to update (or provide additional) wrapper in cases where > building against later Spark version is required. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (SYSTEMML-837) Add additional Explain options to new MLContext API
[ https://issues.apache.org/jira/browse/SYSTEMML-837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson reassigned SYSTEMML-837: --- Assignee: Deron Eriksson > Add additional Explain options to new MLContext API > --- > > Key: SYSTEMML-837 > URL: https://issues.apache.org/jira/browse/SYSTEMML-837 > Project: SystemML > Issue Type: Task > Components: APIs >Reporter: Deron Eriksson >Assignee: Deron Eriksson >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SYSTEMML-565) Documentation frame support
[ https://issues.apache.org/jira/browse/SYSTEMML-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15423331#comment-15423331 ] Deron Eriksson commented on SYSTEMML-565: - I created SYSTEMML-861 to deal with refactoring the DML Language Reference. > Documentation frame support > --- > > Key: SYSTEMML-565 > URL: https://issues.apache.org/jira/browse/SYSTEMML-565 > Project: SystemML > Issue Type: Documentation >Reporter: Matthias Boehm >Assignee: Arvind Surve > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SYSTEMML-861) Restructure DML Language Reference
Deron Eriksson created SYSTEMML-861: --- Summary: Restructure DML Language Reference Key: SYSTEMML-861 URL: https://issues.apache.org/jira/browse/SYSTEMML-861 Project: SystemML Issue Type: Task Components: Documentation Reporter: Deron Eriksson Assignee: Deron Eriksson The DML Language Reference contains lots of great information. However, it needs to be refactored to make it more web-friendly. Additionally, the PYDML Language Reference could potentially either be folded into the DML Language Reference, or at least based on the same format. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SYSTEMML-862) Fail gracefully if Spark version not supported
Deron Eriksson created SYSTEMML-862: --- Summary: Fail gracefully if Spark version not supported Key: SYSTEMML-862 URL: https://issues.apache.org/jira/browse/SYSTEMML-862 Project: SystemML Issue Type: Task Components: Runtime Reporter: Deron Eriksson Assignee: Glenn Weidner If SystemML is run with a Spark version that is not supported, it should fail gracefully. See isSparkVersionSupported() and compareVersion() in MLContextUtil. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SYSTEMML-860) SparkR/HydraR integration with SystemML
[ https://issues.apache.org/jira/browse/SYSTEMML-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson updated SYSTEMML-860: Assignee: Alok Singh > SparkR/HydraR integration with SystemML > --- > > Key: SYSTEMML-860 > URL: https://issues.apache.org/jira/browse/SYSTEMML-860 > Project: SystemML > Issue Type: Improvement > Components: APIs, Runtime >Affects Versions: SystemML 0.11 >Reporter: Alok Singh >Assignee: Alok Singh > > With our effort to create the BigR like R pacakge on the top of the SystemML > and Spark. We have extended the SparkR package and call it HydraR. > However, for that package to work, we have to enhance the MLContext, and > RDDConverterUtilsExt, so that DataFrame and other args can be passed to > SystemML via the SparkR connector. > This effort is for SystemML related java code. > Alok -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SYSTEMML-57) Expand and organize documentation for Python-like language syntax
[ https://issues.apache.org/jira/browse/SYSTEMML-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425647#comment-15425647 ] Deron Eriksson commented on SYSTEMML-57: I will assign this issue to me. Very shortly I will be reworking the DML Language Reference [SYSTEMML-861], and this would be a good time to tackle the PyDML so that everything is symmetrical and consistent across the languages. > Expand and organize documentation for Python-like language syntax > - > > Key: SYSTEMML-57 > URL: https://issues.apache.org/jira/browse/SYSTEMML-57 > Project: SystemML > Issue Type: Task >Reporter: Frederick Reiss >Assignee: Deron Eriksson > Original Estimate: 40h > Remaining Estimate: 40h > > Turn the existing documentation for PyDML into a proper language reference > manual, with a detailed table of contents and a full description of every > language feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SYSTEMML-1229) Add python mlcontext example to engine dev guide
Deron Eriksson created SYSTEMML-1229: Summary: Add python mlcontext example to engine dev guide Key: SYSTEMML-1229 URL: https://issues.apache.org/jira/browse/SYSTEMML-1229 Project: SystemML Issue Type: Task Components: APIs, Documentation Reporter: Deron Eriksson Assignee: Deron Eriksson Priority: Minor Add a basic python mlcontext example to engine dev guide to help developers get started with the python mlcontext api. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-1192) Optionally delete rmvar instructions via MLContext
[
https://issues.apache.org/jira/browse/SYSTEMML-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deron Eriksson resolved SYSTEMML-1192.
--
Resolution: Fixed
Fix Version/s: SystemML 0.13
Fixed by [PR352|https://github.com/apache/incubator-systemml/pull/352].
Optionally can be turned on for troubleshooting by:
{code}
ml.setMaintainSymbolTable(true)
{code}
> Optionally delete rmvar instructions via MLContext
> --
>
> Key: SYSTEMML-1192
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1192
> Project: SystemML
> Issue Type: Improvement
> Components: APIs
>Reporter: Deron Eriksson
>Assignee: Deron Eriksson
> Fix For: SystemML 0.13
>
>
> The symbol table (LocalVariableMap) contains data that is useful for
> determining what happens when a SystemML script executes. Removing all rmvar
> instructions allows all the symbol table values to be displayed through the
> MLContext API, which can be useful when debugging. The symbol table can be
> cleared if needed by calling Script's clearSymbolTable() method.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (SYSTEMML-1192) Optionally delete rmvar instructions via MLContext
[ https://issues.apache.org/jira/browse/SYSTEMML-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson closed SYSTEMML-1192. > Optionally delete rmvar instructions via MLContext > -- > > Key: SYSTEMML-1192 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1192 > Project: SystemML > Issue Type: Improvement > Components: APIs >Reporter: Deron Eriksson >Assignee: Deron Eriksson > Fix For: SystemML 0.13 > > > The symbol table (LocalVariableMap) contains data that is useful for > determining what happens when a SystemML script executes. Removing all rmvar > instructions allows all the symbol table values to be displayed through the > MLContext API, which can be useful when debugging. The symbol table can be > cleared if needed by calling Script's clearSymbolTable() method. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SYSTEMML-1222) MLContext `read` Statement Value Input Error For `cols_in_block` Argument
[
https://issues.apache.org/jira/browse/SYSTEMML-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850636#comment-15850636
]
Deron Eriksson commented on SYSTEMML-1222:
--
Since Long is not a recognized SystemML value type, it makes sense to do the
long to int conversion high up in the API.
Value types in SystemML (from Statement.java):
{code}
public static final String DOUBLE_VALUE_TYPE = "double";
public static final String BOOLEAN_VALUE_TYPE = "boolean";
public static final String INT_VALUE_TYPE = "int";
public static final String STRING_VALUE_TYPE = "string";
{code}
Thank for finding this [~mwdus...@us.ibm.com].
https://github.com/apache/incubator-systemml/pull/370 addresses this.
> MLContext `read` Statement Value Input Error For `cols_in_block` Argument
> -
>
> Key: SYSTEMML-1222
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1222
> Project: SystemML
> Issue Type: Bug
>Reporter: Mike Dusenberry
>Assignee: Deron Eriksson
>Priority: Minor
>
> The {{read}} statement has optional numeric parameters, such as
> {{cols_in_block}} that expect integers. If a Scala {{Long}} is supplied as
> an input variable for one of those parameters via MLContext, and type error
> will occur in which an {{Int}} was expected, but a {{Long}} was received. A
> quick fix is to convert to an integer with {{Long.toInt}} before passing the
> value into the script via MLContext.
> We should automatically cast {{Long}} to {{Int}} internally.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SYSTEMML-1181) Update documentation with changes related to Spark 2.0
[ https://issues.apache.org/jira/browse/SYSTEMML-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850782#comment-15850782 ] Deron Eriksson commented on SYSTEMML-1181: -- [PR371|https://github.com/apache/incubator-systemml/pull/371] addressed sqlContext in the documentation. Are further documentation updates happening under this PR or should it be resolved? [~fschueler] > Update documentation with changes related to Spark 2.0 > -- > > Key: SYSTEMML-1181 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1181 > Project: SystemML > Issue Type: Documentation >Reporter: Arvind Surve >Assignee: Felix Schüler > > Update web page for any changes related to SystemML on Spark 2.0. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SYSTEMML-1181) Update documentation with changes related to Spark 2.0
[ https://issues.apache.org/jira/browse/SYSTEMML-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850698#comment-15850698 ] Deron Eriksson commented on SYSTEMML-1181: -- For Spark 2, I would recommend not spending time updating the docs for the old MLContext API unless it is fairly trivial to update it. This API will be removed in the near future as mentioned in the Roadmap: https://www.mail-archive.com/dev@systemml.incubator.apache.org/msg01199.html > Update documentation with changes related to Spark 2.0 > -- > > Key: SYSTEMML-1181 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1181 > Project: SystemML > Issue Type: Documentation >Reporter: Arvind Surve >Assignee: Felix Schüler > > Update web page for any changes related to SystemML on Spark 2.0. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (SYSTEMML-1226) Add python validation to release process
Deron Eriksson created SYSTEMML-1226: Summary: Add python validation to release process Key: SYSTEMML-1226 URL: https://issues.apache.org/jira/browse/SYSTEMML-1226 Project: SystemML Issue Type: Task Components: Documentation Reporter: Deron Eriksson Assignee: Deron Eriksson Add python tests to the release validation. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (SYSTEMML-1226) Add python validation to release process
[ https://issues.apache.org/jira/browse/SYSTEMML-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson closed SYSTEMML-1226. > Add python validation to release process > > > Key: SYSTEMML-1226 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1226 > Project: SystemML > Issue Type: Task > Components: Documentation >Reporter: Deron Eriksson >Assignee: Deron Eriksson > Fix For: SystemML 0.13 > > > Add python tests to the release validation. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-1230) Add MLContext info functionality to docs
[ https://issues.apache.org/jira/browse/SYSTEMML-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson resolved SYSTEMML-1230. -- Resolution: Fixed Fix Version/s: SystemML 0.13 Fixed by commit https://github.com/apache/incubator-systemml/commit/503307bce84d6fae92f29c01238fc916a7132fca > Add MLContext info functionality to docs > > > Key: SYSTEMML-1230 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1230 > Project: SystemML > Issue Type: Task > Components: APIs, Documentation >Reporter: Deron Eriksson >Assignee: Deron Eriksson >Priority: Minor > Fix For: SystemML 0.13 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SYSTEMML-610) Add Databricks Notebook example to MLContext documentation
[ https://issues.apache.org/jira/browse/SYSTEMML-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852309#comment-15852309 ] Deron Eriksson commented on SYSTEMML-610: - [~fschueler] I believe this would be useful. cc [~freiss] [~reinwald] > Add Databricks Notebook example to MLContext documentation > -- > > Key: SYSTEMML-610 > URL: https://issues.apache.org/jira/browse/SYSTEMML-610 > Project: SystemML > Issue Type: Task >Reporter: Deron Eriksson >Assignee: Deron Eriksson > > Add an example of configuring a Databricks Notebook to use SystemML to the > Spark MLContext Programming Guide. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (SYSTEMML-610) Add Databricks Notebook example to MLContext documentation
[ https://issues.apache.org/jira/browse/SYSTEMML-610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson reassigned SYSTEMML-610: --- Assignee: (was: Deron Eriksson) > Add Databricks Notebook example to MLContext documentation > -- > > Key: SYSTEMML-610 > URL: https://issues.apache.org/jira/browse/SYSTEMML-610 > Project: SystemML > Issue Type: Task >Reporter: Deron Eriksson > > Add an example of configuring a Databricks Notebook to use SystemML to the > Spark MLContext Programming Guide. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (SYSTEMML-1230) Add MLContext info to docs
Deron Eriksson created SYSTEMML-1230: Summary: Add MLContext info to docs Key: SYSTEMML-1230 URL: https://issues.apache.org/jira/browse/SYSTEMML-1230 Project: SystemML Issue Type: Task Components: APIs, Documentation Reporter: Deron Eriksson Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SYSTEMML-1230) Add MLContext info functionality to docs
[ https://issues.apache.org/jira/browse/SYSTEMML-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson updated SYSTEMML-1230: - Summary: Add MLContext info functionality to docs (was: Add MLContext info to docs) > Add MLContext info functionality to docs > > > Key: SYSTEMML-1230 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1230 > Project: SystemML > Issue Type: Task > Components: APIs, Documentation >Reporter: Deron Eriksson >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-1227) Cleanup pyc files
[ https://issues.apache.org/jira/browse/SYSTEMML-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson resolved SYSTEMML-1227. -- Resolution: Fixed Fix Version/s: SystemML 0.13 Fixed by [PR372|https://github.com/apache/incubator-systemml/pull/372]. > Cleanup pyc files > - > > Key: SYSTEMML-1227 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1227 > Project: SystemML > Issue Type: Task > Components: Build >Reporter: Deron Eriksson >Assignee: Deron Eriksson >Priority: Minor > Fix For: SystemML 0.13 > > > Currently *.pyc files can be generated under src/main/python (for instance > when tests are run). > They should be cleaned up when 'mvn clean' is run. > Additionally the *.pyc files should be added to .gitignore. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-1212) Link to main website in header of project documentation
[ https://issues.apache.org/jira/browse/SYSTEMML-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson resolved SYSTEMML-1212. -- Resolution: Fixed Fix Version/s: SystemML 0.13 Fixed by [PR367|https://github.com/apache/incubator-systemml/pull/367]. > Link to main website in header of project documentation > --- > > Key: SYSTEMML-1212 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1212 > Project: SystemML > Issue Type: Improvement > Components: Documentation >Reporter: Deron Eriksson >Assignee: Deron Eriksson > Fix For: SystemML 0.13 > > > Currently there is no link to the main website in the header of the project > documentation. Therefore, once at the project documentation, it can be > difficult to get back to the main website. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SYSTEMML-1232) Migrate stringDataFrameToVectorDataFrame if needed
[
https://issues.apache.org/jira/browse/SYSTEMML-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852566#comment-15852566
]
Deron Eriksson commented on SYSTEMML-1232:
--
cc [~niketanpansare] [~a1singh]
> Migrate stringDataFrameToVectorDataFrame if needed
> --
>
> Key: SYSTEMML-1232
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1232
> Project: SystemML
> Issue Type: Task
> Components: APIs
>Reporter: Deron Eriksson
>
> Restore and migrate RDDConvererUtilsExt.stringDataFrameToVectorDataFrame
> method to Spark 2 (mllib->ml Vector) if needed.
> The RDDConvererUtilsExt.stringDataFrameToVectorDataFrame method was removed
> by commit
> https://github.com/apache/incubator-systemml/commit/578e595fdc506fb8a0c0b18c312fe420a406276d.
> If this method is needed, migrate it to Spark 2.
> Old method:
> {code}
> public static Dataset stringDataFrameToVectorDataFrame(SQLContext
> sqlContext, Dataset inputDF)
> throws DMLRuntimeException {
> StructField[] oldSchema = inputDF.schema().fields();
> //create the new schema
> StructField[] newSchema = new StructField[oldSchema.length];
> for(int i = 0; i < oldSchema.length; i++) {
> String colName = oldSchema[i].name();
> newSchema[i] = DataTypes.createStructField(colName, new
> VectorUDT(), true);
> }
> //converter
> class StringToVector implements Function,
> Row> {
> private static final long serialVersionUID =
> -4733816995375745659L;
> @Override
> public Row call(Tuple2 arg0) throws
> Exception {
> Row oldRow = arg0._1;
> int oldNumCols = oldRow.length();
> if (oldNumCols > 1) {
> throw new DMLRuntimeException("The row
> must have at most one column");
> }
> // parse the various strings. i.e
> // ((1.2,4.3, 3.4)) or (1.2, 3.4, 2.2) or (1.2
> 3.4)
> // [[1.2,34.3, 1.2, 1.2]] or [1.2, 3.4] or [1.3
> 1.2]
> Object [] fields = new Object[oldNumCols];
> ArrayList fieldsArr = new
> ArrayList();
> for (int i = 0; i < oldRow.length(); i++) {
> Object ci=oldRow.get(i);
> if (ci instanceof String) {
> String cis = (String)ci;
> StringBuffer sb = new
> StringBuffer(cis.trim());
> for (int nid=0; i < 2; i++) {
> //remove two level nesting
> if ((sb.charAt(0) ==
> '(' && sb.charAt(sb.length() - 1) == ')') ||
>
> (sb.charAt(0) == '[' && sb.charAt(sb.length() - 1) == ']')
> ) {
>
> sb.deleteCharAt(0);
>
> sb.setLength(sb.length() - 1);
> }
> }
> //have the replace code
> String ncis = "[" +
> sb.toString().replaceAll(" *, *", ",") + "]";
> Vector v = Vectors.parse(ncis);
> fieldsArr.add(v);
> } else {
> throw new
> DMLRuntimeException("Only String is supported");
> }
> }
> Row row =
> RowFactory.create(fieldsArr.toArray());
> return row;
> }
> }
> //output DF
> JavaRDD newRows =
> inputDF.rdd().toJavaRDD().zipWithIndex().map(new StringToVector());
> // DataFrame outDF = sqlContext.createDataFrame(newRows, new
> StructType(newSchema)); //TODO investigate why it doesn't work
> Dataset outDF = sqlContext.createDataFrame(newRows.rdd(),
> DataTypes.createStructType(newSchema));
> return outDF;
> }
> {code}
> Note: the
[jira] [Commented] (SYSTEMML-1181) Update documentation with changes related to Spark 2.1.0
[ https://issues.apache.org/jira/browse/SYSTEMML-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852494#comment-15852494 ] Deron Eriksson commented on SYSTEMML-1181: -- [PR377|https://github.com/apache/incubator-systemml/pull/377] submitted by [~fschueler] removed the old MLContext API from the Spark MLContext Programming Guide. > Update documentation with changes related to Spark 2.1.0 > > > Key: SYSTEMML-1181 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1181 > Project: SystemML > Issue Type: Documentation >Reporter: Arvind Surve >Assignee: Felix Schüler > > Update web page for any changes related to SystemML on Spark 2.1.0 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (SYSTEMML-1231) Migrate pyspark.mllib.common to pyspark.ml.common
Deron Eriksson created SYSTEMML-1231: Summary: Migrate pyspark.mllib.common to pyspark.ml.common Key: SYSTEMML-1231 URL: https://issues.apache.org/jira/browse/SYSTEMML-1231 Project: SystemML Issue Type: Task Components: APIs Affects Versions: SystemML 0.13 Reporter: Deron Eriksson Investigate migrating pyspark.mllib.common in the Python MLContext to the newer pyspark.ml.common, as mentioned in https://github.com/apache/incubator-systemml/pull/369 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SYSTEMML-1231) Migrate pyspark.mllib.common to pyspark.ml.common
[ https://issues.apache.org/jira/browse/SYSTEMML-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson updated SYSTEMML-1231: - Description: Investigate migrating pyspark.mllib.common in the Python MLContext to the newer pyspark.ml.common, as mentioned in https://github.com/apache/incubator-systemml/pull/369 for Spark 2. (was: Investigate migrating pyspark.mllib.common in the Python MLContext to the newer pyspark.ml.common, as mentioned in https://github.com/apache/incubator-systemml/pull/369) > Migrate pyspark.mllib.common to pyspark.ml.common > - > > Key: SYSTEMML-1231 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1231 > Project: SystemML > Issue Type: Task > Components: APIs >Affects Versions: SystemML 0.13 >Reporter: Deron Eriksson > > Investigate migrating pyspark.mllib.common in the Python MLContext to the > newer pyspark.ml.common, as mentioned in > https://github.com/apache/incubator-systemml/pull/369 for Spark 2. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SYSTEMML-1231) Migrate pyspark.mllib.common to pyspark.ml.common
[ https://issues.apache.org/jira/browse/SYSTEMML-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852548#comment-15852548 ] Deron Eriksson commented on SYSTEMML-1231: -- cc [~iyounus] [~mwdus...@us.ibm.com] [~niketanpansare] > Migrate pyspark.mllib.common to pyspark.ml.common > - > > Key: SYSTEMML-1231 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1231 > Project: SystemML > Issue Type: Task > Components: APIs >Affects Versions: SystemML 0.13 >Reporter: Deron Eriksson > > Investigate migrating pyspark.mllib.common in the Python MLContext to the > newer pyspark.ml.common, as mentioned in > https://github.com/apache/incubator-systemml/pull/369 for Spark 2. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (SYSTEMML-1232) Migrate stringDataFrameToVectorDataFrame if needed
Deron Eriksson created SYSTEMML-1232:
Summary: Migrate stringDataFrameToVectorDataFrame if needed
Key: SYSTEMML-1232
URL: https://issues.apache.org/jira/browse/SYSTEMML-1232
Project: SystemML
Issue Type: Task
Components: APIs
Reporter: Deron Eriksson
Restore and migrate RDDConvererUtilsExt.stringDataFrameToVectorDataFrame method
to Spark 2 (mllib->ml Vector) if needed.
The RDDConvererUtilsExt.stringDataFrameToVectorDataFrame method was removed by
commit
https://github.com/apache/incubator-systemml/commit/578e595fdc506fb8a0c0b18c312fe420a406276d.
If this method is needed, migrate it to Spark 2.
Old method:
{code}
public static Dataset stringDataFrameToVectorDataFrame(SQLContext
sqlContext, Dataset inputDF)
throws DMLRuntimeException {
StructField[] oldSchema = inputDF.schema().fields();
//create the new schema
StructField[] newSchema = new StructField[oldSchema.length];
for(int i = 0; i < oldSchema.length; i++) {
String colName = oldSchema[i].name();
newSchema[i] = DataTypes.createStructField(colName, new
VectorUDT(), true);
}
//converter
class StringToVector implements Function,
Row> {
private static final long serialVersionUID =
-4733816995375745659L;
@Override
public Row call(Tuple2 arg0) throws
Exception {
Row oldRow = arg0._1;
int oldNumCols = oldRow.length();
if (oldNumCols > 1) {
throw new DMLRuntimeException("The row
must have at most one column");
}
// parse the various strings. i.e
// ((1.2,4.3, 3.4)) or (1.2, 3.4, 2.2) or (1.2
3.4)
// [[1.2,34.3, 1.2, 1.2]] or [1.2, 3.4] or [1.3
1.2]
Object [] fields = new Object[oldNumCols];
ArrayList fieldsArr = new
ArrayList();
for (int i = 0; i < oldRow.length(); i++) {
Object ci=oldRow.get(i);
if (ci instanceof String) {
String cis = (String)ci;
StringBuffer sb = new
StringBuffer(cis.trim());
for (int nid=0; i < 2; i++) {
//remove two level nesting
if ((sb.charAt(0) ==
'(' && sb.charAt(sb.length() - 1) == ')') ||
(sb.charAt(0) == '[' && sb.charAt(sb.length() - 1) == ']')
) {
sb.deleteCharAt(0);
sb.setLength(sb.length() - 1);
}
}
//have the replace code
String ncis = "[" +
sb.toString().replaceAll(" *, *", ",") + "]";
Vector v = Vectors.parse(ncis);
fieldsArr.add(v);
} else {
throw new
DMLRuntimeException("Only String is supported");
}
}
Row row =
RowFactory.create(fieldsArr.toArray());
return row;
}
}
//output DF
JavaRDD newRows =
inputDF.rdd().toJavaRDD().zipWithIndex().map(new StringToVector());
// DataFrame outDF = sqlContext.createDataFrame(newRows, new
StructType(newSchema)); //TODO investigate why it doesn't work
Dataset outDF = sqlContext.createDataFrame(newRows.rdd(),
DataTypes.createStructType(newSchema));
return outDF;
}
{code}
Note: the org.apache.spark.ml.linalg.Vectors.parse() method does not exist in
Spark 2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (SYSTEMML-57) Expand and organize documentation for Python-like language syntax
[ https://issues.apache.org/jira/browse/SYSTEMML-57?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson reassigned SYSTEMML-57: -- Assignee: (was: Deron Eriksson) > Expand and organize documentation for Python-like language syntax > - > > Key: SYSTEMML-57 > URL: https://issues.apache.org/jira/browse/SYSTEMML-57 > Project: SystemML > Issue Type: Task >Reporter: Frederick Reiss > Original Estimate: 40h > Remaining Estimate: 40h > > Turn the existing documentation for PyDML into a proper language reference > manual, with a detailed table of contents and a full description of every > language feature. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Reopened] (SYSTEMML-1220) dmlFromResource capability in Python (equivalent to Scala
[ https://issues.apache.org/jira/browse/SYSTEMML-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson reopened SYSTEMML-1220: -- Reopened thanks to discussion on PR. > dmlFromResource capability in Python (equivalent to Scala > - > > Key: SYSTEMML-1220 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1220 > Project: SystemML > Issue Type: Improvement > Components: APIs >Affects Versions: SystemML 0.13 >Reporter: Berthold Reinwald >Assignee: Deron Eriksson >Priority: Minor > Fix For: Not Applicable > > > an equivalent mechanism to dmlFromResource in Scala is missing in Python -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (SYSTEMML-1220) dmlFromResource capability in Python (equivalent to Scala
[ https://issues.apache.org/jira/browse/SYSTEMML-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson closed SYSTEMML-1220. > dmlFromResource capability in Python (equivalent to Scala > - > > Key: SYSTEMML-1220 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1220 > Project: SystemML > Issue Type: Improvement > Components: APIs >Affects Versions: SystemML 0.13 >Reporter: Berthold Reinwald >Assignee: Deron Eriksson >Priority: Minor > Fix For: SystemML 0.13 > > > an equivalent mechanism to dmlFromResource in Scala is missing in Python -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SYSTEMML-1204) Deploy 0.12.0 project documentation to main website
[ https://issues.apache.org/jira/browse/SYSTEMML-1204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson updated SYSTEMML-1204: - Summary: Deploy 0.12.0 project documentation to main website (was: Deploy 0.11.0 project documentation to main website) > Deploy 0.12.0 project documentation to main website > --- > > Key: SYSTEMML-1204 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1204 > Project: SystemML > Issue Type: Sub-task > Components: Documentation, Website >Reporter: Deron Eriksson >Assignee: Deron Eriksson > > Generate documentation site for the previous 0.11.0 (make sure _config.yml > version is correct). Deploy to main website following a folder structure > similar to Spark. This should just be committed to svn rather than first to > git and then to svn. > Example Spark URLs: > http://spark.apache.org/docs/2.1.0/ > http://spark.apache.org/docs/2.0.2/ -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-572) Update transform documentation
[ https://issues.apache.org/jira/browse/SYSTEMML-572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson resolved SYSTEMML-572. - Resolution: Won't Fix Fix Version/s: SystemML 0.13 > Update transform documentation > -- > > Key: SYSTEMML-572 > URL: https://issues.apache.org/jira/browse/SYSTEMML-572 > Project: SystemML > Issue Type: Documentation >Reporter: Matthias Boehm >Assignee: Deron Eriksson > Fix For: SystemML 0.13 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SYSTEMML-57) Expand and organize documentation for Python-like language syntax
[ https://issues.apache.org/jira/browse/SYSTEMML-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855130#comment-15855130 ] Deron Eriksson commented on SYSTEMML-57: I am not aware of anyone using PyDML so I don't think I will tackle this issue unless there is a compelling use case. > Expand and organize documentation for Python-like language syntax > - > > Key: SYSTEMML-57 > URL: https://issues.apache.org/jira/browse/SYSTEMML-57 > Project: SystemML > Issue Type: Task >Reporter: Frederick Reiss >Assignee: Deron Eriksson > Original Estimate: 40h > Remaining Estimate: 40h > > Turn the existing documentation for PyDML into a proper language reference > manual, with a detailed table of contents and a full description of every > language feature. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-1220) dmlFromResource capability in Python (equivalent to Scala
[ https://issues.apache.org/jira/browse/SYSTEMML-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson resolved SYSTEMML-1220. -- Resolution: Won't Fix Fix Version/s: Not Applicable > dmlFromResource capability in Python (equivalent to Scala > - > > Key: SYSTEMML-1220 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1220 > Project: SystemML > Issue Type: Improvement > Components: APIs >Affects Versions: SystemML 0.13 >Reporter: Berthold Reinwald >Assignee: Deron Eriksson >Priority: Minor > Fix For: Not Applicable > > > an equivalent mechanism to dmlFromResource in Scala is missing in Python -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (SYSTEMML-1237) Add input float support to MLContext
Deron Eriksson created SYSTEMML-1237: Summary: Add input float support to MLContext Key: SYSTEMML-1237 URL: https://issues.apache.org/jira/browse/SYSTEMML-1237 Project: SystemML Issue Type: Task Components: APIs Reporter: Deron Eriksson Assignee: Deron Eriksson Priority: Minor Currently java.lang.Float is not a supported input type for the new MLContext API. This type can be cast to a double in `public Script in(String name, Object value, Metadata metadata)` to implicitly add float input support to the API. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (SYSTEMML-572) Update transform documentation
[ https://issues.apache.org/jira/browse/SYSTEMML-572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson closed SYSTEMML-572. --- > Update transform documentation > -- > > Key: SYSTEMML-572 > URL: https://issues.apache.org/jira/browse/SYSTEMML-572 > Project: SystemML > Issue Type: Documentation >Reporter: Matthias Boehm >Assignee: Deron Eriksson > Fix For: SystemML 0.13 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (SYSTEMML-1236) Move MLContextProxy out of api package
[ https://issues.apache.org/jira/browse/SYSTEMML-1236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson reassigned SYSTEMML-1236: Assignee: Deron Eriksson > Move MLContextProxy out of api package > -- > > Key: SYSTEMML-1236 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1236 > Project: SystemML > Issue Type: Task > Components: APIs, Runtime >Reporter: Deron Eriksson >Assignee: Deron Eriksson > > org.apache.sysml.api.MLContextProxy is not really a class that API users need > to be exposed to, so move it to another package to keep the api packages > clean. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-1209) Add configurable jekyll option to add API docs link to documentation header
[ https://issues.apache.org/jira/browse/SYSTEMML-1209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson resolved SYSTEMML-1209. -- Resolution: Fixed Fix Version/s: SystemML 0.13 Fixed by https://github.com/apache/incubator-systemml/commit/54b35455752472ec0345828030f860da772942fb > Add configurable jekyll option to add API docs link to documentation header > --- > > Key: SYSTEMML-1209 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1209 > Project: SystemML > Issue Type: Task > Components: Documentation >Reporter: Deron Eriksson >Assignee: Deron Eriksson > Fix For: SystemML 0.13 > > > Add a configurable jekyll option to conditionally allow an API docs link to > be added to the header of the project documentation. This option should be on > when the documentation is generated for the main website since the versioned > project documentation should be able to point to the versioned javadoc > documentation. > The FEEDBACK_LINKS option in _config.yml is an example of allowing such a > condition. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (SYSTEMML-1209) Add configurable jekyll option to add API docs link to documentation header
[ https://issues.apache.org/jira/browse/SYSTEMML-1209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson closed SYSTEMML-1209. > Add configurable jekyll option to add API docs link to documentation header > --- > > Key: SYSTEMML-1209 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1209 > Project: SystemML > Issue Type: Task > Components: Documentation >Reporter: Deron Eriksson >Assignee: Deron Eriksson > Fix For: SystemML 0.13 > > > Add a configurable jekyll option to conditionally allow an API docs link to > be added to the header of the project documentation. This option should be on > when the documentation is generated for the main website since the versioned > project documentation should be able to point to the versioned javadoc > documentation. > The FEEDBACK_LINKS option in _config.yml is an example of allowing such a > condition. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (SYSTEMML-1233) Deprecate old MLContext API
[ https://issues.apache.org/jira/browse/SYSTEMML-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson closed SYSTEMML-1233. > Deprecate old MLContext API > --- > > Key: SYSTEMML-1233 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1233 > Project: SystemML > Issue Type: Task > Components: APIs >Affects Versions: SystemML 0.13 >Reporter: Deron Eriksson >Assignee: Deron Eriksson > Fix For: SystemML 0.13 > > > Deprecate the old MLContext API as it will be removed. It has been replaced > by the new MLContext API. > Most likely it will be removed for the SystemML 1.0 release. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-1233) Deprecate old MLContext API
[ https://issues.apache.org/jira/browse/SYSTEMML-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson resolved SYSTEMML-1233. -- Resolution: Fixed Fix Version/s: SystemML 0.13 Fixed by [PR378|https://github.com/apache/incubator-systemml/pull/378]. > Deprecate old MLContext API > --- > > Key: SYSTEMML-1233 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1233 > Project: SystemML > Issue Type: Task > Components: APIs >Affects Versions: SystemML 0.13 >Reporter: Deron Eriksson >Assignee: Deron Eriksson > Fix For: SystemML 0.13 > > > Deprecate the old MLContext API as it will be removed. It has been replaced > by the new MLContext API. > Most likely it will be removed for the SystemML 1.0 release. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-1235) Migrate GNMFTest to new MLContext
[ https://issues.apache.org/jira/browse/SYSTEMML-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson resolved SYSTEMML-1235. -- Resolution: Fixed Fix Version/s: SystemML 0.13 Fixed by [PR381|https://github.com/apache/incubator-systemml/pull/381]. > Migrate GNMFTest to new MLContext > - > > Key: SYSTEMML-1235 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1235 > Project: SystemML > Issue Type: Task > Components: Test >Reporter: Deron Eriksson >Assignee: Deron Eriksson > Fix For: SystemML 0.13 > > > org.apache.sysml.test.integration.functions.mlcontext.GNMFTest currently uses > the old MLContext API. It should be migrated to the new API. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-1237) Add input float support to MLContext
[ https://issues.apache.org/jira/browse/SYSTEMML-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson resolved SYSTEMML-1237. -- Resolution: Fixed Fix Version/s: SystemML 0.13 Fixed by [PR382|https://github.com/apache/incubator-systemml/pull/382]. > Add input float support to MLContext > > > Key: SYSTEMML-1237 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1237 > Project: SystemML > Issue Type: Task > Components: APIs >Reporter: Deron Eriksson >Assignee: Deron Eriksson >Priority: Minor > Fix For: SystemML 0.13 > > > Currently java.lang.Float is not a supported input type for the new MLContext > API. This type can be cast to a double in `public Script in(String name, > Object value, Metadata metadata)` to implicitly add float input support to > the API. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (SYSTEMML-1237) Add input float support to MLContext
[ https://issues.apache.org/jira/browse/SYSTEMML-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson closed SYSTEMML-1237. > Add input float support to MLContext > > > Key: SYSTEMML-1237 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1237 > Project: SystemML > Issue Type: Task > Components: APIs >Reporter: Deron Eriksson >Assignee: Deron Eriksson >Priority: Minor > Fix For: SystemML 0.13 > > > Currently java.lang.Float is not a supported input type for the new MLContext > API. This type can be cast to a double in `public Script in(String name, > Object value, Metadata metadata)` to implicitly add float input support to > the API. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-1203) Create main Documentation page on project website
[ https://issues.apache.org/jira/browse/SYSTEMML-1203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson resolved SYSTEMML-1203. -- Resolution: Fixed Fix Version/s: SystemML 0.13 Fixed by commit https://github.com/apache/incubator-systemml-website/commit/2b75d93a481e439ead339df5b5b2621ed9d149bd > Create main Documentation page on project website > - > > Key: SYSTEMML-1203 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1203 > Project: SystemML > Issue Type: Sub-task > Components: Documentation, Website >Reporter: Deron Eriksson >Assignee: Deron Eriksson > Fix For: SystemML 0.13 > > > The main website should feature a main Documentation page that can be used to > easily access different versions of the project documentation. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-1202) Deploy versioned documentation to main project website
[ https://issues.apache.org/jira/browse/SYSTEMML-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson resolved SYSTEMML-1202. -- Resolution: Fixed Fix Version/s: SystemML 0.13 Documentation page now exists on main site which links to 0.12.0 docs and the latest docs. 0.12.0 docs deployed to http://systemml.apache.org/docs/0.12.0/index.html. 0.12.0 javadocs deployed to http://systemml.apache.org/docs/0.12.0/api/java/index.html. > Deploy versioned documentation to main project website > -- > > Key: SYSTEMML-1202 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1202 > Project: SystemML > Issue Type: Task > Components: Documentation, Website >Reporter: Deron Eriksson >Assignee: Deron Eriksson > Fix For: SystemML 0.13 > > > The main website should feature versioned documentation that appears when a > SystemML release (at least major releases) is performed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-894) Integrate documentation contents into main SystemML website
[ https://issues.apache.org/jira/browse/SYSTEMML-894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson resolved SYSTEMML-894. - Resolution: Duplicate Fix Version/s: Not Applicable This has been resolved by SYSTEMML-1202. Documentation page now exists at http://systemml.apache.org/documentation on main site which links to 0.12.0 docs and the latest docs. 0.12.0 docs deployed to http://systemml.apache.org/docs/0.12.0/index.html. 0.12.0 javadocs deployed to http://systemml.apache.org/docs/0.12.0/api/java/index.html. > Integrate documentation contents into main SystemML website > --- > > Key: SYSTEMML-894 > URL: https://issues.apache.org/jira/browse/SYSTEMML-894 > Project: SystemML > Issue Type: New Feature > Components: Website >Reporter: Luciano Resende >Assignee: Deron Eriksson > Fix For: Not Applicable > > Attachments: 0001-SYSTEMML-1106-WIP-Sample-new-layout.patch > > > Currently the main SystemML website is hosted at apache, while the > documentation one is hosted at github pages. > Integrating them will provide more consistent look and feel and navigational > links. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-1205) Deploy 0.12.0 javadocs to main website
[ https://issues.apache.org/jira/browse/SYSTEMML-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson resolved SYSTEMML-1205. -- Resolution: Fixed Fix Version/s: SystemML 0.13 Fixed by svn commit 1782251. Version 0.12.0 javadocs now available online at http://systemml.apache.org/docs/0.12.0/api/java/index.html > Deploy 0.12.0 javadocs to main website > -- > > Key: SYSTEMML-1205 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1205 > Project: SystemML > Issue Type: Sub-task > Components: Documentation, Website >Reporter: Deron Eriksson >Assignee: Deron Eriksson > Fix For: SystemML 0.13 > > > Generate javadocs for the previous 0.12.0 release. Deploy the javadocs to the > main website following a folder structure similar to Spark. This should be > committed to svn rather than first to git and then to svn. > Sample URL: http://spark.apache.org/docs/2.1.0/api/java/index.html -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SYSTEMML-1205) Deploy 0.12.0 javadocs to main website
[ https://issues.apache.org/jira/browse/SYSTEMML-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson updated SYSTEMML-1205: - Description: Generate javadocs for the previous 0.12.0 release. Deploy the javadocs to the main website following a folder structure similar to Spark. This should be committed to svn rather than first to git and then to svn. Sample URL: http://spark.apache.org/docs/2.1.0/api/java/index.html was: Generate javadocs for the previous 0.11.0 release. Deploy the javadocs to the main website following a folder structure similar to Spark. This should be committed to svn rather than first to git and then to svn. Sample URL: http://spark.apache.org/docs/2.1.0/api/java/index.html > Deploy 0.12.0 javadocs to main website > -- > > Key: SYSTEMML-1205 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1205 > Project: SystemML > Issue Type: Sub-task > Components: Documentation, Website >Reporter: Deron Eriksson >Assignee: Deron Eriksson > > Generate javadocs for the previous 0.12.0 release. Deploy the javadocs to the > main website following a folder structure similar to Spark. This should be > committed to svn rather than first to git and then to svn. > Sample URL: http://spark.apache.org/docs/2.1.0/api/java/index.html -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-1204) Deploy 0.12.0 project documentation to main website
[ https://issues.apache.org/jira/browse/SYSTEMML-1204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson resolved SYSTEMML-1204. -- Resolution: Fixed Fix Version/s: SystemML 0.13 Fixed by svn commit 1782251. Version 0.12.0 docs now available online at http://systemml.apache.org/docs/0.12.0/index.html > Deploy 0.12.0 project documentation to main website > --- > > Key: SYSTEMML-1204 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1204 > Project: SystemML > Issue Type: Sub-task > Components: Documentation, Website >Reporter: Deron Eriksson >Assignee: Deron Eriksson > Fix For: SystemML 0.13 > > > Generate documentation site for the previous 0.12.0 (make sure _config.yml > version is correct). Deploy to main website following a folder structure > similar to Spark. This should just be committed to svn rather than first to > git and then to svn. > Example Spark URLs: > http://spark.apache.org/docs/2.1.0/ > http://spark.apache.org/docs/2.0.2/ -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (SYSTEMML-1204) Deploy 0.12.0 project documentation to main website
[ https://issues.apache.org/jira/browse/SYSTEMML-1204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson closed SYSTEMML-1204. > Deploy 0.12.0 project documentation to main website > --- > > Key: SYSTEMML-1204 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1204 > Project: SystemML > Issue Type: Sub-task > Components: Documentation, Website >Reporter: Deron Eriksson >Assignee: Deron Eriksson > Fix For: SystemML 0.13 > > > Generate documentation site for the previous 0.12.0 (make sure _config.yml > version is correct). Deploy to main website following a folder structure > similar to Spark. This should just be committed to svn rather than first to > git and then to svn. > Example Spark URLs: > http://spark.apache.org/docs/2.1.0/ > http://spark.apache.org/docs/2.0.2/ -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SYSTEMML-1204) Deploy 0.12.0 project documentation to main website
[ https://issues.apache.org/jira/browse/SYSTEMML-1204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson updated SYSTEMML-1204: - Description: Generate documentation site for the previous 0.12.0 (make sure _config.yml version is correct). Deploy to main website following a folder structure similar to Spark. This should just be committed to svn rather than first to git and then to svn. Example Spark URLs: http://spark.apache.org/docs/2.1.0/ http://spark.apache.org/docs/2.0.2/ was: Generate documentation site for the previous 0.11.0 (make sure _config.yml version is correct). Deploy to main website following a folder structure similar to Spark. This should just be committed to svn rather than first to git and then to svn. Example Spark URLs: http://spark.apache.org/docs/2.1.0/ http://spark.apache.org/docs/2.0.2/ > Deploy 0.12.0 project documentation to main website > --- > > Key: SYSTEMML-1204 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1204 > Project: SystemML > Issue Type: Sub-task > Components: Documentation, Website >Reporter: Deron Eriksson >Assignee: Deron Eriksson > > Generate documentation site for the previous 0.12.0 (make sure _config.yml > version is correct). Deploy to main website following a folder structure > similar to Spark. This should just be committed to svn rather than first to > git and then to svn. > Example Spark URLs: > http://spark.apache.org/docs/2.1.0/ > http://spark.apache.org/docs/2.0.2/ -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SYSTEMML-1203) Create main Documentation page on project website
[ https://issues.apache.org/jira/browse/SYSTEMML-1203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858689#comment-15858689 ] Deron Eriksson commented on SYSTEMML-1203: -- Svn revision 1782274 > Create main Documentation page on project website > - > > Key: SYSTEMML-1203 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1203 > Project: SystemML > Issue Type: Sub-task > Components: Documentation, Website >Reporter: Deron Eriksson >Assignee: Deron Eriksson > Fix For: SystemML 0.13 > > > The main website should feature a main Documentation page that can be used to > easily access different versions of the project documentation. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (SYSTEMML-1203) Create main Documentation page on project website
[ https://issues.apache.org/jira/browse/SYSTEMML-1203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson closed SYSTEMML-1203. > Create main Documentation page on project website > - > > Key: SYSTEMML-1203 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1203 > Project: SystemML > Issue Type: Sub-task > Components: Documentation, Website >Reporter: Deron Eriksson >Assignee: Deron Eriksson > Fix For: SystemML 0.13 > > > The main website should feature a main Documentation page that can be used to > easily access different versions of the project documentation. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (SYSTEMML-1202) Deploy versioned documentation to main project website
[ https://issues.apache.org/jira/browse/SYSTEMML-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson closed SYSTEMML-1202. > Deploy versioned documentation to main project website > -- > > Key: SYSTEMML-1202 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1202 > Project: SystemML > Issue Type: Task > Components: Documentation, Website >Reporter: Deron Eriksson >Assignee: Deron Eriksson > Fix For: SystemML 0.13 > > > The main website should feature versioned documentation that appears when a > SystemML release (at least major releases) is performed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (SYSTEMML-1205) Deploy 0.12.0 javadocs to main website
[ https://issues.apache.org/jira/browse/SYSTEMML-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson closed SYSTEMML-1205. > Deploy 0.12.0 javadocs to main website > -- > > Key: SYSTEMML-1205 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1205 > Project: SystemML > Issue Type: Sub-task > Components: Documentation, Website >Reporter: Deron Eriksson >Assignee: Deron Eriksson > Fix For: SystemML 0.13 > > > Generate javadocs for the previous 0.12.0 release. Deploy the javadocs to the > main website following a folder structure similar to Spark. This should be > committed to svn rather than first to git and then to svn. > Sample URL: http://spark.apache.org/docs/2.1.0/api/java/index.html -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-1222) MLContext `read` Statement Value Input Error For `cols_in_block` Argument
[
https://issues.apache.org/jira/browse/SYSTEMML-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deron Eriksson resolved SYSTEMML-1222.
--
Resolution: Fixed
Fix Version/s: SystemML 0.13
Fixed by [PR370|https://github.com/apache/incubator-systemml/pull/370].
> MLContext `read` Statement Value Input Error For `cols_in_block` Argument
> -
>
> Key: SYSTEMML-1222
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1222
> Project: SystemML
> Issue Type: Bug
>Reporter: Mike Dusenberry
>Assignee: Deron Eriksson
>Priority: Minor
> Fix For: SystemML 0.13
>
>
> The {{read}} statement has optional numeric parameters, such as
> {{cols_in_block}} that expect integers. If a Scala {{Long}} is supplied as
> an input variable for one of those parameters via MLContext, and type error
> will occur in which an {{Int}} was expected, but a {{Long}} was received. A
> quick fix is to convert to an integer with {{Long.toInt}} before passing the
> value into the script via MLContext.
> We should automatically cast {{Long}} to {{Int}} internally.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (SYSTEMML-1222) MLContext `read` Statement Value Input Error For `cols_in_block` Argument
[
https://issues.apache.org/jira/browse/SYSTEMML-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deron Eriksson closed SYSTEMML-1222.
> MLContext `read` Statement Value Input Error For `cols_in_block` Argument
> -
>
> Key: SYSTEMML-1222
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1222
> Project: SystemML
> Issue Type: Bug
>Reporter: Mike Dusenberry
>Assignee: Deron Eriksson
>Priority: Minor
> Fix For: SystemML 0.13
>
>
> The {{read}} statement has optional numeric parameters, such as
> {{cols_in_block}} that expect integers. If a Scala {{Long}} is supplied as
> an input variable for one of those parameters via MLContext, and type error
> will occur in which an {{Int}} was expected, but a {{Long}} was received. A
> quick fix is to convert to an integer with {{Long.toInt}} before passing the
> value into the script via MLContext.
> We should automatically cast {{Long}} to {{Int}} internally.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (SYSTEMML-1232) Migrate stringDataFrameToVectorDataFrame if needed
[
https://issues.apache.org/jira/browse/SYSTEMML-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deron Eriksson reassigned SYSTEMML-1232:
Assignee: Deron Eriksson
> Migrate stringDataFrameToVectorDataFrame if needed
> --
>
> Key: SYSTEMML-1232
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1232
> Project: SystemML
> Issue Type: Task
> Components: APIs
>Reporter: Deron Eriksson
>Assignee: Deron Eriksson
>
> Restore and migrate RDDConvererUtilsExt.stringDataFrameToVectorDataFrame
> method to Spark 2 (mllib->ml Vector) if needed.
> The RDDConvererUtilsExt.stringDataFrameToVectorDataFrame method was removed
> by commit
> https://github.com/apache/incubator-systemml/commit/578e595fdc506fb8a0c0b18c312fe420a406276d.
> If this method is needed, migrate it to Spark 2.
> Old method:
> {code}
> public static Dataset stringDataFrameToVectorDataFrame(SQLContext
> sqlContext, Dataset inputDF)
> throws DMLRuntimeException {
> StructField[] oldSchema = inputDF.schema().fields();
> //create the new schema
> StructField[] newSchema = new StructField[oldSchema.length];
> for(int i = 0; i < oldSchema.length; i++) {
> String colName = oldSchema[i].name();
> newSchema[i] = DataTypes.createStructField(colName, new
> VectorUDT(), true);
> }
> //converter
> class StringToVector implements Function,
> Row> {
> private static final long serialVersionUID =
> -4733816995375745659L;
> @Override
> public Row call(Tuple2 arg0) throws
> Exception {
> Row oldRow = arg0._1;
> int oldNumCols = oldRow.length();
> if (oldNumCols > 1) {
> throw new DMLRuntimeException("The row
> must have at most one column");
> }
> // parse the various strings. i.e
> // ((1.2,4.3, 3.4)) or (1.2, 3.4, 2.2) or (1.2
> 3.4)
> // [[1.2,34.3, 1.2, 1.2]] or [1.2, 3.4] or [1.3
> 1.2]
> Object [] fields = new Object[oldNumCols];
> ArrayList fieldsArr = new
> ArrayList();
> for (int i = 0; i < oldRow.length(); i++) {
> Object ci=oldRow.get(i);
> if (ci instanceof String) {
> String cis = (String)ci;
> StringBuffer sb = new
> StringBuffer(cis.trim());
> for (int nid=0; i < 2; i++) {
> //remove two level nesting
> if ((sb.charAt(0) ==
> '(' && sb.charAt(sb.length() - 1) == ')') ||
>
> (sb.charAt(0) == '[' && sb.charAt(sb.length() - 1) == ']')
> ) {
>
> sb.deleteCharAt(0);
>
> sb.setLength(sb.length() - 1);
> }
> }
> //have the replace code
> String ncis = "[" +
> sb.toString().replaceAll(" *, *", ",") + "]";
> Vector v = Vectors.parse(ncis);
> fieldsArr.add(v);
> } else {
> throw new
> DMLRuntimeException("Only String is supported");
> }
> }
> Row row =
> RowFactory.create(fieldsArr.toArray());
> return row;
> }
> }
> //output DF
> JavaRDD newRows =
> inputDF.rdd().toJavaRDD().zipWithIndex().map(new StringToVector());
> // DataFrame outDF = sqlContext.createDataFrame(newRows, new
> StructType(newSchema)); //TODO investigate why it doesn't work
> Dataset outDF = sqlContext.createDataFrame(newRows.rdd(),
> DataTypes.createStructType(newSchema));
> return outDF;
> }
> {code}
> Note: the
[jira] [Updated] (SYSTEMML-1233) Deprecate old MLContext API
[ https://issues.apache.org/jira/browse/SYSTEMML-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson updated SYSTEMML-1233: - Description: Deprecate the old MLContext API as it will be removed. It has been replaced by the new MLContext API. Most likely it will be removed for the SystemML 1.0 release. was: Deprecate the old MLContext API as it will be removed. It has been replaced by the new MLContext API. > Deprecate old MLContext API > --- > > Key: SYSTEMML-1233 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1233 > Project: SystemML > Issue Type: Task > Components: APIs >Affects Versions: SystemML 0.13 >Reporter: Deron Eriksson >Assignee: Deron Eriksson > > Deprecate the old MLContext API as it will be removed. It has been replaced > by the new MLContext API. > Most likely it will be removed for the SystemML 1.0 release. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (SYSTEMML-1234) Migrate FrameTest to new MLContext
[ https://issues.apache.org/jira/browse/SYSTEMML-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson reassigned SYSTEMML-1234: Assignee: Deron Eriksson > Migrate FrameTest to new MLContext > -- > > Key: SYSTEMML-1234 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1234 > Project: SystemML > Issue Type: Task > Components: Test >Reporter: Deron Eriksson >Assignee: Deron Eriksson > > org.apache.sysml.test.integration.functions.mlcontext.FrameTest currently > uses the old MLContext API. It should be migrated to the new API. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-1224) Migrate vector and labeledpoint classes from mllib to ml
[
https://issues.apache.org/jira/browse/SYSTEMML-1224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deron Eriksson resolved SYSTEMML-1224.
--
Resolution: Fixed
Fix Version/s: SystemML 0.13
Fixed by [PR369|https://github.com/apache/incubator-systemml/pull/369].
> Migrate vector and labeledpoint classes from mllib to ml
>
>
> Key: SYSTEMML-1224
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1224
> Project: SystemML
> Issue Type: Task
> Components: APIs, Runtime
>Affects Versions: SystemML 0.13
>Reporter: Deron Eriksson
>Assignee: Deron Eriksson
> Fix For: SystemML 0.13
>
>
> For Spark 2, execution of test_mllearn_df.py gives SparseVector to Vector
> error:
> {code}
> spark-submit --driver-class-path $SYSTEMML_HOME/SystemML.jar
> test_mllearn_df.py
> {code}
> generates:
> {code}
> Py4JJavaError: An error occurred while calling o206.fit.
> : org.apache.spark.SparkException: Job aborted due to stage failure: Task 1
> in stage 2.0 failed 1 times, most recent failure: Lost task 1.0 in stage 2.0
> (TID 17, localhost, executor driver): java.lang.ClassCastException:
> org.apache.spark.ml.linalg.SparseVector cannot be cast to
> org.apache.spark.mllib.linalg.Vector
> at
> org.apache.sysml.runtime.instructions.spark.utils.RDDConverterUtils.countNnz(RDDConverterUtils.java:314)
> at
> org.apache.sysml.runtime.instructions.spark.utils.RDDConverterUtils.access$400(RDDConverterUtils.java:71)
> at
> org.apache.sysml.runtime.instructions.spark.utils.RDDConverterUtils$DataFrameAnalysisFunction.call(RDDConverterUtils.java:940)
> at
> org.apache.sysml.runtime.instructions.spark.utils.RDDConverterUtils$DataFrameAnalysisFunction.call(RDDConverterUtils.java:921)
> at
> org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1.apply(JavaPairRDD.scala:1040)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
> at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1762)
> {code}
> This can most likely be fixed by migrating relevant classes (typically going
> from mllib package to ml package).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (SYSTEMML-1234) Migrate FrameTest to new MLContext
Deron Eriksson created SYSTEMML-1234: Summary: Migrate FrameTest to new MLContext Key: SYSTEMML-1234 URL: https://issues.apache.org/jira/browse/SYSTEMML-1234 Project: SystemML Issue Type: Task Components: Test Reporter: Deron Eriksson org.apache.sysml.test.integration.functions.mlcontext.FrameTest currently uses the old MLContext API. It should be migrated to the new API. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (SYSTEMML-1235) Migrate GNMFTest to new MLContext
Deron Eriksson created SYSTEMML-1235: Summary: Migrate GNMFTest to new MLContext Key: SYSTEMML-1235 URL: https://issues.apache.org/jira/browse/SYSTEMML-1235 Project: SystemML Issue Type: Task Components: Test Reporter: Deron Eriksson -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SYSTEMML-1235) Migrate GNMFTest to new MLContext
[ https://issues.apache.org/jira/browse/SYSTEMML-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson updated SYSTEMML-1235: - Description: org.apache.sysml.test.integration.functions.mlcontext.GNMFTest currently uses the old MLContext API. It should be migrated to the new API. > Migrate GNMFTest to new MLContext > - > > Key: SYSTEMML-1235 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1235 > Project: SystemML > Issue Type: Task > Components: Test >Reporter: Deron Eriksson > > org.apache.sysml.test.integration.functions.mlcontext.GNMFTest currently uses > the old MLContext API. It should be migrated to the new API. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (SYSTEMML-1236) Move MLContextProxy out of api package
Deron Eriksson created SYSTEMML-1236: Summary: Move MLContextProxy out of api package Key: SYSTEMML-1236 URL: https://issues.apache.org/jira/browse/SYSTEMML-1236 Project: SystemML Issue Type: Task Components: APIs, Runtime Reporter: Deron Eriksson org.apache.sysml.api.MLContextProxy is not really a class that API users need to be exposed to, so move it to another package to keep the api packages clean. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (SYSTEMML-527) Write metadata description with description parameter to write function
[
https://issues.apache.org/jira/browse/SYSTEMML-527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deron Eriksson closed SYSTEMML-527.
---
> Write metadata description with description parameter to write function
> ---
>
> Key: SYSTEMML-527
> URL: https://issues.apache.org/jira/browse/SYSTEMML-527
> Project: SystemML
> Issue Type: Improvement
> Components: APIs
>Reporter: Deron Eriksson
>Assignee: Deron Eriksson
>Priority: Minor
> Fix For: SystemML 0.13
>
>
> The "description" metadata attribute can be used to describe the data in the
> corresponding data file. However, if you try passing in a description via a
> description parameter to the write method, the specified description does not
> appear in the output metadata file.
> For example:
> {code}
> write(b, "n.csv", format="csv", description="this is a test");
> {code}
> generates:
> {code}
> {
> "data_type": "matrix",
> "value_type": "double",
> ...
> "description": {
> "author": "SystemML"
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-527) Write metadata description with description parameter to write function
[
https://issues.apache.org/jira/browse/SYSTEMML-527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deron Eriksson resolved SYSTEMML-527.
-
Resolution: Fixed
Fix Version/s: SystemML 0.13
Fixed by [PR364|https://github.com/apache/incubator-systemml/pull/364].
> Write metadata description with description parameter to write function
> ---
>
> Key: SYSTEMML-527
> URL: https://issues.apache.org/jira/browse/SYSTEMML-527
> Project: SystemML
> Issue Type: Improvement
> Components: APIs
>Reporter: Deron Eriksson
>Assignee: Deron Eriksson
>Priority: Minor
> Fix For: SystemML 0.13
>
>
> The "description" metadata attribute can be used to describe the data in the
> corresponding data file. However, if you try passing in a description via a
> description parameter to the write method, the specified description does not
> appear in the output metadata file.
> For example:
> {code}
> write(b, "n.csv", format="csv", description="this is a test");
> {code}
> generates:
> {code}
> {
> "data_type": "matrix",
> "value_type": "double",
> ...
> "description": {
> "author": "SystemML"
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (SYSTEMML-1233) Deprecate old MLContext API
Deron Eriksson created SYSTEMML-1233: Summary: Deprecate old MLContext API Key: SYSTEMML-1233 URL: https://issues.apache.org/jira/browse/SYSTEMML-1233 Project: SystemML Issue Type: Task Components: APIs Affects Versions: SystemML 0.13 Reporter: Deron Eriksson Assignee: Deron Eriksson Deprecate the old MLContext API as it will be removed. It has been replaced by the new MLContext API. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (SYSTEMML-1194) Replace SQLContext with SparkSession
Deron Eriksson created SYSTEMML-1194: Summary: Replace SQLContext with SparkSession Key: SYSTEMML-1194 URL: https://issues.apache.org/jira/browse/SYSTEMML-1194 Project: SystemML Issue Type: Task Components: APIs, Test Affects Versions: SystemML 1.0 Reporter: Deron Eriksson Assignee: Deron Eriksson SQLContext has been deprecated in Spark 2 and has been replaced by SparkSession. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] (SYSTEMML-1212) Link to main website in header of project documentation
Title: Message Title Deron Eriksson commented on SYSTEMML-1212 Re: Link to main website in header of project documentation The project documentation exists in docs in incubator-systemml, following the Spark model. This is separate from the website. Spark deploys versioned docs to their site but the project docs live in spark rather than spark-website. Adding a link in the header of the docs ensures that you can always get back to main website from the project docs no matter where the project docs are deployed. cc Frederick Reiss Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] [Closed] (SYSTEMML-1201) Update -standalone tgz and zip to -bin
[
https://issues.apache.org/jira/browse/SYSTEMML-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deron Eriksson closed SYSTEMML-1201.
> Update -standalone tgz and zip to -bin
> --
>
> Key: SYSTEMML-1201
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1201
> Project: SystemML
> Issue Type: Task
> Components: Build
>Reporter: Deron Eriksson
>Assignee: Deron Eriksson
> Fix For: SystemML 1.0
>
>
> Currently the -standalone tgz and zip artifacts (generated by 'mvn clean
> package -P distribution') are being renamed during the release process to
> -bin, but when unpacked they still have a root folder of -standalone. They
> should be given the new -bin names during the artifact generation ('mvn clean
> package -P distribution') so that they contain the expected (-bin) root
> folder name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (SYSTEMML-1201) Update -standalone tgz and zip to -bin
[
https://issues.apache.org/jira/browse/SYSTEMML-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deron Eriksson resolved SYSTEMML-1201.
--
Resolution: Fixed
Fix Version/s: SystemML 1.0
Fixed by [PR363|https://github.com/apache/incubator-systemml/pull/363].
> Update -standalone tgz and zip to -bin
> --
>
> Key: SYSTEMML-1201
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1201
> Project: SystemML
> Issue Type: Task
> Components: Build
>Reporter: Deron Eriksson
>Assignee: Deron Eriksson
> Fix For: SystemML 1.0
>
>
> Currently the -standalone tgz and zip artifacts (generated by 'mvn clean
> package -P distribution') are being renamed during the release process to
> -bin, but when unpacked they still have a root folder of -standalone. They
> should be given the new -bin names during the artifact generation ('mvn clean
> package -P distribution') so that they contain the expected (-bin) root
> folder name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SYSTEMML-1211) Verify dependencies for Spark 2
[ https://issues.apache.org/jira/browse/SYSTEMML-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843596#comment-15843596 ] Deron Eriksson commented on SYSTEMML-1211: -- cc [~gweidner] > Verify dependencies for Spark 2 > --- > > Key: SYSTEMML-1211 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1211 > Project: SystemML > Issue Type: Task > Components: Build >Reporter: Deron Eriksson > > With the migration to Spark 2, we should verify that the artifact assemblies > are properly handling all dependencies. > Also, we should verify that that artifact licenses properly include all > dependencies following the Spark 2 migration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SYSTEMML-1211) Verify dependencies for Spark 2
Deron Eriksson created SYSTEMML-1211: Summary: Verify dependencies for Spark 2 Key: SYSTEMML-1211 URL: https://issues.apache.org/jira/browse/SYSTEMML-1211 Project: SystemML Issue Type: Task Components: Build Reporter: Deron Eriksson With the migration to Spark 2, we should verify that the artifact assemblies are properly handling all dependencies. Also, we should verify that that artifact licenses properly include all dependencies following the Spark 2 migration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SYSTEMML-1212) Link to main website in header of project documentation
Deron Eriksson created SYSTEMML-1212: Summary: Link to main website in header of project documentation Key: SYSTEMML-1212 URL: https://issues.apache.org/jira/browse/SYSTEMML-1212 Project: SystemML Issue Type: Improvement Components: Documentation Reporter: Deron Eriksson Assignee: Deron Eriksson Currently there is no link to the main website in the header of the project documentation. Therefore, once at the project documentation, it can be difficult to get back to the main website. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SYSTEMML-1208) Update project documentation styles to be similar to main website
[ https://issues.apache.org/jira/browse/SYSTEMML-1208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson resolved SYSTEMML-1208. -- Resolution: Fixed Fix Version/s: (was: Not Applicable) SystemML 1.0 Fixed by [PR366|https://github.com/apache/incubator-systemml/pull/366]. > Update project documentation styles to be similar to main website > - > > Key: SYSTEMML-1208 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1208 > Project: SystemML > Issue Type: Task > Components: Documentation >Reporter: Deron Eriksson >Assignee: Deron Eriksson >Priority: Minor > Fix For: SystemML 1.0 > > > The current css styles used for the project documentation (see > https://apache.github.io/incubator-systemml/) have not been updated to more > closely match the styles of the main website (see > http://systemml.apache.org/). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (SYSTEMML-1208) Update project documentation styles to be similar to main website
[ https://issues.apache.org/jira/browse/SYSTEMML-1208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson reopened SYSTEMML-1208: -- > Update project documentation styles to be similar to main website > - > > Key: SYSTEMML-1208 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1208 > Project: SystemML > Issue Type: Task > Components: Documentation >Reporter: Deron Eriksson >Assignee: Deron Eriksson >Priority: Minor > Fix For: Not Applicable > > > The current css styles used for the project documentation (see > https://apache.github.io/incubator-systemml/) have not been updated to more > closely match the styles of the main website (see > http://systemml.apache.org/). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (SYSTEMML-1208) Update project documentation styles to be similar to main website
[ https://issues.apache.org/jira/browse/SYSTEMML-1208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15844162#comment-15844162 ] Deron Eriksson edited comment on SYSTEMML-1208 at 1/28/17 7:08 PM: --- https://github.com/apache/incubator-systemml/pull/366 addresses this issue. (colors, etc) was (Author: deron): https://github.com/apache/incubator-systemml/pull/366 addresses this issue. > Update project documentation styles to be similar to main website > - > > Key: SYSTEMML-1208 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1208 > Project: SystemML > Issue Type: Task > Components: Documentation >Reporter: Deron Eriksson >Assignee: Deron Eriksson >Priority: Minor > Fix For: Not Applicable > > > The current css styles used for the project documentation (see > https://apache.github.io/incubator-systemml/) have not been updated to more > closely match the styles of the main website (see > http://systemml.apache.org/). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SYSTEMML-1208) Update project documentation styles to be similar to main website
[ https://issues.apache.org/jira/browse/SYSTEMML-1208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15844162#comment-15844162 ] Deron Eriksson commented on SYSTEMML-1208: -- https://github.com/apache/incubator-systemml/pull/366 addresses this issue. > Update project documentation styles to be similar to main website > - > > Key: SYSTEMML-1208 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1208 > Project: SystemML > Issue Type: Task > Components: Documentation >Reporter: Deron Eriksson >Assignee: Deron Eriksson >Priority: Minor > Fix For: Not Applicable > > > The current css styles used for the project documentation (see > https://apache.github.io/incubator-systemml/) have not been updated to more > closely match the styles of the main website (see > http://systemml.apache.org/). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SYSTEMML-1200) Add LogisticRegressionSuite.scala to integration tests
Deron Eriksson created SYSTEMML-1200:
Summary: Add LogisticRegressionSuite.scala to integration tests
Key: SYSTEMML-1200
URL: https://issues.apache.org/jira/browse/SYSTEMML-1200
Project: SystemML
Issue Type: Task
Components: APIs
Reporter: Deron Eriksson
Priority: Minor
No Scala tests are currently run by the integration tests. Possibly, the
LogisticRegressionSuite class can be annotated with:
{code}
@RunWith(classOf[JUnitRunner])
{code}
with imports such as:
{code}
import org.junit.runner.RunWith
import org.scalatest.FunSuite
import org.scalatest.Matchers
import org.scalatest.junit.JUnitRunner
{code}
Note that at the time of this issue creation, LogisticRegressionSuite fails.
See SYSTEMML-1199.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (SYSTEMML-1155) Use artifact versions from parent pom where possible
[ https://issues.apache.org/jira/browse/SYSTEMML-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson resolved SYSTEMML-1155. -- Resolution: Fixed Fix Version/s: SystemML 1.0 Fixed by [PR356|https://github.com/apache/incubator-systemml/pull/356]. Use Apache parent pom artifact versions for maven-dependency-plugin, maven-jar-plugin, maven-clean-plugin, and maven-antrun-plugin. > Use artifact versions from parent pom where possible > > > Key: SYSTEMML-1155 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1155 > Project: SystemML > Issue Type: Improvement > Components: Build >Reporter: Deron Eriksson >Assignee: Deron Eriksson >Priority: Minor > Fix For: SystemML 1.0 > > > The project pom.xml uses the apache parent pom (currently version 18). Since > the artifact versions in the parent pom are the 'standard' apache version > recommendations, we should use these versions rather than override them where > possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (SYSTEMML-1155) Use artifact versions from parent pom where possible
[ https://issues.apache.org/jira/browse/SYSTEMML-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson closed SYSTEMML-1155. > Use artifact versions from parent pom where possible > > > Key: SYSTEMML-1155 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1155 > Project: SystemML > Issue Type: Improvement > Components: Build >Reporter: Deron Eriksson >Assignee: Deron Eriksson >Priority: Minor > Fix For: SystemML 1.0 > > > The project pom.xml uses the apache parent pom (currently version 18). Since > the artifact versions in the parent pom are the 'standard' apache version > recommendations, we should use these versions rather than override them where > possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
