[jira] [Updated] (SYSTEMML-776) Update SystemML to Support Spark 2.1.0
[
https://issues.apache.org/jira/browse/SYSTEMML-776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvind Surve updated SYSTEMML-776:
--
Summary: Update SystemML to Support Spark 2.1.0 (was: Update SystemML to
Support Spark 2.0.0)
> Update SystemML to Support Spark 2.1.0
> --
>
> Key: SYSTEMML-776
> URL: https://issues.apache.org/jira/browse/SYSTEMML-776
> Project: SystemML
> Issue Type: Improvement
>Affects Versions: SystemML 0.9, SystemML 0.10, SystemML 0.11
>Reporter: Mike Dusenberry
>Assignee: Glenn Weidner
>Priority: Critical
>
> In the upcoming Spark 2.0.0 release, the {{DataFrame}} class has been changed
> to a Scala type pointing to {{DataSet[Row]}}, with no {{DataFrame}} available
> from Java. Therefore, our current build is not compatible with this upcoming
> release. This can be tested by updating the pom to using {{2.0.0-preview}}
> as the Spark version.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (SYSTEMML-1182) Configure cluster to Spark 2.1.0
[ https://issues.apache.org/jira/browse/SYSTEMML-1182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvind Surve updated SYSTEMML-1182: --- Summary: Configure cluster to Spark 2.1.0 (was: Configure cluster to Spark 2.0.2 ) > Configure cluster to Spark 2.1.0 > - > > Key: SYSTEMML-1182 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1182 > Project: SystemML > Issue Type: Sub-task >Reporter: Arvind Surve >Assignee: Berthold Reinwald > > One of the test cluster needs to be configured to Spark 2.1.0 (minimum > version) and Hadoop 2.4.1. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SYSTEMML-1182) Configure cluster to Spark 2.0.2
[ https://issues.apache.org/jira/browse/SYSTEMML-1182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvind Surve updated SYSTEMML-1182: --- Description: One of the test cluster needs to be configured to Spark 2.1.0 (minimum version) and Hadoop 2.4.1. (was: One of the test cluster needs to be configured to Spark 2.0.2 (minimum version) and Hadoop 2.4.1. ) > Configure cluster to Spark 2.0.2 > - > > Key: SYSTEMML-1182 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1182 > Project: SystemML > Issue Type: Sub-task >Reporter: Arvind Surve >Assignee: Berthold Reinwald > > One of the test cluster needs to be configured to Spark 2.1.0 (minimum > version) and Hadoop 2.4.1. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SYSTEMML-1181) Update documentation with changes related to Spark 2.1.0
[ https://issues.apache.org/jira/browse/SYSTEMML-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvind Surve updated SYSTEMML-1181: --- Description: Update web page for any changes related to SystemML on Spark 2.1.0 (was: Update web page for any changes related to SystemML on Spark 2.0.) > Update documentation with changes related to Spark 2.1.0 > > > Key: SYSTEMML-1181 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1181 > Project: SystemML > Issue Type: Documentation >Reporter: Arvind Surve >Assignee: Felix Schüler > > Update web page for any changes related to SystemML on Spark 2.1.0 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SYSTEMML-1181) Update documentation with changes related to Spark 2.1.0
[ https://issues.apache.org/jira/browse/SYSTEMML-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvind Surve updated SYSTEMML-1181: --- Summary: Update documentation with changes related to Spark 2.1.0 (was: Update documentation with changes related to Spark 2.0) > Update documentation with changes related to Spark 2.1.0 > > > Key: SYSTEMML-1181 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1181 > Project: SystemML > Issue Type: Documentation >Reporter: Arvind Surve >Assignee: Felix Schüler > > Update web page for any changes related to SystemML on Spark 2.0. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (SYSTEMML-1212) Link to main website in header of project documentation
[ https://issues.apache.org/jira/browse/SYSTEMML-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson closed SYSTEMML-1212. > Link to main website in header of project documentation > --- > > Key: SYSTEMML-1212 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1212 > Project: SystemML > Issue Type: Improvement > Components: Documentation >Reporter: Deron Eriksson >Assignee: Deron Eriksson > Fix For: SystemML 0.13 > > > Currently there is no link to the main website in the header of the project > documentation. Therefore, once at the project documentation, it can be > difficult to get back to the main website. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-1212) Link to main website in header of project documentation
[ https://issues.apache.org/jira/browse/SYSTEMML-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson resolved SYSTEMML-1212. -- Resolution: Fixed Fix Version/s: SystemML 0.13 Fixed by [PR367|https://github.com/apache/incubator-systemml/pull/367]. > Link to main website in header of project documentation > --- > > Key: SYSTEMML-1212 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1212 > Project: SystemML > Issue Type: Improvement > Components: Documentation >Reporter: Deron Eriksson >Assignee: Deron Eriksson > Fix For: SystemML 0.13 > > > Currently there is no link to the main website in the header of the project > documentation. Therefore, once at the project documentation, it can be > difficult to get back to the main website. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SYSTEMML-1181) Update documentation with changes related to Spark 2.0
[ https://issues.apache.org/jira/browse/SYSTEMML-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850843#comment-15850843 ] Felix Schüler commented on SYSTEMML-1181: - Okay, after some investigation I found out: - pyspark has no SQLContext by default anyways - the SparkContext is there by default So our python docs are fine. We will have to do the switch from SQLContext in some of our function signatures at some point though... > Update documentation with changes related to Spark 2.0 > -- > > Key: SYSTEMML-1181 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1181 > Project: SystemML > Issue Type: Documentation >Reporter: Arvind Surve >Assignee: Felix Schüler > > Update web page for any changes related to SystemML on Spark 2.0. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-1222) MLContext `read` Statement Value Input Error For `cols_in_block` Argument
[
https://issues.apache.org/jira/browse/SYSTEMML-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deron Eriksson resolved SYSTEMML-1222.
--
Resolution: Fixed
Fix Version/s: SystemML 0.13
Fixed by [PR370|https://github.com/apache/incubator-systemml/pull/370].
> MLContext `read` Statement Value Input Error For `cols_in_block` Argument
> -
>
> Key: SYSTEMML-1222
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1222
> Project: SystemML
> Issue Type: Bug
>Reporter: Mike Dusenberry
>Assignee: Deron Eriksson
>Priority: Minor
> Fix For: SystemML 0.13
>
>
> The {{read}} statement has optional numeric parameters, such as
> {{cols_in_block}} that expect integers. If a Scala {{Long}} is supplied as
> an input variable for one of those parameters via MLContext, and type error
> will occur in which an {{Int}} was expected, but a {{Long}} was received. A
> quick fix is to convert to an integer with {{Long.toInt}} before passing the
> value into the script via MLContext.
> We should automatically cast {{Long}} to {{Int}} internally.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (SYSTEMML-1222) MLContext `read` Statement Value Input Error For `cols_in_block` Argument
[
https://issues.apache.org/jira/browse/SYSTEMML-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deron Eriksson closed SYSTEMML-1222.
> MLContext `read` Statement Value Input Error For `cols_in_block` Argument
> -
>
> Key: SYSTEMML-1222
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1222
> Project: SystemML
> Issue Type: Bug
>Reporter: Mike Dusenberry
>Assignee: Deron Eriksson
>Priority: Minor
> Fix For: SystemML 0.13
>
>
> The {{read}} statement has optional numeric parameters, such as
> {{cols_in_block}} that expect integers. If a Scala {{Long}} is supplied as
> an input variable for one of those parameters via MLContext, and type error
> will occur in which an {{Int}} was expected, but a {{Long}} was received. A
> quick fix is to convert to an integer with {{Long.toInt}} before passing the
> value into the script via MLContext.
> We should automatically cast {{Long}} to {{Int}} internally.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-1192) Optionally delete rmvar instructions via MLContext
[
https://issues.apache.org/jira/browse/SYSTEMML-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deron Eriksson resolved SYSTEMML-1192.
--
Resolution: Fixed
Fix Version/s: SystemML 0.13
Fixed by [PR352|https://github.com/apache/incubator-systemml/pull/352].
Optionally can be turned on for troubleshooting by:
{code}
ml.setMaintainSymbolTable(true)
{code}
> Optionally delete rmvar instructions via MLContext
> --
>
> Key: SYSTEMML-1192
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1192
> Project: SystemML
> Issue Type: Improvement
> Components: APIs
>Reporter: Deron Eriksson
>Assignee: Deron Eriksson
> Fix For: SystemML 0.13
>
>
> The symbol table (LocalVariableMap) contains data that is useful for
> determining what happens when a SystemML script executes. Removing all rmvar
> instructions allows all the symbol table values to be displayed through the
> MLContext API, which can be useful when debugging. The symbol table can be
> cleared if needed by calling Script's clearSymbolTable() method.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (SYSTEMML-1192) Optionally delete rmvar instructions via MLContext
[ https://issues.apache.org/jira/browse/SYSTEMML-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson closed SYSTEMML-1192. > Optionally delete rmvar instructions via MLContext > -- > > Key: SYSTEMML-1192 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1192 > Project: SystemML > Issue Type: Improvement > Components: APIs >Reporter: Deron Eriksson >Assignee: Deron Eriksson > Fix For: SystemML 0.13 > > > The symbol table (LocalVariableMap) contains data that is useful for > determining what happens when a SystemML script executes. Removing all rmvar > instructions allows all the symbol table values to be displayed through the > MLContext API, which can be useful when debugging. The symbol table can be > cleared if needed by calling Script's clearSymbolTable() method. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SYSTEMML-1181) Update documentation with changes related to Spark 2.0
[ https://issues.apache.org/jira/browse/SYSTEMML-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850804#comment-15850804 ] Felix Schüler commented on SYSTEMML-1181: - Nevermind, there is python documentation that assumes a SparkContext from the shell. Let me update this and then we can resolve the Jira. > Update documentation with changes related to Spark 2.0 > -- > > Key: SYSTEMML-1181 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1181 > Project: SystemML > Issue Type: Documentation >Reporter: Arvind Surve >Assignee: Felix Schüler > > Update web page for any changes related to SystemML on Spark 2.0. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SYSTEMML-1181) Update documentation with changes related to Spark 2.0
[ https://issues.apache.org/jira/browse/SYSTEMML-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850801#comment-15850801 ] Felix Schüler commented on SYSTEMML-1181: - Our Python documentation still uses the SQLContext but doesn't assume that it is available from the shell. it might be good to update the python parts as well but I have to look into the Spark docs to find out what changed and how to update it. > Update documentation with changes related to Spark 2.0 > -- > > Key: SYSTEMML-1181 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1181 > Project: SystemML > Issue Type: Documentation >Reporter: Arvind Surve >Assignee: Felix Schüler > > Update web page for any changes related to SystemML on Spark 2.0. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SYSTEMML-1181) Update documentation with changes related to Spark 2.0
[ https://issues.apache.org/jira/browse/SYSTEMML-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850788#comment-15850788 ] Felix Schüler commented on SYSTEMML-1181: - Let me check the rest of the docs, so far I only checked the MLContext docs. > Update documentation with changes related to Spark 2.0 > -- > > Key: SYSTEMML-1181 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1181 > Project: SystemML > Issue Type: Documentation >Reporter: Arvind Surve >Assignee: Felix Schüler > > Update web page for any changes related to SystemML on Spark 2.0. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SYSTEMML-1181) Update documentation with changes related to Spark 2.0
[ https://issues.apache.org/jira/browse/SYSTEMML-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850782#comment-15850782 ] Deron Eriksson commented on SYSTEMML-1181: -- [PR371|https://github.com/apache/incubator-systemml/pull/371] addressed sqlContext in the documentation. Are further documentation updates happening under this PR or should it be resolved? [~fschueler] > Update documentation with changes related to Spark 2.0 > -- > > Key: SYSTEMML-1181 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1181 > Project: SystemML > Issue Type: Documentation >Reporter: Arvind Surve >Assignee: Felix Schüler > > Update web page for any changes related to SystemML on Spark 2.0. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (SYSTEMML-1227) Cleanup pyc files
Deron Eriksson created SYSTEMML-1227: Summary: Cleanup pyc files Key: SYSTEMML-1227 URL: https://issues.apache.org/jira/browse/SYSTEMML-1227 Project: SystemML Issue Type: Task Components: Build Reporter: Deron Eriksson Assignee: Deron Eriksson Priority: Minor Currently *.pyc files can be generated under src/main/python (for instance when tests are run). They should be cleaned up when 'mvn clean' is run. Additionally the *.pyc files should be added to .gitignore. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SYSTEMML-1181) Update documentation with changes related to Spark 2.0
[ https://issues.apache.org/jira/browse/SYSTEMML-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850698#comment-15850698 ] Deron Eriksson commented on SYSTEMML-1181: -- For Spark 2, I would recommend not spending time updating the docs for the old MLContext API unless it is fairly trivial to update it. This API will be removed in the near future as mentioned in the Roadmap: https://www.mail-archive.com/dev@systemml.incubator.apache.org/msg01199.html > Update documentation with changes related to Spark 2.0 > -- > > Key: SYSTEMML-1181 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1181 > Project: SystemML > Issue Type: Documentation >Reporter: Arvind Surve >Assignee: Felix Schüler > > Update web page for any changes related to SystemML on Spark 2.0. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SYSTEMML-1222) MLContext `read` Statement Value Input Error For `cols_in_block` Argument
[
https://issues.apache.org/jira/browse/SYSTEMML-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850636#comment-15850636
]
Deron Eriksson commented on SYSTEMML-1222:
--
Since Long is not a recognized SystemML value type, it makes sense to do the
long to int conversion high up in the API.
Value types in SystemML (from Statement.java):
{code}
public static final String DOUBLE_VALUE_TYPE = "double";
public static final String BOOLEAN_VALUE_TYPE = "boolean";
public static final String INT_VALUE_TYPE = "int";
public static final String STRING_VALUE_TYPE = "string";
{code}
Thank for finding this [~mwdus...@us.ibm.com].
https://github.com/apache/incubator-systemml/pull/370 addresses this.
> MLContext `read` Statement Value Input Error For `cols_in_block` Argument
> -
>
> Key: SYSTEMML-1222
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1222
> Project: SystemML
> Issue Type: Bug
>Reporter: Mike Dusenberry
>Assignee: Deron Eriksson
>Priority: Minor
>
> The {{read}} statement has optional numeric parameters, such as
> {{cols_in_block}} that expect integers. If a Scala {{Long}} is supplied as
> an input variable for one of those parameters via MLContext, and type error
> will occur in which an {{Int}} was expected, but a {{Long}} was received. A
> quick fix is to convert to an integer with {{Long.toInt}} before passing the
> value into the script via MLContext.
> We should automatically cast {{Long}} to {{Int}} internally.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SYSTEMML-1181) Update documentation with changes related to Spark 2.0
[ https://issues.apache.org/jira/browse/SYSTEMML-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850597#comment-15850597 ] Felix Schüler commented on SYSTEMML-1181: - Should we update the documentation for the old MLContext, too or will this be removed? > Update documentation with changes related to Spark 2.0 > -- > > Key: SYSTEMML-1181 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1181 > Project: SystemML > Issue Type: Documentation >Reporter: Arvind Surve >Assignee: Felix Schüler > > Update web page for any changes related to SystemML on Spark 2.0. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (SYSTEMML-1222) MLContext `read` Statement Value Input Error For `cols_in_block` Argument
[
https://issues.apache.org/jira/browse/SYSTEMML-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deron Eriksson reassigned SYSTEMML-1222:
Assignee: Deron Eriksson
> MLContext `read` Statement Value Input Error For `cols_in_block` Argument
> -
>
> Key: SYSTEMML-1222
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1222
> Project: SystemML
> Issue Type: Bug
>Reporter: Mike Dusenberry
>Assignee: Deron Eriksson
>Priority: Minor
>
> The {{read}} statement has optional numeric parameters, such as
> {{cols_in_block}} that expect integers. If a Scala {{Long}} is supplied as
> an input variable for one of those parameters via MLContext, and type error
> will occur in which an {{Int}} was expected, but a {{Long}} was received. A
> quick fix is to convert to an integer with {{Long.toInt}} before passing the
> value into the script via MLContext.
> We should automatically cast {{Long}} to {{Int}} internally.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (SYSTEMML-1181) Update documentation with changes related to Spark 2.0
[ https://issues.apache.org/jira/browse/SYSTEMML-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Schüler updated SYSTEMML-1181: Issue Type: Documentation (was: Sub-task) Parent: (was: SYSTEMML-776) > Update documentation with changes related to Spark 2.0 > -- > > Key: SYSTEMML-1181 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1181 > Project: SystemML > Issue Type: Documentation >Reporter: Arvind Surve >Assignee: Felix Schüler > > Update web page for any changes related to SystemML on Spark 2.0. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
