[jira] [Commented] (SPARK-6823) Add a model.matrix like capability to DataFrames (modelDataFrame)
[ https://issues.apache.org/jira/browse/SPARK-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943410#comment-14943410 ] Deborah Siegel commented on SPARK-6823: --- does SPARK-9681 address RFormula supporting identity? eg. (~ . ) for model.matrix of features? would be useful. > Add a model.matrix like capability to DataFrames (modelDataFrame) > - > > Key: SPARK-6823 > URL: https://issues.apache.org/jira/browse/SPARK-6823 > Project: Spark > Issue Type: New Feature > Components: ML, SparkR >Reporter: Shivaram Venkataraman > > Currently Mllib modeling tools work only with double data. However, data > tables in practice often have a set of categorical fields (factors in R), > that need to be converted to a set of 0/1 indicator variables (making the > data actually used in a modeling algorithm completely numeric). In R, this is > handled in modeling functions using the model.matrix function. Similar > functionality needs to be available within Spark. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6823) Add a model.matrix like capability to DataFrames (modelDataFrame)
[ https://issues.apache.org/jira/browse/SPARK-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14875825#comment-14875825 ] Denis commented on SPARK-6823: -- RFormula does it, but it's very limited now. It only supports "+", "-" and ".". It still needs ":" and "*" at least. Not sure if it also supports identity. I have a problem using R's model.matrix with a large dataset. The matrix that I obtain does not fit into memory. Having fully implemented RFormula would be very useful. > Add a model.matrix like capability to DataFrames (modelDataFrame) > - > > Key: SPARK-6823 > URL: https://issues.apache.org/jira/browse/SPARK-6823 > Project: Spark > Issue Type: New Feature > Components: ML, SparkR >Reporter: Shivaram Venkataraman > > Currently Mllib modeling tools work only with double data. However, data > tables in practice often have a set of categorical fields (factors in R), > that need to be converted to a set of 0/1 indicator variables (making the > data actually used in a modeling algorithm completely numeric). In R, this is > handled in modeling functions using the model.matrix function. Similar > functionality needs to be available within Spark. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6823) Add a model.matrix like capability to DataFrames (modelDataFrame)
[ https://issues.apache.org/jira/browse/SPARK-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14875840#comment-14875840 ] Shivaram Venkataraman commented on SPARK-6823: -- I think this is related to https://issues.apache.org/jira/browse/SPARK-9681 - Lets continue the discussion on that JIRA > Add a model.matrix like capability to DataFrames (modelDataFrame) > - > > Key: SPARK-6823 > URL: https://issues.apache.org/jira/browse/SPARK-6823 > Project: Spark > Issue Type: New Feature > Components: ML, SparkR >Reporter: Shivaram Venkataraman > > Currently Mllib modeling tools work only with double data. However, data > tables in practice often have a set of categorical fields (factors in R), > that need to be converted to a set of 0/1 indicator variables (making the > data actually used in a modeling algorithm completely numeric). In R, this is > handled in modeling functions using the model.matrix function. Similar > functionality needs to be available within Spark. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6823) Add a model.matrix like capability to DataFrames (modelDataFrame)
[ https://issues.apache.org/jira/browse/SPARK-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694512#comment-14694512 ] Joseph K. Bradley commented on SPARK-6823: -- Removing target version. Feel free to set to something else. Add a model.matrix like capability to DataFrames (modelDataFrame) - Key: SPARK-6823 URL: https://issues.apache.org/jira/browse/SPARK-6823 Project: Spark Issue Type: New Feature Components: ML, SparkR Reporter: Shivaram Venkataraman Currently Mllib modeling tools work only with double data. However, data tables in practice often have a set of categorical fields (factors in R), that need to be converted to a set of 0/1 indicator variables (making the data actually used in a modeling algorithm completely numeric). In R, this is handled in modeling functions using the model.matrix function. Similar functionality needs to be available within Spark. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6823) Add a model.matrix like capability to DataFrames (modelDataFrame)
[ https://issues.apache.org/jira/browse/SPARK-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14658596#comment-14658596 ] Xiangrui Meng commented on SPARK-6823: -- I think this is implemented in RFormula, but we don't expose RFormula in SparkR directly. We should decide whether we want to add a thin wrapper in SparkR for `model.matrix`. Add a model.matrix like capability to DataFrames (modelDataFrame) - Key: SPARK-6823 URL: https://issues.apache.org/jira/browse/SPARK-6823 Project: Spark Issue Type: New Feature Components: ML, SparkR Reporter: Shivaram Venkataraman Currently Mllib modeling tools work only with double data. However, data tables in practice often have a set of categorical fields (factors in R), that need to be converted to a set of 0/1 indicator variables (making the data actually used in a modeling algorithm completely numeric). In R, this is handled in modeling functions using the model.matrix function. Similar functionality needs to be available within Spark. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6823) Add a model.matrix like capability to DataFrames (modelDataFrame)
[ https://issues.apache.org/jira/browse/SPARK-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14651196#comment-14651196 ] Shivaram Venkataraman commented on SPARK-6823: -- [~ekhliang] [~mengxr] Is this addressed by the StringType PR ? I'm wondering if we can resolve this issue Add a model.matrix like capability to DataFrames (modelDataFrame) - Key: SPARK-6823 URL: https://issues.apache.org/jira/browse/SPARK-6823 Project: Spark Issue Type: New Feature Components: ML, SparkR Reporter: Shivaram Venkataraman Currently Mllib modeling tools work only with double data. However, data tables in practice often have a set of categorical fields (factors in R), that need to be converted to a set of 0/1 indicator variables (making the data actually used in a modeling algorithm completely numeric). In R, this is handled in modeling functions using the model.matrix function. Similar functionality needs to be available within Spark. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6823) Add a model.matrix like capability to DataFrames (modelDataFrame)
[ https://issues.apache.org/jira/browse/SPARK-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14492661#comment-14492661 ] Shivaram Venkataraman commented on SPARK-6823: -- I think the goal of the original JIRA on SparkR was to have a high-level API that'll allow users to express this . We could have this higher-level API in a DataFrame or just provide a wrapper around OneHotEncoder + VectorAssembler in the SparkR ML integration work. I think the second one sounds better to me, but [~cafreeman] and Dan Putler have been looking at this and might be able to add more. Add a model.matrix like capability to DataFrames (modelDataFrame) - Key: SPARK-6823 URL: https://issues.apache.org/jira/browse/SPARK-6823 Project: Spark Issue Type: New Feature Components: ML, SparkR Reporter: Shivaram Venkataraman Currently Mllib modeling tools work only with double data. However, data tables in practice often have a set of categorical fields (factors in R), that need to be converted to a set of 0/1 indicator variables (making the data actually used in a modeling algorithm completely numeric). In R, this is handled in modeling functions using the model.matrix function. Similar functionality needs to be available within Spark. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6823) Add a model.matrix like capability to DataFrames (modelDataFrame)
[ https://issues.apache.org/jira/browse/SPARK-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491885#comment-14491885 ] Joseph K. Bradley commented on SPARK-6823: -- This sounds like it would be covered by the OneHotEncoder + VectorAssembler feature transformers: * [https://issues.apache.org/jira/browse/SPARK-5888] * [https://issues.apache.org/jira/browse/SPARK-5885] Do you think these belong within DataFrame (and that this JIRA should be for SQL instead of ML)? Add a model.matrix like capability to DataFrames (modelDataFrame) - Key: SPARK-6823 URL: https://issues.apache.org/jira/browse/SPARK-6823 Project: Spark Issue Type: New Feature Components: ML, SparkR Reporter: Shivaram Venkataraman Currently Mllib modeling tools work only with double data. However, data tables in practice often have a set of categorical fields (factors in R), that need to be converted to a set of 0/1 indicator variables (making the data actually used in a modeling algorithm completely numeric). In R, this is handled in modeling functions using the model.matrix function. Similar functionality needs to be available within Spark. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
