[GitHub] spark issue #18288: [SPARK-21066][ML] LibSVM load just one input file
Github user facaiy commented on the issue:
https://github.com/apache/spark/pull/18288
Yes.
an example code:
```scala
val df = spark.read.format("libsvm")
.option("numFeatures", "780")
.load("data/mllib/sample_libsvm_data.txt")
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18288: [SPARK-21066][ML] LibSVM load just one input file
Github user darionyaphet commented on the issue: https://github.com/apache/spark/pull/18288 So if I want to load a mulit file LibSVM input should specify the `numFeatures` ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18288: [SPARK-21066][ML] LibSVM load just one input file
Github user facaiy commented on the issue: https://github.com/apache/spark/pull/18288 You might be mistaken. The aim of code here is to encourage user to specify `numFeatures` in any case, rather than encourage user to use only one file. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18288: [SPARK-21066][ML] LibSVM load just one input file
Github user darionyaphet commented on the issue: https://github.com/apache/spark/pull/18288 @srowen @facaiy `numFeatures` is the `number of features` I'm confuse about why features number could control the input path and if just only one input file have some advantage ï¼ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18288: [SPARK-21066][ML] LibSVM load just one input file
Github user facaiy commented on the issue: https://github.com/apache/spark/pull/18288 In my opinion, `numFeatures` is vital for sparse data. Say our feature is 100-dim indeed, while in a small train data their maximum size is 990. It is dangerous (or wrong) to train a 990-dim model as it might fail in the coming test data. Hence, in most cases, `numFeatures` should be given by user. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18288: [SPARK-21066][ML] LibSVM load just one input file
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18288 @darionyaphet what do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18288: [SPARK-21066][ML] LibSVM load just one input file
Github user darionyaphet commented on the issue: https://github.com/apache/spark/pull/18288 @srowen I have update `LibSVMRelationSuite` to adapt libsvm mutli file input :) thanks for you review . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18288: [SPARK-21066][ML] LibSVM load just one input file
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18288 Is it easy to add a simple test for this case? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18288: [SPARK-21066][ML] LibSVM load just one input file
Github user darionyaphet commented on the issue: https://github.com/apache/spark/pull/18288 @srowen It seems OK. `FileInputFormat` will use a comma-separated paths to be set as the list of inputs for the job. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18288: [SPARK-21066][ML] LibSVM load just one input file
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18288 Is that going to work as an argument to parseLibSVMFile ? CC @liancheng --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18288: [SPARK-21066][ML] LibSVM load just one input file
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18288 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
