[ https://issues.apache.org/jira/browse/SPARK-5405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Derrick Burns updated SPARK-5405: --------------------------------- Description: The MLLIB clusterer works well for low (<200) dimensional data. However, performance is linear with the number of dimensions. So, for practical purposes, it is not very useful for high dimensional data. Depending on the data type, one can embed the high dimensional data into lower dimensional spaces in a distance-preserving way. The Spark clusterer should support such embedding. An example implementation that supports high dimensional data is here: https://github.com/derrickburns/generalized-kmeans-clustering was: The MLLIB clusterer works well for low (<200) dimensional data. However, performance is linear with the number of dimensions. So, for practical purposes, it is not very useful for high dimensional data. Depending on the data type, one can embed the high dimensional data into lower dimensional spaces in a distance-preserving way. The Spark clusterer should support such embedding. > Spark clusterer should support high dimensional data > ---------------------------------------------------- > > Key: SPARK-5405 > URL: https://issues.apache.org/jira/browse/SPARK-5405 > Project: Spark > Issue Type: New Feature > Components: MLlib > Affects Versions: 1.2.0 > Reporter: Derrick Burns > Original Estimate: 504h > Remaining Estimate: 504h > > The MLLIB clusterer works well for low (<200) dimensional data. However, > performance is linear with the number of dimensions. So, for practical > purposes, it is not very useful for high dimensional data. > Depending on the data type, one can embed the high dimensional data into > lower dimensional spaces in a distance-preserving way. The Spark clusterer > should support such embedding. > An example implementation that supports high dimensional data is here: > https://github.com/derrickburns/generalized-kmeans-clustering -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org