[jira] [Commented] (SPARK-7334) Implement RandomProjection for Dimensionality Reduction
[ https://issues.apache.org/jira/browse/SPARK-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14996470#comment-14996470 ] Sebastian Alfers commented on SPARK-7334: - It this still relevant? [~josephkb] I saw a discussion about LSH here: https://issues.apache.org/jira/browse/SPARK-5992 > Implement RandomProjection for Dimensionality Reduction > --- > > Key: SPARK-7334 > URL: https://issues.apache.org/jira/browse/SPARK-7334 > Project: Spark > Issue Type: Improvement > Components: MLlib >Reporter: Sebastian Alfers >Priority: Minor > > Implement RandomProjection (RP) for dimensionality reduction > RP is a popular approach to reduce the amount of data while preserving a > reasonable amount of information (pairwise distance) of you data [1][2] > - [1] http://www.yaroslavvb.com/papers/achlioptas-database.pdf > - [2] > http://people.inf.elte.hu/fekete/algoritmusok_msc/dimenzio_csokkentes/randon_projection_kdd.pdf > I compared different implementations of that algorithm: > - https://github.com/sebastian-alfers/random-projection-python -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-7334) Implement RandomProjection for Dimensionality Reduction
Sebastian Alfers created SPARK-7334: --- Summary: Implement RandomProjection for Dimensionality Reduction Key: SPARK-7334 URL: https://issues.apache.org/jira/browse/SPARK-7334 Project: Spark Issue Type: Improvement Components: MLlib Reporter: Sebastian Alfers Priority: Minor Implement RandomProjection (RP) for dimensionality reduction (DR) RP is a popular approach to reduce the amount of data while preserving a reasonable amount of information (pairwise distance) of you data [1][2] - [1] http://www.yaroslavvb.com/papers/achlioptas-database.pdf - [2] http://people.inf.elte.hu/fekete/algoritmusok_msc/dimenzio_csokkentes/randon_projection_kdd.pdf I compared different implementations of that algorithm: - https://github.com/sebastian-alfers/random-projection-python -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-7334) Implement RandomProjection for Dimensionality Reduction
[ https://issues.apache.org/jira/browse/SPARK-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sebastian Alfers updated SPARK-7334: Description: Implement RandomProjection (RP) for dimensionality reduction RP is a popular approach to reduce the amount of data while preserving a reasonable amount of information (pairwise distance) of you data [1][2] - [1] http://www.yaroslavvb.com/papers/achlioptas-database.pdf - [2] http://people.inf.elte.hu/fekete/algoritmusok_msc/dimenzio_csokkentes/randon_projection_kdd.pdf I compared different implementations of that algorithm: - https://github.com/sebastian-alfers/random-projection-python was: Implement RandomProjection (RP) for dimensionality reduction (DR) RP is a popular approach to reduce the amount of data while preserving a reasonable amount of information (pairwise distance) of you data [1][2] - [1] http://www.yaroslavvb.com/papers/achlioptas-database.pdf - [2] http://people.inf.elte.hu/fekete/algoritmusok_msc/dimenzio_csokkentes/randon_projection_kdd.pdf I compared different implementations of that algorithm: - https://github.com/sebastian-alfers/random-projection-python > Implement RandomProjection for Dimensionality Reduction > --- > > Key: SPARK-7334 > URL: https://issues.apache.org/jira/browse/SPARK-7334 > Project: Spark > Issue Type: Improvement > Components: MLlib >Reporter: Sebastian Alfers >Priority: Minor > > Implement RandomProjection (RP) for dimensionality reduction > RP is a popular approach to reduce the amount of data while preserving a > reasonable amount of information (pairwise distance) of you data [1][2] > - [1] http://www.yaroslavvb.com/papers/achlioptas-database.pdf > - [2] > http://people.inf.elte.hu/fekete/algoritmusok_msc/dimenzio_csokkentes/randon_projection_kdd.pdf > I compared different implementations of that algorithm: > - https://github.com/sebastian-alfers/random-projection-python -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-7334) Implement RandomProjection for Dimensionality Reduction
[ https://issues.apache.org/jira/browse/SPARK-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sebastian Alfers updated SPARK-7334: Target Version/s: (was: 1.3.1) > Implement RandomProjection for Dimensionality Reduction > --- > > Key: SPARK-7334 > URL: https://issues.apache.org/jira/browse/SPARK-7334 > Project: Spark > Issue Type: Improvement > Components: MLlib >Reporter: Sebastian Alfers >Priority: Minor > > Implement RandomProjection (RP) for dimensionality reduction > RP is a popular approach to reduce the amount of data while preserving a > reasonable amount of information (pairwise distance) of you data [1][2] > - [1] http://www.yaroslavvb.com/papers/achlioptas-database.pdf > - [2] > http://people.inf.elte.hu/fekete/algoritmusok_msc/dimenzio_csokkentes/randon_projection_kdd.pdf > I compared different implementations of that algorithm: > - https://github.com/sebastian-alfers/random-projection-python -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-7594) Increase maximum amount of columns for covariance matrix for principal components
Sebastian Alfers created SPARK-7594: --- Summary: Increase maximum amount of columns for covariance matrix for principal components Key: SPARK-7594 URL: https://issues.apache.org/jira/browse/SPARK-7594 Project: Spark Issue Type: Improvement Components: MLlib Reporter: Sebastian Alfers Priority: Minor In order to compute a huge dataset, the amount of columns to calculate the covariance matrix is limited: https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala#L129 What is the reason behind this limitation and can it be extended? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7334) Implement RandomProjection for Dimensionality Reduction
[ https://issues.apache.org/jira/browse/SPARK-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587691#comment-14587691 ] Sebastian Alfers commented on SPARK-7334: - I tried to contact [~yuu.ishik...@gmail.com] but got no reply - how can we continue on this? What needs to be done? Maybe we can finish my PR and update the API if necessary? > Implement RandomProjection for Dimensionality Reduction > --- > > Key: SPARK-7334 > URL: https://issues.apache.org/jira/browse/SPARK-7334 > Project: Spark > Issue Type: Improvement > Components: MLlib >Reporter: Sebastian Alfers >Priority: Minor > > Implement RandomProjection (RP) for dimensionality reduction > RP is a popular approach to reduce the amount of data while preserving a > reasonable amount of information (pairwise distance) of you data [1][2] > - [1] http://www.yaroslavvb.com/papers/achlioptas-database.pdf > - [2] > http://people.inf.elte.hu/fekete/algoritmusok_msc/dimenzio_csokkentes/randon_projection_kdd.pdf > I compared different implementations of that algorithm: > - https://github.com/sebastian-alfers/random-projection-python -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7334) Implement RandomProjection for Dimensionality Reduction
[ https://issues.apache.org/jira/browse/SPARK-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589388#comment-14589388 ] Sebastian Alfers commented on SPARK-7334: - I implemented RP as a transformer to be able to serialize the model and re-use it later. Also, the actual implementation of RP is separated and (theoretically) can be used in LSH. I implemented RP as a "stand alone" method as a replacement / comparison to PCA. > Implement RandomProjection for Dimensionality Reduction > --- > > Key: SPARK-7334 > URL: https://issues.apache.org/jira/browse/SPARK-7334 > Project: Spark > Issue Type: Improvement > Components: MLlib >Reporter: Sebastian Alfers >Priority: Minor > > Implement RandomProjection (RP) for dimensionality reduction > RP is a popular approach to reduce the amount of data while preserving a > reasonable amount of information (pairwise distance) of you data [1][2] > - [1] http://www.yaroslavvb.com/papers/achlioptas-database.pdf > - [2] > http://people.inf.elte.hu/fekete/algoritmusok_msc/dimenzio_csokkentes/randon_projection_kdd.pdf > I compared different implementations of that algorithm: > - https://github.com/sebastian-alfers/random-projection-python -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7334) Implement RandomProjection for Dimensionality Reduction
[ https://issues.apache.org/jira/browse/SPARK-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14607994#comment-14607994 ] Sebastian Alfers commented on SPARK-7334: - [~josephkb] any progress on this one? > Implement RandomProjection for Dimensionality Reduction > --- > > Key: SPARK-7334 > URL: https://issues.apache.org/jira/browse/SPARK-7334 > Project: Spark > Issue Type: Improvement > Components: MLlib >Reporter: Sebastian Alfers >Priority: Minor > > Implement RandomProjection (RP) for dimensionality reduction > RP is a popular approach to reduce the amount of data while preserving a > reasonable amount of information (pairwise distance) of you data [1][2] > - [1] http://www.yaroslavvb.com/papers/achlioptas-database.pdf > - [2] > http://people.inf.elte.hu/fekete/algoritmusok_msc/dimenzio_csokkentes/randon_projection_kdd.pdf > I compared different implementations of that algorithm: > - https://github.com/sebastian-alfers/random-projection-python -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
