[
https://issues.apache.org/jira/browse/SPARK-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tarek Elgamal updated SPARK-7856:
---------------------------------
Description:
Currently the PCA implementation has a limitation of fitting d^2
covariance/grammian matrix entries in memory (d is the number of
columns/dimensions of the matrix). We often need only the largest k principal
components. To make pca really scalable, I suggest an implementation where the
memory usage is proportional to the principal components k rather than the full
dimensionality d.
I suggest adopting the solution described in this paper that is published in
SIGMOD 2015 (http://ds.qcri.org/images/profile/tarek_elgamal/sigmod2015.pdf).
The paper offers an implementation for Probabilistic PCA (PPCA) which has less
memory and time complexity and could potentially scale to tall and fat matrices
rather than tall and skinny matrices that is supported by the current PCA
impelmentation.
Probablistic PCA could be potentially added to the set of algorithms supported
by MLlib and it does not necessarily replace the old PCA implementation.
PPCA implementation is adopted in Matlab's Statistics and Machine Learning
Toolbox (http://www.mathworks.com/help/stats/ppca.html)
was:
Currently the PCA implementation has a limitation of fitting d^2
covariance/grammian matrix entries in memory (d is the number of
columns/dimensions of the matrix). We often need only the largest k principal
components. To make pca really scalable, I suggest an implementation where the
memory usage is proportional to the principal components k rather than the full
dimensionality d.
I suggest adopting the solution described in this paper that is published in
SIGMOD 2015 (http://ds.qcri.org/images/profile/tarek_elgamal/sigmod2015.pdf).
The paper offers an implementation for Probabilistic PCA (PPCA) which has less
memory and time complexity and could potentially scale to tall and fat matrices
rather than tall and skinny matrices that is supported by the current PCA
impelmentation.
Probablistic PCA could be potentially added to the set of algorithms supported
by MLlib and it does not necessarily replace the old PCA implementation.
> Scalable PCA implementation for tall and fat matrices
> -----------------------------------------------------
>
> Key: SPARK-7856
> URL: https://issues.apache.org/jira/browse/SPARK-7856
> Project: Spark
> Issue Type: Bug
> Components: MLlib
> Reporter: Tarek Elgamal
>
> Currently the PCA implementation has a limitation of fitting d^2
> covariance/grammian matrix entries in memory (d is the number of
> columns/dimensions of the matrix). We often need only the largest k principal
> components. To make pca really scalable, I suggest an implementation where
> the memory usage is proportional to the principal components k rather than
> the full dimensionality d.
> I suggest adopting the solution described in this paper that is published in
> SIGMOD 2015 (http://ds.qcri.org/images/profile/tarek_elgamal/sigmod2015.pdf).
> The paper offers an implementation for Probabilistic PCA (PPCA) which has
> less memory and time complexity and could potentially scale to tall and fat
> matrices rather than tall and skinny matrices that is supported by the
> current PCA impelmentation.
> Probablistic PCA could be potentially added to the set of algorithms
> supported by MLlib and it does not necessarily replace the old PCA
> implementation.
> PPCA implementation is adopted in Matlab's Statistics and Machine Learning
> Toolbox (http://www.mathworks.com/help/stats/ppca.html)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
