Hi Xiangrui,
Thanks a lot for you answer.
So I fixed my Julia code, also calculated PCA using R as well.
R Code:
-------------
data <- read.csv('/home/upul/Desktop/iris.csv');
X <- data[,1:4]
pca <- prcomp(X, center = TRUE, scale=FALSE)
transformed <- predict(pca, newdata = X)
Julia Code (Fixed)
--------------
data = readcsv("/home/upul/temp/iris.csv");
X = data[:,1:end-1];
meanX = mean(X,1);
m,n = size(X);
X = X - repmat(x, m,1);
u,s,v = svd(X);
transformed = X*v;
Now PCA calculated using Julia and R is identical, but still I can see a
small
difference between PCA values given by Spark and other two.
Thanks,
Upul
On Sat, Jan 10, 2015 at 11:17 AM, Xiangrui Meng <men...@gmail.com> wrote:
> You need to subtract mean values to obtain the covariance matrix
> (http://en.wikipedia.org/wiki/Covariance_matrix).
>
> On Fri, Jan 9, 2015 at 6:41 PM, Upul Bandara <upulband...@gmail.com>
> wrote:
> > Hi Xiangrui,
> >
> > Thanks for the reply.
> >
> > Julia code is also using the covariance matrix:
> > (1/n)*X'*X ;
> >
> > Thanks,
> > Upul
> >
> > On Fri, Jan 9, 2015 at 2:11 AM, Xiangrui Meng <men...@gmail.com> wrote:
> >>
> >> The Julia code is computing the SVD of the Gram matrix. PCA should be
> >> applied to the covariance matrix. -Xiangrui
> >>
> >> On Thu, Jan 8, 2015 at 8:27 AM, Upul Bandara <upulband...@gmail.com>
> >> wrote:
> >> > Hi All,
> >> >
> >> > I tried to do PCA for the Iris dataset
> >> > [https://archive.ics.uci.edu/ml/datasets/Iris] using MLLib
> >> >
> >> > [
> http://spark.apache.org/docs/1.1.1/mllib-dimensionality-reduction.html].
> >> > Also, PCA was calculated in Julia using following method:
> >> >
> >> > Sigma = (1/numRow(X))*X'*X ;
> >> > [U, S, V] = svd(Sigma);
> >> > Ureduced = U(:, 1:k);
> >> > Z = X*Ureduced;
> >> >
> >> > However, I'm seeing a little difference between values given by MLLib
> >> > and
> >> > the method shown above .
> >> >
> >> > Does anyone have any idea about this difference?
> >> >
> >> > Additionally, I have attached two visualizations, related to two
> >> > approaches.
> >> >
> >> > Thanks,
> >> > Upul
> >> >
> >> >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> >> > For additional commands, e-mail: user-h...@spark.apache.org
> >
> >
>
