Thanks. Now I know how to broadcast the dataset but I still wonder after
broadcasting the dataset how can I apply my algorithm to training the model
in the wokers. To describe my question in detail, The following code is used
to train LDA(Latent Dirichlet Allocation) model with JGibbLDA in single
machine, it iterate to sample the topic and train the model. After
broadcasting the dataset, how can I keep the code running in Spark? Thank
you.
LDACmdOption ldaOption = new LDACmdOption(); //to set the
parameters of LDA
ldaOption.est = true;
ldaOption.estc = false;
ldaOption.modelName = "model-final";//the name of the output
file
ldaOption.dir = "/usr/Java";
ldaOption.dfile = "newDoc.dat"//this is the input data file
ldaOption.alpha = 0.5;
ldaOption.beta = 0.1;
ldaOption.K = 10;// the numbers of the topic
ldaOption.niters = 1000;//the times of iteration
topicNum = ldaOption.K;
Estimator estimator = new Estimator();
estimator.init(ldaOption);
estimator.estimate();
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-process-multiple-classification-with-SVM-in-MLlib-tp7174p7368.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
