Hello DB, Thank you! Do you know how to run Linear Regression without SGD on streaming data in spark? I tried SGD but due to step size I was not getting the expected weights.
Best Regards, Arunkumar On Wed, Mar 25, 2015 at 4:33 PM, DB Tsai <dbt...@dbtsai.com> wrote: > Hi Arunkumar, > > I think L-BFGS will not work since L-BFGS algorithm assumes that the > objective function will be always the same (i.e., the data is the > same) for entire optimization process to construct the approximated > Hessian matrix. In the streaming case, the data will be changing, so > it will cause problem for the algorithm. > > Sincerely, > > DB Tsai > ------------------------------------------------------- > Blog: https://www.dbtsai.com > > > On Mon, Mar 16, 2015 at 3:19 PM, EcoMotto Inc. <ecomot...@gmail.com> > wrote: > > Hello, > > > > I am new to spark streaming API. > > > > I wanted to ask if I can apply LBFGS (with LeastSquaresGradient) on > > streaming data? Currently I am using forecahRDD for parsing through > DStream > > and I am generating a model based on each RDD. Am I doing anything > logically > > wrong here? > > Thank you. > > > > Sample Code: > > > > val algorithm = new LBFGS(new LeastSquaresGradient(), new > SimpleUpdater()) > > var initialWeights = > > Vectors.dense(Array.fill(numFeatures)(scala.util.Random.nextDouble())) > > var isFirst = true > > var model = new LinearRegressionModel(null,1.0) > > > > parsedData.foreachRDD{rdd => > > if(isFirst) { > > val weights = algorithm.optimize(rdd, initialWeights) > > val w = weights.toArray > > val intercept = w.head > > model = new LinearRegressionModel(Vectors.dense(w.drop(1)), > intercept) > > isFirst = false > > }else{ > > var ab = ArrayBuffer[Double]() > > ab.insert(0, model.intercept) > > ab.appendAll( model.weights.toArray) > > print("Intercept = "+model.intercept+" :: modelWeights = > > "+model.weights) > > initialWeights = Vectors.dense(ab.toArray) > > print("Initial Weights: "+ initialWeights) > > val weights = algorithm.optimize(rdd, initialWeights) > > val w = weights.toArray > > val intercept = w.head > > model = new LinearRegressionModel(Vectors.dense(w.drop(1)), > intercept) > > } > > > > > > > > Best Regards, > > Arunkumar >
