The ArrayBuffer did the trick! Thanks a lot! I'm learning Scala through spark so these details are still new to me
Sent from my iPhone > On Jan 11, 2016, at 5:18 PM, Jakob Odersky <joder...@gmail.com> wrote: > > Hey, > I just reread your question and saw I overlooked some crucial information. > Here's a solution: > > val data = > model.asInstanceOf[DistributedLDAModel].topicDistributions.sortByKey().collect() > > val tpdist = data.map(doc => doc._2.toArray) > > hope it works this time > >> On 11 January 2016 at 17:14, Jakob Odersky <joder...@gmail.com> wrote: >> Hi Daniel, >> >> You're actually not modifying the original array: `array :+ x ` will give >> you a new array with `x` appended to it. >> In your case the fix is simple: collect() already returns an array, use it >> as the assignment value to your val. >> >> In case you ever want to append values iteratively, search for how to use >> scala "ArrayBuffer"s. Also, keep in mind that RDDs have a foreach method, so >> no need to call collect followed by foreach. >> >> regards, >> --Jakob >> >> >> >>> On 11 January 2016 at 16:55, Daniel Valdivia <h...@danielvaldivia.com> >>> wrote: >>> Hello, >>> >>> I'm trying to put all the values in pair rdd into an array (or list) for >>> later storing, however even if I'm collecting the data then pushing it to >>> the array the array size after the run is 0. >>> >>> Any idea on what I'm missing? >>> >>> Thanks in advance >>> >>> scala> val tpdist: Array[Array[Double]] = Array() >>> tpdist: Array[Array[Double]] = Array() >>> >>> scala> >>> ldaModel.asInstanceOf[DistributedLDAModel].topicDistributions.sortByKey().collect().foreach(doc >>> => tpdist :+ doc._2.toArray ) >>> >>> >>> scala> tpdist.size >>> res27: Int = 0 >
