Re: How to replace user.id to user.names in a file
Hi,
it looks to me as if you need the whole user database on every node, so
maybe put the id-name information as a Map[Id, String] in a broadcast
variable and then do something like
recommendations.map(line = {
line.map(uid = usernames(uid))
})
or so?
Tobias
Re: How to replace user.id to user.names in a file
Hi,
On Wed, Jan 7, 2015 at 10:47 AM, Riginos Samaras samarasrigi...@gmail.com
wrote:
Yes something like this. Can you please give me an example to create a Map?
That depends heavily on the shape of your input file. What about something
like:
(for (line - Source.fromFile(filename).getLines()) {
val items = line.trim.split( )
(items(0).toInt, items(1))
}).toMap
Tobias
Re: How to replace user.id to user.names in a file
Hi,
On Wed, Jan 7, 2015 at 11:13 AM, Riginos Samaras samarasrigi...@gmail.com
wrote:
exactly thats what I'm looking for, my code is like this:
//code
val users_map = users_file.map{ s =
val parts = s.split(,)
(parts(0).toInt, parts(1))
}.distinct
//code
but i get the error:
error: value toMap is not a member of org.apache.spark.rdd.RDD[(Int,
String)]
user_map.toMap
If you want to distribute the Map as a broadcast variable, it must not be
an RDD but a normal Scala map. Make your users_file a regular List, then it
should work.
Tobias
