Re: How to replace user.id to user.names in a file
Hi,
On Wed, Jan 7, 2015 at 11:13 AM, Riginos Samaras
wrote:
> exactly thats what I'm looking for, my code is like this:
> //code
>
> val users_map = users_file.map{ s =>
>
> val parts = s.split(",")
>
> (parts(0).toInt, parts(1))
>
> }.distinct
>
> //code
>
>
> but i get the error:
>
> error: value toMap is not a member of org.apache.spark.rdd.RDD[(Int,
> String)]
>
> user_map.toMap
>
If you want to distribute the Map as a broadcast variable, it must not be
an RDD but a normal Scala map. Make your users_file a regular List, then it
should work.
Tobias
Re: How to replace user.id to user.names in a file
Hi,
On Wed, Jan 7, 2015 at 10:47 AM, Riginos Samaras
wrote:
> Yes something like this. Can you please give me an example to create a Map?
>
That depends heavily on the shape of your input file. What about something
like:
(for (line <- Source.fromFile(filename).getLines()) {
val items = line.trim.split(" ")
(items(0).toInt, items(1))
}).toMap
Tobias
Re: How to replace user.id to user.names in a file
Hi,
it looks to me as if you need the whole user database on every node, so
maybe put the id->name information as a Map[Id, String] in a broadcast
variable and then do something like
recommendations.map(line => {
line.map(uid => usernames(uid))
})
or so?
Tobias
How to replace user.id to user.names in a file
I work on a user to user recommender for a website using mllib.recommendation. I have created a file (recommends.txt) which contains the top 5 recommendations for each user id. The file's form(recommends.txt) is something like this (user::rec1:rec2:rec3:rec4:rec5): /**file's snapshot**/ 5823::944::10030::27::1047::2891::836 14::2097::10030::2427::1874::2018::5804 2726::6557::2776::2959::6619::2018::4466 6367::6557::9359::2726::2542::10159::3574 5804::27::2891::9359::944::1599::1047 1044::944::6557::2542::4737::1866::1874 /**file's snapshot**/ My Question : So I already have the users.sql file which contains the (id,names) and i want to shape recommends.txt to : Bob::Michael::Peter:Steve:Bill::George how can i easily do this? Thank you -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-replace-user-id-to-user-names-in-a-file-tp21006.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
