The typical way to handle that use case would be to join the 3 files together into one RDD and then do the factorization on that. There will definitely be network traffic during the initial join to get everything into one table, and after that there will likely be more network traffic for various shuffle joins that are needed.
So for your individual questions: 1) it's not so much that your algorithm is executed on each node, but that the algorithm is executed on the cluster and small component tasks are run on each node. Each step in the algorithm happens across the whole cluster concurrently. 2) network communication will be needed to move data around during the algorithm. I'm not familiar with MF, but if you need the whole dataset on a machine at once for some reason, then you'll have lots of network computation 3) This question sounds a bit like pre-fetching -- does Spark start loading data that it will need "soon" but does not need now? That does not happen, and Spark RDDs are actually lazy (technical term) in that no computation happens until the end. Think of that like stacking operations on top of data. Stacking operations without evaluating them is cheap, and then once you evaluate the whole thing it can do special pipelining stuff that is able to keep the resident memory set lower. Does that help? Andrew On Wed, Apr 9, 2014 at 9:05 AM, gtanguy <g.tanguy.claravi...@gmail.com>wrote: > Hello everybody, > > I am wondering how Spark handles via HDFS his RDD, what if during a map > phase I need data which are not present locally? > > What I am working on : > I am working on a recommendation algorithm : Matrix Factorization (MF) > using > a stochastic gradient as optimizer. For now my algorithm works locally but > to anticipate further needs I would like to parallelized it using spark > 0.9.0 on HDFS (without yarn). > I saw the regression logistic (RL) SGD example in the MLibs. Matrix > Factorization can be view as multiple regression logistic iteration, so I > will follow the example to implement it. The only difference is : my > dataset > is composed by 3 files : > User.csv -> (UserID age sex..) > Item.csv -> (ItemID color size..) > Obs.csv -> (UserID, ItemID, ratings) > > What I understand : > In the RL example we have only the 'Obs.csv' file. Given that we have 3 > machines, the file will be divided on 3 machines, during the map phase, the > RL algorithm will be respectively executed on the 3 slaves with local data. > So each RL will process 1/3 of the data. During the reduce phase, we just > average the result returned by each slave. No network communication is > needed during the RL process except the reduce step. All the data during > map > phase used/needed are local. > > What I am wondering : > In my case my MF needs on each machine all the informations of the > 'User.csv' file, 1/3 'Item.csv' file and 1/3 obs.csv to operate. When > HDFS > distributes my 3 files, I will have 1/3 of each file on each datanode. > 1.What will happen when my MF algorithm is executed on each node? > 2.Network communications will be needed for the 2/3 of the user.csv, right? > 3.Will network communications be optimized as following : > During an actual computation does the data needed for the next computation > will be loaded (so that the time taking by the network communication won't > affect the computation time)? > > Any help is highly appreciated. > > Best regards, > > Germain Tanguy. > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/How-does-Spark-handle-RDD-via-HDFS-tp4003.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >