Hi Subroto, I cannot visit that video in China Today, I got it somehow. I got a question how can I make the number of Mappers and nodes the same? There is no adjacent matrix of the graph in input files How can I get the "seed" of the graph?
-- Regards! Jun Tan At 2011-08-10 02:51:08,"Subroto Sanyal 00900485" <subrotosan...@huawei.com> wrote: >Hi Jun, > >Basically, as per my understanding what you need is a graph traversal >algorithm in MR framework. >Request you to go through this video tutorial(from Google). This provides >clear illustration of Graph Traversal Algorithm (BreadtFirstSearch) in MR. >http://www.youtube.com/watch?v=BT-piFBP4fE&feature=player_embedded#at=812 >or >http://www.danielblaisdell.com/2008/map-reduce-graph-traversal/ > >Hope the mentioned links answers your questions. > >Regards, >Subroto Sanyal > >Hi Subroto, >Just one more thing >Mapper: > >As “wordcount” mapper consumes line by line; try to consume the hdfs file line >by line. > >The output of Mapper will tell a node is connected directly to which node. > > > >Reducer: > >Reducer will consume mapper output and will hold the output of node (Key) and >all the adjacent node(value) > >The output of Mapper do can tell a node is connected directly to which node >But how can it tell a node is connected indirectly (second neighbor, >neighbor's neighbor) to which node? >If Reducer can tell a node is connected indirectly to which node, how? >To get the adjacent node (value) of a node (key), the database file (gragh) is >needed. >I think that's the problem. > >-- > >Regards! > >Jun Tan > > >At 2011-08-09 21:41:45,"Subroto Sanyal" <subrotosan...@huawei.com> wrote: > >Hi Jun, > > > >I hope you want to find out the adjacent nodes of node in a graph. > > > >I have just gone through the mapper code (not very thoroughly). I feel the >implementation first needs to define the responsibility of Mapper and Reducer. > >I can see the Mapper is doing a file read operation and reading it line by >line….?????? Is it mappers responsibility to read file? > >Mapper implementation is not using the Key and Value??? > > > >Just a simple suggestion: > >Mapper: > >As “wordcount” mapper consumes line by line; try to consume the hdfs file line >by line. > >The output of Mapper will tell a node is connected directly to which node. > > > >Reducer: > >Reducer will consume mapper output and will hold the output of node (Key) and >all the adjacent node(value) > > > >The solution provided may not be optimized. > > > >Regards, >Subroto Sanyal > > >-------------------------------------------------------------------------------- > >From: ?? [mailto:tanjun_2...@163.com] >Sent: Tuesday, August 09, 2011 5:24 PM >To: mapreduce-user@hadoop.apache.org; subroto.san...@huawei.com >Subject: Re:RE: RE: Can reducer get parameters from mapper besides key and >value? > > > >Hi Subroto, > >It's very kind of u to help me. > >I really appreciate it. > >Now I attach the source code of my program. > >It's about find neighbors in a gragh. > >Such as gragh " A---B-----C", we define node B is the first neighbor of node A >while C is the second > >The program is to find the first and second neighbors of A. > >That's the scenario. > >Node A is in the key file and the gragh(s) is in the database file. > >Key file is given by users. Each line of key file is a nodename like > >A > >B > >C > >Database file is in hdfs servers ( in hard disk). Each line is an edge with 2 >nodes like > >A B 0.5 > >B C 0.7 > >0.5 and 0.7 are weights of the edges. > >I can get B according to the key A and get C according to B by mapreduce > >But I want to get B and C according to the key A. > >Now the program attached can finish my job, however I don't think it's good >enough. > >It's too time cost. It's not the program I wanted. > > > >Maybe u can help me. > >Thanks very much! > >-- > >Regards! > >Jun Tan > > >At 2011-08-09 19:16:17,"Subroto Sanyal" <subrotosan...@huawei.com> wrote: > > > >Hi Jun > > > >I mean that if I get some strings in mapper and I want to use them in reducer. > >But they are neither keys nor values. > >As per my understanding, there is no such way to pass an arbitrary reference >from Mapper to Reducer. > >The information written in Output from Mapper is available to Reducer. > >Further more, I don’t feel it will be good idea to keep such dependency. > >Please let me know more about your scenario…may be we/community can suggest >some solution… > > > >By the way, Can reducer get side files in cache? > >Please let me know about “Side Files”….. > > > >Regards, >Subroto Sanyal > > >-------------------------------------------------------------------------------- > >From: ?? [mailto:tanjun_2...@163.com] >Sent: Tuesday, August 09, 2011 12:25 PM >To: mapreduce-user@hadoop.apache.org; subroto.san...@huawei.com >Subject: Re:RE: Can reducer get parameters from mapper besides key and value? > > > >Hi Subroto, > >I mean that if I get some strings in mapper and I want to use them in reducer. > >But they are neither keys nor values. > > > >By the way, Can reducer get side files in cache? > > > >-- > >Regards! > >Jun Tan > > >At 2011-08-09 14:42:10,"Subroto Sanyal" <subrotosan...@huawei.com> wrote: > > > > > > >Hi Jun, > > > >What is the file, list, string [] in this context? > >I mean to say which file or list or string[]. > > > >The new MR (org.apache.hadoop.mapreduce.*) APIs has a parameter “context”. >Request you to browse through the APIs of Context (Inherited from >JobContext->TaskAttemptContext). > >The context parameter may provide you the reference you need. > > > >Regards, >Subroto Sanyal > > >-------------------------------------------------------------------------------- > >From: ?? [mailto:tanjun_2...@163.com] >Sent: Tuesday, August 09, 2011 11:58 AM >To: mapreduce >Subject: Can reducer get parameters from mapper besides key and value? > > > >Hi, > >Can reducer gets parameters from mapper besides key and value? > >Such as files, lists, string[] etc. > >Thanks > > > > > >-- > >Regards! > >Jun Tan > > > > > > > > > > > > > > > > > > > > > > >
