Hi Thomas, As you suggested I have made some chnages to https://github.com/truncs/hello-world/blob/master/src/main/java/edu/sunysb/cs/Icf.java
The custom paritioner that you had for matrix multiplication seems to meet my needs as well. I just want to split each row such that each subsequent row goes to a new task. Would you like to review those changes ? The only thing left now is to take the input and output from hdfs (which is trivial i think ) and to think about ways to generate test matrices for this. On Sun, Jul 15, 2012 at 7:57 PM, Aditya Sarawgi <[email protected]>wrote: > Hi Edward, > > Sorry for the late reply I have been busy with my internship. There are > still some fixes > left before we test it in full distributed mode. I will try to get that > done in the coming > few days. > > Thanks > Aditya Sarawgi > > > On Thu, Jul 5, 2012 at 9:08 AM, Edward J. Yoon <[email protected]>wrote: > >> > Now I would like to test this on a big input and possibly in full >> > distributed mode, so I was wondering how do >> > people usually do these sort of benchmarking. >> >> My 1 thousand cores cluster is ready. >> >> Do you have any plan to contribute your patch to Hama? >> >> On Thu, May 17, 2012 at 4:39 PM, Aditya Sarawgi >> <[email protected]> wrote: >> > Thanks Thomas. >> > I am actually using tags for something else. So for now using >> instanceof is >> > just fine with me. >> > >> > I had a couple of more questions, regarding benchmarking stuff on hama. >> I >> > have a working implementation of >> > Parallel row based icf that given a n x n matrix returns a decomposed n >> x p >> > matrix. >> > >> > >> https://github.com/truncs/hello-world/blob/master/src/main/java/edu/sunysb/cs/Icf.java >> > >> > Now I would like to test this on a big input and possibly in full >> > distributed mode, so I was wondering how do >> > people usually do these sort of benchmarking. >> > >> > Specifically, >> > 1) Do they setup a cluster on AWS ? >> > 2) What is like the standard way to load matrices in different nodes >> with a >> > custom partitioning scheme >> > 3) Is there anything else that I should know >> > >> > On Thu, May 17, 2012 at 3:20 AM, Thomas Jungblut < >> > [email protected]> wrote: >> > >> >> Hi Aditya, >> >> >> >> that's where the concept of Message Tagging comes into play. You have >> tags >> >> in each message which are hardcoded as Strings. >> >> But as Edward told you can use GenericWritable or ObjectWritable >> instead, >> >> so they will tag your messages with the classnames and give you the >> correct >> >> class. >> >> >> >> Is there any way by which I can pop from the receive queue ? >> >> >> >> >> >> peer.getCurrentMessage() is popping from the received queue. >> >> >> >> 2012/5/17 Aditya Sarawgi <[email protected]> >> >> >> >> > Hi, >> >> > >> >> > But thats not the only problem, consider this case >> >> > that there are variable number of messages being sent, so I would >> have to >> >> > maintain >> >> > counts for each peer pointing to the last unread message. >> >> > >> >> > Is there any way by which I can pop from the receive queue ? >> >> > >> >> > >> >> > On Wed, May 16, 2012 at 10:23 PM, Suraj Menon < >> [email protected] >> >> > >wrote: >> >> > >> >> > > Hi, >> >> > > >> >> > > Please take a look at this snippet of code copied and modified from >> >> > > Mapper class to implement your scenario. - >> >> > > >> >> > > >> >> > >> >> >> https://github.com/ssmenon/hama/edit/master/hama-mapreduce/src/org/apache/hama/computemodel/mapreduce/Trials.java >> >> > > Between lines 233 to 245 I am able to send different type of >> messages. >> >> > > With type checks and generics you shouldn't be encountering >> Classcast >> >> > > exception at receiving end too. I am yet to test the next >> superstep, >> >> > > shall update you with sample code for the next superstep mimicking >> >> > > your scenario for receiving. >> >> > > >> >> > > For elegance, we have an experimental Superstep#compute >> >> > > API(org.apache.hama.bsp.Superstep). I have encountered an issue in >> job >> >> > > submission framework with this method in distributed mode; fix for >> >> > > this would be pushed to trunk in next few hours. You can still run >> it >> >> > > using LocalBSPRunner for now. >> >> > > >> >> > > -Suraj >> >> > > >> >> > > On Wed, May 16, 2012 at 9:18 PM, Aditya Sarawgi >> >> > > <[email protected]> wrote: >> >> > > > Hi Edward, >> >> > > > >> >> > > > Yes that is what I did >> >> > > > I wrote an ArrayMessage class (doesn't use generics for now but >> can >> >> be >> >> > > > converted easily) >> >> > > > >> >> > > >> >> > >> >> >> https://github.com/truncs/hello-world/blob/master/src/main/java/edu/sunysb/cs/ArrayMessage.java >> >> > > > >> >> > > > But the problem is that I am sending a IntegerMessage before and >> >> after >> >> > > > reading the IntegerMessage I am sending >> >> > > > an ArrayMessage but the previous IntegerMessage is still there. >> >> > > > >> >> > > > On Wed, May 16, 2012 at 8:34 PM, Edward J. Yoon < >> >> [email protected] >> >> > > >wrote: >> >> > > > >> >> > > >> Hi, >> >> > > >> >> >> > > >> To send or receive multiple Message types, I think you can use >> >> > > >> GenericWritable. You can also implement your own GenericMessage >> and >> >> > > >> contribute it to our project! >> >> > > >> >> >> > > >> Hope this helps you. >> >> > > >> >> >> > > >> On Thu, May 17, 2012 at 7:48 AM, Aditya Sarawgi >> >> > > >> <[email protected]> wrote: >> >> > > >> > Hi Guys, >> >> > > >> > >> >> > > >> > I am wondering how do the receive queues in hama work. >> Consider >> >> this >> >> > > case >> >> > > >> > that I want to sent a different type of BSPMessage in 2 >> >> consecutive >> >> > > >> > superstep. >> >> > > >> > In this first superstep I am sending IntMessage and in the >> next >> >> one >> >> > I >> >> > > am >> >> > > >> > sending a ArrayMessage ( custom message class). >> >> > > >> > >> >> > > >> > Now in the second super step when I do a >> >> > > >> > while ((arrayMessage = (ArrayMessage) >> peer.getCurrentMessage()) >> >> != >> >> > > >> null) { >> >> > > >> > >> >> > > >> > it is throwing a java.lang.ClassCastException, which is >> obvious >> >> > since >> >> > > its >> >> > > >> > trying to cast IntMessage to ArrayMessage. >> >> > > >> > I thought the message is dropped from the queue after it is >> read, >> >> is >> >> > > this >> >> > > >> > not the case ? >> >> > > >> > And if it is not, how can this be handled elegantly ? >> >> > > >> > >> >> > > >> > -- >> >> > > >> > Cheers, >> >> > > >> > Aditya Sarawgi >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> -- >> >> > > >> Best Regards, Edward J. Yoon >> >> > > >> @eddieyoon >> >> > > >> >> >> > > > >> >> > > > >> >> > > > >> >> > > > -- >> >> > > > Cheers, >> >> > > > Aditya Sarawgi >> >> > > >> >> > >> >> > >> >> > >> >> > -- >> >> > Cheers, >> >> > Aditya Sarawgi >> >> > >> >> >> >> >> >> >> >> -- >> >> Thomas Jungblut >> >> Berlin <[email protected]> >> >> >> > >> > >> > >> > -- >> > Cheers, >> > Aditya Sarawgi >> >> >> >> -- >> Best Regards, Edward J. Yoon >> @eddieyoon >> > > > > -- > Cheers, > Aditya Sarawgi > -- Cheers, Aditya Sarawgi
