Hi Edward, Sorry for the late reply I have been busy with my internship. There are still some fixes left before we test it in full distributed mode. I will try to get that done in the coming few days.
Thanks Aditya Sarawgi On Thu, Jul 5, 2012 at 9:08 AM, Edward J. Yoon <[email protected]>wrote: > > Now I would like to test this on a big input and possibly in full > > distributed mode, so I was wondering how do > > people usually do these sort of benchmarking. > > My 1 thousand cores cluster is ready. > > Do you have any plan to contribute your patch to Hama? > > On Thu, May 17, 2012 at 4:39 PM, Aditya Sarawgi > <[email protected]> wrote: > > Thanks Thomas. > > I am actually using tags for something else. So for now using instanceof > is > > just fine with me. > > > > I had a couple of more questions, regarding benchmarking stuff on hama. I > > have a working implementation of > > Parallel row based icf that given a n x n matrix returns a decomposed n > x p > > matrix. > > > > > https://github.com/truncs/hello-world/blob/master/src/main/java/edu/sunysb/cs/Icf.java > > > > Now I would like to test this on a big input and possibly in full > > distributed mode, so I was wondering how do > > people usually do these sort of benchmarking. > > > > Specifically, > > 1) Do they setup a cluster on AWS ? > > 2) What is like the standard way to load matrices in different nodes > with a > > custom partitioning scheme > > 3) Is there anything else that I should know > > > > On Thu, May 17, 2012 at 3:20 AM, Thomas Jungblut < > > [email protected]> wrote: > > > >> Hi Aditya, > >> > >> that's where the concept of Message Tagging comes into play. You have > tags > >> in each message which are hardcoded as Strings. > >> But as Edward told you can use GenericWritable or ObjectWritable > instead, > >> so they will tag your messages with the classnames and give you the > correct > >> class. > >> > >> Is there any way by which I can pop from the receive queue ? > >> > >> > >> peer.getCurrentMessage() is popping from the received queue. > >> > >> 2012/5/17 Aditya Sarawgi <[email protected]> > >> > >> > Hi, > >> > > >> > But thats not the only problem, consider this case > >> > that there are variable number of messages being sent, so I would > have to > >> > maintain > >> > counts for each peer pointing to the last unread message. > >> > > >> > Is there any way by which I can pop from the receive queue ? > >> > > >> > > >> > On Wed, May 16, 2012 at 10:23 PM, Suraj Menon <[email protected] > >> > >wrote: > >> > > >> > > Hi, > >> > > > >> > > Please take a look at this snippet of code copied and modified from > >> > > Mapper class to implement your scenario. - > >> > > > >> > > > >> > > >> > https://github.com/ssmenon/hama/edit/master/hama-mapreduce/src/org/apache/hama/computemodel/mapreduce/Trials.java > >> > > Between lines 233 to 245 I am able to send different type of > messages. > >> > > With type checks and generics you shouldn't be encountering > Classcast > >> > > exception at receiving end too. I am yet to test the next superstep, > >> > > shall update you with sample code for the next superstep mimicking > >> > > your scenario for receiving. > >> > > > >> > > For elegance, we have an experimental Superstep#compute > >> > > API(org.apache.hama.bsp.Superstep). I have encountered an issue in > job > >> > > submission framework with this method in distributed mode; fix for > >> > > this would be pushed to trunk in next few hours. You can still run > it > >> > > using LocalBSPRunner for now. > >> > > > >> > > -Suraj > >> > > > >> > > On Wed, May 16, 2012 at 9:18 PM, Aditya Sarawgi > >> > > <[email protected]> wrote: > >> > > > Hi Edward, > >> > > > > >> > > > Yes that is what I did > >> > > > I wrote an ArrayMessage class (doesn't use generics for now but > can > >> be > >> > > > converted easily) > >> > > > > >> > > > >> > > >> > https://github.com/truncs/hello-world/blob/master/src/main/java/edu/sunysb/cs/ArrayMessage.java > >> > > > > >> > > > But the problem is that I am sending a IntegerMessage before and > >> after > >> > > > reading the IntegerMessage I am sending > >> > > > an ArrayMessage but the previous IntegerMessage is still there. > >> > > > > >> > > > On Wed, May 16, 2012 at 8:34 PM, Edward J. Yoon < > >> [email protected] > >> > > >wrote: > >> > > > > >> > > >> Hi, > >> > > >> > >> > > >> To send or receive multiple Message types, I think you can use > >> > > >> GenericWritable. You can also implement your own GenericMessage > and > >> > > >> contribute it to our project! > >> > > >> > >> > > >> Hope this helps you. > >> > > >> > >> > > >> On Thu, May 17, 2012 at 7:48 AM, Aditya Sarawgi > >> > > >> <[email protected]> wrote: > >> > > >> > Hi Guys, > >> > > >> > > >> > > >> > I am wondering how do the receive queues in hama work. Consider > >> this > >> > > case > >> > > >> > that I want to sent a different type of BSPMessage in 2 > >> consecutive > >> > > >> > superstep. > >> > > >> > In this first superstep I am sending IntMessage and in the next > >> one > >> > I > >> > > am > >> > > >> > sending a ArrayMessage ( custom message class). > >> > > >> > > >> > > >> > Now in the second super step when I do a > >> > > >> > while ((arrayMessage = (ArrayMessage) > peer.getCurrentMessage()) > >> != > >> > > >> null) { > >> > > >> > > >> > > >> > it is throwing a java.lang.ClassCastException, which is obvious > >> > since > >> > > its > >> > > >> > trying to cast IntMessage to ArrayMessage. > >> > > >> > I thought the message is dropped from the queue after it is > read, > >> is > >> > > this > >> > > >> > not the case ? > >> > > >> > And if it is not, how can this be handled elegantly ? > >> > > >> > > >> > > >> > -- > >> > > >> > Cheers, > >> > > >> > Aditya Sarawgi > >> > > >> > >> > > >> > >> > > >> > >> > > >> -- > >> > > >> Best Regards, Edward J. Yoon > >> > > >> @eddieyoon > >> > > >> > >> > > > > >> > > > > >> > > > > >> > > > -- > >> > > > Cheers, > >> > > > Aditya Sarawgi > >> > > > >> > > >> > > >> > > >> > -- > >> > Cheers, > >> > Aditya Sarawgi > >> > > >> > >> > >> > >> -- > >> Thomas Jungblut > >> Berlin <[email protected]> > >> > > > > > > > > -- > > Cheers, > > Aditya Sarawgi > > > > -- > Best Regards, Edward J. Yoon > @eddieyoon > -- Cheers, Aditya Sarawgi
