Re: Issue with MR code not scaling correctly with data sizes

2011-07-15 Thread Robert Evans
Please don't cross post. I put common-user in BCC. I really don't know for sure what is happening especially without the code or more to go on and debugging something remotely over e-mail is extremely difficult. You are essentially doing a cross which is going to be very expensive no matter w

Issue with MR code not scaling correctly with data sizes

2011-07-14 Thread GOEKE, MATTHEW (AG/1000)
All, I have a MR program that I feed in a list of IDs and it generates the unique comparison set as a result. Example: if I have a list {1,2,3,4,5} then the resulting output would be {2x1, 3x2, 3x1, 4x3, 4x2, 4x1, 5x4, 5x3, 5x2, 5x1} or (n^2-n)/2 number of comparisons. My code works just fine o