Re: Reduce shuffle data transfer takes excessively long

2012-01-31 Thread Robert Evans
If just changing the buffer to 4k makes a big difference could you at a minimum file a JIRA to change that buffer size? I know that it is not a final fix but it sure seems like a very nice Band-Aid to put on until we can get to the root of the issues. --Bobby Evans On 1/27/12 9:23 PM, Sven

RE: Reduce shuffle data transfer takes excessively long

2012-01-27 Thread Sven Groot
Hi Nick, Thanks for your reply. I don't think what you are saying is related, as the problem happens when the data is transferred; it's not deserialized or anything during that step. Note that my code isn't involved at all: it's purely Hadoop's own code that's running here. I have done