[
https://issues.apache.org/jira/browse/HADOOP-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12554868
]
Mukund Madhugiri commented on HADOOP-1650:
------------------------------------------
I ran the sort benchmark on 500 nodes and here is the data:
1. Sort 500 using trunk:
- Random writer: 19 mins
- Sort: 79 mins
- Sort validation: 19 mins
2. Sort 500 using trunk + patch
- Random Writer: 24 mins
- Sort: 111 mins
- Sort validation: 17 mins
I see some errors/exceptions in the JT logs:
1. See a lot of these during the random writer job:
2007-12-28 22:31:02,456 INFO org.apache.hadoop.mapred.TaskInProgress: Error
from task_200712282210_0001_m_004256_0: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.dfs
.NotReplicatedYetException: Not replicated
yet:/user/hadoopqa/sortBenchmark500/input/_task_200712282210_0001_m_004256_0/part-04256
2007-12-28 22:31:04,381 INFO org.apache.hadoop.mapred.TaskInProgress: Error
from task_200712282210_0001_m_004279_0: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.dfs
.NotReplicatedYetException: Not replicated
yet:/user/hadoopqa/sortBenchmark500/input/_task_200712282210_0001_m_004279_0/part-04279
2. See a log of these during the random writer job:
2007-12-28 22:31:57,715 INFO org.apache.hadoop.mapred.TaskInProgress: Error
from task_200712282210_0001_m_004485_0: java.net.SocketTimeoutException: timed
out waiting for rpc r
esponse
2007-12-28 22:31:58,368 INFO org.apache.hadoop.mapred.TaskInProgress: Error
from task_200712282210_0001_m_004272_0: java.net.SocketTimeoutException: timed
out waiting for rpc r
esponse
3. See a couple of these during the sort job:
2007-12-28 22:37:43,783 INFO org.apache.hadoop.mapred.TaskInProgress: Error
from task_200712282210_0002_m_002442_0: java.lang.OutOfMemoryError: Java heap
space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.io.BytesWritable.write(BytesWritable.java:137)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:373)
at
org.apache.hadoop.mapred.lib.IdentityMapper.map(IdentityMapper.java:40)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2043)
> Upgrade Jetty to 6.x
> --------------------
>
> Key: HADOOP-1650
> URL: https://issues.apache.org/jira/browse/HADOOP-1650
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Attachments: hadoop-1650-jetty6.1.5.patch,
> hadoop-jetty6.1.4-lib.tar.gz, hadoop-jetty6.1.6-lib.tar.gz,
> jetty-hadoop-6.1.6.patch, jetty-hbase.patch, jetty6.1.4.patch,
> jetty6.1.6.patch
>
>
> This is the third attempt at moving to jetty6. Apparently, the jetty-6.1.4
> has fixed some of the issues we discovered in jetty during HADOOP-736 and
> HADOOP-1273. I'd like to keep this issue open for sometime so that we have
> enough time to test out things.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
