> You would need to click on the map link on jobdetails.jsp and each task > will say something like this: > > 11337 pages, 3748 errors, 1.7 pages/s, 329 kb/s,
Okay. Now I see it. There are 97 map tasks. 96 map tasks state: 0 threads, 0 pages, 0 errors, 0.0 pages/s, 0 kb/s, All the above 96 map tasks state the same thing. 1 map task states: 0 threads, 2888 pages, 54 errors, 3.3 pages/s, 1681 kb/s, There is definitely a problem here. How do I spread out the map tasks to all available slots? I found the errors by clicking on the task. Then, I clicked on the task logs all link. It appeared to be running fine, until near the end, I am seeing error messages like this: ----- -activeThreads=16, spinWaiting=6, fetchQueues.totalSize=800 fetch of http://www.example.com/public/page.asp/1618613 failed with: java.lang.OutOfMemoryError: Java heap space -activeThreads=16, spinWaiting=9, fetchQueues.totalSize=800 -activeThreads=16, spinWaiting=0, fetchQueues.totalSize=800 fetch of http://www.example.com/public/page.asp/118288 failed with: java.lang.OutOfMemoryError: Java heap space fetch of http://www.example.com/public/page.asp/74343 failed with: java.lang.OutOfMemoryError: Java heap space -activeThreads=16, spinWaiting=2, fetchQueues.totalSize=800 fetch of http://www.example.com/public/page.asp/971779 failed with: java.lang.OutOfMemoryError: Java heap space -activeThreads=16, spinWaiting=2, fetchQueues.totalSize=800 fetch of http://www.example.com/public/page.asp/1585170 failed with: java.lang.OutOfMemoryError: Java heap space -activeThreads=16, spinWaiting=1, fetchQueues.totalSize=800 fetch of http://www.example.com/public/page.asp/82747 failed with: java.lang.OutOfMemoryError: Java heap space fetch of http://www.example.com/public/page.asp/687356 failed with: java.lang.OutOfMemoryError: Java heap space -activeThreads=16, spinWaiting=0, fetchQueues.totalSize=798 fetch of http://www.example.com/public/page.asp/1425403 failed with: java.lang.OutOfMemoryError: Java heap space fetch of http://www.example.com/public/page.asp/1461659 failed with: java.lang.OutOfMemoryError: Java heap space -activeThreads=16, spinWaiting=3, fetchQueues.totalSize=800 fetch of http://www.example.com/public/page.asp/1484234 failed with: java.lang.OutOfMemoryError: Java heap space fetch of http://www.example.com/public/page.asp/1096086 failed with: java.lang.OutOfMemoryError: Java heap space -activeThreads=16, spinWaiting=5, fetchQueues.totalSize=800 fetch of http://www.example.com/public/page.asp/12074 failed with: java.lang.OutOfMemoryError: Java heap space fetch of http://www.example.com/public/page.asp/565789 failed with: java.lang.OutOfMemoryError: Java heap space fetch of http://www.example.com/public/page.asp/542157 failed with: java.lang.OutOfMemoryError: Java heap space fetch of http://www.example.com/public/page.asp/522730 failed with: java.lang.OutOfMemoryError: Java heap space fetch of http://www.example.com/public/page.asp/1617438 failed with: java.lang.OutOfMemoryError: Java heap space ----- How do I resolve this error? The master has 4gb, this particular slave has 1.5gb. Another slave has 1gb, another slave has 4gb. Do I just need to add physical memory? Or, is this something else in the configuration? Is this error the cause of only doing 3% of the 100k urls I requested to be done? Or, is it a problem with the other 96 map tasks not doing anything? Thanks again for all of your help. JohnM -- john mendenhall [EMAIL PROTECTED] surf utopia internet services
