I see that it use a while loop to wait for threads to exit and will wait 1
second between each check. so even if fetcher thread was finished, the
whole fetcher process will take little longer to exit.
code structure like this.
do { // wait for threads to exit
pagesLastSec = pages.get();
bytesLastSec = (int)bytes.get();
try {
Thread.sleep(1000);
} catch (InterruptedException e) {}
....
reportStatus(pagesLastSec, bytesLastSec); // your print output is
coming here
LOG.info("-activeThreads=" + activeThreads + ", spinWaiting=" +
spinWaiting.get()
+ ", fetchQueues.totalSize=" + fetchQueues.getTotalSize());
if (!feeder.isAlive() && fetchQueues.getTotalSize() < 5) {
fetchQueues.dump();
}
....
// check timelimit
if (!feeder.isAlive()) {
int hitByTimeLimit = fetchQueues.checkTimelimit();
if (hitByTimeLimit != 0) reporter.incrCounter("FetcherStatus",
"hitByTimeLimit", hitByTimeLimit);
}
// some requests seem to hang, despite all intentions
if ((System.currentTimeMillis() - lastRequestStart.get()) > timeout) {
if (LOG.isWarnEnabled()) {
LOG.warn("Aborting with "+activeThreads+" hung threads.");
}
return;
}
} while (activeThreads.get() > 0);
On Wed, Dec 4, 2013 at 7:57 PM, Amit Sela <[email protected]> wrote:
> In the fetch phase, I notice that some of the mappers take much longer to
> finish.
> In the running task mapreduce admin screen it shows
>
> *1 threads, 1 queues, 0 URLs queued, *
>
> So why those tasks are not complete ?
>
--
Don't Grow Old, Grow Up... :-)
