Lewis John McGibbney created NUTCH-3240:
-------------------------------------------

             Summary: Add retry cause and exhaustion counters
                 Key: NUTCH-3240
                 URL: https://issues.apache.org/jira/browse/NUTCH-3240
             Project: Nutch
          Issue Type: Sub-task
            Reporter: Lewis John McGibbney


The metrics report lists retry counts and patterns as a missing signal. Part of 
that signal already exists, and the rest does not.

{\{FetcherThread}} increments \{{context.getCounter(NutchMetrics.GROUP_FETCHER, 
status.getName())}} once per fetch outcome. \{{ProtocolStatus.getName()}} 
returns \{{retry}} for \{{ProtocolStatus.RETRY}} and \{{exception}} for 
\{{ProtocolStatus.EXCEPTION}}. So a \{{nutch_fetcher}} / \{{retry}} counter 
already counts one of the retry paths. It is a dynamic name, not a 
\{{NutchMetrics}} constant, and it does not say why the retry happened or when 
the crawl gave up.

The paths that write \{{CrawlDatum.STATUS_FETCH_RETRY}} are:

* \{{ProtocolStatus.RETRY}}. No call to \{{ErrorTracker}}. The dynamic counter 
name is \{{retry}}.
* \{{ProtocolStatus.EXCEPTION}}, which falls through into the same output. 
\{{logError(String)}} records \{{ErrorTracker.ErrorType.OTHER}} always, not the 
exception class. HTTP responses that are not 300-307, 400, 401, 404, or 410 
come from \{{HttpBase}} as \{{ProtocolStatus.EXCEPTION}} with the message 
\{{Http code=NNN}}. That includes 403, 429, and every 5xx. The dynamic counter 
name is \{{exception}}.
* the \{{default}} branch, for an unknown protocol status.
* the \{{catch (Throwable)}} around the fetch. \{{logError(Throwable)}} does 
categorize through \{{ErrorTracker}} (timeout, network, or other). The datum is 
still \{{STATUS_FETCH_RETRY}}. \{{ProtocolStatus.STATUS_FAILED}} is what gets 
passed to \{{output}}, and the dynamic status counter is only incremented if 
execution already passed the \{{status.getName()}} line.

{\{CrawlDbReducer}} applies the retry limit. 
\{{AbstractFetchSchedule.setPageRetrySchedule}} increments 
\{{retriesSinceFetch}}, then the reducer marks the record \{{STATUS_DB_GONE}} 
when the count is no longer below \{{db.fetch.retry.max}} (default 3). The 
status counter for the resulting datum is \{{db_gone}}, the same counter used 
for a permanent failure. Nothing counts "retries exhausted."

h2. Proposal

Add constants to \{{NutchMetrics}} and cache the references.

In \{{FetcherThread}}, whenever the datum is set to \{{STATUS_FETCH_RETRY}}, 
increment \{{retries_total}} in \{{nutch_fetcher}} and exactly one cause 
counter:

* \{{catch (Throwable)}}: \{{retries_timeout_total}}, 
\{{retries_network_total}}, or \{{retries_other_total}}, from 
\{{ErrorTracker.categorize}}
* \{{ProtocolStatus.EXCEPTION}} whose message matches \{{Http code=}} with a 
code from 500 to 599: \{{retries_http_5xx_total}}. Parse the code from the 
message. Do not treat any other HTTP code as a 5xx
* \{{ProtocolStatus.EXCEPTION}} otherwise, and the unknown-status branch: 
\{{retries_other_total}}
* \{{ProtocolStatus.RETRY}}: \{{retries_protocol_retry_total}}. Leave the 
existing dynamic \{{retry}} counter in place

In \{{CrawlDbReducer}}, when the record is marked gone because 
\{{retriesSinceFetch >= retryMax}}, increment \{{retries_exhausted_total}} in 
\{{nutch_crawldb}}. This is a CrawlDb fact. Do not put it in \{{nutch_fetcher}}.

Keep the cause set fixed. Do not add a counter per status code or per host.

h2. Acceptance

* \{{ProtocolStatus.RETRY}} increments \{{retries_total}} and 
\{{retries_protocol_retry_total}}, and does not increment a timeout, network, 
or 5xx counter.
* An HTTP 500 carried as \{{ProtocolStatus.EXCEPTION}} with message \{{Http 
code=500}} increments \{{retries_total}} and \{{retries_http_5xx_total}}. An 
HTTP 403 or 429 on the same path increments \{{retries_other_total}} instead.
* A \{{SocketTimeoutException}} on the throwable path increments 
\{{retries_total}} and \{{retries_timeout_total}}.
* A record whose \{{retriesSinceFetch}} reaches \{{db.fetch.retry.max}} 
increments \{{nutch_crawldb}} / \{{retries_exhausted_total}}.
* A successful first fetch increments none of these counters.
* A unit test covers each of those five cases.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to