[
https://issues.apache.org/jira/browse/NUTCH-247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12474259
]
Andrzej Bialecki commented on NUTCH-247:
-----------------------------------------
One of the previous comments was that the check should be preferably done
_before_ the job is submitted, and I agree with this notion. This is more
user-friendly, because job doesn't have to be started (allocating cluster
resources), it produces a clear message on the console instead of producing
multiple log outputs, and also this specific condition can be checked without
starting a map-reduce job.
> robot parser to restrict.
> -------------------------
>
> Key: NUTCH-247
> URL: https://issues.apache.org/jira/browse/NUTCH-247
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Affects Versions: 0.8
> Reporter: Stefan Groschupf
> Assigned To: Dennis Kubes
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: agent-names.patch, agent-names3.patch.txt
>
>
> If the agent name and the robots agents are not proper configure the Robot
> rule parser uses LOG.severe to log the problem but solve it also.
> Later on the fetcher thread checks for severe errors and stop if there is one.
> RobotRulesParser:
> if (agents.size() == 0) {
> agents.add(agentName);
> LOG.severe("No agents listed in 'http.robots.agents' property!");
> } else if (!((String)agents.get(0)).equalsIgnoreCase(agentName)) {
> agents.add(0, agentName);
> LOG.severe("Agent we advertise (" + agentName
> + ") not listed first in 'http.robots.agents' property!");
> }
> Fetcher.FetcherThread:
> if (LogFormatter.hasLoggedSevere()) // something bad happened
> break;
> I suggest to use warn or something similar instead of severe to log this
> problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
