[jira] Commented: (NUTCH-489) URLFilter-suffix management of the url path when the url contains some query parameters
[ https://issues.apache.org/jira/browse/NUTCH-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12497770 ] Doğacan Güney commented on NUTCH-489: - This is obviously useful but: * Your patches both in this issue and in NUTCH-488 have whitespace issues. Nutch uses (or at least, tries to use) 2-space indentation. You seem to use a tab-indentation, so your patches end up removing code only to re-add it with wrong indentation. * I don't understand why new P option only works after I. What if I want this filter to be case-sensitive and to filter on path? * (Not related to your patch) It seems even a "-I" line makes SuffixURLFilter ignore case. Is that intended? > URLFilter-suffix management of the url path when the url contains some query > parameters > --- > > Key: NUTCH-489 > URL: https://issues.apache.org/jira/browse/NUTCH-489 > Project: Nutch > Issue Type: Improvement >Affects Versions: 0.9.0 > Environment: Windows, Java 1.5 >Reporter: Emmanuel Joke > Attachments: suffix-urlfilter.txt.patch, SuffixURLFilter.java.patch > > > The current filter compares only on string level. It try to apply the filter > to the full URL (path + query parameters). > So, even if we have define in our filter to exclude all js extension, it > won't exclude this URL http://www.toto.com/from.js?id=5. > I've added a new parameter in the filter which can be use to configure the > filter to exclude URL based on the url path. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-489) URLFilter-suffix management of the url path when the url contains some query parameters
[ https://issues.apache.org/jira/browse/NUTCH-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12498113 ] Doğacan Güney commented on NUTCH-489: - Hmm.. Won't it now cause Nutch to filter on path on a line like this: -(jpg|JPG) Because, you just check if line contains a 'P', and line above does. Perhaps, a better option would be to require P and I be the only characters on their line, like: +I +P # but not +IP > URLFilter-suffix management of the url path when the url contains some query > parameters > --- > > Key: NUTCH-489 > URL: https://issues.apache.org/jira/browse/NUTCH-489 > Project: Nutch > Issue Type: Improvement >Affects Versions: 0.9.0 > Environment: Windows, Java 1.5 >Reporter: Emmanuel Joke > Attachments: suffix-urlfilter.txt.patch, SuffixURLFilter.java.patch, > SuffixURLFilter_v2.java.patch > > > The current filter compares only on string level. It try to apply the filter > to the full URL (path + query parameters). > So, even if we have define in our filter to exclude all js extension, it > won't exclude this URL http://www.toto.com/from.js?id=5. > I've added a new parameter in the filter which can be use to configure the > filter to exclude URL based on the url path. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-489) URLFilter-suffix management of the url path when the url contains some query parameters
[ https://issues.apache.org/jira/browse/NUTCH-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499777 ] Doğacan Güney commented on NUTCH-489: - Please ignore my last comment. I don't know what I was on when I wrote that. Your patch seems to be correct. Andrzej, (hope you are reading this:) isn't javadoc in SuffixURLFilter wrong? It says ", you should use "+.jpg" instead.", but AFAICS, that line would have no effect except possibly switching mode to accept. Since, first character after '+' is not 'I', filter will just skip rest of the line. Am I missing something here? > URLFilter-suffix management of the url path when the url contains some query > parameters > --- > > Key: NUTCH-489 > URL: https://issues.apache.org/jira/browse/NUTCH-489 > Project: Nutch > Issue Type: Improvement >Affects Versions: 0.9.0 > Environment: Windows, Java 1.5 >Reporter: Emmanuel Joke > Attachments: suffix-urlfilter.txt.patch, SuffixURLFilter.java.patch, > SuffixURLFilter_v2.java.patch > > > The current filter compares only on string level. It try to apply the filter > to the full URL (path + query parameters). > So, even if we have define in our filter to exclude all js extension, it > won't exclude this URL http://www.toto.com/from.js?id=5. > I've added a new parameter in the filter which can be use to configure the > filter to exclude URL based on the url path. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
