[jira] Updated: (NUTCH-706) Url regex normalizer

[Julien Nioche (JIRA)]

     [ 
https://issues.apache.org/jira/browse/NUTCH-706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julien Nioche updated NUTCH-706:
--------------------------------

    Fix Version/s:     (was: 1.1)

Both variants of the substitution rule above break existing tests. More work 
will be needed to get a pattern which covers the case described by Meghna *and* 
is compatible with the existing test cases.
Moving it to post-1.1

> Url regex normalizer
> --------------------
>
>                 Key: NUTCH-706
>                 URL: https://issues.apache.org/jira/browse/NUTCH-706
>             Project: Nutch
>          Issue Type: Bug
>    Affects Versions: 1.0.0
>            Reporter: Meghna Kukreja
>            Priority: Minor
>
> Hey,
> I encountered the following problem while trying to crawl a site using
> nutch-trunk. In the file regex-normalize.xml, the following regex is
> used to remove session ids:
> <pattern>([;_]?((?i)l|j|bv_)?((?i)sid|phpsessid|sessionid)=.*?)(\?|&amp;|#|$)</pattern>.
> This pattern also transforms a url, such as,
> "&newsId=2000484784794&newsLang=en" into "&new&newsLang=en" (since it
> matches 'sId' in the 'newsId'), which is incorrect and hence does not
> get fetched. This expression needs to be changed to prevent this.
> Thanks,
> Meghna

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.