[jira] [Updated] (NUTCH-1314) Impose a limit on the length of outlink target urls
[ https://issues.apache.org/jira/browse/NUTCH-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lewis John McGibbney updated NUTCH-1314: Fix Version/s: (was: 2.5) 2.4 > Impose a limit on the length of outlink target urls > --- > > Key: NUTCH-1314 > URL: https://issues.apache.org/jira/browse/NUTCH-1314 > Project: Nutch > Issue Type: Improvement >Reporter: Ferdy Galema >Assignee: Lewis John McGibbney > Fix For: 2.4 > > Attachments: NUTCH-1314-trunk.patch, NUTCH-1314-v2.patch, > NUTCH-1314-v3.patch, NUTCH-1314-v4.patch, NUTCH-1314.patch > > > In the past we have encountered situations where crawling specific broken > sites resulted in ridiciously long urls that caused the stalling of tasks. > The regex plugins (normalizing/filtering) processed single urls for hours, if > not indefinitely hanging. > My suggestion is to limit the outlink url target length as soon possible. It > is a configurable limit, the default is 3000. This should be reasonably long > enough for most uses. But sufficienly strict enough to make sure regex > plugins do not choke on urls that are too long. Please see attached patch for > the Nutchgora implementation. > I'd like to hear what you think about this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NUTCH-1314) Impose a limit on the length of outlink target urls
[ https://issues.apache.org/jira/browse/NUTCH-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Canan Girgin updated NUTCH-1314: Attachment: NUTCH-1314-v4.patch > Impose a limit on the length of outlink target urls > --- > > Key: NUTCH-1314 > URL: https://issues.apache.org/jira/browse/NUTCH-1314 > Project: Nutch > Issue Type: Improvement >Reporter: Ferdy Galema > Fix For: 2.4, 1.12 > > Attachments: NUTCH-1314-trunk.patch, NUTCH-1314-v2.patch, > NUTCH-1314-v3.patch, NUTCH-1314-v4.patch, NUTCH-1314.patch > > > In the past we have encountered situations where crawling specific broken > sites resulted in ridiciously long urls that caused the stalling of tasks. > The regex plugins (normalizing/filtering) processed single urls for hours, if > not indefinitely hanging. > My suggestion is to limit the outlink url target length as soon possible. It > is a configurable limit, the default is 3000. This should be reasonably long > enough for most uses. But sufficienly strict enough to make sure regex > plugins do not choke on urls that are too long. Please see attached patch for > the Nutchgora implementation. > I'd like to hear what you think about this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NUTCH-1314) Impose a limit on the length of outlink target urls
[ https://issues.apache.org/jira/browse/NUTCH-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sebastian Nagel updated NUTCH-1314: --- Fix Version/s: 1.12 > Impose a limit on the length of outlink target urls > --- > > Key: NUTCH-1314 > URL: https://issues.apache.org/jira/browse/NUTCH-1314 > Project: Nutch > Issue Type: Improvement >Reporter: Ferdy Galema > Fix For: 2.4, 1.12 > > Attachments: NUTCH-1314-trunk.patch, NUTCH-1314-v2.patch, > NUTCH-1314-v3.patch, NUTCH-1314.patch > > > In the past we have encountered situations where crawling specific broken > sites resulted in ridiciously long urls that caused the stalling of tasks. > The regex plugins (normalizing/filtering) processed single urls for hours, if > not indefinitely hanging. > My suggestion is to limit the outlink url target length as soon possible. It > is a configurable limit, the default is 3000. This should be reasonably long > enough for most uses. But sufficienly strict enough to make sure regex > plugins do not choke on urls that are too long. Please see attached patch for > the Nutchgora implementation. > I'd like to hear what you think about this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NUTCH-1314) Impose a limit on the length of outlink target urls
[ https://issues.apache.org/jira/browse/NUTCH-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Canan Girgin updated NUTCH-1314: Attachment: NUTCH-1314-v3.patch > Impose a limit on the length of outlink target urls > --- > > Key: NUTCH-1314 > URL: https://issues.apache.org/jira/browse/NUTCH-1314 > Project: Nutch > Issue Type: Improvement >Reporter: Ferdy Galema > Fix For: 2.3 > > Attachments: NUTCH-1314.patch, NUTCH-1314-trunk.patch, > NUTCH-1314-v2.patch, NUTCH-1314-v3.patch > > > In the past we have encountered situations where crawling specific broken > sites resulted in ridiciously long urls that caused the stalling of tasks. > The regex plugins (normalizing/filtering) processed single urls for hours, if > not indefinitely hanging. > My suggestion is to limit the outlink url target length as soon possible. It > is a configurable limit, the default is 3000. This should be reasonably long > enough for most uses. But sufficienly strict enough to make sure regex > plugins do not choke on urls that are too long. Please see attached patch for > the Nutchgora implementation. > I'd like to hear what you think about this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1314) Impose a limit on the length of outlink target urls
[ https://issues.apache.org/jira/browse/NUTCH-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lewis John McGibbney updated NUTCH-1314: Attachment: NUTCH-1314-trunk.patch NUTCH-1314-v2.patch Fresh patches for 2.x and trunk branches respectively. [~markus17] I tried to accommodate your suggestions but please let me know if there is something we can work on. If someone could test it would be great. Thanks Lewis > Impose a limit on the length of outlink target urls > --- > > Key: NUTCH-1314 > URL: https://issues.apache.org/jira/browse/NUTCH-1314 > Project: Nutch > Issue Type: Improvement >Reporter: Ferdy Galema > Fix For: 1.7, 2.2 > > Attachments: NUTCH-1314.patch, NUTCH-1314-trunk.patch, > NUTCH-1314-v2.patch > > > In the past we have encountered situations where crawling specific broken > sites resulted in ridiciously long urls that caused the stalling of tasks. > The regex plugins (normalizing/filtering) processed single urls for hours, if > not indefinitely hanging. > My suggestion is to limit the outlink url target length as soon possible. It > is a configurable limit, the default is 3000. This should be reasonably long > enough for most uses. But sufficienly strict enough to make sure regex > plugins do not choke on urls that are too long. Please see attached patch for > the Nutchgora implementation. > I'd like to hear what you think about this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1314) Impose a limit on the length of outlink target urls
[ https://issues.apache.org/jira/browse/NUTCH-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Nioche updated NUTCH-1314: - Fix Version/s: 2.2 1.7 > Impose a limit on the length of outlink target urls > --- > > Key: NUTCH-1314 > URL: https://issues.apache.org/jira/browse/NUTCH-1314 > Project: Nutch > Issue Type: Improvement >Reporter: Ferdy Galema > Fix For: 1.7, 2.2 > > Attachments: NUTCH-1314.patch > > > In the past we have encountered situations where crawling specific broken > sites resulted in ridiciously long urls that caused the stalling of tasks. > The regex plugins (normalizing/filtering) processed single urls for hours, if > not indefinitely hanging. > My suggestion is to limit the outlink url target length as soon possible. It > is a configurable limit, the default is 3000. This should be reasonably long > enough for most uses. But sufficienly strict enough to make sure regex > plugins do not choke on urls that are too long. Please see attached patch for > the Nutchgora implementation. > I'd like to hear what you think about this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1314) Impose a limit on the length of outlink target urls
[ https://issues.apache.org/jira/browse/NUTCH-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdy Galema updated NUTCH-1314: Attachment: NUTCH-1314.patch > Impose a limit on the length of outlink target urls > --- > > Key: NUTCH-1314 > URL: https://issues.apache.org/jira/browse/NUTCH-1314 > Project: Nutch > Issue Type: Improvement >Reporter: Ferdy Galema > Attachments: NUTCH-1314.patch > > > In the past we have encountered situations where crawling specific broken > sites resulted in ridiciously long urls that caused the stalling of tasks. > The regex plugins (normalizing/filtering) processed single urls for hours, if > not indefinitely hanging. > My suggestion is to limit the outlink url target length as soon possible. It > is a configurable limit, the default is 3000. This should be reasonably long > enough for most uses. But sufficienly strict enough to make sure regex > plugins do not choke on urls that are too long. Please see attached patch for > the Nutchgora implementation. > I'd like to hear what you think about this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
