sebastian-nagel commented on code in PR #796:
URL: https://github.com/apache/nutch/pull/796#discussion_r1384536930
##########
src/plugin/urlfilter-fast/src/java/org/apache/nutch/urlfilter/fast/FastURLFilter.java:
##########
@@ -97,9 +97,17 @@ public class FastURLFilter implements URLFilter {
private Configuration conf;
public static final String URLFILTER_FAST_FILE = "urlfilter.fast.file";
+ public static final String URLFILTER_FAST_PATH_MAX_LENGTH =
"urlfilter.fast.url.path.max.length";
+ public static final String URLFILTER_FAST_QUERY_MAX_LENGTH =
"urlfilter.fast.url.query.max.length";
+
Review Comment:
What about adding a third limit for path and query combined
([URL.getFile()](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/net/URL.html#getFile())?
- if somebody defined two generous but reasonable limits (for example, 2048)
for both path and query, the resulting URL may still get quite long and cause
troubles
- also the HTTP GET request includes both path and query
- for many use cases it should sufficient to just set this limit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@nutch.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
