TamimEhsan commented on code in PR #1631:
URL: https://github.com/apache/stormcrawler/pull/1631#discussion_r2280256931
##########
core/src/main/java/org/apache/stormcrawler/filtering/basic/BasicURLFilter.java:
##########
@@ -48,7 +48,11 @@ public String filter(
public final String filterPathRepeat(String urlToFilter) {
// check whether a path element is repeated N times
String[] paths = urlToFilter.split("/");
- if (paths.length <= 4) return urlToFilter;
+
+ final int minimumPathLength = 5;
Review Comment:
@sigee I looked up for other instances like this and followed those. for
example
https://github.com/apache/stormcrawler/blob/9d4b0302a7859f9f24caa1fe3cdabb498f968112/core/src/main/java/org/apache/stormcrawler/bolt/FeedParserBolt.java#L93-L96
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
