rzo1 commented on issue #1597: URL: https://github.com/apache/stormcrawler/issues/1597#issuecomment-3685969443
I pushed an initial migration on a separate branch (1597). We’re now seeing some failures in BasicURLNormalizerTest because URI enforces RFC 3986 compliance strictly, whereas URL is more tolerant of invalid input. We need to decide how we want to handle this going forward. In practice, a crawler may encounter malformed URIs but still attempt to normalize them into a valid URL. The open question is how much recovery or sanitization we want to perform versus simply failing when an invalid URI is encountered. WDYT @sebastian-nagel @jnioche ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
