[
https://issues.apache.org/jira/browse/NUTCH-1807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18092566#comment-18092566
]
Hudson commented on NUTCH-1807:
-------------------------------
SUCCESS: Integrated in Jenkins build Nutch ยป Nutch-trunk #247 (See
[https://ci-builds.apache.org/job/Nutch/job/Nutch-trunk/247/])
NUTCH-1807 Avoid methods relying on system-specific default locale / charset
(snagel:
[https://github.com/apache/nutch/commit/32cee5212ba5310eaa5916bd8886c11d24a1d3d5])
* (edit) default.properties
* (add) src/testresources/forbidden-apis-signatures.txt
* (edit) build.xml
NUTCH-1807 Avoid methods relying on system-specific default locale / charset
(snagel:
[https://github.com/apache/nutch/commit/c180c0d2961ff593de9767a0586f856f66eca0d0])
* (edit) src/java/org/apache/nutch/protocol/RobotRulesParser.java
* (edit) src/java/org/apache/nutch/tools/AbstractCommonCrawlFormat.java
* (edit) src/java/org/apache/nutch/indexer/IndexerMapReduce.java
* (edit) src/java/org/apache/nutch/segment/SegmentReader.java
NUTCH-1807 Avoid methods relying on system-specific default locale / charset
(snagel:
[https://github.com/apache/nutch/commit/c77975cadf6a5d5ea7a96339cfc8bdb1a3f2a368])
* (edit) build.xml
NUTCH-1807 Avoid methods relying on system-specific default locale / charset
(snagel:
[https://github.com/apache/nutch/commit/226fa5a30e4c4a114198203df3d11290c266b86f])
* (edit) src/testresources/forbidden-apis-signatures.txt
NUTCH-1807 Avoid methods relying on system-specific default locale / charset
(snagel:
[https://github.com/apache/nutch/commit/c451051b91b8457897500046c5712be7504998b0])
* (edit) build.xml
NUTCH-1807 Avoid methods relying on system-specific default locale / charset
(snagel:
[https://github.com/apache/nutch/commit/9068999c0e992bf954ab2f70d85fd6def5b1e66a])
* (add) src/test/host-protocol-mapping.txt
* (edit) src/java/org/apache/nutch/protocol/ProtocolFactory.java
* (edit) build.xml
* (edit) src/test/org/apache/nutch/protocol/TestProtocolFactory.java
NUTCH-1807 Avoid methods relying on system-specific default locale / charset
(snagel:
[https://github.com/apache/nutch/commit/04dc40e78799187effca1e67134c53801ffe95b3])
* (edit) build.xml
NUTCH-1807 Avoid methods relying on system-specific default locale / charset
(snagel:
[https://github.com/apache/nutch/commit/f2c7555d09e830bc1b8c27908c99bd94b1f57a49])
* (edit) src/plugin/parse-ext/src/java/org/apache/nutch/parse/ext/ExtParser.java
* (edit)
src/plugin/protocol-okhttp/src/test/org/apache/nutch/protocol/okhttp/TestBadServerResponses.java
* (edit)
src/plugin/subcollection/src/java/org/apache/nutch/indexer/subcollection/SubcollectionIndexingFilter.java
* (edit)
src/plugin/urlnormalizer-regex/src/java/org/apache/nutch/net/urlnormalizer/regex/RegexURLNormalizer.java
* (edit)
src/plugin/urlfilter-suffix/src/java/org/apache/nutch/urlfilter/suffix/SuffixURLFilter.java
* (edit) src/java/org/apache/nutch/scoring/webgraph/LinkRank.java
* (edit)
src/plugin/parse-tika/src/test/org/apache/nutch/parse/tika/TestMetadataExtraction.java
* (edit) src/java/org/apache/nutch/util/AbstractChecker.java
* (edit)
src/plugin/indexer-csv/src/java/org/apache/nutch/indexwriter/csv/CSVIndexWriter.java
* (edit) src/test/org/apache/nutch/plugin/TestPluginSystem.java
* (edit) src/plugin/protocol-foo/src/java/org/apache/nutch/protocol/foo/Foo.java
* (edit) src/test/org/apache/nutch/util/TestGZIPUtils.java
* (edit)
src/plugin/lib-http/src/test/org/apache/nutch/protocol/http/api/TestRobotRulesParser.java
* (edit) src/java/org/apache/nutch/tools/ShowProperties.java
* (edit) src/java/org/apache/nutch/util/EncodingDetector.java
* (edit) src/java/org/apache/nutch/segment/SegmentReader.java
* (edit)
src/plugin/urlnormalizer-protocol/src/java/org/apache/nutch/net/urlnormalizer/protocol/ProtocolURLNormalizer.java
* (edit)
src/plugin/urlfilter-prefix/src/java/org/apache/nutch/urlfilter/prefix/PrefixURLFilter.java
* (edit) src/java/org/apache/nutch/fetcher/Fetcher.java
* (edit) src/java/org/apache/nutch/tools/CommonCrawlFormatWARC.java
* (edit)
src/plugin/parse-html/src/java/org/apache/nutch/parse/html/HTMLMetaProcessor.java
* (edit)
src/plugin/protocol-okhttp/src/java/org/apache/nutch/protocol/okhttp/OkHttpResponse.java
* (edit)
src/plugin/urlnormalizer-slash/src/java/org/apache/nutch/net/urlnormalizer/slash/SlashURLNormalizer.java
* (edit) src/java/org/apache/nutch/crawl/CrawlDbReader.java
* (edit)
src/plugin/parse-html/src/java/org/apache/nutch/parse/html/DOMContentUtils.java
* (edit)
src/plugin/subcollection/src/test/org/apache/nutch/collection/TestSubcollection.java
* (edit) src/test/org/apache/nutch/segment/TestSegmentMerger.java
* (edit)
src/plugin/urlnormalizer-regex/src/test/org/apache/nutch/net/urlnormalizer/regex/TestRegexURLNormalizer.java
* (edit)
src/plugin/protocol-ftp/src/java/org/apache/nutch/protocol/ftp/FtpResponse.java
* (edit)
src/plugin/protocol-okhttp/src/java/org/apache/nutch/protocol/okhttp/IPFilterRules.java
* (edit)
src/plugin/protocol-http/src/java/org/apache/nutch/protocol/http/HttpResponse.java
* (edit) src/test/org/apache/nutch/crawl/CrawlDBTestUtil.java
* (edit)
src/plugin/parse-tika/src/java/org/apache/nutch/parse/tika/DOMContentUtils.java
* (edit)
src/plugin/mimetype-filter/src/test/org/apache/nutch/indexer/filter/MimeTypeIndexingFilterTest.java
* (edit)
src/plugin/mimetype-filter/src/java/org/apache/nutch/indexer/filter/MimeTypeIndexingFilter.java
* (edit)
src/plugin/protocol-file/src/java/org/apache/nutch/protocol/file/FileResponse.java
* (edit) src/test/org/apache/nutch/protocol/TestProtocolFactory.java
* (edit)
src/plugin/indexer-rabbit/src/java/org/apache/nutch/indexwriter/rabbit/RabbitMessage.java
* (edit) src/plugin/protocol-ftp/src/java/org/apache/nutch/protocol/ftp/Ftp.java
* (edit)
src/plugin/protocol-http/src/test/org/apache/nutch/protocol/http/TestBadServerResponses.java
* (edit)
src/plugin/urlnormalizer-ajax/src/java/org/apache/nutch/net/urlnormalizer/ajax/AjaxURLNormalizer.java
* (edit) src/java/org/apache/nutch/util/URLUtil.java
* (edit)
src/plugin/publish-rabbitmq/src/java/org/apache/nutch/publisher/rabbitmq/RabbitMQPublisherImpl.java
* (edit)
src/plugin/urlfilter-domaindenylist/src/java/org/apache/nutch/urlfilter/domaindenylist/DomainDenylistURLFilter.java
* (edit)
src/plugin/microformats-reltag/src/java/org/apache/nutch/microformats/reltag/RelTagParser.java
* (edit)
src/plugin/headings/src/test/org/apache/nutch/parse/headings/TestHeadingsParseFilter.java
* (edit)
src/plugin/index-links/src/java/org/apache/nutch/indexer/links/LinksIndexingFilter.java
* (edit) src/plugin/feed/src/java/org/apache/nutch/parse/feed/FeedParser.java
* (edit)
src/plugin/urlfilter-domain/src/java/org/apache/nutch/urlfilter/domain/DomainURLFilter.java
* (edit) src/java/org/apache/nutch/tools/arc/ArcSegmentCreator.java
* (edit)
src/plugin/indexer-cloudsearch/src/java/org/apache/nutch/indexwriter/cloudsearch/CloudSearchIndexWriter.java
* (edit) src/java/org/apache/nutch/tools/warc/WARCExporter.java
* (edit) src/java/org/apache/nutch/crawl/DeduplicationJob.java
* (edit)
src/plugin/language-identifier/src/test/org/apache/nutch/analysis/lang/TestHTMLLanguageParser.java
* (edit)
src/plugin/lib-regex-filter/src/test/org/apache/nutch/urlfilter/api/RegexURLFilterBaseTest.java
* (edit)
src/plugin/protocol-selenium/src/java/org/apache/nutch/protocol/selenium/HttpResponse.java
* (edit)
src/plugin/parsefilter-naivebayes/src/java/org/apache/nutch/parsefilter/naivebayes/Classify.java
* (edit)
src/plugin/parsefilter-naivebayes/src/java/org/apache/nutch/parsefilter/naivebayes/Train.java
* (edit)
src/plugin/parse-tika/src/test/org/apache/nutch/parse/tika/TestOOParser.java
* (edit)
src/plugin/indexer-rabbit/src/java/org/apache/nutch/indexwriter/rabbit/RabbitIndexWriter.java
* (edit) src/java/org/apache/nutch/segment/ContentAsTextInputFormat.java
* (edit)
src/plugin/protocol-ftp/src/java/org/apache/nutch/protocol/ftp/Client.java
* (edit)
src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer/host/HostURLNormalizer.java
* (edit)
src/plugin/index-anchor/src/java/org/apache/nutch/indexer/anchor/AnchorIndexingFilter.java
* (edit)
src/plugin/parse-tika/src/java/org/apache/nutch/parse/tika/DOMBuilder.java
* (edit) src/java/org/apache/nutch/parse/ParseOutputFormat.java
* (edit)
src/plugin/parse-html/src/test/org/apache/nutch/parse/html/TestRobotsMetaProcessor.java
* (edit) src/java/org/apache/nutch/util/PrefixStringMatcher.java
* (edit)
src/plugin/protocol-ftp/src/java/org/apache/nutch/protocol/ftp/FtpRobotRulesParser.java
* (edit) src/java/org/apache/nutch/tools/arc/ArcRecordReader.java
* (edit)
src/plugin/indexer-dummy/src/java/org/apache/nutch/indexwriter/dummy/DummyIndexWriter.java
* (edit)
src/plugin/protocol-interactiveselenium/src/java/org/apache/nutch/protocol/interactiveselenium/HttpResponse.java
* (edit)
src/plugin/parse-tika/src/test/org/apache/nutch/parse/tika/TestBoilerpipeExtraction.java
* (edit)
src/plugin/lib-http/src/java/org/apache/nutch/protocol/http/api/HttpRobotRulesParser.java
* (edit) src/test/org/apache/nutch/util/TestEncodingDetector.java
* (edit)
src/plugin/parse-tika/src/test/org/apache/nutch/parse/tika/TestLinkExtractionEdgeCases.java
* (edit) src/java/org/apache/nutch/crawl/MD5Signature.java
* (edit) src/java/org/apache/nutch/tools/FileDumper.java
* (edit) src/java/org/apache/nutch/crawl/LinkDb.java
* (edit) src/java/org/apache/nutch/tools/DmozParser.java
* (edit)
src/plugin/urlfilter-fast/src/java/org/apache/nutch/urlfilter/fast/FastURLFilter.java
* (edit) src/java/org/apache/nutch/tools/CommonCrawlFormatJackson.java
* (edit) src/java/org/apache/nutch/hostdb/HostDatum.java
* (edit) src/java/org/apache/nutch/tools/CommonCrawlDataDumper.java
* (edit) src/test/org/apache/nutch/segment/TestSegmentMergerCrawlDatums.java
* (edit) src/test/org/apache/nutch/util/TestNodeWalker.java
* (edit)
src/plugin/parsefilter-regex/src/test/org/apache/nutch/parsefilter/regex/TestRegexParseFilter.java
* (edit)
src/plugin/protocol-httpclient/src/java/org/apache/nutch/protocol/httpclient/HttpBasicAuthentication.java
* (edit)
src/plugin/lib-http/src/java/org/apache/nutch/protocol/http/api/HttpBase.java
* (edit)
src/plugin/parse-html/src/test/org/apache/nutch/parse/html/TestDOMContentUtils.java
* (edit) src/test/org/apache/nutch/fetcher/TestFetcher.java
* (edit)
src/plugin/language-identifier/src/java/org/apache/nutch/analysis/lang/HTMLLanguageParser.java
* (edit)
src/plugin/protocol-httpclient/src/test/org/apache/nutch/protocol/httpclient/TestProtocolHttpClient.java
* (edit) src/java/org/apache/nutch/crawl/TextProfileSignature.java
* (edit) src/java/org/apache/nutch/fetcher/FetchItem.java
* (edit) src/java/org/apache/nutch/util/SitemapProcessor.java
* (edit) src/java/org/apache/nutch/util/NutchJob.java
* (edit) src/java/org/apache/nutch/util/TimingUtil.java
* (edit)
src/plugin/parse-tika/src/test/org/apache/nutch/parse/tika/TestRobotsMetaProcessor.java
* (edit) src/test/org/apache/nutch/util/TestMimeUtil.java
* (edit)
src/plugin/indexer-rabbit/src/java/org/apache/nutch/indexwriter/rabbit/RabbitDocument.java
* (edit) src/java/org/apache/nutch/crawl/AdaptiveFetchSchedule.java
* (edit) src/java/org/apache/nutch/crawl/Generator.java
* (edit) src/java/org/apache/nutch/util/DumpFileUtil.java
* (edit) src/java/org/apache/nutch/fetcher/FetcherThread.java
* (edit)
src/plugin/protocol-okhttp/src/test/org/apache/nutch/protocol/okhttp/TestIPAddressFiltering.java
* (edit)
src/plugin/parsefilter-regex/src/java/org/apache/nutch/parsefilter/regex/RegexParseFilter.java
* (edit)
src/plugin/parse-tika/src/test/org/apache/nutch/parse/tika/TestParserFailureHandling.java
* (edit) src/java/org/apache/nutch/tools/ResolveUrls.java
* (edit)
src/plugin/protocol-file/src/java/org/apache/nutch/protocol/file/File.java
* (edit)
src/plugin/lib-regex-filter/src/java/org/apache/nutch/urlfilter/api/RegexURLFilterBase.java
* (edit)
src/plugin/protocol-htmlunit/src/java/org/apache/nutch/protocol/htmlunit/HttpResponse.java
* (edit)
src/plugin/parse-js/src/java/org/apache/nutch/parse/js/JSParseFilter.java
NUTCH-1807 Avoid methods relying on system-specific default locale / charset
(snagel:
[https://github.com/apache/nutch/commit/e807f54d68619c49fb85132e8ef75a0facd2239b])
* (edit) src/test/host-protocol-mapping.txt
* (edit) src/testresources/forbidden-apis-signatures.txt
* (edit) .yetus/blanks-tabs.txt
NUTCH-1807 Avoid methods relying on system-specific default locale / charset
(snagel:
[https://github.com/apache/nutch/commit/4319278a488807597bec969eacbe6942538ba89c])
* (edit) build.xml
* (edit) .github/workflows/master-build.yml
> avoid methods relying on system-specific default locale / charset
> -----------------------------------------------------------------
>
> Key: NUTCH-1807
> URL: https://issues.apache.org/jira/browse/NUTCH-1807
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.8, 2.2.1
> Reporter: Sebastian Nagel
> Assignee: Sebastian Nagel
> Priority: Minor
> Fix For: 1.23
>
>
> Many methods in Java (and libraries) used to convert Strings, Numbers, Dates
> rely on the system-specific default locale / character set. This may cause
> strange behaviour and errors impossible to reproduce on other systems, see
> [~thetaphi]'s [blog
> post|http://blog.thetaphi.de/2012/07/default-locales-default-charsets-and.html],
> and discussions in NUTCH-1693 and NUTCH-1554.
> A search with the [forbidden-apis
> client|https://code.google.com/p/forbidden-apis/wiki/CliUsage] shows 120
> calls of such methods in trunk (without test classes):
> {code}
> # compile Nutch before check: all tested class files
> # are then located in build/ (including plugins)
> % CLASSPATH=`find build/ -name '*.jar' | tr '\n' ':'`
> % java -jar forbiddenapis-1.5.1.jar -d build/ -c $CLASSPATH \
> -b jdk-unsafe-1.8 -b commons-io-unsafe-2.4
> {code}
> It is also possible to [integrate the check into the ant
> build|https://code.google.com/p/forbidden-apis/wiki/AntUsage] (to avoid that
> "forbidden" calls slip into the code again).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
