Added: dev/nutch/1.21/CHANGES.md
==============================================================================
--- dev/nutch/1.21/CHANGES.md (added)
+++ dev/nutch/1.21/CHANGES.md Tue Jul 15 18:47:22 2025
@@ -0,0 +1,3404 @@
+# Nutch Change Log
+
+## Nutch 1.21 Release 15/07/2025 (dd/mm/yyyy)
+Release Report: https://s.apache.org/bs58y
+
+### Breaking Changes
+
+- LinkDB records now can hold metadata, see 
[NUTCH-3101](https://issues.apache.org/jira/browse/NUTCH-3101).
+  This requires that existing LinkDBs are created anew starting from the 
segments. Nutch 1.21 cannot process LinkDBs written with prior Nutch versions.
+
+
+### New Feature
+
+- [NUTCH-2856](https://issues.apache.org/jira/browse/NUTCH-2856) - Implement a 
protocol-smb plugin based on hierynomus/smbj
+- [NUTCH-3063](https://issues.apache.org/jira/browse/NUTCH-3063) - Support for 
"addBinaryContent" from REST API
+
+### Sub-task
+
+- [NUTCH-2812](https://issues.apache.org/jira/browse/NUTCH-2812) - Methods 
returning array may expose internal representation
+
+### Bug
+
+- [NUTCH-3039](https://issues.apache.org/jira/browse/NUTCH-3039) - Failure to 
handle ftp:// URLs
+- [NUTCH-3044](https://issues.apache.org/jira/browse/NUTCH-3044) - Generator: 
NPE when extracting the host part of a URL fails
+- [NUTCH-3055](https://issues.apache.org/jira/browse/NUTCH-3055) - README: fix 
Github "hub" commands
+- [NUTCH-3057](https://issues.apache.org/jira/browse/NUTCH-3057) - Arbitrary 
indexer "leaks" previous value into a field processed after an exception
+- [NUTCH-3060](https://issues.apache.org/jira/browse/NUTCH-3060) - Javadoc 
link broken on website
+- [NUTCH-3066](https://issues.apache.org/jira/browse/NUTCH-3066) - Protocol 
plugin unit tests fail randomly
+- [NUTCH-3067](https://issues.apache.org/jira/browse/NUTCH-3067) - Improve 
performance of FetchItemQueues if error state is preserved
+- [NUTCH-3072](https://issues.apache.org/jira/browse/NUTCH-3072) - Fetcher to 
stop QueueFeeder if aborting with "hung threads"
+- [NUTCH-3075](https://issues.apache.org/jira/browse/NUTCH-3075) - tld plugin 
makes injector crash
+- [NUTCH-3078](https://issues.apache.org/jira/browse/NUTCH-3078) - Database is 
not unlocked when injector fails
+- [NUTCH-3079](https://issues.apache.org/jira/browse/NUTCH-3079) - Dumping a 
segment fails unless it has been fetched and parsed
+- [NUTCH-3087](https://issues.apache.org/jira/browse/NUTCH-3087) - Nutch 
crawling inconsistent on URLs with userinfo
+- [NUTCH-3092](https://issues.apache.org/jira/browse/NUTCH-3092) - Replace all 
imports of commons-lang by commons-lang3
+- [NUTCH-3093](https://issues.apache.org/jira/browse/NUTCH-3093) - Ant target 
test-plugins to depend on compile-core-test
+- [NUTCH-3094](https://issues.apache.org/jira/browse/NUTCH-3094) - Github 
tests to run if build configuration changes
+- [NUTCH-3096](https://issues.apache.org/jira/browse/NUTCH-3096) - HostDB 
ResolverThread can create too many job counters
+- [NUTCH-3097](https://issues.apache.org/jira/browse/NUTCH-3097) - Plugin 
indexer-elastic throws ClassNotFoundException due to invalid dependencies
+- [NUTCH-3103](https://issues.apache.org/jira/browse/NUTCH-3103) - Improper 
fetch interval given as example
+- [NUTCH-3106](https://issues.apache.org/jira/browse/NUTCH-3106) - Issue with 
SSLHandshakeException in v1.20 using protocol-http plugin and proxy
+- [NUTCH-3108](https://issues.apache.org/jira/browse/NUTCH-3108) - Fix SLF4J 
Class Loader Conflict in language-identifier
+- [NUTCH-3114](https://issues.apache.org/jira/browse/NUTCH-3114) - Avoid stale 
fetching when only URLs from queues blocked by the exponential backoff remain
+
+### Improvement
+
+- [NUTCH-1806](https://issues.apache.org/jira/browse/NUTCH-1806) - Delegate 
processing of URL domains to crawler commons
+- [NUTCH-2157](https://issues.apache.org/jira/browse/NUTCH-2157) - Parent 
Issue for Addressing Miredot REST API Warnings
+- [NUTCH-2771](https://issues.apache.org/jira/browse/NUTCH-2771) - Tests in 
nightly builds: speed up long runners
+- [NUTCH-2976](https://issues.apache.org/jira/browse/NUTCH-2976) - 
SitemapProcessor: verify sitemap values added from sitemap to CrawlDB 
(priority, modification time and change frequency)
+- [NUTCH-3043](https://issues.apache.org/jira/browse/NUTCH-3043) - Generator: 
count URLs rejected by URL filters
+- [NUTCH-3058](https://issues.apache.org/jira/browse/NUTCH-3058) - Fetcher: 
counter for hung threads
+- [NUTCH-3061](https://issues.apache.org/jira/browse/NUTCH-3061) - URL filters 
to log name of the rule file rules are read from
+- [NUTCH-3062](https://issues.apache.org/jira/browse/NUTCH-3062) - 
protocol-okhttp: optionally record HTTP and SSL/TLS versions
+- [NUTCH-3065](https://issues.apache.org/jira/browse/NUTCH-3065) - Format 
changelog as Markdown
+- [NUTCH-3073](https://issues.apache.org/jira/browse/NUTCH-3073) - Address 
Java compiler warnings
+- [NUTCH-3083](https://issues.apache.org/jira/browse/NUTCH-3083) - Add 
RobotRulesParser to bin/nutch
+- [NUTCH-3086](https://issues.apache.org/jira/browse/NUTCH-3086) - Consolidate 
plugin extension names and IDs
+- [NUTCH-3095](https://issues.apache.org/jira/browse/NUTCH-3095) - Update 
.gitignore to ignore Hadoop native libraries
+- [NUTCH-3100](https://issues.apache.org/jira/browse/NUTCH-3100) - HostDB to 
support minimum records per host
+- [NUTCH-3101](https://issues.apache.org/jira/browse/NUTCH-3101) - LinkDb's 
Inlink class to support metadata
+- [NUTCH-3112](https://issues.apache.org/jira/browse/NUTCH-3112) - Utilize 
parameterized logging
+- [NUTCH-3113](https://issues.apache.org/jira/browse/NUTCH-3113) - Group 
commands in bin/nutch command-line help
+- [NUTCH-3115](https://issues.apache.org/jira/browse/NUTCH-3115) - Allow POJO 
in Arbitrary Indexer to access indexing objects in filter constrctor
+- [NUTCH-3116](https://issues.apache.org/jira/browse/NUTCH-3116) - Minor 
dependency upgrades and update of license list and notice file
+
+### Task
+
+- [NUTCH-1942](https://issues.apache.org/jira/browse/NUTCH-1942) - Remove 
TopLevelDomain
+- [NUTCH-3041](https://issues.apache.org/jira/browse/NUTCH-3041) - Address 
confusing logging in o.a.n.net.URLExemptionFilters
+- [NUTCH-3054](https://issues.apache.org/jira/browse/NUTCH-3054) - Address 
deprecation of Node16 for all GitHub Actions
+- [NUTCH-3084](https://issues.apache.org/jira/browse/NUTCH-3084) - Improve CI 
by filtering and separating plugin and core test execution
+
+
+## Nutch 1.20 Release 09/04/2024 (dd/mm/yyyy)
+Release Report: https://s.apache.org/ovjf3
+
+### Sub-task
+
+- [NUTCH-2596](https://issues.apache.org/jira/browse/NUTCH-2596) - Upgrade 
from org.mortbay.jetty to org.eclipse.jetty
+- [NUTCH-2852](https://issues.apache.org/jira/browse/NUTCH-2852) - Method 
invokes System.exit(...) 9 bugs
+- [NUTCH-2972](https://issues.apache.org/jira/browse/NUTCH-2972) - Javadoc 
build fails using JDK 17
+- [NUTCH-3007](https://issues.apache.org/jira/browse/NUTCH-3007) - Fix 
impossible casts
+
+### Bug
+
+- [NUTCH-2634](https://issues.apache.org/jira/browse/NUTCH-2634) - Some links 
marked as "nofollow" are followed anyway.
+- [NUTCH-2820](https://issues.apache.org/jira/browse/NUTCH-2820) - Review 
sample files used in any23 unit tests
+- [NUTCH-2924](https://issues.apache.org/jira/browse/NUTCH-2924) - Generate 
maxCount expr evaluated only once
+- [NUTCH-2937](https://issues.apache.org/jira/browse/NUTCH-2937) - parse-tika: 
review dependency exclusions and avoid dependency conflicts in distributed mode
+- [NUTCH-2973](https://issues.apache.org/jira/browse/NUTCH-2973) - Single 
domain names (eg https://localnet) can't be crawled - filtering fails
+- [NUTCH-2974](https://issues.apache.org/jira/browse/NUTCH-2974) - Ant build 
fails with "Unparseable date" on certain platforms
+- [NUTCH-2979](https://issues.apache.org/jira/browse/NUTCH-2979) - Upgrade 
Commons Text to 1.10.0
+- [NUTCH-2982](https://issues.apache.org/jira/browse/NUTCH-2982) - Generator: 
parameter for URL normalization not passed forward
+- [NUTCH-2985](https://issues.apache.org/jira/browse/NUTCH-2985) - Disable 
plugin urlfilter-validator by default
+- [NUTCH-2992](https://issues.apache.org/jira/browse/NUTCH-2992) - Fetcher: 
always block fetch queues when exceptions threshold is reached
+- [NUTCH-3000](https://issues.apache.org/jira/browse/NUTCH-3000) - 
protocol-selenium returns only the body, strips off the <head/> element
+- [NUTCH-3001](https://issues.apache.org/jira/browse/NUTCH-3001) - 
protocol-selenium requires Content-Type header 
+- [NUTCH-3002](https://issues.apache.org/jira/browse/NUTCH-3002) - 
Protocol-okhttp HttpResponse: HTTP header metadata lookup should be 
case-insensitive
+- [NUTCH-3008](https://issues.apache.org/jira/browse/NUTCH-3008) - 
indexer-elastic: downgrade to ES 7.10.2 to address licensing issues
+- [NUTCH-3012](https://issues.apache.org/jira/browse/NUTCH-3012) - 
SegmentReader when dumping with option -recode: NPE on unparsed documents
+- [NUTCH-3027](https://issues.apache.org/jira/browse/NUTCH-3027) - Trivial 
resource leak patch in DomainSuffixes.java
+- [NUTCH-3035](https://issues.apache.org/jira/browse/NUTCH-3035) - Update 
license and notice file for release of 1.20 
+
+### New Feature
+
+- [NUTCH-2832](https://issues.apache.org/jira/browse/NUTCH-2832) - Create 
tutorial on sending Nutch logs to Elasticsearch
+- [NUTCH-2888](https://issues.apache.org/jira/browse/NUTCH-2888) - Selenium 
Protocol: Support for Selenium 4
+- [NUTCH-2920](https://issues.apache.org/jira/browse/NUTCH-2920) - Implement a 
indexer-opensearch plugin
+- [NUTCH-2991](https://issues.apache.org/jira/browse/NUTCH-2991) - Support 
HTTP/S Header Authorization for Solr connections
+- [NUTCH-3029](https://issues.apache.org/jira/browse/NUTCH-3029) - Host 
specific max. and min. intervals in adaptive scheduler
+    
+### Improvement
+
+- [NUTCH-2853](https://issues.apache.org/jira/browse/NUTCH-2853) - bin/nutch: 
remove deprecated commands solrindex, solrdedup, solrclean
+- [NUTCH-2883](https://issues.apache.org/jira/browse/NUTCH-2883) - Provide 
means to run server as a persistent service in Docker container
+- [NUTCH-2897](https://issues.apache.org/jira/browse/NUTCH-2897) - Do not 
supress deprecated API warnings
+- [NUTCH-2961](https://issues.apache.org/jira/browse/NUTCH-2961) - Upgrade 
dependencies of parsefilter-naivebayes
+- [NUTCH-2980](https://issues.apache.org/jira/browse/NUTCH-2980) - Upgrade 
Selenium Java to 4.7.2
+- [NUTCH-2983](https://issues.apache.org/jira/browse/NUTCH-2983) - 
nutch-default.xml improvements
+- [NUTCH-2990](https://issues.apache.org/jira/browse/NUTCH-2990) - 
HttpRobotRulesParser to follow 5 redirects as specified by RFC 9309
+- [NUTCH-2993](https://issues.apache.org/jira/browse/NUTCH-2993) - 
ScoringDepth plugin to skip depth check based on URL Pattern
+- [NUTCH-2995](https://issues.apache.org/jira/browse/NUTCH-2995) - Upgrade to 
crawler-commons 1.4
+- [NUTCH-2996](https://issues.apache.org/jira/browse/NUTCH-2996) - Use new 
SimpleRobotRulesParser API entry point (crawler-commons 1.4)
+- [NUTCH-2997](https://issues.apache.org/jira/browse/NUTCH-2997) - Add 
Override annotations where applicable
+- [NUTCH-3004](https://issues.apache.org/jira/browse/NUTCH-3004) - Avoid NPE 
in HttpResponse
+- [NUTCH-3005](https://issues.apache.org/jira/browse/NUTCH-3005) - Upgrade 
selenium as needed
+- [NUTCH-3009](https://issues.apache.org/jira/browse/NUTCH-3009) - Upgrade to 
Hadoop 3.3.6
+- [NUTCH-3010](https://issues.apache.org/jira/browse/NUTCH-3010) - Injector: 
count unique number of injected URLs
+- [NUTCH-3011](https://issues.apache.org/jira/browse/NUTCH-3011) - 
HttpRobotRulesParser: handle HTTP 429 Too Many Requests same as server errors 
(HTTP 5xx)
+- [NUTCH-3013](https://issues.apache.org/jira/browse/NUTCH-3013) - Employ 
commons-lang3's StopWatch to simplify timing logic
+- [NUTCH-3014](https://issues.apache.org/jira/browse/NUTCH-3014) - Standardize 
Job names
+- [NUTCH-3015](https://issues.apache.org/jira/browse/NUTCH-3015) - Add more CI 
steps to GitHub master-build.yml
+- [NUTCH-3017](https://issues.apache.org/jira/browse/NUTCH-3017) - Allow 
fast-urlfilter to load from HDFS/S3 and support gzipped input
+- [NUTCH-3025](https://issues.apache.org/jira/browse/NUTCH-3025) - 
urlfilter-fast to filter based on the length of the URL
+- [NUTCH-3031](https://issues.apache.org/jira/browse/NUTCH-3031) - 
ProtocolFactory host mapper to support domains
+- [NUTCH-3032](https://issues.apache.org/jira/browse/NUTCH-3032) - Indexing 
plugin as an adapter for end user's own POJO instances
+- [NUTCH-3036](https://issues.apache.org/jira/browse/NUTCH-3036) - Upgrade 
org.seleniumhq.selenium:selenium-java dependency in lib-selenium
+
+### Task
+
+- [NUTCH-2959](https://issues.apache.org/jira/browse/NUTCH-2959) - Upgrade to 
Apache Tika 2.9.0
+- [NUTCH-2977](https://issues.apache.org/jira/browse/NUTCH-2977) - Support for 
showing dependency tree
+- [NUTCH-2978](https://issues.apache.org/jira/browse/NUTCH-2978) - Move to 
slf4j2 and remove log4j1 and reload4j
+- [NUTCH-2984](https://issues.apache.org/jira/browse/NUTCH-2984) - Drop test 
proxy server and benchmark tool
+- [NUTCH-2989](https://issues.apache.org/jira/browse/NUTCH-2989) - Can't have 
username/pw AND https on elastic-indexer?!
+- [NUTCH-2998](https://issues.apache.org/jira/browse/NUTCH-2998) - Remove the 
Any23 plugin
+- [NUTCH-2999](https://issues.apache.org/jira/browse/NUTCH-2999) - Update 
Lucene version to latest 8.x
+- [NUTCH-3016](https://issues.apache.org/jira/browse/NUTCH-3016) - Upgrade 
Apache Ivy to 2.5.2
+- [NUTCH-3019](https://issues.apache.org/jira/browse/NUTCH-3019) - Upgrade to 
Apache Tika 2.9.1
+- [NUTCH-3020](https://issues.apache.org/jira/browse/NUTCH-3020) - 
ParseSegment should check for protocol's flags for truncation
+- [NUTCH-3024](https://issues.apache.org/jira/browse/NUTCH-3024) - Remove 
flaky 'dependency check' target
+- [NUTCH-3033](https://issues.apache.org/jira/browse/NUTCH-3033) - Upgrade Ivy 
to v2.5.2
+- [NUTCH-3037](https://issues.apache.org/jira/browse/NUTCH-3037) - Upgrade 
org.apache.kafka:kafka_2.12: to v3.7.0
+- [NUTCH-3038](https://issues.apache.org/jira/browse/NUTCH-3038) - Address 
issues discovered during 1.20 release management dryrun
+                                                                               
                                                                         
+
+## Nutch 1.19 Release 22/08/2022 (dd/mm/yyyy)
+Release Report: https://s.apache.org/lf6li
+
+### Breaking Changes
+
+- Nutch is built on JDK 11 (NUTCH-2857)
+- the Nutch WebApp was moved to a separate repository (NUTCH-2886)
+  - see https://github.com/apache/nutch-webapp
+  - or  https://gitbox.apache.org/repos/asf?p=nutch-webapp.git
+- the plugin parse-swf for parsing Shockwave/Adobe Flash content was removed 
(NUTCH-2861)
+
+### Sub-task
+
+- [NUTCH-2819](https://issues.apache.org/jira/browse/NUTCH-2819) - Move 
spotbugs "installation" directory to avoid that spotbugs is shipped in Nutch 
runtime
+- [NUTCH-2846](https://issues.apache.org/jira/browse/NUTCH-2846) - Fix various 
bugs spotted by NUTCH-2815
+- [NUTCH-2850](https://issues.apache.org/jira/browse/NUTCH-2850) - Method 
ignores exceptional return value
+- [NUTCH-2851](https://issues.apache.org/jira/browse/NUTCH-2851) - Random 
object created and used only once
+- [NUTCH-2855](https://issues.apache.org/jira/browse/NUTCH-2855) - Update 
org.elasticsearch.client
+
+### Bug
+
+- [NUTCH-2290](https://issues.apache.org/jira/browse/NUTCH-2290) - Update 
licenses of bundled libraries
+- [NUTCH-2512](https://issues.apache.org/jira/browse/NUTCH-2512) - Nutch does 
not build under JDK9
+- [NUTCH-2821](https://issues.apache.org/jira/browse/NUTCH-2821) - Deduplicate 
licenses in LICENSE.txt file
+- [NUTCH-2822](https://issues.apache.org/jira/browse/NUTCH-2822) - Split the 
LICENSE.txt file into two files for source resp. binary releases
+- [NUTCH-2831](https://issues.apache.org/jira/browse/NUTCH-2831) - Elastic 
indexer does not support SSL
+- [NUTCH-2843](https://issues.apache.org/jira/browse/NUTCH-2843) - Duplicate 
declaration of dependencies in ivy.xml
+- [NUTCH-2858](https://issues.apache.org/jira/browse/NUTCH-2858) - 
urlnormalizer-protocol: URL port is lost during normalization
+- [NUTCH-2862](https://issues.apache.org/jira/browse/NUTCH-2862) - Do not 
include Ivy jar in source release package
+- [NUTCH-2863](https://issues.apache.org/jira/browse/NUTCH-2863) - Injector to 
parse command-line flags case-insensitive
+- [NUTCH-2866](https://issues.apache.org/jira/browse/NUTCH-2866) - 
MetaData.toString() should return "key=value ..."
+- [NUTCH-2868](https://issues.apache.org/jira/browse/NUTCH-2868) - 
urlnormalizer-protocol fails with StringIndexOutOfBoundsException when reading 
invalid line in configuration file
+- [NUTCH-2881](https://issues.apache.org/jira/browse/NUTCH-2881) - bug in 
'nutch' symlink in docker container
+- [NUTCH-2889](https://issues.apache.org/jira/browse/NUTCH-2889) - nutch 
indexer-elasticsearch plugin, doesn't work with https protocol
+- [NUTCH-2890](https://issues.apache.org/jira/browse/NUTCH-2890) - 
Protocol-okhttp: upgrade okhttp to 4.9.1 to address infinite connection retries
+- [NUTCH-2894](https://issues.apache.org/jira/browse/NUTCH-2894) - Java plugin 
compilation classpath: priorize plugin dependencies
+- [NUTCH-2899](https://issues.apache.org/jira/browse/NUTCH-2899) - Remove 
needless warning about missing o/a/rat/anttasks/antlib.xml
+- [NUTCH-2902](https://issues.apache.org/jira/browse/NUTCH-2902) - Jexl 
parsing error on statements
+- [NUTCH-2905](https://issues.apache.org/jira/browse/NUTCH-2905) - Mask 
sensitive strings in log output of index writers
+- [NUTCH-2910](https://issues.apache.org/jira/browse/NUTCH-2910) - 
FetchItemQueues overloaded constructor also interprets fetcher timeout as -1 
e.g. no-timeout.
+- [NUTCH-2915](https://issues.apache.org/jira/browse/NUTCH-2915) - Upgrade to 
log4j 2.15.0
+- [NUTCH-2916](https://issues.apache.org/jira/browse/NUTCH-2916) - Fix log 
file rotation / rename default log file
+- [NUTCH-2917](https://issues.apache.org/jira/browse/NUTCH-2917) - Remove 
transitive dependency to log4j 1.x
+- [NUTCH-2922](https://issues.apache.org/jira/browse/NUTCH-2922) - Upgrade to 
log4j 2.17.0
+- [NUTCH-2935](https://issues.apache.org/jira/browse/NUTCH-2935) - 
DeduplicationJob: failure on URLs with invalid percent encoding
+- [NUTCH-2936](https://issues.apache.org/jira/browse/NUTCH-2936) - Early 
registration of URL stream handlers provided by plugins may fail Hadoop jobs 
running in distributed mode if protocol-okhttp is used
+- [NUTCH-2945](https://issues.apache.org/jira/browse/NUTCH-2945) - Solr Index 
Writer pluging schema.xml missing a copyToField
+- [NUTCH-2947](https://issues.apache.org/jira/browse/NUTCH-2947) - Fetcher: 
keep state of empty fetch queues unless queue feeder is finished
+- [NUTCH-2949](https://issues.apache.org/jira/browse/NUTCH-2949) - Tasks of a 
multi-threaded map runner may fail because of slow creation of URL stream 
handlers
+- [NUTCH-2951](https://issues.apache.org/jira/browse/NUTCH-2951) - Crawl datum 
with metadata WRITABLE_GENERATE_TIME_KEY awaits fetching forever
+- [NUTCH-2955](https://issues.apache.org/jira/browse/NUTCH-2955) - 
indexer-solr: replace deprecated/removed field type solr.LatLonType
+- [NUTCH-2969](https://issues.apache.org/jira/browse/NUTCH-2969) - Javadoc: 
Javascript search is not working when built on JDK 11
+
+### New Feature
+
+- [NUTCH-2901](https://issues.apache.org/jira/browse/NUTCH-2901) - migrate to 
maven or gradle
+
+### Improvement
+
+- [NUTCH-1403](https://issues.apache.org/jira/browse/NUTCH-1403) - Add default 
ScoringFilter for manipulating metadata
+- [NUTCH-2429](https://issues.apache.org/jira/browse/NUTCH-2429) - Fix Plugin 
System to allow protocol plugins to bundle their URLStreamHandlers
+- [NUTCH-2449](https://issues.apache.org/jira/browse/NUTCH-2449) - Usage of 
Tika LanguageIdentifier in language-identifier plugin
+- [NUTCH-2573](https://issues.apache.org/jira/browse/NUTCH-2573) - Suspend 
crawling if robots.txt fails to fetch with 5xx status
+- [NUTCH-2795](https://issues.apache.org/jira/browse/NUTCH-2795) - 
CrawlDbReader: compress CrawlDb dumps if configured
+- [NUTCH-2807](https://issues.apache.org/jira/browse/NUTCH-2807) - 
SitemapProcessor to warn that ignoring robots.txt affects detection of sitemaps
+- [NUTCH-2808](https://issues.apache.org/jira/browse/NUTCH-2808) - Document 
side effects of ignoring robots.txt
+- [NUTCH-2840](https://issues.apache.org/jira/browse/NUTCH-2840) - Fix 
'report-vulnerabilities' ant target in build.xml
+- [NUTCH-2842](https://issues.apache.org/jira/browse/NUTCH-2842) - Fix Javadoc 
warnings, errors and add Javadoc check to Github Action and Jenkins
+- [NUTCH-2845](https://issues.apache.org/jira/browse/NUTCH-2845) - Update 
urlfilter-suffix rules
+- [NUTCH-2847](https://issues.apache.org/jira/browse/NUTCH-2847) - 
HttpDateFormat: Simplify based on new Java 8 DateTime API
+- [NUTCH-2849](https://issues.apache.org/jira/browse/NUTCH-2849) - Replace 
remaining package.html files with package-info.java
+- [NUTCH-2857](https://issues.apache.org/jira/browse/NUTCH-2857) - Upgrade 
from JDK1.8 --> JDK11
+- [NUTCH-2859](https://issues.apache.org/jira/browse/NUTCH-2859) - 
urlnormalizer-protocol: allow to normalize domains
+- [NUTCH-2861](https://issues.apache.org/jira/browse/NUTCH-2861) - Remove 
parse-swf
+- [NUTCH-2864](https://issues.apache.org/jira/browse/NUTCH-2864) - Upgrade 
Dockerfile to use JDK 11
+- [NUTCH-2865](https://issues.apache.org/jira/browse/NUTCH-2865) - WARC 
exporter support for metadata and dropping empty responses
+- [NUTCH-2867](https://issues.apache.org/jira/browse/NUTCH-2867) - Support for 
custom HostDb aggregators
+- [NUTCH-2869](https://issues.apache.org/jira/browse/NUTCH-2869) - Add 
@Override annotations to Nutch plugins
+- [NUTCH-2879](https://issues.apache.org/jira/browse/NUTCH-2879) - fireant 
upgrade dependency hadoop-hdfs in ivy/ivy.xml from 3.1.3 to 3.3.1
+- [NUTCH-2882](https://issues.apache.org/jira/browse/NUTCH-2882) - Configure 
NutchUiServer for DEPLOYMENT and improve logging
+- [NUTCH-2885](https://issues.apache.org/jira/browse/NUTCH-2885) - Upgrade to 
Log4j2
+- [NUTCH-2886](https://issues.apache.org/jira/browse/NUTCH-2886) - Move Nutch 
WebApp to separate repository
+- [NUTCH-2891](https://issues.apache.org/jira/browse/NUTCH-2891) - Upgrade to 
Tika 2.1
+- [NUTCH-2892](https://issues.apache.org/jira/browse/NUTCH-2892) - Upgrade to 
Any23 2.5
+- [NUTCH-2893](https://issues.apache.org/jira/browse/NUTCH-2893) - fireant 
upgrade dependency elasticsearch-rest-high-level-client in 
src/plugin/indexer-elastic/ivy.xml from 7.11.1 to 7.13.2
+- [NUTCH-2896](https://issues.apache.org/jira/browse/NUTCH-2896) - 
Protocol-okhttp: make connection pool configurable
+- [NUTCH-2898](https://issues.apache.org/jira/browse/NUTCH-2898) - IDE Setup 
for nutch with Intellij IDEA is not well documented
+- [NUTCH-2903](https://issues.apache.org/jira/browse/NUTCH-2903) - Unable to 
Connect to Elasticsearch over HTTPS
+- [NUTCH-2904](https://issues.apache.org/jira/browse/NUTCH-2904) - Upgrade to 
crawler-commons 1.2
+- [NUTCH-2908](https://issues.apache.org/jira/browse/NUTCH-2908) - Log 
mapreduce job messages and counters in local mode
+- [NUTCH-2911](https://issues.apache.org/jira/browse/NUTCH-2911) - Add cleanup 
call in Fetcher.java
+- [NUTCH-2914](https://issues.apache.org/jira/browse/NUTCH-2914) - 
nutch-default.xml: remove obsolete and unused properties
+- [NUTCH-2918](https://issues.apache.org/jira/browse/NUTCH-2918) - Upgrade to 
log4j 2.16.0
+- [NUTCH-2919](https://issues.apache.org/jira/browse/NUTCH-2919) - Upgrade to 
Tika 2.2.1 and Any23 2.6
+- [NUTCH-2923](https://issues.apache.org/jira/browse/NUTCH-2923) - Add Job Id 
in Job Failure messages
+- [NUTCH-2929](https://issues.apache.org/jira/browse/NUTCH-2929) - Fetcher: 
start threads slowly to avoid that resources are temporarily exhausted
+- [NUTCH-2930](https://issues.apache.org/jira/browse/NUTCH-2930) - 
Protocol-okhttp: implement IP filter
+- [NUTCH-2946](https://issues.apache.org/jira/browse/NUTCH-2946) - Fetcher: 
optionally slow down fetching from hosts with repeated exceptions
+- [NUTCH-2948](https://issues.apache.org/jira/browse/NUTCH-2948) - Upgrade 
dependencies to Any23 2.7 and Tika 2.3.0
+- [NUTCH-2950](https://issues.apache.org/jira/browse/NUTCH-2950) - 
UpdateHostDb: performance improvements
+- [NUTCH-2952](https://issues.apache.org/jira/browse/NUTCH-2952) - Upgrade 
core dependencies (Hadoop 3.3.3, log4j 2.17.2)
+- [NUTCH-2953](https://issues.apache.org/jira/browse/NUTCH-2953) - Indexer 
Elastic to ignore SSL issues
+- [NUTCH-2956](https://issues.apache.org/jira/browse/NUTCH-2956) - 
index-geoip: dependency upgrades and improvements
+- [NUTCH-2957](https://issues.apache.org/jira/browse/NUTCH-2957) - 
indexer-solr / Solr schema: add fall-back field definitions for unknown index 
fields
+- [NUTCH-2958](https://issues.apache.org/jira/browse/NUTCH-2958) - Upgrade to 
crawler-commons 1.3
+- [NUTCH-2962](https://issues.apache.org/jira/browse/NUTCH-2962) - Update and 
complete package info of protocol plugins
+- [NUTCH-2963](https://issues.apache.org/jira/browse/NUTCH-2963) - Upgrade 
dependencies before release of 1.19
+
+### Task
+
+- [NUTCH-2826](https://issues.apache.org/jira/browse/NUTCH-2826) - Migrate 
Nutch Site from Apache CMS to Hugo
+- [NUTCH-2870](https://issues.apache.org/jira/browse/NUTCH-2870) - fireant 
upgrade dependency junit in ivy/ivy.xml from 4.13.1 to 4.13.2
+
+
+## Nutch 1.18 Release 14/01/2021 (dd/mm/yyyy)
+Release Report: https://s.apache.org/lqara
+
+### Breaking Changes
+
+- As part of NUTCH-2805, the plugin urlfilter-domainblacklist has been renamed 
to urlfilter-domaindenylist. And the fields required for the plugin 
urlfilter.domainblacklist.rules and urlfilter.domainblacklist.file has been 
replaced with urlfilter.domaindenylist.rules and urlfilter.domaindenylist.file 
respectively. See 
[NUTCH-2802](https://issues.apache.org/jira/browse/NUTCH-2802) for more details.
+
+### Sub-task
+
+- [NUTCH-2671](https://issues.apache.org/jira/browse/NUTCH-2671) - Upgrade ant 
ivy library
+- [NUTCH-2672](https://issues.apache.org/jira/browse/NUTCH-2672) - Ant build 
erronously installs *-test.jar instead *.jar for target "nightly"
+- [NUTCH-2805](https://issues.apache.org/jira/browse/NUTCH-2805) - Rename 
plugin urlfilter-domainblacklist
+- [NUTCH-2809](https://issues.apache.org/jira/browse/NUTCH-2809) - Upgrade 
any23 plugin dependency to 2.4
+- [NUTCH-2816](https://issues.apache.org/jira/browse/NUTCH-2816) - Add 
Spotbugs target to ant build
+- [NUTCH-2817](https://issues.apache.org/jira/browse/NUTCH-2817) - Avoid check 
for equality of URL path and file part using ==/!=
+- [NUTCH-2829](https://issues.apache.org/jira/browse/NUTCH-2829) - Fix ant 
target "clean-cache"
+
+### Bug
+
+- [NUTCH-2669](https://issues.apache.org/jira/browse/NUTCH-2669) - Reliable 
solution for javax.ws packaging.type
+- [NUTCH-2697](https://issues.apache.org/jira/browse/NUTCH-2697) - Upgrade Ivy 
to fix the issue of an unset packaging.type property
+- [NUTCH-2801](https://issues.apache.org/jira/browse/NUTCH-2801) - 
RobotsRulesParser command-line checker to use http.robots.agents as fall-back
+- [NUTCH-2810](https://issues.apache.org/jira/browse/NUTCH-2810) - 
FreeGenerator to actually apply configured number of fetch lists
+- [NUTCH-2813](https://issues.apache.org/jira/browse/NUTCH-2813) - 
MoreIndexingFilter - can't parse erroneous date - 2019-07-03T10:28:14
+- [NUTCH-2814](https://issues.apache.org/jira/browse/NUTCH-2814) - 
HttpDateFormat's internal time zone may change after parsing a date
+- [NUTCH-2818](https://issues.apache.org/jira/browse/NUTCH-2818) - Ant build: 
upgrade Apache Rat report task
+- [NUTCH-2823](https://issues.apache.org/jira/browse/NUTCH-2823) - 
IllegalStateException in IndexWriters.describe() when validating url param for 
SolrIndexer
+- [NUTCH-2824](https://issues.apache.org/jira/browse/NUTCH-2824) - 
urlnormalizer-basic to unescape percent-encoded host names
+
+### Improvement
+
+- [NUTCH-1190](https://issues.apache.org/jira/browse/NUTCH-1190) - 
MoreIndexingFilter refactor: move data formats used to parse "lastModified" to 
a config file.
+- [NUTCH-2582](https://issues.apache.org/jira/browse/NUTCH-2582) - Set pool 
size of XML SAX parsers used for MIME detection in Tika 1.19
+- [NUTCH-2730](https://issues.apache.org/jira/browse/NUTCH-2730) - 
SitemapProcessor to treat sitemap URLs as Set instead of List
+- [NUTCH-2782](https://issues.apache.org/jira/browse/NUTCH-2782) - 
protocol-http / lib-http: support TLSv1.3
+- [NUTCH-2796](https://issues.apache.org/jira/browse/NUTCH-2796) - Upgrade to 
crawler-commons 1.1
+- [NUTCH-2799](https://issues.apache.org/jira/browse/NUTCH-2799) - Add 
.asf.yaml file
+- [NUTCH-2833](https://issues.apache.org/jira/browse/NUTCH-2833) - Upgrade to 
Tika 1.25
+- [NUTCH-2835](https://issues.apache.org/jira/browse/NUTCH-2835) - Upgrade 
commons-jexl from 2 --> 3
+- [NUTCH-2836](https://issues.apache.org/jira/browse/NUTCH-2836) - Upgrade 
various commons dependencies
+- [NUTCH-2837](https://issues.apache.org/jira/browse/NUTCH-2837) - Update 
multiple dependencies
+- [NUTCH-2841](https://issues.apache.org/jira/browse/NUTCH-2841) - Upgrade 
xercesImpl dependency
+
+### Wish
+
+- [NUTCH-2834](https://issues.apache.org/jira/browse/NUTCH-2834) - 
Deduplication mode via command line in crawl script
+
+### Task
+
+- [NUTCH-2830](https://issues.apache.org/jira/browse/NUTCH-2830) - Upgrade 
any23 to v2.4
+
+
+## Nutch 1.17 Release 18/06/2020 (dd/mm/yyyy)
+Release Report: https://s.apache.org/ovhry
+
+### Bug
+
+- [NUTCH-1559](https://issues.apache.org/jira/browse/NUTCH-1559) - 
parse-metatags duplicates extracted metatags
+- [NUTCH-2379](https://issues.apache.org/jira/browse/NUTCH-2379) - crawl 
script dedup's crawldb update is slow
+- [NUTCH-2419](https://issues.apache.org/jira/browse/NUTCH-2419) - Some URL 
filters and normalizers do not respect command-line override for rule file
+- [NUTCH-2507](https://issues.apache.org/jira/browse/NUTCH-2507) - 
NutchTutorial wiki pages as a lot of outdated command line calls when it starts 
with the solr interaction
+- [NUTCH-2511](https://issues.apache.org/jira/browse/NUTCH-2511) - 
SitemapProcessor limited by http.content.limit
+- [NUTCH-2525](https://issues.apache.org/jira/browse/NUTCH-2525) - Metadata 
indexer cannot handle uppercase parse metadata
+- [NUTCH-2567](https://issues.apache.org/jira/browse/NUTCH-2567) - 
parse-metatags writes all meta tags twice
+- [NUTCH-2720](https://issues.apache.org/jira/browse/NUTCH-2720) - ROBOTS 
metatag ignored when capitalized
+- [NUTCH-2745](https://issues.apache.org/jira/browse/NUTCH-2745) - Solr 
schema.xml not shipped in binary release
+- [NUTCH-2748](https://issues.apache.org/jira/browse/NUTCH-2748) - Fetch 
status gone (redirect exceeded) not to overwrite existing items in CrawlDb
+- [NUTCH-2751](https://issues.apache.org/jira/browse/NUTCH-2751) - nutch clean 
does not work with secured solr cloud
+- [NUTCH-2753](https://issues.apache.org/jira/browse/NUTCH-2753) - Add -listen 
option to command-line help of CrawlDbReader and LinkDbReader
+- [NUTCH-2754](https://issues.apache.org/jira/browse/NUTCH-2754) - 
fetcher.max.crawl.delay ignored if exceeding 5 min. / 300 sec.
+- [NUTCH-2760](https://issues.apache.org/jira/browse/NUTCH-2760) - 
protocol-okhttp: properly record HTTP version in request message header
+- [NUTCH-2761](https://issues.apache.org/jira/browse/NUTCH-2761) - ivy jar 
fails to download
+- [NUTCH-2763](https://issues.apache.org/jira/browse/NUTCH-2763) - 
protocol-okhttp (store.http.headers): add whitespace in status line after 
status code also when message is empty
+- [NUTCH-2770](https://issues.apache.org/jira/browse/NUTCH-2770) - 
Subcollection logic allows empty string as a whitelist value, thus matching 
every incoming document.
+- [NUTCH-2778](https://issues.apache.org/jira/browse/NUTCH-2778) - 
indexer-elastic to properly log errors
+- [NUTCH-2787](https://issues.apache.org/jira/browse/NUTCH-2787) - CrawlDb 
JSON dump does not export metadata primitive data types correctly
+- [NUTCH-2789](https://issues.apache.org/jira/browse/NUTCH-2789) - 
Documentation: update links to point to cwiki
+- [NUTCH-2790](https://issues.apache.org/jira/browse/NUTCH-2790) - 
CSVIndexWriter does not escape leading quotes properly
+- [NUTCH-2791](https://issues.apache.org/jira/browse/NUTCH-2791) - 
domainstats, protocolstats and crawlcomplete do not handle GCS URLs
+
+### New Feature
+
+- [NUTCH-1863](https://issues.apache.org/jira/browse/NUTCH-1863) - Add JSON 
format dump output to readdb command
+
+### Improvement
+
+- [NUTCH-1194](https://issues.apache.org/jira/browse/NUTCH-1194) - Generator: 
CrawlDB lock should be released earlier
+- [NUTCH-2002](https://issues.apache.org/jira/browse/NUTCH-2002) - 
ParserChecker and IndexingFiltersChecker to check robots.txt
+- [NUTCH-2184](https://issues.apache.org/jira/browse/NUTCH-2184) - Enable 
IndexingJob to function with no crawldb
+- [NUTCH-2495](https://issues.apache.org/jira/browse/NUTCH-2495) - Use 
-deleteGone instead of clean job in crawler script while indexing
+- [NUTCH-2496](https://issues.apache.org/jira/browse/NUTCH-2496) - Speed up 
link inversion step in crawling script
+- [NUTCH-2501](https://issues.apache.org/jira/browse/NUTCH-2501) - allow to 
set Java heap size when using crawl script in distributed mode
+- [NUTCH-2649](https://issues.apache.org/jira/browse/NUTCH-2649) - Optionally 
skip TLS/SSL certificate validation for protocol-selenium and protocol-htmlunit
+- [NUTCH-2733](https://issues.apache.org/jira/browse/NUTCH-2733) - 
protocol-okhttp: add support for Brotli compression (Content-Encoding)
+- [NUTCH-2739](https://issues.apache.org/jira/browse/NUTCH-2739) - 
indexer-elastic: Upgrade ES and migrate to REST client
+- [NUTCH-2743](https://issues.apache.org/jira/browse/NUTCH-2743) - Add list of 
Nutch properties (nutch-default.xml) to documentation
+- [NUTCH-2746](https://issues.apache.org/jira/browse/NUTCH-2746) - Basic URL 
normalizer to normalize Unicode domain names
+- [NUTCH-2747](https://issues.apache.org/jira/browse/NUTCH-2747) - Replace 
remaining o.a.commons.logging by org.slf4j
+- [NUTCH-2750](https://issues.apache.org/jira/browse/NUTCH-2750) - Improve 
CrawlDbReader & LinkDbReader reader handling
+- [NUTCH-2752](https://issues.apache.org/jira/browse/NUTCH-2752) - 
indexer-solr: Upgrade to latest Solr version
+- [NUTCH-2755](https://issues.apache.org/jira/browse/NUTCH-2755) - Remove 
obsolete plugin indexer-elastic-rest
+- [NUTCH-2757](https://issues.apache.org/jira/browse/NUTCH-2757) - 
indexer-elastic: add authentication options
+- [NUTCH-2758](https://issues.apache.org/jira/browse/NUTCH-2758) - Add plugin 
READMEs to binary release packages
+- [NUTCH-2759](https://issues.apache.org/jira/browse/NUTCH-2759) - bin/crawl: 
Rename option --num-slaves
+- [NUTCH-2762](https://issues.apache.org/jira/browse/NUTCH-2762) - Replace 
http:// URLs by https:// (build files and documentation)
+- [NUTCH-2767](https://issues.apache.org/jira/browse/NUTCH-2767) - Fetcher to 
stop filling queues skipped due to repeated exceptions
+- [NUTCH-2768](https://issues.apache.org/jira/browse/NUTCH-2768) - 
FetcherThread: unnecessary usage of class casts
+- [NUTCH-2772](https://issues.apache.org/jira/browse/NUTCH-2772) - Debugging 
parse filter to show serialized DOM tree
+- [NUTCH-2773](https://issues.apache.org/jira/browse/NUTCH-2773) - 
SegmentReader (-dump or -get): show HTML content as UTF-8
+- [NUTCH-2774](https://issues.apache.org/jira/browse/NUTCH-2774) - Annotate 
methods implementing the Hadoop API by @Override
+- [NUTCH-2775](https://issues.apache.org/jira/browse/NUTCH-2775) - Fetcher to 
guarantee minimum delay even if robots.txt defines shorter Crawl-delay
+- [NUTCH-2776](https://issues.apache.org/jira/browse/NUTCH-2776) - Fetcher to 
temporarily deduplicate followed redirects
+- [NUTCH-2777](https://issues.apache.org/jira/browse/NUTCH-2777) - Upgrade to 
Hadoop 3.1
+- [NUTCH-2779](https://issues.apache.org/jira/browse/NUTCH-2779) - Upgrade to 
Tika 1.24.1
+- [NUTCH-2780](https://issues.apache.org/jira/browse/NUTCH-2780) - Upgrade 
index-solr to use Solr 8.5.1
+- [NUTCH-2781](https://issues.apache.org/jira/browse/NUTCH-2781) - Increase 
default Java heap size
+- [NUTCH-2783](https://issues.apache.org/jira/browse/NUTCH-2783) - Use (more) 
parametrized logging
+- [NUTCH-2784](https://issues.apache.org/jira/browse/NUTCH-2784) - Add tool to 
list Nutch and Hadoop properties
+- [NUTCH-2785](https://issues.apache.org/jira/browse/NUTCH-2785) - 
FreeGenerator: command-line option to define number of generated fetch lists
+- [NUTCH-2788](https://issues.apache.org/jira/browse/NUTCH-2788) - ParseData: 
improve presentation of Metadata in method toString()
+- [NUTCH-2794](https://issues.apache.org/jira/browse/NUTCH-2794) - Add 
additional ciphers to HTTP base's default cipher suite
+
+### Test
+
+- [NUTCH-1945](https://issues.apache.org/jira/browse/NUTCH-1945) - Test for 
XLSX parser
+
+### Task
+
+- [NUTCH-2434](https://issues.apache.org/jira/browse/NUTCH-2434) - Add methods 
to reset parameters HTMLMetaTags
+
+### Sub-task
+
+- [NUTCH-2735](https://issues.apache.org/jira/browse/NUTCH-2735) - Update the 
indexer-solr documentation about the schema.xml usage
+
+
+## Nutch 1.16 Release 02/10/2019 (dd/mm/yyyy)
+Release Report: https://s.apache.org/l2j94
+
+### Comments
+
+- schema.xml has been moved to indexer-solr plugin directory. This file is 
provided as a
+  reference/guide for Solr users (NUTCH-2654)
+
+### Breaking Changes
+
+- The value of crawl.gen.delay is now read in milliseconds as stated in the 
description in nutch-default.xml.
+  Previously, the value has been read in days, see 
[NUTCH-1842](https://issues.apache.org/jira/browse/NUTCH-1842)
+  for further information.
+
+- HostDB entries have been moved from Integer to Long in order to accomodate 
very large
+  hosts. Remove your existing HostDB and recreate it with bin/nutch 
updatehostdb, see
+  [NUTCH-2694](https://issues.apache.org/jira/browse/NUTCH-2694) for 
additional information.
+
+- The signature class TextProfileSignature has been improved to be stable over
+  consecutive runs by sorting tokens by frequency first and secondarily in 
lexicographic
+  order.  If an existing CrawlDb contains signatures generated by 
TextProfileSignature
+  these are likely to change when upgrading to Nutch 1.16.  The previous 
behavior relying
+  on a semi-stable pseudo-random hash sorting could be restored setting the 
property
+  `db.signature.text_profile.sec_sort_lex` to `false`. See also NUTCH-2381.
+
+### Bug
+
+- [NUTCH-1063](https://issues.apache.org/jira/browse/NUTCH-1063) - 
OutlinkExtractor test generates an exception but does not fail
+- [NUTCH-1842](https://issues.apache.org/jira/browse/NUTCH-1842) - 
crawl.gen.delay has a wrong default value in nutch-default.xml or is being 
parsed incorrectly
+- [NUTCH-2279](https://issues.apache.org/jira/browse/NUTCH-2279) - LinkRank 
fails when using Hadoop MR output compression
+- [NUTCH-2381](https://issues.apache.org/jira/browse/NUTCH-2381) - In some 
situations the class TextProfileSignature gives different signatures for the 
same text "profile" page.
+- [NUTCH-2387](https://issues.apache.org/jira/browse/NUTCH-2387) - Nutch 
should not index document with "noindex" meta
+- [NUTCH-2457](https://issues.apache.org/jira/browse/NUTCH-2457) - Embedded 
documents likely not correctly parsed by Tika
+- [NUTCH-2475](https://issues.apache.org/jira/browse/NUTCH-2475) - If and 
else-if branches has the same condition
+- [NUTCH-2482](https://issues.apache.org/jira/browse/NUTCH-2482) - index-geoip 
not to add null values to document fields
+- [NUTCH-2585](https://issues.apache.org/jira/browse/NUTCH-2585) - NPE in 
TrieStringMatcher
+- [NUTCH-2598](https://issues.apache.org/jira/browse/NUTCH-2598) - 
URLNormalizerChecker fails on invalid URLs in input
+- [NUTCH-2606](https://issues.apache.org/jira/browse/NUTCH-2606) - MIME 
detection is wrong for plain-text documents send as Content-Type 
"application/msword"
+- [NUTCH-2635](https://issues.apache.org/jira/browse/NUTCH-2635) - Generator 
writes unneeded temporary output
+- [NUTCH-2639](https://issues.apache.org/jira/browse/NUTCH-2639) - bin/nutch 
fails to set native library path on Cygwin causing jobs to fail with 
UnsatisfiedLinkError
+- [NUTCH-2641](https://issues.apache.org/jira/browse/NUTCH-2641) - 
ClassCastException in webui
+- [NUTCH-2642](https://issues.apache.org/jira/browse/NUTCH-2642) - 
MoreIndexingFilter parses ISO 8601 UTC dates in local time zone
+- [NUTCH-2643](https://issues.apache.org/jira/browse/NUTCH-2643) - ant target 
"resolve-default" to depend on "init"
+- [NUTCH-2644](https://issues.apache.org/jira/browse/NUTCH-2644) - 
CrawlDbReader -dump ignores filter options
+- [NUTCH-2645](https://issues.apache.org/jira/browse/NUTCH-2645) - Webgraph 
tools ignore command-line options
+- [NUTCH-2650](https://issues.apache.org/jira/browse/NUTCH-2650) - 
-addBinaryContent -base64 flags are causing "String length must be a multiple 
of four" error in IndexingJob
+- [NUTCH-2652](https://issues.apache.org/jira/browse/NUTCH-2652) - Fetcher 
launches more fetch tasks than fetch lists
+- [NUTCH-2655](https://issues.apache.org/jira/browse/NUTCH-2655) - Update Solr 
schema.xml for Solr 7.x
+- [NUTCH-2656](https://issues.apache.org/jira/browse/NUTCH-2656) - Update 
description to configure Solr 7.x in tutorial
+- [NUTCH-2673](https://issues.apache.org/jira/browse/NUTCH-2673) - 
EOFException protocol-http
+- [NUTCH-2674](https://issues.apache.org/jira/browse/NUTCH-2674) - HostDb: 
dump shows wrong column headers
+- [NUTCH-2680](https://issues.apache.org/jira/browse/NUTCH-2680) - 
Documentation: https supported by multiple protocol plugins not only httpclient
+- [NUTCH-2687](https://issues.apache.org/jira/browse/NUTCH-2687) - Regex for 
reading title from Content-Disposition is wrong
+- [NUTCH-2694](https://issues.apache.org/jira/browse/NUTCH-2694) - HostDB to 
aggregate by long instead of integer
+- [NUTCH-2696](https://issues.apache.org/jira/browse/NUTCH-2696) - Nutch 
SegmentReader does not dump non-ASCII characters with Hadoop 3.x
+- [NUTCH-2699](https://issues.apache.org/jira/browse/NUTCH-2699) - 
Protocol-okhttp: needless loops to increment requested bytes counter when more 
content is already buffered
+- [NUTCH-2703](https://issues.apache.org/jira/browse/NUTCH-2703) - parse-tika: 
Boilerpipe should not run for non-(X)HTML pages
+- [NUTCH-2706](https://issues.apache.org/jira/browse/NUTCH-2706) - 
-addBinaryContent flag can cause "String length must be a multiple of four" 
error in IndexingJob
+- [NUTCH-2715](https://issues.apache.org/jira/browse/NUTCH-2715) - 
WARCExporter fails on large records
+- [NUTCH-2716](https://issues.apache.org/jira/browse/NUTCH-2716) - 
protocol-http: Response headers are not stored for a compressed response
+- [NUTCH-2717](https://issues.apache.org/jira/browse/NUTCH-2717) - Generator 
cannot open hostDB
+- [NUTCH-2722](https://issues.apache.org/jira/browse/NUTCH-2722) - Fetch 
dependencies via https
+- [NUTCH-2723](https://issues.apache.org/jira/browse/NUTCH-2723) - Indexer 
Solr not to decode URLs before deletion
+- [NUTCH-2724](https://issues.apache.org/jira/browse/NUTCH-2724) - Metadata 
indexer not to emit empty values
+- [NUTCH-2729](https://issues.apache.org/jira/browse/NUTCH-2729) - 
protocol-okhttp: fix marking of truncated content
+- [NUTCH-2731](https://issues.apache.org/jira/browse/NUTCH-2731) - Solr 
Cleanup Step Fails when Authentication is Required
+- [NUTCH-2738](https://issues.apache.org/jira/browse/NUTCH-2738) - Generator: 
document property generate.restrict.status
+- [NUTCH-2740](https://issues.apache.org/jira/browse/NUTCH-2740) - Generator: 
generate.max.count overflow not logged
+
+### New Feature
+
+- [NUTCH-2676](https://issues.apache.org/jira/browse/NUTCH-2676) - Update to 
the latest selenium and add code to use chrome and firefox headless mode with 
the remote web driver
+
+### Improvement
+
+- [NUTCH-1014](https://issues.apache.org/jira/browse/NUTCH-1014) - Migrate 
from Apache ORO to java.util.regex
+- [NUTCH-1021](https://issues.apache.org/jira/browse/NUTCH-1021) - Migrate 
OutlinkExtractor from Apache ORO to java.util.regex
+- [NUTCH-1982](https://issues.apache.org/jira/browse/NUTCH-1982) - Make Git 
ignore IDE project files and add note about IDE setup
+- [NUTCH-2460](https://issues.apache.org/jira/browse/NUTCH-2460) - use the 
headless option of firefox and chrome in protocol-selenium
+- [NUTCH-2602](https://issues.apache.org/jira/browse/NUTCH-2602) - 
Configuration values in the description of index writers
+- [NUTCH-2612](https://issues.apache.org/jira/browse/NUTCH-2612) - Support for 
sitemap processing by hostname
+- [NUTCH-2623](https://issues.apache.org/jira/browse/NUTCH-2623) - Fetcher to 
guarantee delay for same host/domain/ip independent of http/https protocol
+- [NUTCH-2625](https://issues.apache.org/jira/browse/NUTCH-2625) - 
ProtocolFactory.getProtocol(url) may create multiple plugin instances
+- [NUTCH-2626](https://issues.apache.org/jira/browse/NUTCH-2626) - bin/crawl: 
remove option -noParsing from fetch command
+- [NUTCH-2627](https://issues.apache.org/jira/browse/NUTCH-2627) - Fetcher to 
optionally filter URLs
+- [NUTCH-2628](https://issues.apache.org/jira/browse/NUTCH-2628) - Fetcher: 
optionally generate signature of unparsed content
+- [NUTCH-2629](https://issues.apache.org/jira/browse/NUTCH-2629) - 
Documentation for CSV Index Writer
+- [NUTCH-2630](https://issues.apache.org/jira/browse/NUTCH-2630) - Fetcher to 
log skipped records by robots.txt
+- [NUTCH-2631](https://issues.apache.org/jira/browse/NUTCH-2631) - 
KafkaIndexWriter
+- [NUTCH-2632](https://issues.apache.org/jira/browse/NUTCH-2632) - 
protocol-okhttp doesn't accept proxy authentication
+- [NUTCH-2633](https://issues.apache.org/jira/browse/NUTCH-2633) - Fix 
deprecation warnings when building Nutch master branch under JDK 10.0.2+13
+- [NUTCH-2647](https://issues.apache.org/jira/browse/NUTCH-2647) - Skip TLS 
certificate checks in protocol-http plugin
+- [NUTCH-2648](https://issues.apache.org/jira/browse/NUTCH-2648) - Make 
configurable whether TLS/SSL certificates are checked by protocol plugins
+- [NUTCH-2651](https://issues.apache.org/jira/browse/NUTCH-2651) - Upgrade to 
Tika 1.19.1 (from 1.18)
+- [NUTCH-2653](https://issues.apache.org/jira/browse/NUTCH-2653) - 
ProtocolFactory.getProtocol(url) creates separate plugin instances for 
http/https
+- [NUTCH-2654](https://issues.apache.org/jira/browse/NUTCH-2654) - Remove 
obsolete index-writer configuration in conf/
+- [NUTCH-2657](https://issues.apache.org/jira/browse/NUTCH-2657) - 
Protocol-http to store HTTP response header with "\r\n"
+- [NUTCH-2658](https://issues.apache.org/jira/browse/NUTCH-2658) - Add README 
file to all plugins in src/plugin
+- [NUTCH-2659](https://issues.apache.org/jira/browse/NUTCH-2659) - Add missing 
Apache license headers
+- [NUTCH-2660](https://issues.apache.org/jira/browse/NUTCH-2660) - Unit tests 
of plugins parse-js, headings, index-jexl-filter to be executed during build
+- [NUTCH-2661](https://issues.apache.org/jira/browse/NUTCH-2661) - Move 
TestOutlinks to the proper path
+- [NUTCH-2663](https://issues.apache.org/jira/browse/NUTCH-2663) - Improve 
index-jexl-filter syntax for scripts
+- [NUTCH-2666](https://issues.apache.org/jira/browse/NUTCH-2666) - Increase 
default value for http.content.limit / ftp.content.limit / file.content.limit
+- [NUTCH-2668](https://issues.apache.org/jira/browse/NUTCH-2668) - Integrate 
OWASP dependency checks as ant target
+- [NUTCH-2678](https://issues.apache.org/jira/browse/NUTCH-2678) - Allow for 
per-host configurable protocol plugin
+- [NUTCH-2682](https://issues.apache.org/jira/browse/NUTCH-2682) - Upgrade to 
Tika 1.20
+- [NUTCH-2683](https://issues.apache.org/jira/browse/NUTCH-2683) - 
DeduplicationJob: add option to prefer https:// over http://
+- [NUTCH-2686](https://issues.apache.org/jira/browse/NUTCH-2686) - Separate 
field for mime types mapped by index-more plugin
+- [NUTCH-2688](https://issues.apache.org/jira/browse/NUTCH-2688) - Unify the 
licence headers
+- [NUTCH-2689](https://issues.apache.org/jira/browse/NUTCH-2689) - Speed up 
urlfilter-regex and urlfilter-automaton
+- [NUTCH-2690](https://issues.apache.org/jira/browse/NUTCH-2690) - 
Configurable and fast URL filter
+- [NUTCH-2691](https://issues.apache.org/jira/browse/NUTCH-2691) - Improve 
logging from scoring-depth plugin
+- [NUTCH-2692](https://issues.apache.org/jira/browse/NUTCH-2692) - 
Subcollection to support case-insensitive white and black lists
+- [NUTCH-2693](https://issues.apache.org/jira/browse/NUTCH-2693) - Misspelled 
configuration property names in documentation
+- [NUTCH-2695](https://issues.apache.org/jira/browse/NUTCH-2695) - Fix some 
alerts raised by LGTM
+- [NUTCH-2700](https://issues.apache.org/jira/browse/NUTCH-2700) - 
Indexchecker: improve command-line help
+- [NUTCH-2701](https://issues.apache.org/jira/browse/NUTCH-2701) - Fetcher: 
log dates and times also in human-readable form
+- [NUTCH-2702](https://issues.apache.org/jira/browse/NUTCH-2702) - Fetcher: 
suppress stack for frequent exceptions
+- [NUTCH-2704](https://issues.apache.org/jira/browse/NUTCH-2704) - Upgrade 
crawler-commons dependency to 1.0
+- [NUTCH-2708](https://issues.apache.org/jira/browse/NUTCH-2708) - 
urlfilter-automaton: update library dependency (dk.brics.automaton)
+- [NUTCH-2709](https://issues.apache.org/jira/browse/NUTCH-2709) - Remove 
unused properties and code related to HTTP protocol
+- [NUTCH-2718](https://issues.apache.org/jira/browse/NUTCH-2718) - Names of 
index writers and exchanges configuration files to be configurable
+- [NUTCH-2719](https://issues.apache.org/jira/browse/NUTCH-2719) - NPE if 
exchanges.xml uses index writer not available
+- [NUTCH-2725](https://issues.apache.org/jira/browse/NUTCH-2725) - Plugin 
lib-http to support per-host configurable cookies
+- [NUTCH-2726](https://issues.apache.org/jira/browse/NUTCH-2726) - Upgrade to 
Tika 1.22
+- [NUTCH-2727](https://issues.apache.org/jira/browse/NUTCH-2727) - Upgrade 
Hadoop dependencies to 2.9.2
+- [NUTCH-2728](https://issues.apache.org/jira/browse/NUTCH-2728) - 
protocol-okhttp: upgrade okhttp dependency to 3.14.2
+- [NUTCH-2732](https://issues.apache.org/jira/browse/NUTCH-2732) - Ignored and 
tracked configuration files by git
+- [NUTCH-2736](https://issues.apache.org/jira/browse/NUTCH-2736) - Upgrade 
Dockerfile to be based on recent Ubuntu LTS version
+- [NUTCH-2737](https://issues.apache.org/jira/browse/NUTCH-2737) - Generator: 
count and log reason of rejections during selection
+
+### Task
+
+- [NUTCH-2192](https://issues.apache.org/jira/browse/NUTCH-2192) - Get rid of 
oro
+- [NUTCH-2613](https://issues.apache.org/jira/browse/NUTCH-2613) - 
Documentation for exchange component
+- [NUTCH-2698](https://issues.apache.org/jira/browse/NUTCH-2698) - Remove 
sonar build task from build.xml
+
+### Sub-task
+
+- [NUTCH-1121](https://issues.apache.org/jira/browse/NUTCH-1121) - JUnit test 
for parse-js
+- [NUTCH-2621](https://issues.apache.org/jira/browse/NUTCH-2621) - Generate 
report of third-party licenses
+- [NUTCH-2684](https://issues.apache.org/jira/browse/NUTCH-2684) - Add 
README.md file to all indexer writers plugins
+- [NUTCH-2685](https://issues.apache.org/jira/browse/NUTCH-2685) - Add 
README.md file to all exchange plugins
+
+
+## Nutch 1.15 Release (25/07/2018)
+Release Report: https://s.apache.org/nczS
+
+### Breaking Changes
+
+- indexer plugins are now configured in a single XML file 
(conf/index-writers.xml),
+  see https://cwiki.apache.org/confluence/display/NUTCH/IndexWriters - setting 
or overwriting configuration
+  parameters via Nutch properties is not possible anymore.
+
+### Bug
+
+- [NUTCH-1993](https://issues.apache.org/jira/browse/NUTCH-1993) - Nutch does 
not use backup parsers
+- [NUTCH-2071](https://issues.apache.org/jira/browse/NUTCH-2071) - A parser 
failure on a single document may fail crawling job if parser.timeout=-1
+- [NUTCH-2145](https://issues.apache.org/jira/browse/NUTCH-2145) - parse/index 
checker fail to fetch valid percent-encoded URLs
+- [NUTCH-2161](https://issues.apache.org/jira/browse/NUTCH-2161) - Interrupted 
failed and/or killed tasks fail to clean up temp directories in HDFS
+- [NUTCH-2273](https://issues.apache.org/jira/browse/NUTCH-2273) - Selenium 
and InteractiveSelenium Do Not Support HTTPS
+- [NUTCH-2310](https://issues.apache.org/jira/browse/NUTCH-2310) - 
Protocol-Selenium does not support HTTPS protocol
+- [NUTCH-2321](https://issues.apache.org/jira/browse/NUTCH-2321) - Indexing 
filter checker leaks threads
+- [NUTCH-2324](https://issues.apache.org/jira/browse/NUTCH-2324) - Issue in 
setting default linkdb path
+- [NUTCH-2447](https://issues.apache.org/jira/browse/NUTCH-2447) - Work-around 
SSLProtocolException: handshake alert: unrecognized_name
+- [NUTCH-2454](https://issues.apache.org/jira/browse/NUTCH-2454) - REST API 
fix for usage of hostdb in generator
+- [NUTCH-2461](https://issues.apache.org/jira/browse/NUTCH-2461) - Generate 
passes the data to when maxCount == 0
+- [NUTCH-2466](https://issues.apache.org/jira/browse/NUTCH-2466) - Sitemap 
processor to follow redirects
+- [NUTCH-2467](https://issues.apache.org/jira/browse/NUTCH-2467) - Sitemap 
type field can be null
+- [NUTCH-2485](https://issues.apache.org/jira/browse/NUTCH-2485) - 
ParserFactory swallows exception
+- [NUTCH-2486](https://issues.apache.org/jira/browse/NUTCH-2486) - Compiler 
Warning: Unchecked / unsafe operations in MimeTypeIndexingFilter
+- [NUTCH-2489](https://issues.apache.org/jira/browse/NUTCH-2489) - Dependency 
collision with lucene-analyzers-common in scoring-similarity plugin
+- [NUTCH-2490](https://issues.apache.org/jira/browse/NUTCH-2490) - Sitemap 
processing: Sitemap index files not working
+- [NUTCH-2494](https://issues.apache.org/jira/browse/NUTCH-2494) - Fetcher: 
java.lang.IllegalArgumentException: Wrong FS: s3
+- [NUTCH-2499](https://issues.apache.org/jira/browse/NUTCH-2499) - Elastic 
REST Indexer: Duplicate values
+- [NUTCH-2505](https://issues.apache.org/jira/browse/NUTCH-2505) - nutch does 
not delete the .locked file, when the generator partition got an exception
+- [NUTCH-2508](https://issues.apache.org/jira/browse/NUTCH-2508) - Misleading 
documentation about http.proxy.exception.list
+- [NUTCH-2509](https://issues.apache.org/jira/browse/NUTCH-2509) - 
Inconsistent behavior in SitemapProcessor
+- [NUTCH-2513](https://issues.apache.org/jira/browse/NUTCH-2513) - ant eclipse 
target fails with "protocol switch unsafe"
+- [NUTCH-2517](https://issues.apache.org/jira/browse/NUTCH-2517) - mergesegs 
corrupts segment data
+- [NUTCH-2518](https://issues.apache.org/jira/browse/NUTCH-2518) - Must check 
return value of job.waitForCompletion()
+- [NUTCH-2520](https://issues.apache.org/jira/browse/NUTCH-2520) - Wrong 
Accept-Charset sent when http.accept.charset is not defined
+- [NUTCH-2521](https://issues.apache.org/jira/browse/NUTCH-2521) - 
SitemapProcessor to use property sitemap.redir.max
+- [NUTCH-2523](https://issues.apache.org/jira/browse/NUTCH-2523) - 
UpdateHostDB blocks usage of plugins unintentionally
+- [NUTCH-2524](https://issues.apache.org/jira/browse/NUTCH-2524) - bin/crawl: 
fix check for HostDb in distributed mode
+- [NUTCH-2533](https://issues.apache.org/jira/browse/NUTCH-2533) - Injector: 
NullPointerException if seed URL dir contains non-file entries
+- [NUTCH-2535](https://issues.apache.org/jira/browse/NUTCH-2535) - 
CrawlDbReader -stats: ClassCastException
+- [NUTCH-2544](https://issues.apache.org/jira/browse/NUTCH-2544) - Nutch 1.15 
no longer compatible with AWS EMR and S3
+- [NUTCH-2547](https://issues.apache.org/jira/browse/NUTCH-2547) - 
urlnormalizer-basic fails on special characters in path/query
+- [NUTCH-2549](https://issues.apache.org/jira/browse/NUTCH-2549) - 
protocol-http does not behave the same as browsers
+- [NUTCH-2550](https://issues.apache.org/jira/browse/NUTCH-2550) - Fetcher 
fails to follow redirects
+- [NUTCH-2551](https://issues.apache.org/jira/browse/NUTCH-2551) - 
NullPointerException in generator
+- [NUTCH-2552](https://issues.apache.org/jira/browse/NUTCH-2552) - 
CrawlDbReader -topN fails
+- [NUTCH-2553](https://issues.apache.org/jira/browse/NUTCH-2553) - Fetcher not 
to modify URLs to be fetched
+- [NUTCH-2554](https://issues.apache.org/jira/browse/NUTCH-2554) - 
parserchecker can't fetch some URLs
+- [NUTCH-2565](https://issues.apache.org/jira/browse/NUTCH-2565) - MergeDB 
incorrectly handles unfetched CrawlDatums
+- [NUTCH-2568](https://issues.apache.org/jira/browse/NUTCH-2568) - Caught 
exception is immediately rethrown
+- [NUTCH-2569](https://issues.apache.org/jira/browse/NUTCH-2569) - 
ClassNotFoundException when running in (pseudo-)distributed mode
+- [NUTCH-2570](https://issues.apache.org/jira/browse/NUTCH-2570) - 
Deduplication job fails to install deduplicated CrawlDb
+- [NUTCH-2571](https://issues.apache.org/jira/browse/NUTCH-2571) - 
SegmentReader -list fails to read segment
+- [NUTCH-2572](https://issues.apache.org/jira/browse/NUTCH-2572) - HostDb: 
updatehostdb does not set values
+- [NUTCH-2574](https://issues.apache.org/jira/browse/NUTCH-2574) - Generator: 
hostCount >= maxCount comparison wrong
+- [NUTCH-2581](https://issues.apache.org/jira/browse/NUTCH-2581) - Caching of 
redirected robots.txt may overwrite correct robots.txt rules
+- [NUTCH-2589](https://issues.apache.org/jira/browse/NUTCH-2589) - HTML 
redirections are not followed when using parse-tika
+- [NUTCH-2590](https://issues.apache.org/jira/browse/NUTCH-2590) - 
SegmentReader -get fails
+- [NUTCH-2592](https://issues.apache.org/jira/browse/NUTCH-2592) - Fetcher to 
log reason of failed fetches
+- [NUTCH-2593](https://issues.apache.org/jira/browse/NUTCH-2593) - Single mode 
doesn't work in RabbitMQ indexer
+- [NUTCH-2597](https://issues.apache.org/jira/browse/NUTCH-2597) - NPE in 
updatehostdb
+- [NUTCH-2601](https://issues.apache.org/jira/browse/NUTCH-2601) - 
Elasticsearch Rest and Amazon CloudSearch have the same implementation class in 
indexer-writers.xml
+- [NUTCH-2607](https://issues.apache.org/jira/browse/NUTCH-2607) - 
ParserChecker should call ScoringFilters.passScoreAfterParsing() on all parses
+- [NUTCH-2609](https://issues.apache.org/jira/browse/NUTCH-2609) - 
urlnormalizer-basic to normalize path of file: URLs
+- [NUTCH-2614](https://issues.apache.org/jira/browse/NUTCH-2614) - NPE in 
CrawlDbReader -stats on empty CrawlDb
+- [NUTCH-2616](https://issues.apache.org/jira/browse/NUTCH-2616) - Review 
routing of deletions by Exchange component
+- [NUTCH-2618](https://issues.apache.org/jira/browse/NUTCH-2618) - 
protocol-okhttp not to use http.timeout for max duration to fetch document
+- [NUTCH-2620](https://issues.apache.org/jira/browse/NUTCH-2620) - 
urlfilter-validator incorrectly assumes that top-level domains are not longer 
than 4 characters
+- [NUTCH-2624](https://issues.apache.org/jira/browse/NUTCH-2624) - 
protocol-okhttp resource leak
+
+### New Feature
+
+- [NUTCH-1129](https://issues.apache.org/jira/browse/NUTCH-1129) - Any23 Nutch 
plugin
+- [NUTCH-1541](https://issues.apache.org/jira/browse/NUTCH-1541) - Indexer 
plugin to write CSV
+- [NUTCH-2412](https://issues.apache.org/jira/browse/NUTCH-2412) - Exchange 
component for indexing job
+- [NUTCH-2492](https://issues.apache.org/jira/browse/NUTCH-2492) - Add more 
configuration parameters to crawl script
+
+### Improvement
+
+- [NUTCH-1106](https://issues.apache.org/jira/browse/NUTCH-1106) - Options to 
skip url's based on length
+- [NUTCH-1480](https://issues.apache.org/jira/browse/NUTCH-1480) - SolrIndexer 
to write to multiple servers.
+- [NUTCH-2012](https://issues.apache.org/jira/browse/NUTCH-2012) - Merge 
parsechecker and indexchecker
+- [NUTCH-2375](https://issues.apache.org/jira/browse/NUTCH-2375) - Upgrade the 
code base from org.apache.hadoop.mapred to org.apache.hadoop.mapreduce
+- [NUTCH-2390](https://issues.apache.org/jira/browse/NUTCH-2390) - No 
documentation on pluggable indexing
+- [NUTCH-2411](https://issues.apache.org/jira/browse/NUTCH-2411) - 
Index-metadata to support indexing multiple values for a field
+- [NUTCH-2416](https://issues.apache.org/jira/browse/NUTCH-2416) - Fetcher to 
log thread ID
+- [NUTCH-2432](https://issues.apache.org/jira/browse/NUTCH-2432) - Protocol 
httpclient to disable cookies if http.enable.cookie.header is false
+- [NUTCH-2441](https://issues.apache.org/jira/browse/NUTCH-2441) - ARG_SEGMENT 
usage
+- [NUTCH-2491](https://issues.apache.org/jira/browse/NUTCH-2491) - Integrate 
sitemap processing and HostDB into crawl script
+- [NUTCH-2493](https://issues.apache.org/jira/browse/NUTCH-2493) - Add 
configuration parameter for sitemap processing to crawler script
+- [NUTCH-2497](https://issues.apache.org/jira/browse/NUTCH-2497) - Elastic 
REST Indexer: Allow multiple hosts
+- [NUTCH-2502](https://issues.apache.org/jira/browse/NUTCH-2502) - Any23 
Plugin: Add Content-Type filtering
+- [NUTCH-2503](https://issues.apache.org/jira/browse/NUTCH-2503) - Add option 
to run tests for a single plugin
+- [NUTCH-2510](https://issues.apache.org/jira/browse/NUTCH-2510) - Crawl 
script modification. HostDb : generate, optional usage and description
+- [NUTCH-2516](https://issues.apache.org/jira/browse/NUTCH-2516) - Hadoop 
imports use wildcards
+- [NUTCH-2519](https://issues.apache.org/jira/browse/NUTCH-2519) - Log 
mapreduce job counters in local mode
+- [NUTCH-2526](https://issues.apache.org/jira/browse/NUTCH-2526) - NPE in 
scoring-opic when indexing document without CrawlDb datum
+- [NUTCH-2527](https://issues.apache.org/jira/browse/NUTCH-2527) - URL filter: 
provide rules to exclude localhost and private address spaces
+- [NUTCH-2530](https://issues.apache.org/jira/browse/NUTCH-2530) - Rename 
property db.max.anchor.length > linkdb.max.anchor.length
+- [NUTCH-2534](https://issues.apache.org/jira/browse/NUTCH-2534) - 
CrawlDbReader -stats: make score quantiles configurable
+- [NUTCH-2539](https://issues.apache.org/jira/browse/NUTCH-2539) - Not correct 
naming of db.url.filters and db.url.normalizers in nutch-default.xml
+- [NUTCH-2543](https://issues.apache.org/jira/browse/NUTCH-2543) - readdb & 
readlinkdb to implement AbstractChecker
+- [NUTCH-2545](https://issues.apache.org/jira/browse/NUTCH-2545) - Upgrade to 
Any23 2.2
+- [NUTCH-2566](https://issues.apache.org/jira/browse/NUTCH-2566) - Fix 
exception log messages
+- [NUTCH-2576](https://issues.apache.org/jira/browse/NUTCH-2576) - HTTP 
protocol plugin based on okhttp
+- [NUTCH-2577](https://issues.apache.org/jira/browse/NUTCH-2577) - 
protocol-selenium can't handle https
+- [NUTCH-2578](https://issues.apache.org/jira/browse/NUTCH-2578) - Avoid lock 
by MimeUtil in constructor of protocol.Content
+- [NUTCH-2579](https://issues.apache.org/jira/browse/NUTCH-2579) - Fetcher to 
use parsed URL to call ProtocolFactory.getProtocol(url)
+- [NUTCH-2580](https://issues.apache.org/jira/browse/NUTCH-2580) - 
Improvements for Rabbitmq support
+- [NUTCH-2583](https://issues.apache.org/jira/browse/NUTCH-2583) - Upgrading 
Nutch's dependencies
+- [NUTCH-2584](https://issues.apache.org/jira/browse/NUTCH-2584) - Upgrade 
parse-tika to use Tika 1.18
+- [NUTCH-2594](https://issues.apache.org/jira/browse/NUTCH-2594) - 
Documentation for indexer plugins
+- [NUTCH-2595](https://issues.apache.org/jira/browse/NUTCH-2595) - Upgrade 
crawler-commons dependency to 0.10
+- [NUTCH-2600](https://issues.apache.org/jira/browse/NUTCH-2600) - Refactoring 
indexer-solr
+- [NUTCH-2611](https://issues.apache.org/jira/browse/NUTCH-2611) - Add 
line-breaks when parsing HTML block-level elements
+- [NUTCH-2617](https://issues.apache.org/jira/browse/NUTCH-2617) - Disable 
Exchange component by default
+- [NUTCH-2619](https://issues.apache.org/jira/browse/NUTCH-2619) - 
protocol-okhttp: allow to keep partially fetched docs as truncated
+
+### Task
+
+- [NUTCH-1219](https://issues.apache.org/jira/browse/NUTCH-1219) - Upgrade all 
jobs to new MapReduce API
+- [NUTCH-1228](https://issues.apache.org/jira/browse/NUTCH-1228) - Change 
mapred.task.timeout to mapreduce.task.timeout in fetcher
+
+### Sub-task
+
+- [NUTCH-1223](https://issues.apache.org/jira/browse/NUTCH-1223) - Migrate 
WebGraph to MapReduce API
+- [NUTCH-1224](https://issues.apache.org/jira/browse/NUTCH-1224) - Migrate 
FreeGenerator to MapReduce API
+- [NUTCH-1226](https://issues.apache.org/jira/browse/NUTCH-1226) - Migrate 
CrawlDbReader to MapReduce API
+- [NUTCH-2152](https://issues.apache.org/jira/browse/NUTCH-2152) - CommonCrawl 
dump via Service endpoint
+- [NUTCH-2555](https://issues.apache.org/jira/browse/NUTCH-2555) - URL 
normalization problem: path not starting with a '/'
+- [NUTCH-2556](https://issues.apache.org/jira/browse/NUTCH-2556) - 
protocol-http makes invalid HTTP/1.0 requests
+- [NUTCH-2557](https://issues.apache.org/jira/browse/NUTCH-2557) - 
protocol-http fails to follow redirections when an HTTP response body is invalid
+- [NUTCH-2558](https://issues.apache.org/jira/browse/NUTCH-2558) - 
protocol-http cannot handle a missing HTTP status line
+- [NUTCH-2559](https://issues.apache.org/jira/browse/NUTCH-2559) - 
protocol-http cannot handle colons after the HTTP status code
+- [NUTCH-2560](https://issues.apache.org/jira/browse/NUTCH-2560) - 
protocol-http throws an error when an http header spans over multiple lines
+- [NUTCH-2561](https://issues.apache.org/jira/browse/NUTCH-2561) - 
protocol-http can be made to read arbitrarily large HTTP responses
+- [NUTCH-2562](https://issues.apache.org/jira/browse/NUTCH-2562) - 
protocol-http fails to read large chunked HTTP responses
+- [NUTCH-2563](https://issues.apache.org/jira/browse/NUTCH-2563) - HTTP header 
spellchecking issues
+- [NUTCH-2575](https://issues.apache.org/jira/browse/NUTCH-2575) - 
protocol-http does not respect the maximum content-size for chunked responses
+- [NUTCH-2622](https://issues.apache.org/jira/browse/NUTCH-2622) - Unbundle 
LGPL-licensed jars from binary release
+
+
+## Nutch 1.14 Release 18/12/2017 (dd/mm/yyyy)
+
+### Breaking Changes
+
+- the bin/crawl script now expects the path to the seed to be preceded by -s
+  ([NUTCH-2046](https://issues.apache.org/jira/browse/NUTCH-2046))
+
+### Bug
+
+- [NUTCH-2071](https://issues.apache.org/jira/browse/NUTCH-2071) - A parser 
failure on a single document may fail crawling job
+- [NUTCH-2235](https://issues.apache.org/jira/browse/NUTCH-2235) - Classpath 
discrepancy with protocol-selenium in deploy mode
+- [NUTCH-2269](https://issues.apache.org/jira/browse/NUTCH-2269) - Clean not 
working after crawl
+- [NUTCH-2295](https://issues.apache.org/jira/browse/NUTCH-2295) - Nutch 
master docker container broken
+- [NUTCH-2297](https://issues.apache.org/jira/browse/NUTCH-2297) - 
CrawlDbReader -stats wrong values for earliest fetch time and shortest interval
+- [NUTCH-2316](https://issues.apache.org/jira/browse/NUTCH-2316) - Library 
conflict with Parser-Tika Plugin and Lib Folder
+- [NUTCH-2317](https://issues.apache.org/jira/browse/NUTCH-2317) - Plugin jars 
don't get added to classpath while running in local
+- [NUTCH-2322](https://issues.apache.org/jira/browse/NUTCH-2322) - URL not 
available for Jexl operations
+- [NUTCH-2354](https://issues.apache.org/jira/browse/NUTCH-2354) - Upgrade 
Hadoop dependencies to 2.7.4
+- [NUTCH-2365](https://issues.apache.org/jira/browse/NUTCH-2365) - HTTP 
Redirects to SubDomains don't get crawled if db.ignore.external.links.mode == 
byDomain
+- [NUTCH-2371](https://issues.apache.org/jira/browse/NUTCH-2371) - Injector to 
support noFilter and noNormalize
+- [NUTCH-2372](https://issues.apache.org/jira/browse/NUTCH-2372) - Javadocs 
build failing.
+- [NUTCH-2386](https://issues.apache.org/jira/browse/NUTCH-2386) - 
BasicURLNormalizer does not encode curly braces
+- [NUTCH-2391](https://issues.apache.org/jira/browse/NUTCH-2391) - Spurious 
Duplications for MD5
+- [NUTCH-2394](https://issues.apache.org/jira/browse/NUTCH-2394) - Possible 
bugs in the source code
+- [NUTCH-2398](https://issues.apache.org/jira/browse/NUTCH-2398) - Fetcher 
saving redirected robots.txt under redirect target URL
+- [NUTCH-2399](https://issues.apache.org/jira/browse/NUTCH-2399) - 
indexer-elastic does not index multi-value fields (only the first value is 
indexed)
+- [NUTCH-2401](https://issues.apache.org/jira/browse/NUTCH-2401) - headings 
plugin does not trim values
+- [NUTCH-2403](https://issues.apache.org/jira/browse/NUTCH-2403) - Nutch 
Selenium: Wrong documentation about PhantomJS
+- [NUTCH-2413](https://issues.apache.org/jira/browse/NUTCH-2413) - Parsing 
fetcher to respect property "parse.filter.urls"
+- [NUTCH-2420](https://issues.apache.org/jira/browse/NUTCH-2420) - Bug in 
variable generate.max.count and fetcher.server.delay
+- [NUTCH-2436](https://issues.apache.org/jira/browse/NUTCH-2436) - Remove 
empty comment, and redundant semicolon from CommandRunner
+- [NUTCH-2442](https://issues.apache.org/jira/browse/NUTCH-2442) - Injector to 
stop if job fails to avoid loss of CrawlDb
+- [NUTCH-2444](https://issues.apache.org/jira/browse/NUTCH-2444) - HostDB CSV 
dumper to emit field header by default
+- [NUTCH-2446](https://issues.apache.org/jira/browse/NUTCH-2446) - 
URLFiltersCheck fix
+- [NUTCH-2448](https://issues.apache.org/jira/browse/NUTCH-2448) - Allow 
Sending an empty http.agent.version
+- [NUTCH-2451](https://issues.apache.org/jira/browse/NUTCH-2451) - 
protocol-ftp to resolve relative URL when following redirects
+- [NUTCH-2452](https://issues.apache.org/jira/browse/NUTCH-2452) - Problem 
retrieving encoded URLs via FTP?
+- [NUTCH-2456](https://issues.apache.org/jira/browse/NUTCH-2456) - Allow to 
index pages/URLs not contained in CrawlDb
+- [NUTCH-2458](https://issues.apache.org/jira/browse/NUTCH-2458) - TikaParser 
doesn't work with tika-config.xml set
+- [NUTCH-2464](https://issues.apache.org/jira/browse/NUTCH-2464) - Plugin 
headings: Headers That Contain HTML Elements Are Not Parsed
+- [NUTCH-2465](https://issues.apache.org/jira/browse/NUTCH-2465) - Broken 
Eclipse project. Classpaths and interactiveselenium should be fixed.
+- [NUTCH-2472](https://issues.apache.org/jira/browse/NUTCH-2472) - Sitemap 
processor does not honour db.ignore.external.links
+- [NUTCH-2473](https://issues.apache.org/jira/browse/NUTCH-2473) - 
Elasticsearch REST Indexer broken due to wrong depenency
+- [NUTCH-2474](https://issues.apache.org/jira/browse/NUTCH-2474) - 
CrawlDbReader -stats fails with ClassCastException
+- [NUTCH-2478](https://issues.apache.org/jira/browse/NUTCH-2478) - // is not a 
valid base URL
+- [NUTCH-2483](https://issues.apache.org/jira/browse/NUTCH-2483) - 
Remove/replace indirect dependencies to org.json
+
+### Improvement
+
+- [NUTCH-1763](https://issues.apache.org/jira/browse/NUTCH-1763) - Improving 
comments on the Injector Class
+- [NUTCH-2034](https://issues.apache.org/jira/browse/NUTCH-2034) - CrawlDB 
filtered documents counter.
+- [NUTCH-2035](https://issues.apache.org/jira/browse/NUTCH-2035) - Regex 
filter using case sensitive rules.
+- [NUTCH-2046](https://issues.apache.org/jira/browse/NUTCH-2046) - The crawl 
script should be able to skip an initial injection.
+- [NUTCH-2135](https://issues.apache.org/jira/browse/NUTCH-2135) - Ant Eclipse 
build does not include protocol-interactiveselenium
+- [NUTCH-2193](https://issues.apache.org/jira/browse/NUTCH-2193) - Upgrade 
feed parser plugin to use rome 1.5
+- [NUTCH-2216](https://issues.apache.org/jira/browse/NUTCH-2216) - 
db.ignore.*.links to optionally follow internal redirects
+- [NUTCH-2281](https://issues.apache.org/jira/browse/NUTCH-2281) - Support 
non-default FileSystem
+- [NUTCH-2296](https://issues.apache.org/jira/browse/NUTCH-2296) - 
Elasticsearch Indexing Over Rest
+- [NUTCH-2320](https://issues.apache.org/jira/browse/NUTCH-2320) - 
URLFilterChecker to run as TCP Telnet service
+- [NUTCH-2335](https://issues.apache.org/jira/browse/NUTCH-2335) - Injector 
not to filter and normalize existing URLs in CrawlDb
+- [NUTCH-2362](https://issues.apache.org/jira/browse/NUTCH-2362) - Upgrade 
MaxMind GeoIP version in index-geoip
+- [NUTCH-2368](https://issues.apache.org/jira/browse/NUTCH-2368) - Variable 
generate.max.count and fetcher.server.delay
+- [NUTCH-2370](https://issues.apache.org/jira/browse/NUTCH-2370) - FileDumper: 
save JSON mapping file -> URL
+- [NUTCH-2376](https://issues.apache.org/jira/browse/NUTCH-2376) - Improve 
configurability of HTTP Accept* header fields
+- [NUTCH-2378](https://issues.apache.org/jira/browse/NUTCH-2378) - ChildFirst 
plugin classloader
+- [NUTCH-2380](https://issues.apache.org/jira/browse/NUTCH-2380) - 
indexer-elastic version upgrade to 5.3.0
+- [NUTCH-2397](https://issues.apache.org/jira/browse/NUTCH-2397) - Parser to 
add paragraph line breaks
+- [NUTCH-2400](https://issues.apache.org/jira/browse/NUTCH-2400) - Solr 6.6.0 
compatibility
+- [NUTCH-2406](https://issues.apache.org/jira/browse/NUTCH-2406) - Sum up 
constants, make minor changes
+- [NUTCH-2408](https://issues.apache.org/jira/browse/NUTCH-2408) - CrawlDb: 
allow update from unparsed segments
+- [NUTCH-2409](https://issues.apache.org/jira/browse/NUTCH-2409) - Injector: 
complete command-line help and counters
+- [NUTCH-2414](https://issues.apache.org/jira/browse/NUTCH-2414) - Allow 
LanguageIndexingFilter to actually filter documents by language.
+- [NUTCH-2430](https://issues.apache.org/jira/browse/NUTCH-2430) - Complete 
plugin build configuration
+- [NUTCH-2431](https://issues.apache.org/jira/browse/NUTCH-2431) - 
URLFilterchecker to implement Tool-interface
+- [NUTCH-2439](https://issues.apache.org/jira/browse/NUTCH-2439) - Upgrade to 
Apache Tika 1.17
+- [NUTCH-2443](https://issues.apache.org/jira/browse/NUTCH-2443) - Extract 
links from the video tag with the parse-html plugin
+- [NUTCH-2445](https://issues.apache.org/jira/browse/NUTCH-2445) - Fetcher 
following outlinks to keep track of already fetched items
+- [NUTCH-2463](https://issues.apache.org/jira/browse/NUTCH-2463) - Enable 
sampling CrawlDB
+- [NUTCH-2468](https://issues.apache.org/jira/browse/NUTCH-2468) - should 
filter out invalid URLs by default
+- [NUTCH-2470](https://issues.apache.org/jira/browse/NUTCH-2470) - 
CrawlDbReader -stats to show quantiles of score
+- [NUTCH-2477](https://issues.apache.org/jira/browse/NUTCH-2477) - Refactor 
*Checker classes to use base class for common code
+- [NUTCH-2480](https://issues.apache.org/jira/browse/NUTCH-2480) - Upgrade 
crawler-commons dependency to 0.9
+
+### New Feature
+
+- [NUTCH-1465](https://issues.apache.org/jira/browse/NUTCH-1465) - Support 
sitemaps in Nutch
+- [NUTCH-1932](https://issues.apache.org/jira/browse/NUTCH-1932) - 
Automatically remove orphaned pages
+- [NUTCH-2333](https://issues.apache.org/jira/browse/NUTCH-2333) - Indexer for 
RabbitMQ
+- [NUTCH-2338](https://issues.apache.org/jira/browse/NUTCH-2338) - 
URLNormalizerChecker to run as TCP Telnet service
+- [NUTCH-2415](https://issues.apache.org/jira/browse/NUTCH-2415) - Create a 
JEXL based IndexingFilter
+- [NUTCH-2433](https://issues.apache.org/jira/browse/NUTCH-2433) - Html 
Parser: keep htmltag where the outlinks are found
+- [NUTCH-2435](https://issues.apache.org/jira/browse/NUTCH-2435) - New 
configuration allowing to choose whether to store 'parse_text' directory or not.
+- [NUTCH-2484](https://issues.apache.org/jira/browse/NUTCH-2484) - Extend 
indexer-elastic-rest to support languages
+
+### Task
+
+- [NUTCH-2181](https://issues.apache.org/jira/browse/NUTCH-2181) - Add Webpage 
for 3rd Party Connectors/Libraries to Apache Nutch
+
+
+## Nutch 1.13 Release 28/03/2017 (dd/mm/yyyy)
+Release Report: https://s.apache.org/wq3x
+
+### Sub-task
+
+- [NUTCH-2246](https://issues.apache.org/jira/browse/NUTCH-2246) - Refactor 
/seed endpoint for backward compatibility
+
+### Bug
+
+- [NUTCH-1553](https://issues.apache.org/jira/browse/NUTCH-1553) - Property 
'indexer.delete.robots.noindex' not working when using parser-html.
+- [NUTCH-2242](https://issues.apache.org/jira/browse/NUTCH-2242) - 
lastModified not always set
+- [NUTCH-2291](https://issues.apache.org/jira/browse/NUTCH-2291) - Fix mrunit 
dependencies
+- [NUTCH-2337](https://issues.apache.org/jira/browse/NUTCH-2337) - 
urlnormalizer-basic to strip empty port
+- [NUTCH-2345](https://issues.apache.org/jira/browse/NUTCH-2345) - 
FetchItemQueue logs are logged with wrong class name
+- [NUTCH-2349](https://issues.apache.org/jira/browse/NUTCH-2349) - 
urlnormalizer-basic NPE for ill-formed URL "http:/"
+- [NUTCH-2357](https://issues.apache.org/jira/browse/NUTCH-2357) - Index 
metadata throw Exception because writable object cannot be cast to Text
+- [NUTCH-2359](https://issues.apache.org/jira/browse/NUTCH-2359) - 
Parsefilter-regex raises IndexOutOfBoundsException when rules are ill-formed
+- [NUTCH-2364](https://issues.apache.org/jira/browse/NUTCH-2364) - 
http.agent.rotate: IllegalArgumentException / last element of agent names 
ignored
+- [NUTCH-2366](https://issues.apache.org/jira/browse/NUTCH-2366) - Deprecated 
Job constructor in hostdb/ReadHostDb.java
+
+### Improvement
+
+- [NUTCH-1308](https://issues.apache.org/jira/browse/NUTCH-1308) - Add main() 
to ZipParser
+- [NUTCH-2164](https://issues.apache.org/jira/browse/NUTCH-2164) - 
Inconsistent 'Modified Time' in crawl db
+- [NUTCH-2234](https://issues.apache.org/jira/browse/NUTCH-2234) - Upgrade to 
elasticsearch 2.3.3
+- [NUTCH-2236](https://issues.apache.org/jira/browse/NUTCH-2236) - Upgrade to 
Hadoop 2.7.2
+- [NUTCH-2262](https://issues.apache.org/jira/browse/NUTCH-2262) - Utilize 
parameterized logging notation across Fetcher
+- [NUTCH-2272](https://issues.apache.org/jira/browse/NUTCH-2272) - Index 
checker server to optionally keep client connection open
+- [NUTCH-2286](https://issues.apache.org/jira/browse/NUTCH-2286) - 
CrawlDbReader -stats to show fetch time and interval
+- [NUTCH-2287](https://issues.apache.org/jira/browse/NUTCH-2287) - 
Indexer-elastic plugin should use Elasticsearch BulkProcessor and BackoffPolicy
+- [NUTCH-2299](https://issues.apache.org/jira/browse/NUTCH-2299) - Remove 
obsolete properties protocol.plugin.check.*
+- [NUTCH-2300](https://issues.apache.org/jira/browse/NUTCH-2300) - Fetcher to 
optionally save robots.txt
+- [NUTCH-2327](https://issues.apache.org/jira/browse/NUTCH-2327) - Seeds 
injected in REST workflow must be ingested into HDFS
+- [NUTCH-2329](https://issues.apache.org/jira/browse/NUTCH-2329) - Update 
Slf4j logging for Java 8 and upgrade miredot plugin version
+- [NUTCH-2336](https://issues.apache.org/jira/browse/NUTCH-2336) - 
SegmentReader to implement Tool
+- [NUTCH-2352](https://issues.apache.org/jira/browse/NUTCH-2352) - Log with 
Generic Class Name at Nutch 1.x
+- [NUTCH-2355](https://issues.apache.org/jira/browse/NUTCH-2355) - Protocol 
plugins to set cookie if Cookie metadata field is present
+- [NUTCH-2367](https://issues.apache.org/jira/browse/NUTCH-2367) - Get single 
record from HostDB
+
+### New Feature
+
+- [NUTCH-2132](https://issues.apache.org/jira/browse/NUTCH-2132) - 
Publisher/Subscriber model for Nutch to emit events
+
+### Task
+
+- [NUTCH-2171](https://issues.apache.org/jira/browse/NUTCH-2171) - Upgrade 
Nutch Trunk to Java 1.8
+
+ 
+## Nutch 1.12 Release 28/05/2016 (dd/mm/yyyy)
+Release Report: https://s.apache.org/nutch1.12
+
+### Breaking Changes
+
+Fellow committers, Nutch 1.12 contains a breaking change 
[NUTCH-2220](https://issues.apache.org/jira/browse/NUTCH-2220).
+Please use the note below and in the release announcement and keep it on top 
in this CHANGES.txt for the Nutch 1.12 release.
+
+* replace your old conf/nutch-default.xml with the conf/nutch-default.xml from 
Nutch 1.12 release
+* if you use LinkDB (e.g. invertlinks) and modified parameters db.max.inlinks 
and/or db.max.anchor.length
+  and/or db.ignore.internal.links, rename those parameters to 
linkdb.max.inlinks and
+  linkdb.max.anchor.length and linkdb.ignore.internal.links
+* db.ignore.internal.links and db.ignore.external.links now operate on the 
CrawlDB only
+* linkdb.ignore.internal.links and linkdb.ignore.external.links now operate on 
the LinkDB only
+
+### Sub-task
+
+- [NUTCH-2250](https://issues.apache.org/jira/browse/NUTCH-2250) - 
CommonCrawlDumper : Invalid format + skipped parts
+
+### Bug
+
+- [NUTCH-2042](https://issues.apache.org/jira/browse/NUTCH-2042) - parse-html 
increase chunk size used to detect charset
+- [NUTCH-2180](https://issues.apache.org/jira/browse/NUTCH-2180) - FileDumper 
dumps data, but breaks midway on corrupt segments
+- [NUTCH-2189](https://issues.apache.org/jira/browse/NUTCH-2189) - Domain 
filter must deactivate if no rules are present
+- [NUTCH-2203](https://issues.apache.org/jira/browse/NUTCH-2203) - Suffix URL 
filter can't handle trailing/leading whitespaces
+- [NUTCH-2206](https://issues.apache.org/jira/browse/NUTCH-2206) - Provide 
example scoring.similarity.stopword.file
+- [NUTCH-2213](https://issues.apache.org/jira/browse/NUTCH-2213) - 
CommonCrawlDataDumper saves gzipped body in extracted form
+- [NUTCH-2223](https://issues.apache.org/jira/browse/NUTCH-2223) - Upgrade 
xercesImpl to 2.11.0 to fix hang on issue in tika mimetype detection
+- [NUTCH-2224](https://issues.apache.org/jira/browse/NUTCH-2224) - Average 
bytes/second calculated incorrectly in fetcher
+- [NUTCH-2225](https://issues.apache.org/jira/browse/NUTCH-2225) - Parsed time 
calculated incorrectly
+- [NUTCH-2228](https://issues.apache.org/jira/browse/NUTCH-2228) - Plugin 
index-replace unit test broken on Java 8
+- [NUTCH-2232](https://issues.apache.org/jira/browse/NUTCH-2232) - 
DeduplicationJob should decode URL's before length is compared
+- [NUTCH-2241](https://issues.apache.org/jira/browse/NUTCH-2241) - Unstable 
Selenium plugin in Nutch. Fixed bugs and enhanced configuration
+- [NUTCH-2256](https://issues.apache.org/jira/browse/NUTCH-2256) - 
Inconsistent log level practice
+
+### Improvement
+
+- [NUTCH-1233](https://issues.apache.org/jira/browse/NUTCH-1233) - Rely on 
Tika for outlink extraction
+- [NUTCH-1712](https://issues.apache.org/jira/browse/NUTCH-1712) - Use 
MultipleInputs in Injector to make it a single mapreduce job
+- [NUTCH-2172](https://issues.apache.org/jira/browse/NUTCH-2172) - index-more: 
document format of contenttype-mapping.txt
+- [NUTCH-2178](https://issues.apache.org/jira/browse/NUTCH-2178) - 
DeduplicationJob to optionally group on host or domain
+- [NUTCH-2182](https://issues.apache.org/jira/browse/NUTCH-2182) - Make 
reverseUrlDirs file dumper option hash the URL for consistency
+- [NUTCH-2183](https://issues.apache.org/jira/browse/NUTCH-2183) - Improvement 
to SegmentChecker for skipping non-segments present in segments directory
+- [NUTCH-2187](https://issues.apache.org/jira/browse/NUTCH-2187) - Change 
FileDumper SHAs to all uppercase
+- [NUTCH-2195](https://issues.apache.org/jira/browse/NUTCH-2195) - 
IndexingFilterChecker to optionally follow N redirects
+- [NUTCH-2196](https://issues.apache.org/jira/browse/NUTCH-2196) - 
IndexingFilterChecker to optionally normalize
+- [NUTCH-2197](https://issues.apache.org/jira/browse/NUTCH-2197) - Add solr5 
solrcloud indexer support
+- [NUTCH-2204](https://issues.apache.org/jira/browse/NUTCH-2204) - Remove 
junit lib from runtime
+- [NUTCH-2218](https://issues.apache.org/jira/browse/NUTCH-2218) - Switch 
CrawlCompletion arg parsing to Commons CLI
+- [NUTCH-2221](https://issues.apache.org/jira/browse/NUTCH-2221) - Introduce 
db.ignore.internal.links to FetcherThread
+- [NUTCH-2229](https://issues.apache.org/jira/browse/NUTCH-2229) - Allow Jexl 
expressions on CrawlDatum's fixed attributes
+- [NUTCH-2231](https://issues.apache.org/jira/browse/NUTCH-2231) - Jexl 
support in generator job
+- [NUTCH-2252](https://issues.apache.org/jira/browse/NUTCH-2252) - Allow 
phantomjs as a browser for selenium options
+- [NUTCH-2263](https://issues.apache.org/jira/browse/NUTCH-2263) - Support for 
mingram and maxgram at Unigram Cosine Similarity Model
+
+### New Feature
+
+- [NUTCH-961](https://issues.apache.org/jira/browse/NUTCH-961) - Expose Tika's 
boilerpipe support
+- [NUTCH-1325](https://issues.apache.org/jira/browse/NUTCH-1325) - HostDB for 
Nutch
+- [NUTCH-2144](https://issues.apache.org/jira/browse/NUTCH-2144) - Plugin to 
override db.ignore.external to exempt interesting external domain URLs
+- [NUTCH-2190](https://issues.apache.org/jira/browse/NUTCH-2190) - Protocol 
normalizer
+- [NUTCH-2191](https://issues.apache.org/jira/browse/NUTCH-2191) - Add 
protocol-htmlunit
+- [NUTCH-2194](https://issues.apache.org/jira/browse/NUTCH-2194) - Run 
IndexingFilterChecker as simple Telnet server
+- [NUTCH-2219](https://issues.apache.org/jira/browse/NUTCH-2219) - Criteria 
order to be configurable in DeduplicationJob
+- [NUTCH-2227](https://issues.apache.org/jira/browse/NUTCH-2227) - 
RegexParseFilter
+- [NUTCH-2245](https://issues.apache.org/jira/browse/NUTCH-2245) - Developed 
the NGram Model on the existing Unigram Cosine Similarity Model
+
+### Task
+
+- [NUTCH-2201](https://issues.apache.org/jira/browse/NUTCH-2201) - Remove 
loops program from webgraph package
+- [NUTCH-2211](https://issues.apache.org/jira/browse/NUTCH-2211) - Filter and 
normalizer checkers missing in bin/nutch
+- [NUTCH-2220](https://issues.apache.org/jira/browse/NUTCH-2220) - Rename db.* 
options used only by the linkdb to linkdb.*
+
+
+## Nutch 1.11 Release 03/12/2015 (dd/mm/yyyy)
+Release Report: http://s.apache.org/nutch11
+
+- [NUTCH-2176](https://issues.apache.org/jira/browse/NUTCH-2176) - Clean up of 
log4j.properties (markus)
+
+- [NUTCH-2107](https://issues.apache.org/jira/browse/NUTCH-2107) - plugin.xml 
to validate against plugin.dtd (snagel)
+
+- [NUTCH-2177](https://issues.apache.org/jira/browse/NUTCH-2177) - Generator 
produces only one partition even in distributed mode (jnioche, snagel)
+
+- [NUTCH-2158](https://issues.apache.org/jira/browse/NUTCH-2158) - Upgrade to 
Tika 1.11 (jnioche, snagel)
+
+- [NUTCH-2175](https://issues.apache.org/jira/browse/NUTCH-2175) - Typos in 
property descriptions in nutch-default.xml (Roannel Fernández Hernández via 
snagel)
+
+- [NUTCH-2069](https://issues.apache.org/jira/browse/NUTCH-2069) - Ignore 
external links based on domain (jnioche)
+
+- [NUTCH-2173](https://issues.apache.org/jira/browse/NUTCH-2173) - String.join 
in FileDumper breaks the build (joyce)
+
+- [NUTCH-2166](https://issues.apache.org/jira/browse/NUTCH-2166) - Add reverse 
URL format to dump tool (joyce)
+
+- [NUTCH-2157](https://issues.apache.org/jira/browse/NUTCH-2157) - Addressing 
Miredot REST API Warnings (Sujen Shah)
+
+- [NUTCH-2165](https://issues.apache.org/jira/browse/NUTCH-2165) - FileDumper 
Util hard codes part-# folder name (joyce)
+
+- [NUTCH-2167](https://issues.apache.org/jira/browse/NUTCH-2167) - Backport 
TableUtil from 2.x for URL reversing (joyce)
+
+- [NUTCH-2160](https://issues.apache.org/jira/browse/NUTCH-2160) - Upgrade 
Selenium Java to 2.48.2 (lewismc, kwhitehall)
+
+- [NUTCH-2120](https://issues.apache.org/jira/browse/NUTCH-2120) - Remove 
MapWritable from trunk codebase (lewismc)
+
+- [NUTCH-1911](https://issues.apache.org/jira/browse/NUTCH-1911) - Improve 
DomainStatistics tool command line parsing (joyce)
+
+- [NUTCH-2064](https://issues.apache.org/jira/browse/NUTCH-2064) - 
URLNormalizer basic to encode reserved chars and decode non-reserved chars 
(markus, snagel)
+
+- [NUTCH-2159](https://issues.apache.org/jira/browse/NUTCH-2159) - Ensure that 
all WebApp files are copied into generated artifacts for 1.X Webapp (lewismc)
+
+- [NUTCH-2154](https://issues.apache.org/jira/browse/NUTCH-2154) - Nutch REST 
API (DB) suffering NullPointerException (Aron Ahmadia, Sujen Shah via mattmann)
+
+- [NUTCH-2150](https://issues.apache.org/jira/browse/NUTCH-2150) - Add 
protocolstats utility (Michael Joyce via mattmann)
+
+- [NUTCH-2146](https://issues.apache.org/jira/browse/NUTCH-2146) - hashCode on 
the Outlink class (jorgelbg via mattmann)
+
+- [NUTCH-2155](https://issues.apache.org/jira/browse/NUTCH-2155) - Create a 
"crawl completeness" utility (Michael Joyce via mattmann)
+
+- [NUTCH-1988](https://issues.apache.org/jira/browse/NUTCH-1988) - Make nested 
output directory dump optional... again (Michael Joyce via lewismc)
+
+- [NUTCH-1800](https://issues.apache.org/jira/browse/NUTCH-1800) - 
Documentation for Nutch 1.X and 2.X REST APIs (lewismc)
+
+- [NUTCH-2149](https://issues.apache.org/jira/browse/NUTCH-2149) - REST 
endpoint to read Nutch sequence files (Sujen Shah)
+
+- [NUTCH-2139](https://issues.apache.org/jira/browse/NUTCH-2139) - Basic 
plugin to index inlinks and outlinks (jorgelbg)
+
+- [NUTCH-2128](https://issues.apache.org/jira/browse/NUTCH-2128) - Review and 
update mapred --> mapreduce config params in crawl script (lewismc)
+
+- [NUTCH-2141](https://issues.apache.org/jira/browse/NUTCH-2141) - Change the 
InteractiveSelenium plugin handler Interface to return page content (Balaji 
Gurumurthy via mattmann)
+
+- [NUTCH-2129](https://issues.apache.org/jira/browse/NUTCH-2129) - Add 
protocol status tracking to crawl datum (Michael Joyce via mattmann)
+
+- [NUTCH-2142](https://issues.apache.org/jira/browse/NUTCH-2142) - Nutch File 
Dump - FileNotFoundException (Invalid Argument) Error (Karanjeet Singh via 
mattmann)
+
+- [NUTCH-2136](https://issues.apache.org/jira/browse/NUTCH-2136) - Implement a 
different version of Naive Bayes Parse Filter (Asitang Mishra)
+
+- [NUTCH-2109](https://issues.apache.org/jira/browse/NUTCH-2109) - Create a 
brute force click-all-ajax-links utility fucntion for selenium interactive 
plugin (Asitang Mishra)
+
+- [NUTCH-2108](https://issues.apache.org/jira/browse/NUTCH-2108) - Add a 
function to the selenium interactive plugin interface to do multiple 
manipulation of driver and then return the data (Asitang Mishra)
+
+- [NUTCH-2124](https://issues.apache.org/jira/browse/NUTCH-2124) - Fetcher 
following same redirect again and again (Yogendra Kumar Soni via snagel)
+
+- [NUTCH-2123](https://issues.apache.org/jira/browse/NUTCH-2123) - Seed List 
REST API returns Text but headers indicate/require JSON (Aron Ahmadia, Sujen 
Shah via mattmann)
+
+- [NUTCH-2086](https://issues.apache.org/jira/browse/NUTCH-2086) - Nutch 1.X 
Webui (Sujen Shah, mattmann via lewismc)
+
+- [NUTCH-2121](https://issues.apache.org/jira/browse/NUTCH-2121) - Update 
javadoc link for Hadoop 2.4.0 in default.properties (Sujen Shah)
+

[... 2416 lines stripped ...]


Reply via email to