GitHub user thammegowda opened a pull request:
https://github.com/apache/nutch/pull/103
NUTCH-2250 : CommonCrawlDumper : Invalid format and skipped parts
### Summary
+ Reads all parts of segments
+ writes single document per dump file
### What was the fix?
1. Lists all files insides segments and selects all the parts matching to
pattern part-[0-9]{5} under the content directory.
2. CommonCrawlFormat implementation for "JACKSON" type does not clear
state, instead it keeps appending documents. This fix uses a new instance for
each document (actually it was hard to resolve the state issue because of too
many options such as compressing and archiving).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/thammegowda/nutch NUTCH-2250
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nutch/pull/103.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #103
----
commit 47cc4e27a41a78753ed431685010da1987bca269
Author: Thamme Gowda <tgow...@gmail.com>
Date: 2016-04-14T09:38:30Z
NUTCH-2250 : CommonCrawlDumper : Invalid format and skipped parts
+ Reads all parts of segements
+ FIX : writes only one document to dump file
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
