Sebastian Nagel created NUTCH-3250:
--------------------------------------
Summary: CommonCrawlDataDumper fails to read segment
Key: NUTCH-3250
URL: https://issues.apache.org/jira/browse/NUTCH-3250
Project: Nutch
Issue Type: Bug
Components: dumpers
Affects Versions: 1.23
Reporter: Sebastian Nagel
Fix For: 1.24
(seen while testing / reproducing NUTCH-3249)
CommonCrawlDataDumper uses a fixed pattern how SequenceFile partitions are
named:
{noformat}
part-[0-9]{5}
{noformat}
This is the pattern used in the "old" MapReduce API which has been replaced
already in 2018 (Nutch 1.15, NUTCH-1219).
Effectively, all segments written with Nutch 1.15 and upwards fail to be
processed by CommonCrawlDataDumper.
Ideally, no assumptions about naming of partitions should be made and the
Hadoop classes should be used to read the input.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)