[ https://issues.apache.org/jira/browse/NUTCH-2865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17446676#comment-17446676 ]
Sebastian Nagel commented on NUTCH-2865: ---------------------------------------- Hi [~markus17], looks good and tests went successfully. I've made two changes in [6bb30c7|https://github.com/sebastian-nagel/nutch/commit/6bb30c719550f6d7b8efc127c052ed1a21816321]: - add the Content-Type to metadata and text records. So, it's now: -- metadata: {{WARC-Type: metadata\r\nContent-Type: application/json}}, cf. [WARC "metadata" record|https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/#metadata] -- text: {{WARC-Type: conversion\r\nContent-Type: text/plain}}, cf. [WARC "conversion" record|https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/#conversion] - apply code formats If there are no objections I'll commit this. > WARC exporter support for metadata and dropping empty responses > --------------------------------------------------------------- > > Key: NUTCH-2865 > URL: https://issues.apache.org/jira/browse/NUTCH-2865 > Project: Nutch > Issue Type: Improvement > Reporter: Markus Jelsma > Assignee: Markus Jelsma > Priority: Minor > Fix For: 1.19 > > Attachments: NUTCH-2865.patch, NUTCH-2865.patch, NUTCH-2865.patch > > > WARCExporter is a handy tool to dump the segments. Unfortunately it also > emits WARC records for status' other than success of notmodified, which > accounts for a decent number in each crawl cycle. It also doesn't emit parsed > metadata or extracted text. It does now. > > This patch adds three switches: > * -includeOnlySuccessfulResponses to only emit records of success or > notmodified > * -includeParseData to also emit parse metadata as WARC metadata record > * -includeParseText to also emit extracted text as WARC metadata > Both metadata objects are stored in the same WARC metadata record to save > space. -- This message was sent by Atlassian Jira (v8.20.1#820001)