[jira] [Updated] (NUTCH-2865) WARC exporter support for metadata and dropping empty responses
[ https://issues.apache.org/jira/browse/NUTCH-2865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2865: - Description: WARCExporter is a handy tool to dump the segments. Unfortunately it also emits WARC records for status' other than success of notmodified, which accounts for a decent number in each crawl cycle. It also doesn't emit parsed metadata or extracted text. It does now. This patch adds three switches: * -includeOnlySuccessfulResponses to only emit records of success or notmodified * -includeParseData to also emit parse metadata as WARC metadata record * -includeParseText to also emit extracted text as WARC metadata Both metadata objects are stored in the same WARC metadata record to save space. was: WARCExporter is a handy tool to dump the segments. Unfortunately it also emits WARC records for status' other than success of notmodified, which accounts for a decent number in each crawl cycle. It also doesn't emit parsed metadata or extracted text. It does now. This patch adds three switches: * -omitEmptyResponses to only emit records of success or notmodified * -includeParseData to also emit parse metadata as WARC metadata record * -includeParseText to also emit extracted text as WARC metadata Both metadata objects are stored in the same WARC metadata record to save space. > WARC exporter support for metadata and dropping empty responses > --- > > Key: NUTCH-2865 > URL: https://issues.apache.org/jira/browse/NUTCH-2865 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.19 > > Attachments: NUTCH-2865.patch, NUTCH-2865.patch > > > WARCExporter is a handy tool to dump the segments. Unfortunately it also > emits WARC records for status' other than success of notmodified, which > accounts for a decent number in each crawl cycle. It also doesn't emit parsed > metadata or extracted text. It does now. > > This patch adds three switches: > * -includeOnlySuccessfulResponses to only emit records of success or > notmodified > * -includeParseData to also emit parse metadata as WARC metadata record > * -includeParseText to also emit extracted text as WARC metadata > Both metadata objects are stored in the same WARC metadata record to save > space. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NUTCH-2865) WARC exporter support for metadata and dropping empty responses
[ https://issues.apache.org/jira/browse/NUTCH-2865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2865: - Attachment: NUTCH-2865.patch > WARC exporter support for metadata and dropping empty responses > --- > > Key: NUTCH-2865 > URL: https://issues.apache.org/jira/browse/NUTCH-2865 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.19 > > Attachments: NUTCH-2865.patch, NUTCH-2865.patch, NUTCH-2865.patch > > > WARCExporter is a handy tool to dump the segments. Unfortunately it also > emits WARC records for status' other than success of notmodified, which > accounts for a decent number in each crawl cycle. It also doesn't emit parsed > metadata or extracted text. It does now. > > This patch adds three switches: > * -includeOnlySuccessfulResponses to only emit records of success or > notmodified > * -includeParseData to also emit parse metadata as WARC metadata record > * -includeParseText to also emit extracted text as WARC metadata > Both metadata objects are stored in the same WARC metadata record to save > space. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NUTCH-2865) WARC exporter support for metadata and dropping empty responses
[ https://issues.apache.org/jira/browse/NUTCH-2865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17359353#comment-17359353 ] Markus Jelsma commented on NUTCH-2865: -- * separated the parseText and parseData * changed the response flag * contentMeta and parseMeta are now added as valid JSON using GSON * Refers-To field added It 's all not very beautiful code, and some partial duplicate pieces, but it works. > WARC exporter support for metadata and dropping empty responses > --- > > Key: NUTCH-2865 > URL: https://issues.apache.org/jira/browse/NUTCH-2865 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.19 > > Attachments: NUTCH-2865.patch, NUTCH-2865.patch, NUTCH-2865.patch > > > WARCExporter is a handy tool to dump the segments. Unfortunately it also > emits WARC records for status' other than success of notmodified, which > accounts for a decent number in each crawl cycle. It also doesn't emit parsed > metadata or extracted text. It does now. > > This patch adds three switches: > * -includeOnlySuccessfulResponses to only emit records of success or > notmodified > * -includeParseData to also emit parse metadata as WARC metadata record > * -includeParseText to also emit extracted text as WARC metadata > Both metadata objects are stored in the same WARC metadata record to save > space. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (NUTCH-2865) WARC exporter support for metadata and dropping empty responses
[ https://issues.apache.org/jira/browse/NUTCH-2865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17359353#comment-17359353 ] Markus Jelsma edited comment on NUTCH-2865 at 6/8/21, 1:28 PM: --- * separated the parseText and parseData * changed the response flag * contentMeta and parseMeta are now added as valid JSON using GSON * Refers-To field added It 's all not very beautiful code, and some partial duplicate pieces, but it works. But with GSON we can choose to remove that as a dependency for some of our plugins. was (Author: markus17): * separated the parseText and parseData * changed the response flag * contentMeta and parseMeta are now added as valid JSON using GSON * Refers-To field added It 's all not very beautiful code, and some partial duplicate pieces, but it works. > WARC exporter support for metadata and dropping empty responses > --- > > Key: NUTCH-2865 > URL: https://issues.apache.org/jira/browse/NUTCH-2865 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.19 > > Attachments: NUTCH-2865.patch, NUTCH-2865.patch, NUTCH-2865.patch > > > WARCExporter is a handy tool to dump the segments. Unfortunately it also > emits WARC records for status' other than success of notmodified, which > accounts for a decent number in each crawl cycle. It also doesn't emit parsed > metadata or extracted text. It does now. > > This patch adds three switches: > * -includeOnlySuccessfulResponses to only emit records of success or > notmodified > * -includeParseData to also emit parse metadata as WARC metadata record > * -includeParseText to also emit extracted text as WARC metadata > Both metadata objects are stored in the same WARC metadata record to save > space. -- This message was sent by Atlassian Jira (v8.3.4#803005)