Hi Sebastian, Thank you for your response. It was a great help. I didn't realise that it is intended for users to edit the bin/crawl file. Although looking at it now it's clear.
This makes it easier for me to access the html content within my plugin, thanks again On Fri, May 28, 2021 at 8:36 PM Sebastian Nagel <[email protected]> wrote: > Hi Kieran, > > see the command-line options > > -addBinaryContent > index raw/binary content in field `binaryContent` > -base64 > use Base64 encoding for binary content > > of the Nutch index job [1]. Note that the content maybe indeed > binary, eg. for PDF documents but also for HTML pages which use > a different encoding than UTF-8. > > Best, > Sebastian > > [1] > https://wiki.apache.org/confluence/pages/viewpage.action?pageId=122916842 > > > On 5/28/21 5:28 PM, Kieran Munday wrote: > > Hi users@, > > > > I am new to Nutch (v.1.17) and my current project requires the indexing > of > > the html of crawled pages. It also requires fields that can be derived > from > > the raw html such as image count, and charset. > > > > I have looked on StackOverflow for how to achieve this and most people > from > > my understanding seem to be recommending processing the segments to > extract > > the html and modify the documents post-crawl. This doesn't fit my use > case > > as I need to calculate these fields at crawl time before they are indexed > > into Elasticsearch. > > > > The other recommendations I have seen mention creating a plugin to > override > > the parse-html plugin. However, I have found rather limited documentation > > on how to do this correctly and am not sure on how to return from the > > plugin in a way that the field propagates into the NutchDocument which > will > > be processed in the Indexers' write method. > > > > Do any of you have any advice or links to documentation that explains how > > to modify what gets set in the NutchDocument? > > > > Thank you in advance > > > >
