Use XML then. Although you will need to escape the XML special characters as I did in the pattern.

The point is simply: Quickly and simply try to find the simple test scenario that illustrates the problem.

-- Jack Krupansky

-----Original Message----- From: Andreas Owen
Sent: Monday, September 09, 2013 7:05 PM
To: solr-user@lucene.apache.org
Subject: Re: charfilter doesn't do anything

i tried but that isn't working either, it want a data-stream, i'll have to check how to post json instead of xml

On 10. Sep 2013, at 12:52 AM, Jack Krupansky wrote:

Did you at least try the pattern I gave you?

The point of the curl was the data, not how you send the data. You can just use the standard Solr simple post tool.

-- Jack Krupansky

-----Original Message----- From: Andreas Owen
Sent: Monday, September 09, 2013 6:40 PM
To: solr-user@lucene.apache.org
Subject: Re: charfilter doesn't do anything

i've downloaded curl and tried it in the comman prompt and power shell on my win 2008r2 server, thats why i used my dataimporter with a single line html file and copy/pastet the lines into schema.xml


On 9. Sep 2013, at 11:20 PM, Jack Krupansky wrote:

Did you in fact try my suggested example? If not, please do so.

-- Jack Krupansky

-----Original Message----- From: Andreas Owen
Sent: Monday, September 09, 2013 4:42 PM
To: solr-user@lucene.apache.org
Subject: Re: charfilter doesn't do anything

i index html pages with a lot of lines and not just a string with the body-tag. it doesn't work with proper html files, even though i took all the new lines out.

html-file:
<html>nav-content<body> nur das will ich sehen</body>footer-content</html>

solr update debug output:
"text_html": ["<html>\r\n\r\n<meta name=\"Content-Encoding\" content=\"ISO-8859-1\">\r\n<meta name=\"Content-Type\" content=\"text/html; charset=ISO-8859-1\">\r\n<title></title>\r\n\r\n<body>nav-content nur das will ich sehenfooter-content</body></html>"]



On 8. Sep 2013, at 3:28 PM, Jack Krupansky wrote:

I tried this and it seems to work when added to the standard Solr example in 4.4:

<field name="body" type="text_html_body" indexed="true" stored="true" />

<fieldType name="text_html_body" class="solr.TextField" positionIncrementGap="100" >
<analyzer>
<charFilter class="solr.PatternReplaceCharFilterFactory" pattern="^.*&lt;body&gt;(.*)&lt;/body&gt;.*$" replacement="$1" />
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>

That char filter retains only text between <body> and </body>. Is that what you wanted?

Indexing this data:

curl 'localhost:8983/solr/update?commit=true' -H 'Content-type:application/json' -d '
[{"id":"doc-1","body":"abc <body>A test.</body> def"}]'

And querying with these commands:

curl "http://localhost:8983/solr/select/?q=*:*&indent=true&wt=json";
Shows all data

curl "http://localhost:8983/solr/select/?q=body:test&indent=true&wt=json";
shows the body text

curl "http://localhost:8983/solr/select/?q=body:abc&indent=true&wt=json";
shows nothing (outside of body)

curl "http://localhost:8983/solr/select/?q=body:def&indent=true&wt=json";
shows nothing (outside of body)

curl "http://localhost:8983/solr/select/?q=body:body&indent=true&wt=json";
Shows nothing, HTML tag stripped

In your original query, you didn't show us what your default field, df parameter, was.

-- Jack Krupansky

-----Original Message----- From: Andreas Owen
Sent: Sunday, September 08, 2013 5:21 AM
To: solr-user@lucene.apache.org
Subject: Re: charfilter doesn't do anything

yes but that filter html and not the specific tag i want.

On 7. Sep 2013, at 7:51 PM, Erick Erickson wrote:

Hmmm, have you looked at:
http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.HTMLStripCharFilterFactory

Not quite the <body>, perhaps, but might it help?


On Fri, Sep 6, 2013 at 11:33 AM, Andreas Owen <a...@conx.ch> wrote:

ok i have html pages with <html>.....<!--body-->content i
want....<!--/body-->.....</html>. i want to extract (index, store) only that between the body-comments. i thought regexTransformer would be the
best because xpath doesn't work in tika and i cant nest a
xpathEntetyProcessor to use xpath. what i have also found out is that the
htmlparser from tika cuts my body-comments out and tries to make well
formed html, which i would like to switch off.

On 6. Sep 2013, at 5:04 PM, Shawn Heisey wrote:

On 9/6/2013 7:09 AM, Andreas Owen wrote:
i've managed to get it working if i use the regexTransformer and string
is on the same line in my tika entity. but when the string is multilined it
isn't working even though i tried ?s to set the flag dotall.

<entity name="tika" processor="TikaEntityProcessor" url="${rec.url}"
dataSource="dataUrl" onError="skip" htmlMapper="identity" format="html"
transformer="RegexTransformer">
 <field column="text_html" regex="&lt;body&gt;(.+)&lt;/body&gt;"
replaceWith="QQQQQQQQQQQQQQQ" sourceColName="text"  />
</entity>

then i tried it like this and i get a stackoverflow

<field column="text_html" regex="&lt;body&gt;((.|\n|\r)+)&lt;/body&gt;"
replaceWith="QQQQQQQQQQQQQQQ" sourceColName="text"  />

in javascript this works but maybe because i only used a small string.

Sounds like we've got an XY problem here.

http://people.apache.org/~hossman/#xyproblem

How about you tell us *exactly* what you'd actually like to have happen
and then we can find a solution for you?

It sounds a little bit like you're interested in stripping all the HTML
tags out.  Perhaps the HTMLStripCharFilter?


http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.HTMLStripCharFilterFactory

Something that I already said: By using the KeywordTokenizer, you won't be able to search for individual words on your HTML input. The entire
input string is treated as a single token, and therefore ONLY exact
entire-field matches (or certain wildcard matches) will be possible.


http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.KeywordTokenizerFactory

Note that no matter what you do to your data with the analysis chain,
Solr will always return the text that was originally indexed in search
results.  If you need to affect what gets stored as well, perhaps you
need an Update Processor.

Thanks,
Shawn

Reply via email to