On Wed, 7 Nov 2007 20:18:25 -0800 (PST)
David Neubert <[EMAIL PROTECTED]> wrote:
> I am sure this is 101 question, but I am bit confused about indexing xml data
> using SOLR.
>
> I have rich xml content (books) that need to searched at granular levels
> (specifically paragraph and sentence levels very accurately, no
> approximations). My source text has exact <p></p> and <s></s> tags for this
> purpose. I have built this app in previous versions (using other search
> engines) indexing the text twice, (1) where every paragraph was a virtual
> document and (2) where every sentence was a virtual document -- both
> extracted from the source file (which was a singe xml file for the entire
> book). I have of course thought about using an XML engine eXists or
> Xindices, but I am prefer to the stability and user base and performance that
> Lucene/SOLR seems to have, and also there is a large body of text that is
> regular documents and not well formed XML as well.
>
> I am brand new to SOLR (one day) and at a basic level understand SOLR's nice
> simple xml scheme to add documents:
>
> <add>
> <doc>
> <field name="foo1">foo value 1</field>
> <field name="foo2">foo value 2</field>
> </doc>
> <doc>...</doc>
> </add>
>
> But my problem is that I believe I need to perserve the xml markup at the
> paragraph and sentence levels, so I was hoping to create a content field that
> could just contain the source xml for the paragraph or sentence respectively.
> There are reasons for this that I won't go into -- alot of granular work in
> this app, accessing pars and sens.
>
> Obviously an XML mechanism that could leverage the xml structure (via XPath
> or XPointers) would work great. Still I think Lucene can do this in a field
> level way-- and I also can't imagine that users who are indexing XML
> documents have to go through the trouble of striping all the markup before
> indexing? Hopefully I missing something basic?
>
> It would be great to pointed in the right direction on this matter?
>
> I think I need something along this line:
>
> <add>
> <doc>
> <field name="foo1">value 1</field>
> <field name="foo2">value 2</field>
> ....
> <field name="content"><an xml stream with embedded source markup></field>
> </doc>
> </add>
>
> Maybe the overall question -- is what is the best way to index XML content
> using SOLR -- is all this tag stripping really necessary?
crazy/silly idea maybe... could you use dynamic fields, each containing a
sentence, and a reference to the paragraph it belongs to ?
eg, (not sure if the syntax is correct..)
<dynamicField name="s_*" type="string" />
Then when you create your document you can define
<doc>
<field name="s_1_p1">{Sentence #1, Para#1}</field>
<field name="s_2_p1">{Sentence #2, Para#1}</field>
<field name="s_3_p1">{Sentence #3, Para#1}</field>
<field name="s_1_p2">{Sentence #1, Para#2}</field>
[...]
</doc>
I have no idea how scalable that would be.
cheers,
B
_________________________
{Beto|Norberto|Numard} Meijome
Immediate success shouldn't be necessary as a motivation to do the right thing.
I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.
