Well, it's hard to say what "correctly" would be. Remove all XML? Preserve attributes? Preserve tags? Put the attributes and values into fields in the document? My point is that there's no obviously "correct" parsing.
But if you just want to strip out all the <....>, it seems like PatternTokenizer might work for you... HTH Erick 2010/10/15 Christoph Hermann <[email protected]> > Hi, > > is there a Tokenizer in Lucene, that tokenizes XML correctly? > > I.e. that one gets from the following XML: > <span>this is <span attr="foo">example</span>text.</span> > > Tokens (or similar): > <span> | this | is | <span attr="foo"> | example | </span> | text. | > </span> > > Or would i need to write such a Tokenizer myself? > > regards > Christoph Hermann > > -- > Christoph Hermann > Institut für Informatik > Tel: +49 761-203-8171 Fax: +49 761-203-8162 > e-mail: [email protected] > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
