gus-asf commented on pull request #1965: URL: https://github.com/apache/lucene-solr/pull/1965#issuecomment-708688007
> I don't fully understand why this flags handling is needed in the tokenfilter, because i have not yet reviewed the new Solr feature. LUCENE-9575 (patch waiting for the test changes here) will show you where the flags are gained. The basic idea is that the flags are used to identify the source tokens that gave rise to synonym tokens and (sometimes) remove the source tokens *after* they participate in further (arbitrary) processing... the canonical example is to allow '401k' and '401(k)' to match the same documents about retirement plans and also NOT match documents regarding "401 Unauthorized" or "Vitamin K" ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
