[
https://issues.apache.org/jira/browse/SOLR-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191418#comment-14191418
]
Trey Grainger commented on SOLR-6492:
-------------------------------------
Hi Sharon,
In terms of your suggestion, I do think that using local params to pass in
the language could be a more user-friendly solution than requiring them to
put the params on the field name: i.e. q={!langs=en|de}hello world&df=text
vs. q=hello world&df=text|en,de, though the syntax may get a bit weird if
you want to specify different languages for different fields. For example,
if using the edsimax query parser, you would need to do something like
q={!langs=text1:en,de|text2:en,zh}hello world&qf=text1 text2 vs. just
q=hello world&qf=text1|en,de text2|en|zh.
For the most simple use-case (every field uses the same language), or for
the use-case where you don't know what fields the user is querying on
up-front, I think the local params syntax would be preferred for end-users.
There is a big down-side to doing this, however: it requires you to
implement a qparser to parse this data and put it somewhere that the
Analyzer can see. This means that your multi-lingual field would only be
searchable with your custom query parser (whereas if the determination of
the language is passed in as part of the field name or content as I
described, it should work seamlessly with all of the query parsers, since
the data gets passed through all the way to the Analyzer).
Your solution with the ThreadLocal storage of the data is interesting...
I'm not positive whether it will work or not (i.e. does the analyzer always
run on the same thread as the incoming request for both queries and
indexing, and will that also continue to be the case into the future)? I
know that threads are at least re-used across requests and that the
TokenStreamComponents for analyzers are re-used in a threadlocal pool, but
that just means you'd have to be very careful about not caching or reusing
languages across requests, not that it couldn't work. Also, just out of
curiosity, how do you plan to pass the languages in at index time?
The Analyzer/Tokenizers only accept the fieldname (string) and the field
content (reader) as parameters, so passing in additional parameters through
a threadlocal seems like a bit of a hack that violates the design there
(though arguably that design is too restrictive and should change). I'd be
curious if anyone else thinks this would work...
Thanks,
Trey Grainger
Co-author, Solr in Action
Director of Engineering, Search & Analytics @CareerBuilder
> Solr field type that supports multiple, dynamic analyzers
> ---------------------------------------------------------
>
> Key: SOLR-6492
> URL: https://issues.apache.org/jira/browse/SOLR-6492
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Trey Grainger
> Fix For: 5.0
>
>
> A common request - particularly for multilingual search - is to be able to
> support one or more dynamically-selected analyzers for a field. For example,
> someone may have a "content" field and pass in a document in Greek (using an
> Analyzer with Tokenizer/Filters for German), a separate document in English
> (using an English Analyzer), and possibly even a field with mixed-language
> content in Greek and English. This latter case could pass the content
> separately through both an analyzer defined for Greek and another Analyzer
> defined for English, stacking or concatenating the token streams based upon
> the use-case.
> There are some distinct advantages in terms of index size and query
> performance which can be obtained by stacking terms from multiple analyzers
> in the same field instead of duplicating content in separate fields and
> searching across multiple fields.
> Other non-multilingual use cases may include things like switching to a
> different analyzer for the same field to remove a feature (i.e. turning
> on/off query-time synonyms against the same field on a per-query basis).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
