[
https://issues.apache.org/jira/browse/SOLR-11917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16341692#comment-16341692
]
Hoss Man commented on SOLR-11917:
---------------------------------
h2. *S2.1*: Easy Multi-Language Querying (SOLR-6492)
h3. *S2.1G*: Goal
Simplified indexing & querying of text in diff languages w/o the query clients
being _required_ to know about a lot of language specific variant field names.
At index time we want things to be "easy" for clients wending documents,
regardless of whether they already know the lang of each field value in
advance, or if they want solr to do langauge detection.
h3. *S2.1A*: Suggested Approach
{panel:title=Refresher: Summary of Solr In Action (SIA) code linked to from}
SOLR-6492
*What's included & how it works...*
* custom update process & custom field type
** processor is subclass of existing lang detect update processor
*** super class normally adds a field with languages in doc, or renames fields
to include language (ie: text => text_de)
** field type is subclass of TextField
*** goes out of it's way to override any Analyzer config with a custom one
(details below)
*** configured with a list of mappings from langid to other (existing) field
types
* Index Time:
** update processor delegates to super to detect languages but instead of (in
addition to?) super class's behavior of adding a language field to doc, or
renaming the field with suffix, the custom processor "decorates" the values
with the detected language(s)...
** for any field where the field type is our custom type:
*** "decorate" each of the field values with either:
**** the langs of the whole doc
**** the langs of the field (after re-running lang detect on all values in
just that field)
**** the langs of the individual field value (after re-running lang detect on
just that field value)
** other processors can then run as normal, and eventually the IndexSchema is
asked to build up the IndexableFields for this doc, and it delegates to the
(custom) field type for these "decorated" fields...
** field type's custom analyzer looks for these lang "decorations" on each
field value
*** for every lang found, go fetch the analyzer from the mapped field type
it's configured with
*** create a token stream that delegates to all the other analyzers & merges
the resulting token streams
*** all this custom delegation/merging tokenstream stuff is
(optionally/wisely) wrapped in RemoveDuplicatesTokenFilter since there can be
lots of dup tokens for similar languages.
* Query Time:
** the query string provided by the user can be "decorated" with a list of
languages
** the normal plumbing of TextField analyzes the query string, delegating to
the various analyzers
*** AFAICT: this means MultiTermPhraseQueries are frequently produced?
* NOTE: as mentioned in Trey's LR talk for 2014, a "perk" of this solution
(over using diff fields per languages) is that mixing languages in one field
value can – in theory – still produce useful phrase queries, even if the
non-correct analyzers butcher the terms in other languages such that a single
phrase produced by either language analyzer wouldn't match the original string
** [https://www.youtube.com/watch?v=MQ6WtBw8T_U]
** BUT: it's not really clear if/how useful/important this is. _Does any one
have any actual usecases for this???_
*The Fiddly / Awkward / Problematic Bits Of All This Existing Code*
* language "decoration" is super hackish
** index time:
*** the update processor prepends them as a string
*** not a lot of easy improvements currently possible given the current
SolrInputDocument / UpdateProcessor / DocumentBuilder structure / code paths
**** fixing this "THE RIGHT WAY" would probably require some pretty big
changes to all this code so SolrInputField could support arbitrary metadata
(instead of just "boost" like it does today) and passing the SolrInputFields
all the way to the FieldType's createFields method
**** the hackish way to do this might be to follow in the footsteps of atomic
update with "field value may be a map containing magic keys", but...
***** this would probably break Atomic Updates (unexpected keys in the Maps it
thinks it owns)
***** this was already a super heinous API hack and hacks this heinous should
not be reworded by being copied
***** Even if we did this, i'm not certain the FieldType's createFields()
would get the full Map w/o a bunch of other changes in the middle – if we're
going to have to change existing DocumentBuilder/IndexSchema code to make this
work, let's not be heinous about it.
** query time:
*** user must prefixing the terms _inside_ the query strings – _after_ the
field name
*** example: {{q=my_multi_lang_field:"en,es|Hello there compadre"}}
*** fixing this in a sane way should be really straight forward...
**** all of the "public Query getFoo(...)" methods a FieldType must implement
take in the QParser originating the query
**** we can ask the QParser for the local/req params
**** so syntax like "q= \{!field f=body langs='en,es'}Hello there compadre"
would be easy to support
* the tokenstream merging slurps in the entire Reader as a String on first
use, then pre-analyzes using every analyzer and builds up an in memory
LinkedList<Token>
** why is this needed? why can't we just cache _one_ "Token" per Analyzer?
ie...
*** each call to incrementTokens calls incrementTokens on any delegate
analyzer where we don't have a cached token (and says it's not done with the
input)
*** then return (and null out) whichever cached token has the lowest position
** also: since this is super custom code and we know the way our analyzer is
getting used is from our custom FieldType, why mess with a Reader -> String at
all since we know for certain the Reader is a StringReader
*** ie: bypass the normal "this.getAnalyzer()" and just give the "Analyzer"
the original String ?{panel}
h4. *S2.1.STRAW1*: Straw Man Proposal #1 – aka "Complex For Users"
Existing SIA Code + query time local params
* keep most of the existing SOLR-6492 code as is
** all indexing code and update processor sub class stay the same
*** including the hackish way we have to prefix-decorate the langs on field
values at index time
*** hopefully fix the analyzer to be more efficient
** at query time:
*** in the FieldType: override the "public Query getFoo(...)" methods to look
at the local/req params for a langs
*** use those langs when using our custom (wrapper) analyzer
*** *NOTE:* This cleaner query time API still has a hitch – see *S2.1.HITCH*
below
* This approach seems more "complex" to explain to users then the strawman #2
(*S2.1.STRAW2*) below
** particularly given the dependency on the new update processor (or users
adding magic field value decoration at index time)
** and especially if/when they gain more experience with solr and want to
understand more what's happening under the covers and how to tweak/customize
behavior.
** see full pro/con list below
h4. *S2.1E.STRAW1*: Hypothetical Example Usage of this *S2.1.STRAW1* Strawman...
{code:xml}
<field name="title" type="langaware" />
<field name="body" type="langaware" />
<fieldType name="langaware" class="solr.MultiLangAwareTextField"
defaultFieldType="text_general"
fieldMappings="en:text_english,
es:text_latin,
fr:text_french"/>
<fieldType name="text_general" ... />
<fieldType name="text_english" ... />
<fieldType name="text_french" ... />
<fieldType name="text_latin" ... />
{code}
{code:xml}
<!-- doc sent by client using new custom update processor -->
<doc>
<field name="title">Solr In Action</field>
<field name="body">Ipsum Lorem ... thousands of pages of text</field>
<doc>
<!-- doc sent by client that knows what lang these fields are
<doc>
<field name="title">en|Solr In Action</field>
<field name="body">la|Ipsum Lorem ... thousands of pages of text</field>
<doc>
{code}
{noformat}
# Uses the lang specific analysis the user asked for
/query?q={!lang=la}body:Lorem&fq={!field f=title lang=en}Action
# Falls back to the text_general analysis since no lang is known
/query?q=body:Lorem&fq={!field f=title}Action
{noformat}
h4. *S2.1.STRAW2*: Straw Man Proposal #2 – aka "Simple for Users"
Override only the Query parsing bits of TextField (or huper duper text field)
* continue using diff fields per lang ague (either dynamic or explicitly) in
schema
* continue using the existing clone / lang detect update processors to
processors copy/rename fields (ie: title => title + title_es)
* let the analyzer for types like "text_es" do it's regular analysis and
indexing into the underlying fields like "title_es"
* let types like "text_multilang" for fields like "title" be a new
QueryLangAwareProxyTextField that extends TextField
** still supports a direct analyzer configuration for it's "default" behavior
(ie: something simple that is as lang agnostic as possible, aka: text_general)
** at index time, just does it's regular indexing with it's configured analyzer
** at query time:
*** if the QParser's params don't indicate a lang ague, do a normal query
against the specified field
*** if the QParser does specify some languages:
**** build up a list lang specific field names using the current field name +
the languages (ie "title" + "_" + "es")
***** fetch the FieldType's for each of those field names from the IndexSchema
***** delegate to the equivalent "public Query getFoo" for each of those
FieldTypes, wrap the results in a DisjunctionMaxQuery
*** *NOTE:* This use of QParser params still has the same hitch as strawman #1
(*S2.1.STRAW1*) – see *S2.1.HITCH* below
* This approach seems simpler to explain to new users then strawman #1
(*S2.1.STRAW1*)
** particularly given that it can be useful (in a clean way) even w/o any
(new) langid update processors when users already know the language of the
fields for each doc, but just want simplified querying.
** see full pro/con list below
h4. *S2.1E.STRAW2*: Hypothetical Examples of this *S2.1.STRAW2* Strawman #2...
{code:xml}
<field name="title" type="langaware" />
<field name="body" type="langaware" />
<fieldType name="langaware" class="solr.QueryLangAwareProxyTextField">
<!-- no special mappings needed, just simple lang agnostic default analyzers
-->
<analyzer type="index" ... />
<analyzer type="query" ... />
</fieldType>
<dynamicField name="*_en" type="text_english" ... />
<dynamicField name="*_fr" type="text_french" ... />
<dynamicField name="*_la" type="text_latin" ... />
<fieldType name="text_english" ... />
<fieldType name="text_french" ... />
<fieldType name="text_latin" ... />
{code}
{code:xml}
<!-- sample doc sent by client using langid update processor -->
<!-- title copied to title_en, body copied to body_la -->
<doc>
<field name="title">Solr In Action</field>
<field name="body">Ipsum Lorem ... thousands of pages of text</field>
<doc>
<!-- sample doc sent by client that knows what lang these fields are -->
<!-- CloneFieldUpdateProcessor or something simple like can copy these to
"title" & "body" -->
<doc>
<field name="title_en">Solr In Action</field>
<field name="body_en">Ipsum Lorem ... thousands of pages of text</field>
<doc>
{code}
{noformat}
# rewrites the queries against the lang specific versions using the langs the
user asked for
/query?q={!lang=la}body:Lorem&fq={!field f=title lang=en}Action
# Falls back to the default analysis (configured on 'langaware' type) since no
'lang' is specified
/query?q=body:Lorem&fq={!field f=title}Action
# user can still choose to sort on, or filter against, the existence of data in
specific language fields
/query?q=body_la:Lorem&sort=title_la asc
{noformat}
h4. *S2.1.HITCH*: One Hitch @ Query Time To Both Strawmen
Currently, SolrQueryParserBase/QueryBuilder sometimes uses the "Analyzer"
(IndexSchema's per field wrapper) directly w/o delegating to
ft.getFieldQuery(...).
*Best solution I can think of:*
* SolrQueryParserBase should override createFieldQuery(Analyzer,...) in a way
that it can delegate to the FieldType
** must happen in such a way that the FieldType can make a callback to the low
level QueryBuilder.createFieldQuery – otherwise we'll have to copy/paste a lot
of existing code.
** NOTE: QueryBuilder.createFieldQuery currently protected.
* This callback should involve a QParser (like the existing "public Query
getFoo" methods on FieldType) to access the flags/params to capture some of the
QueryBuilder state / variables passed to createFieldQuery
** in our special case, we ignore the specified Analyzer and pick one at query
time
* GENERAL IMPROVEMENT IDEA:
** maybe QParser should extend SolrQueryParserBase/QueryBuilder and
automatically call some QueryBuilder setter methods based on common local
params (like "df", "f", "q.op", etc...)
** some existing QParsers (like LuceneQParser and ExtendedDismaxQParser could
then be refactored to do their query parsing directly (instead of the QParser
instantiating a custom subclass of SolrQueryParser)
** this would potentially simplify a variety of existing QParser subclasses
** could also simplify some FieldType.getFoo methods that currently call "new
FooQuery" – they could instead delegate back to QParser.newFooQuery
** if we did this, then the callback mechanism needed for these strawmen ideas
would be (mostly?) straight forward:
*** QParser would override QueryBuilder.createFieldQuery(Analyzer,...) to
delegate to the FieldType's getFieldQuery, passing in a nested/sub-QParser with
the various method call specific options included as state/params
*** QParser would also expose a new public method that the FieldType could
call back to that would ultimately call super.createFieldQuery(Analyzer,...)
*Hypothetical (Broken) Alternatively:*
* we could consider eliminating the analyzers "cache" that IndexSchema uses
(only helpful for non-dynamic fields) and change getQueryAnalyzer to take in a
QParser can can capture some query/request state so that the FieldType can
customize the Analyzer behavior
* then our special field type can delegate to a completely diff FieldType
* The Problem With This Alternative:
** the field *name* that QueryBuilder then uses in the underlying Query
objects would still be "wrong"
** This would _not_ be a problem with the callback approach discussed above,
because our new FieldType could call callback to the QueryBuilder methods w/any
field name + analyzer pair it wanted.
h4. *S2.1.PROCON*: Pros/Cons of the two Strawmen
* The *S2.1.STRAW2* approach seems simpler to understand/explain to users
** no special/magic field types they have to declare and reference from
another field
*** ie: they declare/manage title, title_en, title_es, title_de fields – the
only thing special is that querying the "title" field can proxy to the others
as well _when the query requests it_
*** this means this approach is also automatically compatible with people who
want to explicitly index multiple fields for each language:
**** ie: they already have/know an "english title" and a "translated spanish
title" in the source docs, and don't need any index side (langdetect/copyfield)
help – our new field type just helps make the query side simple/easy to use.
** also plays nicely if we decide to do the SortableTextField described above
(*S1.1*) and want to extend it here:
*** the user still has distinct fields for "title", "title_es", etc... and can
choose to sort on any of them
**** even in the trivial case, where they only have one original field value
per doc (which lang detect also copied to title_XX), they probably always want
to sort on the general "title" field – "keep simple stuff simple"
** can be implemented completely independently/orthogonally from all the ideas
discussed here
** the downsides of *S2.1.STRAW2* are:
*** doesn't give us the "mixed languages in a single field value" phrase query
benefit (which seemed out of scope? do we have usecases like this we care
about?)
*** doesn't "save space" like single field approach when multi languages
produce same tokens
**** Although i'm not convinced that's fundamentally true – or even
beneficial: since any space savings from diff languages producing the same
underlying "term" text may be offset by potential false positives in phrase
matches (since we're assuming that even if multiple (guessed) languages may be
specified at query time, the query string is expected to be in a single
language and searching across those multiple languages should be done
independently
* the *S2.1.STRAW1* approach seems like it would be harder to explain to
novice users
** ie: the special configuration of refering to (otherwise seemingly unused)
fieldtypes from special fieldtypes
*** we've had this in the psat with some things like ExternalFileField &
CurrencyField and it's always confusing
** the special prefix decoration of langs at indexing time also means this
approach either *requires* users learn about & use the new update processor
(ie: the features are locked together), or require some explanation of how
clients must decorate the field values
** which also means that this approach would also not play very nicely with
people who have pre-translated field values at index time
*** we could potentially offer an "prepend lang code update processor" to make
it easy to massage their data for them
**** ex: {{title_es:"Hola Juan", title_en:"Hello John"}} ==> {{title:["es|Hola
Juan", "en|Hello John"]}}
** if we extend the SortableTextField (*S1.1*) idea...
*** only the trivial usecases (each logical field is only in one language)
plays nicely with sorting
*** if a user starts with multiple different "translated" fields – and has to
consolidate them as multiple field values in a single field (with our
hypothetical "prepend lang code update processor") then they don't really have
any way to "sort on the spanish title" with this approach
**** unless of course they *also* redundantly index every lang variant as it's
own field – but then most of the benefits of this approach are out the window
(ie: there are no fieldname/configuration/disk savings as compared to the other
strawman)
** The key upside i can think of for *S2.1.STRAW1*:
*** If we first focus on *S2.2* (see below), then the schema syntax could
potentially be simplified to remove the "lang -> some other fieldType name"
mapping and instead use lots of nested analyzers named after each langauge
*** this might still be a bit confusing however if people want diff
index/query(/multiTerm) analyzers for each langague ... would have to use some
sort of regid naming convention?
{panel}
*NOTE:* If either strawman is implemented, we should strongly consider
including an additional option/subclass of this new "*LangAwareTextField" to
automatically use the langid plugin code at query time to try and "guess" the
lang if it isn't specified in a 'lang' local/request params
* at least for language-detect (latest version), there are special models
built just for short inputs
* we could potentially make the code use the guessed lang at query time only
if above some configured confidence:
** or: if explicit 'lang' param, use only that lang – but if the langauge is
guessed, query using both the field/analyzer for that specific lang as well as
the 'default' field/analyzer{panel}
> A Potential Roadmap for robust multi-analyzer TextFields w/various options
> for configuring docValues
> ----------------------------------------------------------------------------------------------------
>
> Key: SOLR-11917
> URL: https://issues.apache.org/jira/browse/SOLR-11917
> Project: Solr
> Issue Type: Wish
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Hoss Man
> Assignee: Hoss Man
> Priority: Major
>
> A while back, I was tasked at my day job to brainstorm & design some "smarter
> field types" in Solr. In particular to think about:
> # How to simplify some of the "special things" people have to know about
> Solr behavior when creating their schemas
> # How to reduce the number of situations where users have to copy/clone one
> "logical field" into multiple "schema felds in order to meet diff use cases
> The main result of this thought excercise is a handful of usecases/goals that
> people seem to have - many of which are already tracked in existing jiras -
> along with a high level design/roadmap of potential solutions for these goals
> that can be implemented incrementally to leverage some common changes (and
> what those changes might look like).
> My intention is to use this jira as a place to share these ideas for broader
> community discussion, and as a central linkage point for the related jiras.
> (details to follow in a very looooooong comment)
> ----
> NOTE: I am not (at this point) personally committing to following through on
> implementing every aspect of these ideas :)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
