I've made some fixes to the MySQL search backend for Chinese and other
languages using variants.
Some languages don’t use word spacing, like Chinese and Japanese. To let
the search index know where word boundaries are, we have to internally
insert spaces between some characters:
维基百科 -> 维 基 百 科
Then to add insult to injury, we need to fudge the Unicode characters to
ensure things work reliably with older and newer versions of MySQL:
维 基 百 科 -> u8e7bbb4 u8e59fba u8e799be u8e7a791
For a long time, this word segmentation wasn’t being handled correctly
for Chinese in our default MySQL search backend, so searching for a
multi-character word often gave false matches where the characters were
all present, but not together.
This should now be fixed in r52338: the intermediate query
representation passed to the search backend internally treats your
multi-character Chinese input as a phrase, which will only match actual
adjacent characters:
维基百科 -> +"u8e7bbb4 u8e59fba u8e799be u8e7a791"
Variants for eg Serbian are also now using parens internally so they
should match more usefully.
Note that Wikimedia’s sites such as Wikipedia run on a fancier, but more
demanding, search backend with a separate Java-based engine built around
Apache Lucene. Sometimes we have to remind ourselves that third-party
users will mostly be using the MySQL-based default, and oh boy it still
needs some lovin’! :)
-- brion
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
