Hi Mike, Thanks for the response! I'm actually not super familiar with UserDictionaries, but looking at the code, it seems like a single line in the user provided user dictionary corresponds to a single entry? In that case, here is the line (or entry) that does have both widths that I believe is causing the problem:
ﾚｺｰﾀﾞｰ,レコーダー,レコーダー,JA名詞 I'm guess here the surface is ﾚｺｰﾀﾞｰ and the concatentated segment is the first occurrence of レコーダー. I'm what surface or concatenated segment means though, and what it would mean semantically to replace the surface with the full width version or the concatenated segment with the half width version. Thanks, Marc On Thu, Jan 13, 2022 at 7:18 AM Michael Sokolov <msoko...@gmail.com> wrote: > HI Marc, I wonder if there is a workaround for this issue: eg, could > we have entries for both widths? I wonder if there is some interaction > with an analysis chain that is doing half-width -> full-width > conversion (or vice versa)? I think the UserDictionary has to operate > on pre-analyzed tokens ... although maybe *after* char filtering, > which presumably could handle width conversions. A bunch of rambling, > but maybe the point is - can you share some more information -- what > is the full entry in the dictionary that causes the problem? > > On Wed, Jan 12, 2022 at 7:04 PM Marc D'Mello <marcd2...@gmail.com> wrote: > > > > Hi, > > > > I had a question about the Japanese user dictionary. We have a user > > dictionary that used to work but after attempting to upgrade Lucene, it > > fails with the following error: > > > > Caused by: java.lang.RuntimeException: Illegal user dictionary entry > ﾚｺｰﾀﾞｰ > > - the concatenated segmentation (レコーダー) does not match the surface form > > (ﾚｺｰﾀﾞｰ) > > at > > > org.apache.lucene.analysis.ja.dict.UserDictionary.<init>(UserDictionary.java:123) > > > > The specific commit causing this error is here > > < > https://github.com/apache/lucene/commit/73ba88a50dec64f367caa88d277c26dfd1d8883b#diff-75fd48fadfd3d011e9c34c4310ef66e9009edfbc738fd82deb5661a8edb5c5d9 > >. > > The only thing that seems to differ is that the characters are full-width > > vs half-width, so I was wondering if this is intended behavior or a > bug/too > > restrictive. Any suggestions for fixing this would be greatly > appreciated! > > Thanks! > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >
