Alexandre - i am sorry if i was not clear, this is about queries, this all 
happens at query time. Yes we can do the substitution in with the regex replace 
filter, but i would propose this weird exception to be added to 
WhitespaceTokenizer so Lucene deals with this by itself.

Markus
 
-----Original message-----
> From:Alexandre Rafalovitch <arafa...@gmail.com>
> Sent: Wednesday 8th October 2014 16:12
> To: solr-user <solr-user@lucene.apache.org>
> Subject: Re: WhitespaceTokenizer to consider incorrectly encoded c2a0?
> 
> Is this a suggestion for JIRA ticket? Or a question on how to solve
> it? If the later, you could probably stick a RegEx replacement in the
> UpdateRequestProcessor chain and be done with it.
> 
> As to why? I would look for the rest of the MSWord-generated
> artifacts, such as "smart" quotes, extra-long dashes, etc.
> 
> Regards,
>    Alex.
> Personal: http://www.outerthoughts.com/ and @arafalov
> Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
> 
> 
> On 8 October 2014 09:59, Markus Jelsma <markus.jel...@openindex.io> wrote:
> > Hi,
> >
> > For some crazy reason, some users somehow manage to substitute a perfectly 
> > normal space with a badly encoded non-breaking space, properly URL encoded 
> > this then becomes %c2a0 and depending on the encoding you use to view you 
> > probably see  followed by a space. For example:
> >
> > Because c2a0 is not considered whitespace (indeed, it is not real 
> > whitespace, that is 00a0) by the Java Character class, the 
> > WhitespaceTokenizer won't split on it, but the WordDelimiterFilter still 
> > does, somehow mitigating the problem as it becomes:
> >
> > HTMLSCF een abonnement
> > WT een abonnement
> > WDF een eenabonnement abonnement
> >
> > Should the WhitespaceTokenizer not include this weird edge case?
> >
> > Cheers,
> > Markus
> 

Reply via email to