I have spent days trying to figure out how to get UTF-8 working with my
site.
Here's my environment:
Linux version 2.6.16.29-xen_3.0.3.0
Ruby 1.8.4 (2005-12-24 [i386-linux]
Rails 1.2.3
mongrel (1.0.1)
mongrel_cluster (1.0.2, 0.2.1)
ferret (0.11.4)
acts_as_ferret stable plugin
Ferret DRB server
When I don't use an analyzer with my acts_as_ferret declaration,
everything works fine. However, I can't expect users to enter "Álex
Rodríguez" when searching.. they're going to put "alex rodriguez" (or
some variation of his name, which I handle using a fuzzy search)
So then call an analyzer in my acts_as_ferret declaration:
acts_as_ferret({ :fields => {:first_name => {:store => :no},
:last_name => {:store => :no},
:db_state => {:index =>
:untokenized_omit_norms, :term_vector => :no}},
:remote => true}, {:analyzer => UtfAnalyzer.new})
Here's the analyzer I'm using... pretty much taken from from here:
http://ferret.davebalmain.com/api/classes/Ferret/Analysis/MappingFilter.html
-----
class UtfAnalyzer < Ferret::Analysis::Analyzer
include Ferret::Analysis
CHARACTER_MAPPINGS = {
['à','á','â','ã','ä','å','ā','ă'] => 'a',
'æ' => 'ae',
['ď','đ'] => 'd',
['ç','ć','č','ĉ','ċ'] => 'c',
['è','é','ê','ë','ē','ę','ě','ĕ','ė',] => 'e',
['ƒ'] => 'f',
['ĝ','ğ','ġ','ģ'] => 'g',
['ĥ','ħ'] => 'h',
['ì','ì','í','î','ï','ī','ĩ','ĭ'] => 'i',
['į','ı','ij','ĵ'] => 'j',
['ķ','ĸ'] => 'k',
['ł','ľ','ĺ','ļ','ŀ'] => 'l',
['ñ','ń','ň','ņ','ʼn','ŋ'] => 'n',
['ò','ó','ô','õ','ö','ø','ō','ő','ŏ','ŏ'] => 'o',
['œ'] => 'oek',
['ą'] => 'q',
['ŕ','ř','ŗ'] => 'r',
['ś','š','ş','ŝ','ș'] => 's',
['ť','ţ','ŧ','ț'] => 't',
['ù','ú','û','ü','ū','ů','ű','ŭ','ũ','ų'] => 'u',
['ŵ'] => 'w',
['ý','ÿ','ŷ'] => 'y',
['ž','ż','ź'] => 'z'
}
def token_stream(field, str)
MappingFilter.new(StandardTokenizer.new(str), CHARACTER_MAPPINGS)
end
end
I think Ferret is working fine... because when I run some tests, the
mapping filter correctly pulls out the accented characters... exactly as
it should.
However, when something is persisted via the model (acts_as_ferret and
DRB server), I get unexpected behavior...
- using a model with ONE field declared in acts_as_ferret, and a string
with accented characters -- I can search it as expected - with either
accented or non-accented character, adn I get the results returned;
however, I don't get any other results for the non-accented records.
ONLY the accented records get returned when searching.
- using a model with multiple characters defined (as in Player model
above) -- nothing gets returned, neither accented or non-accented
records, or any combination
My ferret_server.log file shows characters that are very different from
the accented characters I'm trying to search on...
Search entered in form: Álex Rodríguez
ferret_server.log: Ãlex rodrÃguez
Not sure why this is occuring, but I've also redisplayed the submitted
text on a web page and it displays correctly. This leads me to believe
that Ruby/Rails is successfully getting the information, and that html
page encoding is correct, along with environment variables, etc.. As I
stated earlier, my Ferret test takes the string "Rodríguez" and returns
token["Rodriguez":0:10:1] demonstrating the UtfAnalyzer works fine
outside of acts_as_ferret...
So any help here would be much appreciated.
Thanks,
Brandon
--
Posted via http://www.ruby-forum.com/.
_______________________________________________
Ferret-talk mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/ferret-talk
