[jira] Commented: (LUCENE-1029) Illegal character replacements in ISOLatin1AccentFilter
[ https://issues.apache.org/jira/browse/LUCENE-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12597883#action_12597883 ] Marko Asplund commented on LUCENE-1029: --- I've switched projects some time ago so it'll take while for me to setup the tests. I'll try to get around doing this at a later time. > Illegal character replacements in ISOLatin1AccentFilter > --- > > Key: LUCENE-1029 > URL: https://issues.apache.org/jira/browse/LUCENE-1029 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Affects Versions: 2.2 >Reporter: Marko Asplund > Attachments: ISOLatin1AccentFilter-by-Collator.patch, > ISOLatin1AccentFilter-javadoc.patch > > > The ISOLatin1AccentFilter class is responsible for replacing "accented > characters in the ISO Latin 1 character set by their unaccented equivalent". > Some of the replacements performed for scandinavian characters (used e.g. in > the finnish, swedish, danish languages etc.) are illegal. The scandinavian > characters are different from the accented characters used e.g. in latin > based languages such as french in that these characters (ä, ö, å) represent > entirely independent sounds in the language and therefore cannot be > represented with any other sound without change of meaning. It is therefore > illegal to replace these characters with any other character. > This means for example that you can't change the finnish word sää (weather) > to saa (will have) because these are two entirely different words with > different meaning. The same applies to scandinavian languages as well. > There's no connection between the sounds represented by ä and a; ö and o or å > and a. > In addition to the three characters mentioned above danish and norwegian use > other special characters such as ø and æ. It should be checked if the > replacement is legal for these characters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1029) Illegal character replacements in ISOLatin1AccentFilter
[ https://issues.apache.org/jira/browse/LUCENE-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12597703#action_12597703 ] Otis Gospodnetic commented on LUCENE-1029: -- I only skimmed the comments. Has anyone tried Hiroaki's patch? Does it satisfy your needs? Marko, Uwe, Karl, Mark? > Illegal character replacements in ISOLatin1AccentFilter > --- > > Key: LUCENE-1029 > URL: https://issues.apache.org/jira/browse/LUCENE-1029 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Affects Versions: 2.2 >Reporter: Marko Asplund > Attachments: ISOLatin1AccentFilter-by-Collator.patch, > ISOLatin1AccentFilter-javadoc.patch > > > The ISOLatin1AccentFilter class is responsible for replacing "accented > characters in the ISO Latin 1 character set by their unaccented equivalent". > Some of the replacements performed for scandinavian characters (used e.g. in > the finnish, swedish, danish languages etc.) are illegal. The scandinavian > characters are different from the accented characters used e.g. in latin > based languages such as french in that these characters (ä, ö, å) represent > entirely independent sounds in the language and therefore cannot be > represented with any other sound without change of meaning. It is therefore > illegal to replace these characters with any other character. > This means for example that you can't change the finnish word sää (weather) > to saa (will have) because these are two entirely different words with > different meaning. The same applies to scandinavian languages as well. > There's no connection between the sounds represented by ä and a; ö and o or å > and a. > In addition to the three characters mentioned above danish and norwegian use > other special characters such as ø and æ. It should be checked if the > replacement is legal for these characters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1029) Illegal character replacements in ISOLatin1AccentFilter
[ https://issues.apache.org/jira/browse/LUCENE-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12578195#action_12578195 ] Hiroaki Kawai commented on LUCENE-1029: --- I'd like to comment that we have another tool for this. :-) java.text.Collator can collate the texts, and the instance is base on Locale, wow! So, if we use this collator, you might get a better query result, i.e, more low search noise that German "ä" might hit with "ae". I'd like to submit a patch later. > Illegal character replacements in ISOLatin1AccentFilter > --- > > Key: LUCENE-1029 > URL: https://issues.apache.org/jira/browse/LUCENE-1029 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Affects Versions: 2.2 >Reporter: Marko Asplund > Attachments: ISOLatin1AccentFilter-javadoc.patch > > > The ISOLatin1AccentFilter class is responsible for replacing "accented > characters in the ISO Latin 1 character set by their unaccented equivalent". > Some of the replacements performed for scandinavian characters (used e.g. in > the finnish, swedish, danish languages etc.) are illegal. The scandinavian > characters are different from the accented characters used e.g. in latin > based languages such as french in that these characters (ä, ö, å) represent > entirely independent sounds in the language and therefore cannot be > represented with any other sound without change of meaning. It is therefore > illegal to replace these characters with any other character. > This means for example that you can't change the finnish word sää (weather) > to saa (will have) because these are two entirely different words with > different meaning. The same applies to scandinavian languages as well. > There's no connection between the sounds represented by ä and a; ö and o or å > and a. > In addition to the three characters mentioned above danish and norwegian use > other special characters such as ø and æ. It should be checked if the > replacement is legal for these characters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1029) Illegal character replacements in ISOLatin1AccentFilter
[ https://issues.apache.org/jira/browse/LUCENE-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536589 ] Marko Asplund commented on LUCENE-1029: --- Perhaps it could be expressed more accurately using the following sentence: It will be impossible to search for a word in its original form without also matching filtered forms of the word at the same time. > Illegal character replacements in ISOLatin1AccentFilter > --- > > Key: LUCENE-1029 > URL: https://issues.apache.org/jira/browse/LUCENE-1029 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Affects Versions: 2.2 >Reporter: Marko Asplund > Attachments: ISOLatin1AccentFilter-javadoc.patch > > > The ISOLatin1AccentFilter class is responsible for replacing "accented > characters in the ISO Latin 1 character set by their unaccented equivalent". > Some of the replacements performed for scandinavian characters (used e.g. in > the finnish, swedish, danish languages etc.) are illegal. The scandinavian > characters are different from the accented characters used e.g. in latin > based languages such as french in that these characters (ä, ö, å) represent > entirely independent sounds in the language and therefore cannot be > represented with any other sound without change of meaning. It is therefore > illegal to replace these characters with any other character. > This means for example that you can't change the finnish word sää (weather) > to saa (will have) because these are two entirely different words with > different meaning. The same applies to scandinavian languages as well. > There's no connection between the sounds represented by ä and a; ö and o or å > and a. > In addition to the three characters mentioned above danish and norwegian use > other special characters such as ø and æ. It should be checked if the > replacement is legal for these characters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1029) Illegal character replacements in ISOLatin1AccentFilter
[ https://issues.apache.org/jira/browse/LUCENE-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536510 ] Digy commented on LUCENE-1029: -- I think , the phrase + * Please note that the replacements performed by this filter will result in words changing their original semantic meaning in many cases. + * It will also be impossible to search for the word in its original form. < is wrong. If you index and then search the text "kön" using the same analyzer that uses ISOLatin1AccentFilter, you will get the result. Who cares that it is stored as "kon" in the index.( of course searching "kön" will also return results containing "kon" , but there are a lot of cases where it is better getting more than getting nothing). DIGY > Illegal character replacements in ISOLatin1AccentFilter > --- > > Key: LUCENE-1029 > URL: https://issues.apache.org/jira/browse/LUCENE-1029 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Affects Versions: 2.2 >Reporter: Marko Asplund > Attachments: ISOLatin1AccentFilter-javadoc.patch > > > The ISOLatin1AccentFilter class is responsible for replacing "accented > characters in the ISO Latin 1 character set by their unaccented equivalent". > Some of the replacements performed for scandinavian characters (used e.g. in > the finnish, swedish, danish languages etc.) are illegal. The scandinavian > characters are different from the accented characters used e.g. in latin > based languages such as french in that these characters (ä, ö, å) represent > entirely independent sounds in the language and therefore cannot be > represented with any other sound without change of meaning. It is therefore > illegal to replace these characters with any other character. > This means for example that you can't change the finnish word sää (weather) > to saa (will have) because these are two entirely different words with > different meaning. The same applies to scandinavian languages as well. > There's no connection between the sounds represented by ä and a; ö and o or å > and a. > In addition to the three characters mentioned above danish and norwegian use > other special characters such as ø and æ. It should be checked if the > replacement is legal for these characters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [jira] Commented: (LUCENE-1029) Illegal character replacements in ISOLatin1AccentFilter
If you are to compare with stemmers, consider that these creates unique tokens that does not interfere with semantic meanings. Not starting anything here again, but it took me so darn long to find something that porter stems and kills the semantic meaning that I had to share. That damn algorithm is amazing...I was coming to the conclusion that it was absolutely perfect on the English language...until after a couple days of searching I found international goes to intern. Eureka! Though a hollow victory at best. That algorithm is pretty amazing... - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1029) Illegal character replacements in ISOLatin1AccentFilter
[ https://issues.apache.org/jira/browse/LUCENE-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535049 ] Mark Miller commented on LUCENE-1029: - My comment about stemming was not meant to compare a stemmer to a diacritical stripper, but rather to point out that the result of such an operation does not necessarily have to create something 'legal' (just as a stemmer does not create 'legal' words). This was in response to the comment 'Some of the ISOLatin1AccentFilter are legal while others are illegal. ' Your point about semantic meaning is well taken, but was not intended to be part of the comparison I was going for. My bad. I think that the fact that ripping diacriticals can change the meaning of words goes without saying...otherwise, why even have them in the language? As Uwe said, the main motivating factor is to allow easy entry with the keyboard of another language. Of course this must come with a compromise. Other search engines I have seen offer the exact functionality of this class. (CPL, SearchServer, etc) Literally, this thing is called an accent filter...letters go in, accents come off. Doing more really does seem like a job for another class. If I can borrow a word I didn't know from DM Smith, transliteration seems to go beyond an ISOLatin1AccentFilter. This is a tough sell I know -- programmers seem to push the definition of filter to its limits and IMHO into the realm of transform/translate. Anyhow...I apologize for beating a dead horse... > Illegal character replacements in ISOLatin1AccentFilter > --- > > Key: LUCENE-1029 > URL: https://issues.apache.org/jira/browse/LUCENE-1029 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Affects Versions: 2.2 >Reporter: Marko Asplund > > The ISOLatin1AccentFilter class is responsible for replacing "accented > characters in the ISO Latin 1 character set by their unaccented equivalent". > Some of the replacements performed for scandinavian characters (used e.g. in > the finnish, swedish, danish languages etc.) are illegal. The scandinavian > characters are different from the accented characters used e.g. in latin > based languages such as french in that these characters (ä, ö, å) represent > entirely independent sounds in the language and therefore cannot be > represented with any other sound without change of meaning. It is therefore > illegal to replace these characters with any other character. > This means for example that you can't change the finnish word sää (weather) > to saa (will have) because these are two entirely different words with > different meaning. The same applies to scandinavian languages as well. > There's no connection between the sounds represented by ä and a; ö and o or å > and a. > In addition to the three characters mentioned above danish and norwegian use > other special characters such as ø and æ. It should be checked if the > replacement is legal for these characters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1029) Illegal character replacements in ISOLatin1AccentFilter
[ https://issues.apache.org/jira/browse/LUCENE-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535047 ] DM Smith commented on LUCENE-1029: -- One could maintain compatibility by adding a constructor that supplies a transliteration, where a transliteration is an implementation of an interface Transliteration. The default would be the current behavior. But I don't think that buys very much. It is kind of like saying a filter can contain a filter. > Illegal character replacements in ISOLatin1AccentFilter > --- > > Key: LUCENE-1029 > URL: https://issues.apache.org/jira/browse/LUCENE-1029 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Affects Versions: 2.2 >Reporter: Marko Asplund > > The ISOLatin1AccentFilter class is responsible for replacing "accented > characters in the ISO Latin 1 character set by their unaccented equivalent". > Some of the replacements performed for scandinavian characters (used e.g. in > the finnish, swedish, danish languages etc.) are illegal. The scandinavian > characters are different from the accented characters used e.g. in latin > based languages such as french in that these characters (ä, ö, å) represent > entirely independent sounds in the language and therefore cannot be > represented with any other sound without change of meaning. It is therefore > illegal to replace these characters with any other character. > This means for example that you can't change the finnish word sää (weather) > to saa (will have) because these are two entirely different words with > different meaning. The same applies to scandinavian languages as well. > There's no connection between the sounds represented by ä and a; ö and o or å > and a. > In addition to the three characters mentioned above danish and norwegian use > other special characters such as ø and æ. It should be checked if the > replacement is legal for these characters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1029) Illegal character replacements in ISOLatin1AccentFilter
[ https://issues.apache.org/jira/browse/LUCENE-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535040 ] Karl Wettin commented on LUCENE-1029: - Hoss Man - 15/Oct/07 02:47 PM > the equivalence described in the javadocs is one of visual character > equivalence, not of semantic word equivalence - that would be a lot more > complicated. if anyone would like to submit a patch contianing a new filter > that is capable of doing that, i'm sure the community would certianly welcome > it. I think you missunderstand why I focused on the stemmer. My point was that this filter can not be compared with stmmer as in earlier posts. I do not think that the documentation is missleadning, nor do I think there is any need to break the backwards compability. All I say is that I welcome a solution that makes this filter more configurable. Not sure what a smart way to do that would be though. I'm open for a discussion. Perhaps one could feed it with exceptions, perhaps a per language definition, perhaps something else? > Illegal character replacements in ISOLatin1AccentFilter > --- > > Key: LUCENE-1029 > URL: https://issues.apache.org/jira/browse/LUCENE-1029 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Affects Versions: 2.2 >Reporter: Marko Asplund > > The ISOLatin1AccentFilter class is responsible for replacing "accented > characters in the ISO Latin 1 character set by their unaccented equivalent". > Some of the replacements performed for scandinavian characters (used e.g. in > the finnish, swedish, danish languages etc.) are illegal. The scandinavian > characters are different from the accented characters used e.g. in latin > based languages such as french in that these characters (ä, ö, å) represent > entirely independent sounds in the language and therefore cannot be > represented with any other sound without change of meaning. It is therefore > illegal to replace these characters with any other character. > This means for example that you can't change the finnish word sää (weather) > to saa (will have) because these are two entirely different words with > different meaning. The same applies to scandinavian languages as well. > There's no connection between the sounds represented by ä and a; ö and o or å > and a. > In addition to the three characters mentioned above danish and norwegian use > other special characters such as ø and æ. It should be checked if the > replacement is legal for these characters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1029) Illegal character replacements in ISOLatin1AccentFilter
[ https://issues.apache.org/jira/browse/LUCENE-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534987 ] Hoss Man commented on LUCENE-1029: -- The functionality of ISOLatin1AccentFilter shouldn't change in a way that wouldn't be bckward compatible. if people feel the documentation is misleading and doesn't accurately reflect what the Filter does, then by all means please submit a documentation patch. first and foremost the purpose of this filter is to replace accented characters with non-accented characters ... the equivalence described in the javadocs is one of visual character equivalence, not of semantic word equivalence -- that would be a lot more complicated. if anyone would like to submit a patch contianing a new filter that is capable of doing that, i'm sure the community would certianly welcome it. > Illegal character replacements in ISOLatin1AccentFilter > --- > > Key: LUCENE-1029 > URL: https://issues.apache.org/jira/browse/LUCENE-1029 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Affects Versions: 2.2 >Reporter: Marko Asplund > > The ISOLatin1AccentFilter class is responsible for replacing "accented > characters in the ISO Latin 1 character set by their unaccented equivalent". > Some of the replacements performed for scandinavian characters (used e.g. in > the finnish, swedish, danish languages etc.) are illegal. The scandinavian > characters are different from the accented characters used e.g. in latin > based languages such as french in that these characters (ä, ö, å) represent > entirely independent sounds in the language and therefore cannot be > represented with any other sound without change of meaning. It is therefore > illegal to replace these characters with any other character. > This means for example that you can't change the finnish word sää (weather) > to saa (will have) because these are two entirely different words with > different meaning. The same applies to scandinavian languages as well. > There's no connection between the sounds represented by ä and a; ö and o or å > and a. > In addition to the three characters mentioned above danish and norwegian use > other special characters such as ø and æ. It should be checked if the > replacement is legal for these characters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1029) Illegal character replacements in ISOLatin1AccentFilter
[ https://issues.apache.org/jira/browse/LUCENE-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534839 ] DM Smith commented on LUCENE-1029: -- Transliteration rules are language dependent. I suggest that the documentation for the ISOLatin1AccentFilter be adjusted to match it's behavior, stating that it strips diacritics from characters and does further substitutions (giving the precise list) and that it does not do transliteration. Further give examples as stated in the above comments that the results for such a stripping may result in examples that are entirely inappropriate. ICU4J can be used to do per language transliteration. IIRC, dependency on third party code is allowed in contrib. So, it would be appropriate for such filters to be in contrib. > Illegal character replacements in ISOLatin1AccentFilter > --- > > Key: LUCENE-1029 > URL: https://issues.apache.org/jira/browse/LUCENE-1029 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Affects Versions: 2.2 >Reporter: Marko Asplund > > The ISOLatin1AccentFilter class is responsible for replacing "accented > characters in the ISO Latin 1 character set by their unaccented equivalent". > Some of the replacements performed for scandinavian characters (used e.g. in > the finnish, swedish, danish languages etc.) are illegal. The scandinavian > characters are different from the accented characters used e.g. in latin > based languages such as french in that these characters (ä, ö, å) represent > entirely independent sounds in the language and therefore cannot be > represented with any other sound without change of meaning. It is therefore > illegal to replace these characters with any other character. > This means for example that you can't change the finnish word sää (weather) > to saa (will have) because these are two entirely different words with > different meaning. The same applies to scandinavian languages as well. > There's no connection between the sounds represented by ä and a; ö and o or å > and a. > In addition to the three characters mentioned above danish and norwegian use > other special characters such as ø and æ. It should be checked if the > replacement is legal for these characters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1029) Illegal character replacements in ISOLatin1AccentFilter
[ https://issues.apache.org/jira/browse/LUCENE-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534829 ] Marko Asplund commented on LUCENE-1029: --- It's also very easy to find examples in the finnish language where the meaning of the word changes when you make the character replacements done by the filter class. Just to give you a some examples: - sää (weather) ==> saa (will have) - pässi (goat) ==> passi (passport) ... The filter class Javadoc says the following: "A filter that replaces accented characters in the ISO Latin 1 character set (ISO-8859-1) by their unaccented equivalent. The case will not be altered." In my opinion changing the meaning of a word does not qualify as an "equivalent" replacement. > Illegal character replacements in ISOLatin1AccentFilter > --- > > Key: LUCENE-1029 > URL: https://issues.apache.org/jira/browse/LUCENE-1029 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Affects Versions: 2.2 >Reporter: Marko Asplund > > The ISOLatin1AccentFilter class is responsible for replacing "accented > characters in the ISO Latin 1 character set by their unaccented equivalent". > Some of the replacements performed for scandinavian characters (used e.g. in > the finnish, swedish, danish languages etc.) are illegal. The scandinavian > characters are different from the accented characters used e.g. in latin > based languages such as french in that these characters (ä, ö, å) represent > entirely independent sounds in the language and therefore cannot be > represented with any other sound without change of meaning. It is therefore > illegal to replace these characters with any other character. > This means for example that you can't change the finnish word sää (weather) > to saa (will have) because these are two entirely different words with > different meaning. The same applies to scandinavian languages as well. > There's no connection between the sounds represented by ä and a; ö and o or å > and a. > In addition to the three characters mentioned above danish and norwegian use > other special characters such as ø and æ. It should be checked if the > replacement is legal for these characters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1029) Illegal character replacements in ISOLatin1AccentFilter
[ https://issues.apache.org/jira/browse/LUCENE-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534818 ] Karl Wettin commented on LUCENE-1029: - >> With the accent filter, running the Swedish word "kön" through the filter >> would >> create "kon". The first means "gender" and the second "cow". That would not >> be accetable. > > I am feeling lazy right now, but it seems to me you could find a similar rare > stemming > example (eg something that means something else in its stemmed form). The > process > is algorithmic after all, and there are many language with plenty of words > out there. Just to point out, pretty much any small (less than say 6 letters or so) in Swedish containing å, ä or ö would get a complete different meaning if you replace the letters. > Illegal character replacements in ISOLatin1AccentFilter > --- > > Key: LUCENE-1029 > URL: https://issues.apache.org/jira/browse/LUCENE-1029 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Affects Versions: 2.2 >Reporter: Marko Asplund > > The ISOLatin1AccentFilter class is responsible for replacing "accented > characters in the ISO Latin 1 character set by their unaccented equivalent". > Some of the replacements performed for scandinavian characters (used e.g. in > the finnish, swedish, danish languages etc.) are illegal. The scandinavian > characters are different from the accented characters used e.g. in latin > based languages such as french in that these characters (ä, ö, å) represent > entirely independent sounds in the language and therefore cannot be > represented with any other sound without change of meaning. It is therefore > illegal to replace these characters with any other character. > This means for example that you can't change the finnish word sää (weather) > to saa (will have) because these are two entirely different words with > different meaning. The same applies to scandinavian languages as well. > There's no connection between the sounds represented by ä and a; ö and o or å > and a. > In addition to the three characters mentioned above danish and norwegian use > other special characters such as ø and æ. It should be checked if the > replacement is legal for these characters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1029) Illegal character replacements in ISOLatin1AccentFilter
[ https://issues.apache.org/jira/browse/LUCENE-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534814 ] Mark Miller commented on LUCENE-1029: - > With the accent filter, running the Swedish word "kön" through the filter > would create "kon". The first means "gender" and the second "cow". That would > not be accetable. I am feeling lazy right now, but it seems to me you could find a similar rare stemming example (eg something that means something else in its stemmed form). The process is algorithmic after all, and there are many language with plenty of words out there. Regardless, it doesn't seem this filter claims it will maintain the meaning of "kön"...rather it will strip the '..' off the top of the 'o'. Its a brute force and somewhat dangerous filter from the get go...stripping accents its not a valid language operation that I know of. I'll leave at that from my side of the argument Let the Lucene gods speak. > Illegal character replacements in ISOLatin1AccentFilter > --- > > Key: LUCENE-1029 > URL: https://issues.apache.org/jira/browse/LUCENE-1029 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Affects Versions: 2.2 >Reporter: Marko Asplund > > The ISOLatin1AccentFilter class is responsible for replacing "accented > characters in the ISO Latin 1 character set by their unaccented equivalent". > Some of the replacements performed for scandinavian characters (used e.g. in > the finnish, swedish, danish languages etc.) are illegal. The scandinavian > characters are different from the accented characters used e.g. in latin > based languages such as french in that these characters (ä, ö, å) represent > entirely independent sounds in the language and therefore cannot be > represented with any other sound without change of meaning. It is therefore > illegal to replace these characters with any other character. > This means for example that you can't change the finnish word sää (weather) > to saa (will have) because these are two entirely different words with > different meaning. The same applies to scandinavian languages as well. > There's no connection between the sounds represented by ä and a; ö and o or å > and a. > In addition to the three characters mentioned above danish and norwegian use > other special characters such as ø and æ. It should be checked if the > replacement is legal for these characters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1029) Illegal character replacements in ISOLatin1AccentFilter
[ https://issues.apache.org/jira/browse/LUCENE-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534810 ] Karl Wettin commented on LUCENE-1029: - I'm on Markos line here. If you are to compare with stemmers, consider that these creates unique tokens that does not interfere with semantic meanings. With the accent filter, running the Swedish word "kön" through the filter would create "kon". The first means "gender" and the second "cow". That would not be accetable. I say this filter needs to be more configurable. > Illegal character replacements in ISOLatin1AccentFilter > --- > > Key: LUCENE-1029 > URL: https://issues.apache.org/jira/browse/LUCENE-1029 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Affects Versions: 2.2 >Reporter: Marko Asplund > > The ISOLatin1AccentFilter class is responsible for replacing "accented > characters in the ISO Latin 1 character set by their unaccented equivalent". > Some of the replacements performed for scandinavian characters (used e.g. in > the finnish, swedish, danish languages etc.) are illegal. The scandinavian > characters are different from the accented characters used e.g. in latin > based languages such as french in that these characters (ä, ö, å) represent > entirely independent sounds in the language and therefore cannot be > represented with any other sound without change of meaning. It is therefore > illegal to replace these characters with any other character. > This means for example that you can't change the finnish word sää (weather) > to saa (will have) because these are two entirely different words with > different meaning. The same applies to scandinavian languages as well. > There's no connection between the sounds represented by ä and a; ö and o or å > and a. > In addition to the three characters mentioned above danish and norwegian use > other special characters such as ø and æ. It should be checked if the > replacement is legal for these characters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1029) Illegal character replacements in ISOLatin1AccentFilter
[ https://issues.apache.org/jira/browse/LUCENE-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534804 ] Mark Miller commented on LUCENE-1029: - I think Uwe nailed this one. Stripping accents in general is just not "legal". But many times it is desirable. This filter does that for you. It goes without saying that if you strip the accent you change the meaning...likewise, when you stem a word you create illegal words... > Illegal character replacements in ISOLatin1AccentFilter > --- > > Key: LUCENE-1029 > URL: https://issues.apache.org/jira/browse/LUCENE-1029 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Affects Versions: 2.2 >Reporter: Marko Asplund > > The ISOLatin1AccentFilter class is responsible for replacing "accented > characters in the ISO Latin 1 character set by their unaccented equivalent". > Some of the replacements performed for scandinavian characters (used e.g. in > the finnish, swedish, danish languages etc.) are illegal. The scandinavian > characters are different from the accented characters used e.g. in latin > based languages such as french in that these characters (ä, ö, å) represent > entirely independent sounds in the language and therefore cannot be > represented with any other sound without change of meaning. It is therefore > illegal to replace these characters with any other character. > This means for example that you can't change the finnish word sää (weather) > to saa (will have) because these are two entirely different words with > different meaning. The same applies to scandinavian languages as well. > There's no connection between the sounds represented by ä and a; ö and o or å > and a. > In addition to the three characters mentioned above danish and norwegian use > other special characters such as ø and æ. It should be checked if the > replacement is legal for these characters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1029) Illegal character replacements in ISOLatin1AccentFilter
[ https://issues.apache.org/jira/browse/LUCENE-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534800 ] Marko Asplund commented on LUCENE-1029: --- I have to disagree, I think it's a problem that the filter makes illegal character replacements. Soundex match is different since by definition it's all about non-exact or approximate matching. In some languages accented characters may have equivalent unaccented characters with which the accented ones may be replaced without change or loss of meaning. Some of the ISOLatin1AccentFilter are legal while others are illegal. The illegal ones should be fixed. > Illegal character replacements in ISOLatin1AccentFilter > --- > > Key: LUCENE-1029 > URL: https://issues.apache.org/jira/browse/LUCENE-1029 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Affects Versions: 2.2 >Reporter: Marko Asplund > > The ISOLatin1AccentFilter class is responsible for replacing "accented > characters in the ISO Latin 1 character set by their unaccented equivalent". > Some of the replacements performed for scandinavian characters (used e.g. in > the finnish, swedish, danish languages etc.) are illegal. The scandinavian > characters are different from the accented characters used e.g. in latin > based languages such as french in that these characters (ä, ö, å) represent > entirely independent sounds in the language and therefore cannot be > represented with any other sound without change of meaning. It is therefore > illegal to replace these characters with any other character. > This means for example that you can't change the finnish word sää (weather) > to saa (will have) because these are two entirely different words with > different meaning. The same applies to scandinavian languages as well. > There's no connection between the sounds represented by ä and a; ö and o or å > and a. > In addition to the three characters mentioned above danish and norwegian use > other special characters such as ø and æ. It should be checked if the > replacement is legal for these characters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1029) Illegal character replacements in ISOLatin1AccentFilter
[ https://issues.apache.org/jira/browse/LUCENE-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534797 ] Uwe Schindler commented on LUCENE-1029: --- This is true for other european languages, too. In Germany it is also a difference between "ä" and "a" (it sounds different). A correct replacement in German would be to replace "ä" by "ae" (two chars). But I think it is not a problem. The real use of this filter is to enable people coming from other countries without the keys on their keyboard to search in a lucene index. Many americans for example search for the German last name "Müller" always by typing "Muller", because they cannot enter the umlaut. In Scandianian languages it will be the same, they would enter "o" instead of "ø". The accent filter is just to enable this. If you create an index just for one scandinavian country, just leave this filter out. And in principle it is no problem to find documents that does not match the entered keywords exact. The filter is the same like the Soundex filter. After a transformation to soundex the word lokks different and has never his original meaning :) > Illegal character replacements in ISOLatin1AccentFilter > --- > > Key: LUCENE-1029 > URL: https://issues.apache.org/jira/browse/LUCENE-1029 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Affects Versions: 2.2 >Reporter: Marko Asplund > > The ISOLatin1AccentFilter class is responsible for replacing "accented > characters in the ISO Latin 1 character set by their unaccented equivalent". > Some of the replacements performed for scandinavian characters (used e.g. in > the finnish, swedish, danish languages etc.) are illegal. The scandinavian > characters are different from the accented characters used e.g. in latin > based languages such as french in that these characters (ä, ö, å) represent > entirely independent sounds in the language and therefore cannot be > represented with any other sound without change of meaning. It is therefore > illegal to replace these characters with any other character. > This means for example that you can't change the finnish word sää (weather) > to saa (will have) because these are two entirely different words with > different meaning. The same applies to scandinavian languages as well. > There's no connection between the sounds represented by ä and a; ö and o or å > and a. > In addition to the three characters mentioned above danish and norwegian use > other special characters such as ø and æ. It should be checked if the > replacement is legal for these characters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
