Yes, this browse problem has been discovered before, and there is a solution to it ;)
The ordering is dependent on the 'sort_' columns of the browse table. By altering the normalisation of the data that is stored there, you can affect the way the ordering is done, without the LC_COLLATE side effects. Applying this patch: http://sourceforge.net/tracker/index.php?func=detail&aid=1672065&group_id=19984&atid=319984 (remember to add the ICU4J jar), will decompose the non-ascii characters into the ascii main character and a separate combining character before writing it to the database. It also makes it easy to customize the normalization in other ways - for example, you could easily strip all the non-ascii combining characters after decomposition if you wanted. Or do something more advanced - like obey the correct ordering rules for a specified locale (although this would be 'baked in' to the database table for everyone, not be per user based). The patch above has minimal impact on the existing codebase - IIRC, there are only three lines in existing code that it alters. As noted in the comments, there is also Richard Jones' patch: http://sourceforge.net/tracker/index.php?func=detail&aid=1702233&group_id=19984&atid=319984 which incorporates the above ordering customisation / fixes. It's well worth considering using, although it would take a little more effort to switch over to. G On Tue, 2007-07-03 at 22:18 +0930, Steve Thomas wrote: > I’m still investigating this problem – and I can’t imagine that others have > not also run into this, although it is possible they have simply not > noticed. > > In our Author browse list we have things like this sequence: > > Hill, Robert Stephen > Hill, Roderick Jeffrey > Hills, Brian Andrew > Hills, F > Hill, S. G. > Hill, S. J. > Hills, K. > Hill, S. M. > Hill, T. > > -- you can see that the sequence seems to be ignoring punctuation AND > spaces, which is not good. The unix sort utility also produces the same > sequence. > > I suspect this is something to do with the Unix locale settings, > specifically the environment variable LC_COLLATE. > On our RedHat system, this is LC_COLLATE="en_AU.UTF-8" > On the old Solaris box (where the collation sequence is correct) we have > LC_COLLATE=en_AU.ISO8859-1 > > However, on RedHat the collation from sort is only correct if I set > LC_COLLATE=C > > > So the problem appears to be two-fold: how do we get the browse sort order > to be correct, including non-ASCII characters, AND how do we search words > with non-ASCII characters? > > > Any and all ideas welcomed. > > > > Stephen Thomas, > Senior Systems Analyst, > University of Adelaide Library > UNIVERSITY OF ADELAIDE SA 5005 AUSTRALIA > Phone: +61 8 830 35190 > Fax: +61 8 830 34369 > Email: [EMAIL PROTECTED] > URL: http://www.adelaide.edu.au/directory/stephen.thomas > CRICOS Provider Number 00123M > > ----------------------------------------------------------- > This email message is intended only for the addressee(s) and contains > information that may be confidential and/or copyright. If you are not the > intended recipient please notify the sender by reply email and immediately > delete this email. Use, disclosure or reproduction of this email by anyone > other than the intended recipient(s) is strictly prohibited. No > representation is made that this email or any attachments are free of > viruses. Virus scanning is recommended and is the responsibility of the > recipient. > ________________________________________ > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Steve Thomas > Sent: Monday, 2 July 2007 3:28 pm > To: [email protected] > Subject: [Dspace-tech] Non-ASCII Unicode and searching in DSpace > > Hi. > > We recently moved from a Solaris box to an HP running RHEL. Now I find that > search and browse no longer work with words containing non-ASCII Unicode > characters, e.g. the name Mühlhäusler. This doesn’t show up in the browse > list, and searches fail to find it. > > Was there something we missed in the installation, or is this an artefact of > using RHEL? > > We’re still on DSpace 1.3.2 > > Display is fine, just the searching is a problem. > > > Cheers, > Steve > > Stephen Thomas, > Senior Systems Analyst, > University of Adelaide Library > UNIVERSITY OF ADELAIDE SA 5005 AUSTRALIA > Phone: +61 8 830 35190 > Fax: +61 8 830 34369 > Email: [EMAIL PROTECTED] > URL: http://www.adelaide.edu.au/directory/stephen.thomas > CRICOS Provider Number 00123M > > ----------------------------------------------------------- > This email message is intended only for the addressee(s) and contains > information that may be confidential and/or copyright. If you are not the > intended recipient please notify the sender by reply email and immediately > delete this email. Use, disclosure or reproduction of this email by anyone > other than the intended recipient(s) is strictly prohibited. No > representation is made that this email or any attachments are free of > viruses. Virus scanning is recommended and is the responsibility of the > recipient. > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > DSpace-tech mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/dspace-tech This email has been scanned by Postini. For more information please visit http://www.postini.com ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ DSpace-tech mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/dspace-tech
