[
https://issues.apache.org/jira/browse/LUCENE-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615766#comment-14615766
]
Ramkumar Aiyengar commented on LUCENE-6563:
-------------------------------------------
bq. I honestly think this is a quirk/bug in the JVM... and perhaps should be
reported. Setting LANG=C shouldn't be affecting how filenames are (mis)handled
(and it currently does).
I digged a bit, and it appears that Java is kind of doing the right thing given
it's current API. Certain newer versions/FSs of Windows and MacOSX guarantee
that all files are in UTF-16/8 respectively. Linux/Solaris etc. (aka the more
traditional Unix systems) tend not to care about the encoding at all. As a
matter of fact, a filename is just a sequence of bytes, it's not even a string.
How that byte array is displayed comes down to the locale. This is probably why
this works.
{code}
$ LANG=C touch 中国
{code}
touch doesn't care about the input, the shell maps it into a set of UTF-8
bytes, and is stored as a filename. ls, then, in an UTF-8 locale shows the
correct thing
{code}
$ ls
build.xml ivy.xml lib lucene-test-framework.iml src 中国
{code}
And if I try to read in the C locale, I get a bunch of unreadable chars
{code}
$ LANG=C ls
build.xml ivy.xml lib lucene-test-framework.iml src ??????
{code}
Java APIs on the other hand treats filenames as strings in all it's APIs, and
as a result is forced to need an encoding even when it is using it only for I/O
and not for display. As a result, it is forced to choose some encoding, and it
goes with the locale.. In platforms where the filename encoding is guaranteed
to be UTF-8, it goes with that -- see
http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8003228 for MacOSX.
Looks like this issue is not specific to Java -- see
https://bugs.python.org/issue19846 this for example.
> MockFileSystemTestCase.testURI should be improved to handle cases where
> OS/JVM cannot create non-ASCII filenames
> ----------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-6563
> URL: https://issues.apache.org/jira/browse/LUCENE-6563
> Project: Lucene - Core
> Issue Type: Wish
> Reporter: Christine Poerschke
> Assignee: Dawid Weiss
> Priority: Minor
>
> {{ant test -Dtestcase=TestVerboseFS -Dtests.method=testURI
> -Dtests.file.encoding=UTF-8}} fails (for example) with 'Oracle Corporation
> 1.8.0_45 (64-bit)' when the default {{sun.jnu.encoding}} system property is
> (for example) {{ANSI_X3.4-1968}}
> [details to follow]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
