[ https://issues.apache.org/jira/browse/SOLR-2381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003980#comment-13003980 ]
Bernd Fehling commented on SOLR-2381: ------------------------------------- Robert, unfortunately I wasn't able to build a reproducible test so I decided to debug it on my server. The bug is in Jetty and has been fixed with jetty-7.3.1.v20110307. Because I started debugging during weekend I used the older jetty.7.3.0 with the bug included, located the bug and recognized today that it had just been fixed in the new version from yesterday. Nevertheless here is the description because I went through all the bits and bytes. In jetty-7 there is jetty-server with org.eclipse.jetty.server.HttpWriter.java. That is the OutputWriter which extends Writer and does the UTF-8 encoding. The buffer comes of size 8192 bytes and is chunked and encoded with HttpWriter in sizes of 512 bytes. The encoding is that in java it is UTF-16 and is read as integer. If the code is above BMP ist has a surrogate which is read first and thereafter the next integer. Excample: 55349(dec) and 56320(dec) is converted to 119808(10) which is U+1D400 Remember that the buffer is of size 512 bytes. But what if the counter is at 510 and a Unicode above BMP comes up? The solution is to write the current buffer to output, reset it and start over with an empty buffer. And here is/was the bug. The "surrogate reminder" was cleared to early at a wrong place and got lost. If I find a svn with jetty-6.1.26 sources I will look into that one also. Otherwise use jetty-7.3.1-v20110307 that is fixed. May be we should setup a xml page for testing that has at least more than 512 characters of UTF-8 code above BMP in a row for testing? > The included jetty server does not support UTF-8 > ------------------------------------------------ > > Key: SOLR-2381 > URL: https://issues.apache.org/jira/browse/SOLR-2381 > Project: Solr > Issue Type: Bug > Reporter: Robert Muir > Assignee: Robert Muir > Priority: Blocker > Fix For: 3.1, 4.0 > > Attachments: SOLR-2381.patch, SOLR-ServletOutputWriter.patch, > jetty-6.1.26-patched-JETTY-1340.jar, jetty-util-6.1.26-patched-JETTY-1340.jar > > > Some background here: > http://www.lucidimagination.com/search/document/6babe83bd4a98b64/which_unicode_version_is_supported_with_lucene > Some possible solutions: > * wait and see if we get resolution on > http://jira.codehaus.org/browse/JETTY-1340. To be honest, I am not even sure > where jetty is being maintained (there is a separate jetty project at > eclipse.org with another bugtracker, but the older releases are at codehaus). > * include a patched version of jetty with correct utf-8, using that patch. > * remove jetty and include a different container instead. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org