Hi Mathias,

> > > I assume that the char[] returned form
> > > IndexableBinaryStringTools.encode is encoded in UTF-8 again
> > > and then stored. At some point the information is lost and
> > > cannot be recovered.
> >
> > Can you give an example?  This should not happen.
> 
> My character array returned by IndexableBinaryStringTools.encode looks
> like following:
> 
> char[] encoded = new char[] {0, 8508, 3392, 64, 0, 8, 0, 0};
[...]
> BTW: I've tested it with EmbeddedSolrServer and Solr/Lucene trunk.
> 
> Why has the string representation changed? From the changed string I
> cannot decode the correct ID.

Looks to me like the returned value is in a Solr-internal form of XML character 
escaping: \u0000 is represented as "#0;" and \u0008 is represented as "#8;".  
(The escaping code is in solr/src/java/org/apache/common/util/XML.java.)  

You can get the value back in its original binary form by unescaping the 
/#[0-9]+;/ format.  Here is a test illustrating this fix that I added to 
SolrExampleTests, then ran from SolrExampleEmbeddedTest:

==============
  @Test
  public void testIndexableBinary() throws Exception {
    // Empty the database...
    server.deleteByQuery( "*:*" );// delete everything!
    server.commit();
    assertNumFound( "*:*", 0 ); // make sure it got in
 
    byte[] binary = new byte[] 
      { (byte)0, (byte)0, (byte)0x84, (byte)0xF0, (byte)0x6A, (byte)0, 
        (byte)4, (byte)0, (byte)0,    (byte)0,    (byte)2,    (byte)0 };
    int encodedLen = IndexableBinaryStringTools.getEncodedLength
      (binary, 0, binary.length);
    char encoded[] = new char[encodedLen];
    IndexableBinaryStringTools.encode
      (binary, 0, binary.length, encoded, 0, encoded.length);
    final String encodedString = new String(encoded);
    log.info("Encoded: " + stringToIntSequence(encodedString));
    // Expected encoded: {         0, 8508, 3392,   64,    0,    8,    0,    0 }
    String expectedEncoded = "\u0000\u213C\u0D40\u0040\u0000\u0008\u0000\u0000";
    assertEquals(stringToIntSequence(expectedEncoded),
                 stringToIntSequence(encodedString));
      
    SolrInputDocument doc = new SolrInputDocument();
    doc.addField("id", encodedString);
    server.add(doc);
    server.commit();
        
    SolrQuery query = new SolrQuery();
    query.setQuery("*:*");
    QueryResponse rsp = server.query(query);
    SolrDocument retrievedDoc = rsp.getResults().get(0);
    String retrievedEncoded = (String)retrievedDoc.getFieldValue("id");
    String unescapedRetrievedEncoded = 
unescapeSolrXMLEscaping(retrievedEncoded);
    assertEquals(stringToIntSequence(encodedString), 
                 stringToIntSequence(unescapedRetrievedEncoded));
  }
    
  String stringToIntSequence(String str) {
    StringBuilder builder = new StringBuilder();
    for (int chnum = 0 ; chnum < str.length() ; ++chnum) {
      if (chnum > 0) {
        builder.append(", ");
      }
      builder.append((int)str.charAt(chnum))
        .append(" (").append(str.charAt(chnum)).append(")");
    }
    return builder.toString();
  }
  String unescapeSolrXMLEscaping(String escaped) {
    StringBuffer unescaped = new StringBuffer();
    Matcher matcher = Pattern.compile("#(\\d+);").matcher(escaped);
    while (matcher.find()) {
      String replacement = String.format
        ("%c",(char)Integer.parseInt(matcher.group(1)));
      matcher.appendReplacement(unescaped, replacement); 
    }
    matcher.appendTail(unescaped);
    return unescaped.toString();
  }
==============

Steve

Reply via email to