Re: failed tests because of wrong locale

[Stefan Oestreicher]

Shalin Shekhar Mangar schrieb:

Hi Stefan,

Do you mind opening a jira issue and attaching these patches there?

not at all: https://issues.apache.org/jira/browse/SOLR-700

For NumberFormatTransformer, I guess we should add another attribute for
locale which should be used for parsing. The reason is that the data you are
processing may be coming from a different locale than the system locale.
What do you think?

I totally agree.

On Thu, Aug 14, 2008 at 2:33 AM, Stefan Oestreicher <
[EMAIL PROTECTED]> wrote:

Hi,

I just ran the unit tests on my system (uses a german locale) and the
DateFieldTest and LegacyDateFieldTest failed because the expected decimal
format used "." as separator but the actual result was formatted using the
"," separator.
This happens because the ISO8601CanonicalDateFormat class which is defined
inside the DateField class doesn't specify the US locale when creating the
formatter used for milliseconds.
I attached a patch which fixes that problem
(schema.DateField-locale.patch.txt).

TestNumberFormatTransformer in contrib/dataimporthandler also failed
because handler.dataimport.NumberFormatTransformer uses a NumberFormatter
which relies on the system locale. But I think in this case it's
intentional, so I modified the test case to use the grouping separator of
the system locale
(handler.dataimport.TestNumberFormatTransformer-locale.patch.txt).

After applying those changes all tests succeeded.

kind regards,

Stefan

### Eclipse Workspace Patch 1.0
#P solr
Index:
contrib/dataimporthandler/src/test/java/org/apache/solr/handler/dataimport/TestNumberFormatTransformer.java
===================================================================
---
contrib/dataimporthandler/src/test/java/org/apache/solr/handler/dataimport/TestNumberFormatTransformer.java
(revision 685675)
+++
contrib/dataimporthandler/src/test/java/org/apache/solr/handler/dataimport/TestNumberFormatTransformer.java
(working copy)
@@ -19,6 +19,7 @@
 import org.junit.Assert;
 import org.junit.Test;

+import java.text.DecimalFormatSymbols;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
@@ -32,6 +33,9 @@
 * @since solr 1.3
 */
 public class TestNumberFormatTransformer {
+  private static final char GRP_SEP = new DecimalFormatSymbols()
+    .getGroupingSeparator();
+
  @Test
  @SuppressWarnings("unchecked")
  public void testTransformRow_SingleNumber() {
@@ -40,7 +44,7 @@
            NumberFormatTransformer.FORMAT_STYLE,
NumberFormatTransformer.NUMBER));
    Context c = AbstractDataImportHandlerTest.getContext(null, null, null,
0,
            l, null);
-    Map m = AbstractDataImportHandlerTest.createMap("num", "123,567");
+    Map m = AbstractDataImportHandlerTest.createMap("num",
"123"+GRP_SEP+"567");
    new NumberFormatTransformer().transformRow(m, c);
    Assert.assertEquals(new Long(123567), m.get("num"));
  }
@@ -56,8 +60,8 @@
            NumberFormatTransformer.FORMAT_STYLE,
NumberFormatTransformer.NUMBER));

    List inputs = new ArrayList();
-    inputs.add("123,567");
-    inputs.add("245,678");
+    inputs.add("123"+GRP_SEP+"567");
+    inputs.add("245"+GRP_SEP+"678");
    Map row = AbstractDataImportHandlerTest.createMap("inputs", inputs);

    VariableResolverImpl resolver = new VariableResolverImpl();

### Eclipse Workspace Patch 1.0
#P solr
Index: src/java/org/apache/solr/schema/DateField.java
===================================================================
--- src/java/org/apache/solr/schema/DateField.java      (revision 685675)
+++ src/java/org/apache/solr/schema/DateField.java      (working copy)
@@ -31,6 +31,7 @@
 import java.util.Date;
 import java.util.TimeZone;
 import java.util.Locale;
+import java.text.DecimalFormatSymbols;
 import java.text.SimpleDateFormat;
 import java.text.DateFormat;
 import java.text.NumberFormat;
@@ -245,7 +246,8 @@
    protected NumberFormat millisParser
      = NumberFormat.getIntegerInstance(Locale.US);

-    protected NumberFormat millisFormat = new DecimalFormat(".###");
+    protected NumberFormat millisFormat = new DecimalFormat(".###",
+      new DecimalFormatSymbols(Locale.US));

    public ISO8601CanonicalDateFormat() {
      super("yyyy-MM-dd'T'HH:mm:ss", Locale.US);
@@ -295,7 +297,8 @@
      ISO8601CanonicalDateFormat c
        = (ISO8601CanonicalDateFormat) super.clone();
      c.millisParser = NumberFormat.getIntegerInstance(Locale.US);
-      c.millisFormat = new DecimalFormat(".###");
+      c.millisFormat = new DecimalFormat(".###",
+        new DecimalFormatSymbols(Locale.US));
      return c;
    }
  }