Re: RFC: Make test failed because of the locale LANG

[David Holmes]

On 29/03/2019 1:59 am, Naoto Sato wrote:
Hi,
I don't think there is any *official* rule for the regression tests to 
succeed in any locale, but if the test case is locale sensitive such as 
in this case, I'd suggest it should correctly specify the locale 
beforehand, or quit gracefully in general.

For this specific case, I'd suggest not set the environment variable (as 
it won't work with Windows), using runtime system properties 
(user.language/user.country) would help.

Which specific case are you referring to? :)

Given there are well over a thousand failures in total I think we need 
to set a reasonable initial locale as part of the test setup. Many of 
the hotspot tests check actual and expected output and we, in general, 
have no idea what kinds of output may be subject to locale specific changes.

Cheers,
David

Naoto

On 3/27/19 10:47 PM, David Holmes wrote:
Hi Jing,

On 28/03/2019 3:22 pm, Jing Tian wrote:
Hi,

When I am doing the 'make test'.If the local LANG is set as 
'zh_CN.UTF-8', Test cases will have a lot of error messages.

==============================
Test summary
==============================
    TEST                                              TOTAL  PASS 
FAIL ERROR
 >> jtreg:test/hotspot/jtreg:tier1                     1373 1371 2 0 <<
 >> jtreg:test/jdk:tier1                               1867 1860 7 0 <<
 >> jtreg:test/langtools:tier1                         3922 2470 
1450     2 <<
    jtreg:test/nashorn:tier1                              0 0     
0     0
    jtreg:test/jaxp:tier1                                 0 0     
0     0
 >> jtreg:test/jdk:tier2                               3334 3305 29 0 <<
 >> jtreg:test/langtools:tier2 11     9     2     0 <<
    jtreg:test/nashorn:tier2                             35 35 0     0
 >> jtreg:test/jaxp:tier2                               438 437 1 0 <<
 >> jtreg:test/jdk:tier3                               1104 1068 36 0 <<
    jtreg:test/langtools:tier3                            0 0     
0     0
    jtreg:test/nashorn:tier3                              0 0     
0     0
    jtreg:test/jaxp:tier3                                 0 0     
0     0
==============================

Given most of these are not hotspot issues and given the number of 
failures, I think this is something that would need to be discussed 
much more broadly. So I've cc'd core-libs-dev and compiler-dev. I 
recall there was a very recent discussion regarding some tests failing 
for the same reason, but don't recall the outcome.

David
-----

On the same machine,when i set LANG=en_US.UTF-8.

==============================
Test summary
==============================
    TEST                                              TOTAL  PASS 
FAIL ERROR
 >> jtreg:test/hotspot/jtreg:tier1                     1388 1386 2 0 <<
 >> jtreg:test/jdk:tier1                               1867 1843 19 5 <<
    jtreg:test/langtools:tier1                         3920 3920 0     0
    jtreg:test/nashorn:tier1                              0 0     
0     0
    jtreg:test/jaxp:tier1                                 0 0     
0     0
 >> jtreg:test/jdk:tier2                               3328 3290 31 7 <<
    jtreg:test/langtools:tier2                           11 11 0     0
    jtreg:test/nashorn:tier2                             35 35 0     0
    jtreg:test/jaxp:tier2                               438 438 0     0
 >> jtreg:test/jdk:tier3                               1104 1067 37 0 <<
    jtreg:test/langtools:tier3                            0 0     
0     0
    jtreg:test/nashorn:tier3                              0 0     
0     0
    jtreg:test/jaxp:tier3                                 0 0     
0     0


By comparison we can find, lots of(1000+) langtools tests will get 
fail, and other(about 30+, exclude some X11 and sanity that

result problem) test cases will also fail because of local LANG.


such as in the test/hotspot/jtreg/compiler/c2/Test8062950.java,

shouldContain("Error: Could not find or load main class " + CLASSNAME)


When in the zh_CN.UTF-8 environment, because some of the output 
information is changed to Chinese by some properties file,

the English cannot be matched, which will result in an fail.

When I set  LANG=en_US/LC_ALL=C, this test will pass.


I think there are some possible solutions.


1.we set LC_ALL=C/LANG=EN_us in the Runtests.gmk, but this 
modification is more violent because he will affect all test cases.


2.We modify each individual test，E.g 
test/hotspot/jtreg/compiler/c2/Test8062950.java

diff -r 0421d49b6217 test/hotspot/jtreg/compiler/c2/Test8062950.java
  package compiler.c2;
  import jdk.test.lib.process.ProcessTools;
+import java.util.Map;
+import jdk.test.lib.process.OutputAnalyzer;

  public class Test8062950 {
      private static final String CLASSNAME = "DoesNotExist";
      public static void main(String[] args) throws Exception {
- ProcessTools.executeTestJvm("-Xcomp",
- "-XX:-TieredCompilation",
- "-XX:-UseOptoBiasInlining",
- CLASSNAME)
- .shouldHaveExitValue(1)
-                    .shouldContain("Error: Could not find or load 
main class " + CLASSNAME)
-                    .shouldNotContain("A fatal error has been 
detected")
-                    .shouldNotContain("Internal Error")
-                    .shouldNotContain("HotSpot Virtual Machine Error");
+        final ProcessBuilder pb = 
ProcessTools.createJavaProcessBuilder(true,
+ "-Xcomp",
+ "-XX:-TieredCompilation",
+ "-XX:-UseOptoBiasInlining",
+ CLASSNAME);
+        final Map<String, String> env = pb.environment();
+        env.put("LC_ALL", "en_US.UTF-8");
+        OutputAnalyzer output = new OutputAnalyzer(pb.start());
+ output.shouldHaveExitValue(1);
+        output.shouldContain("Error: Could not find or load main 
class " + CLASSNAME);
+        output.shouldNotContain("A fatal error has been detected");
+        output.shouldNotContain("Internal Error");
+        output.shouldNotContain("HotSpot Virtual Machine Error");
}
  }

But I don't know if this will change the test program too much, 
because the related problems are a lot in langtools tests.


3.And i find that there is a function can judge the locale

  if (!isEnglishLocale()) { // only english version
return;
  }

But in this case, I think in many test cases, we may not be able to 
return so directly, because some test cases may have other test 
purposes.


So I don't know if you have any ideas or some suggestions to solve 
the problem that the output information and English do not match in this

non-English environment.


Cheers,

Jing Tian