Eliaaazzz commented on issue #39332:
URL: https://github.com/apache/beam/issues/39332#issuecomment-4977221400
Some follow-up, since the report above was based on a single run.
### It is deterministic, not flaky
Three runs of the portable Spark batch VR on Windows, every test failing the
same way at `TestPipeline.create()`:
| run | filter | result |
|---|---|---|
| 1 | `*SplittableDoFnTest*` | 9 tests, 9 failed, 9 `JsonParseException` |
| 2 | `*SplittableDoFnTest*` | 9 tests, 9 failed, 9 `JsonParseException` |
| 3 | `*ParDoTest*` | 59 tests, 59 failed, 59 `JsonParseException` |
77 of 77, and every failure carries the same `through reference chain:
java.lang.Object[][0]`. It is not specific to a test class, and nothing passes.
### Minimal reproduction of the mechanism
The quote loss can be reproduced without Gradle or Beam. This mimics how the
Test task forks a worker: a `-D` property whose value is compact JSON, passed
through `ProcessBuilder`.
```java
// PropPrinter.java
public class PropPrinter {
public static void main(String[] args) {
System.out.println("CHILD_RECEIVED=" +
System.getProperty("beamTestPipelineOptions"));
}
}
// Launcher.java
import java.io.File;
import java.util.Arrays;
public class Launcher {
public static void main(String[] args) throws Exception {
String json =
"[\"--runner=org.apache.beam.runners.portability.testing.TestPortableRunner\",\"--experiments=beam_fn_api\"]";
System.out.println("PARENT_SENT =" + json);
String java = System.getProperty("java.home") + File.separator + "bin" +
File.separator + "java";
ProcessBuilder pb = new ProcessBuilder(
Arrays.asList(java, "-DbeamTestPipelineOptions=" + json, "-cp",
System.getProperty("user.dir"), "PropPrinter"));
pb.redirectErrorStream(true);
Process p = pb.start();
System.out.print(new String(p.getInputStream().readAllBytes()));
p.waitFor();
}
}
```
On Windows with JDK 11:
```
PARENT_SENT
=["--runner=org.apache.beam.runners.portability.testing.TestPortableRunner","--experiments=beam_fn_api"]
CHILD_RECEIVED=[--runner=org.apache.beam.runners.portability.testing.TestPortableRunner,--experiments=beam_fn_api]
```
Every quote is gone, which is exactly the `[--runner=...` that Jackson
rejects at the start of the array. `ProcessBuilder` only quotes arguments
containing whitespace, and `JsonOutput.toJson` emits no whitespace, so the
argument goes through unquoted and the C runtime strips the quotes. The same
value survives on Linux, which is why CI never sees this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
