Eliaaazzz opened a new issue, #39332:
URL: https://github.com/apache/beam/issues/39332
### What happened?
On Windows, every Java ValidatesRunner test fails during
`TestPipeline.create()` before any pipeline runs, because the
`beamTestPipelineOptions` system property arrives at the test JVM with its JSON
quotes stripped.
Reproduce on Windows (any runner's VR task, this is the portable Spark one):
```
gradlew.bat :runners:spark:3:job-server:validatesPortableRunnerBatch --tests
"*SplittableDoFnTest*"
```
All tests fail identically:
```
java.lang.RuntimeException at SplittableDoFnTest.java:140
Caused by: com.fasterxml.jackson.databind.JsonMappingException
Caused by: com.fasterxml.jackson.core.JsonParseException:
Unexpected character ('-' (code 45)) in numeric value: expected digit
(0-9) to follow minus sign
at [Source: UNKNOWN; line: 1, column: 4] (through reference chain:
java.lang.Object[][0])
```
`SplittableDoFnTest.java:140` is `@Rule public final transient TestPipeline
p = TestPipeline.create()`.
### Cause
`BeamModulePlugin` passes the options as compact JSON:
```groovy
config.systemProperties.put("beamTestPipelineOptions",
JsonOutput.toJson(beamTestPipelineOptions))
```
`JsonOutput.toJson` emits
`["--runner=org.apache.beam...","--jobServerDriver=..."]` with no spaces.
Gradle forks the test worker with that as a `-D` argument, and Java's
`ProcessBuilder` on Windows only quotes arguments that contain whitespace. The
argument is therefore passed unquoted and the Windows C runtime strips the
inner `"` characters, so `TestPipeline` reads:
```
[--runner=org.apache.beam...,--jobServerDriver=...]
```
`TestPipeline.testingPipelineOptions()` then does
`MAPPER.readValue(beamTestPipelineOptions, List.class)`
(TestPipeline.java:394-397). Jackson reads `[`, `-`, `-`, and reports "expected
digit to follow minus sign" at column 4, which matches the error above.
This does not affect CI, which runs on Linux.
### Prior art in the same file
`BeamModulePlugin` already handles this for one code path, wrapping each
option so a layer of quoting survives:
```groovy
// Windows handles quotation marks differently
if (pipelineOptionsString &&
System.properties['os.name'].toLowerCase().contains('windows')) {
def allOptionsListFormatted = allOptionsList.collect { "\"$it\"" }
pipelineOptionsStringFormatted = JsonOutput.toJson(allOptionsListFormatted)
} else if (pipelineOptionsString) {
pipelineOptionsStringFormatted = JsonOutput.toJson(allOptionsList)
}
```
The ValidatesRunner task paths do not do this:
- `createPortableValidatesRunnerTask`,
`config.systemProperties.put("beamTestPipelineOptions",
JsonOutput.toJson(beamTestPipelineOptions))`
- `createCrossLanguageValidatesRunnerTask`, `systemProperty
"beamTestPipelineOptions", JsonOutput.toJson(config.javaPipelineOptions)`
Applying the same treatment in those paths looks like it would fix it,
though writing the options to a file and pointing the property at it would
avoid the command-line quoting question entirely.
I ran into this while working on #19468 and could not get any local Java VR
signal on Windows, so I had to rely on CI. Happy to put up a patch if the
approach sounds right.
### Issue Priority
Priority: 3 (minor)
### Issue Components
- [x] Component: Java SDK
- [x] Component: Infrastructure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
