Re: Running direct runner test on windows
Hi Tobiasz, It seems the cross platform pipeline option string handling is only applied to gradle tasks with "enableJavaPerformanceTesting" applied, and it's currently not applied to direct runner tests. I created this PR, do you mind reviewing it? https://github.com/apache/beam/pull/13449 Thank you, Haizhou On Mon, Nov 30, 2020 at 2:21 AM Tobiasz Kędzierski < tobiasz.kedzier...@polidea.com> wrote: > Hi Haizhou, > > I suppose these quotations mark are handled here: > > https://github.com/apache/beam/blob/8cdb8075b7d28e84591b13fb01d0144d941a5ef2/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1518 > > I was struggling with them while doing GA workflows which runs on > linux/mac/windows: > https://github.com/apache/beam/blob/master/.github/workflows/java_tests.yml > > BR > Tobiasz > > On Sun, Nov 29, 2020 at 8:45 AM Haizhou Zhao > wrote: > >> Hello Folks, >> >> I'm Haizhou, and I'm new to Beam code base. When I was running >> 'needsRunnerTests' with 'DirectRunner' on Windows 10, I found that the >> pipeline option[1] could not be parsed on Windows 10 but was running >> perfectly on my ubuntu desktop. >> >> It seems after groovy setting system property and java getting system >> property at the time of pipeline construction, Windows will drop the quotes >> so that the string >> >> ["--runner=DirectRunner", "--runnerDeterminedSharding=false"] >> >> becomes >> >> [--runner=DirectRunner, --runnerDeterminedSharding=false] >> >> which will fail the object mapper parsing. What solved the issue for me >> on Windows 10 was adding single quotes around, like >> >> ['"--runner=DirectRunner"', '"--runnerDeterminedSharding=false"'] >> >> But, the above modification does not work on ubuntu/linux. Not an expert >> on OS encoding, I was wondering if anyone has run into the same issue >> before, and is there a good way to support this test on both Operating >> Systems. >> >> Thank you, >> Haizhou Zhao >> >> [1] >> https://github.com/apache/beam/blob/master/runners/direct-java/build.gradle#L104 >> >> >>
Re: Running direct runner test on windows
Hi Haizhou, I suppose these quotations mark are handled here: https://github.com/apache/beam/blob/8cdb8075b7d28e84591b13fb01d0144d941a5ef2/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1518 I was struggling with them while doing GA workflows which runs on linux/mac/windows: https://github.com/apache/beam/blob/master/.github/workflows/java_tests.yml BR Tobiasz On Sun, Nov 29, 2020 at 8:45 AM Haizhou Zhao wrote: > Hello Folks, > > I'm Haizhou, and I'm new to Beam code base. When I was running > 'needsRunnerTests' with 'DirectRunner' on Windows 10, I found that the > pipeline option[1] could not be parsed on Windows 10 but was running > perfectly on my ubuntu desktop. > > It seems after groovy setting system property and java getting system > property at the time of pipeline construction, Windows will drop the quotes > so that the string > > ["--runner=DirectRunner", "--runnerDeterminedSharding=false"] > > becomes > > [--runner=DirectRunner, --runnerDeterminedSharding=false] > > which will fail the object mapper parsing. What solved the issue for me on > Windows 10 was adding single quotes around, like > > ['"--runner=DirectRunner"', '"--runnerDeterminedSharding=false"'] > > But, the above modification does not work on ubuntu/linux. Not an expert > on OS encoding, I was wondering if anyone has run into the same issue > before, and is there a good way to support this test on both Operating > Systems. > > Thank you, > Haizhou Zhao > > [1] > https://github.com/apache/beam/blob/master/runners/direct-java/build.gradle#L104 > > >
Running direct runner test on windows
Hello Folks, I'm Haizhou, and I'm new to Beam code base. When I was running 'needsRunnerTests' with 'DirectRunner' on Windows 10, I found that the pipeline option[1] could not be parsed on Windows 10 but was running perfectly on my ubuntu desktop. It seems after groovy setting system property and java getting system property at the time of pipeline construction, Windows will drop the quotes so that the string ["--runner=DirectRunner", "--runnerDeterminedSharding=false"] becomes [--runner=DirectRunner, --runnerDeterminedSharding=false] which will fail the object mapper parsing. What solved the issue for me on Windows 10 was adding single quotes around, like ['"--runner=DirectRunner"', '"--runnerDeterminedSharding=false"'] But, the above modification does not work on ubuntu/linux. Not an expert on OS encoding, I was wondering if anyone has run into the same issue before, and is there a good way to support this test on both Operating Systems. Thank you, Haizhou Zhao [1] https://github.com/apache/beam/blob/master/runners/direct-java/build.gradle#L104