Thanks Udi, is there a good example for either of these? #1 - seems like you have to rewrite your assertion logic without the PAssert? Is there some way to capture the pipeline output and iterate over it? The pattern I have seen for this in the past also has thread safety issues (Using a DoFn at the end of the pipeline to add the output to a collection is not safe since the collection can be executed concurrently) #2 - Would BigqueryMatcher be a good example for this? which is used in BigQueryTornadoesIT.java Or is there another example you would suggest looking at for reference?
- I guess to this you need to implement the SerializableMatcher interface and use the matcher as an option in the pipeline options. On Tue, Jan 22, 2019 at 4:28 PM Udi Meiri <[email protected]> wrote: > Some options: > - You could wait to assert until after p.waitForFinish(). > - You could PAssert using SerializableMatcher and allow any > lastModifiedTime. > > On Tue, Jan 22, 2019 at 3:56 PM Alex Amato <[email protected]> wrote: > >> +Jeff, Eugene, >> >> Hi Jeff and Eugene, >> >> I've noticed that Jeff's PR >> <https://github.com/apache/beam/commit/410d6c7b5f933dcb0280894553c1e576ee4e4884> >> introduced >> a race condition in this test, but its not clear exactly how to add Jeff's >> test check in a thread safe way. I believe this to be the source of the >> flakeyness Do you have any suggestions Eugene (since you authored this >> test)? >> >> I added some details to this JIRA issue explaining in full >> https://jira.apache.org/jira/browse/BEAM-6491?filter=-2 >> >> >> On Tue, Jan 22, 2019 at 3:34 PM Alex Amato <[email protected]> wrote: >> >>> I've seen this fail in a few different PRs for different contributors, >>> and its causing some issues during the presubmit process.. This is a >>> multithreadred test with a lot of sleeps, so it looks a bit suspicious as >>> the source of the problem. >>> >>> https://builds.apache.org/job/beam_PreCommit_Java_Commit/3688/testReport/org.apache.beam.sdk.io/FileIOTest/testMatchWatchForNewFiles/ >>> >>> I filed a JIRA for this issue: >>> https://jira.apache.org/jira/browse/BEAM-6491?filter=-2 >>> >>> >>>
