I don't think any of the gotchas apply to me (at the bottom of this link). https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/testing.html#junit-rule-miniclusterwithclientresource
I'm assuming for a batch job that I don't have to do anything for: "You can implement a custom parallel source function for emitting watermarks if your job uses event time timers." On Tue, Oct 6, 2020 at 2:42 PM Dan Hill <quietgol...@gmail.com> wrote: > I've tried to enable additional logging for a few hours today. I think > something with junit5 is swallowing the logs. I'm using Bazel and junit5. > I setup MiniClusterResourceConfiguration using a custom extension. Are > there any known issues with Flink and junit5? I can try switching to > junit4. > > When I've binary searched this issue, this failure happens if my query in > step 3 has a join it. If I remove the join, I can remove step 4 and the > code still works. I've renamed a bunch of my tables too and the problem > still exists. > > > > > > On Tue, Oct 6, 2020, 00:42 Aljoscha Krettek <aljos...@apache.org> wrote: > >> Hi Dan, >> >> there were some bugs and quirks in the MiniCluster that we recently fixed: >> >> - https://issues.apache.org/jira/browse/FLINK-19123 >> - https://issues.apache.org/jira/browse/FLINK-19264 >> >> But I think they are probably unrelated to your case. Could you enable >> logging and see from the logs whether the 2) and 3) jobs execute >> correctly on the MiniCluster? >> >> Best, >> Aljoscha >> >> On 06.10.20 08:08, Dan Hill wrote: >> > I'm writing a test for a batch job using >> MiniClusterResourceConfiguration. >> > >> > Here's a simple description of my working test case: >> > 1) I use TableEnvironment.executeSql(...) to create a source and sink >> table >> > using tmp filesystem directory. >> > 2) I use executeSql to insert some test data into the source tabel. >> > 3) I use executeSql to select from source and insert into sink. >> > 4) I use executeSql from the same source to a different sink. >> > >> > When I do these steps, it works. If I remove step 4, no data gets >> written >> > to the sink. My actual code is more complex than this (has create view, >> > join and more tables). This is a simplified description but highlights >> the >> > weird error. >> > >> > Has anyone hit issues like this? I'm assuming I have a small code bug >> in >> > my queries that's causing issues. These queries appear to work in >> > production so I'm confused. Are there ways of viewing failed jobs or >> > queries with MiniClusterResourceConfiguration? >> > >> > Thanks! >> > - Dan >> > >> >>
