Hey Till, Thanks for the reply -- I'll try to see if I can reproduce this in a small repo and share it with you.
Best, Austin On Tue, Sep 29, 2020 at 3:58 AM Till Rohrmann <trohrm...@apache.org> wrote: > Hi Austin, > > could you share with us the exact job you are running (including the > custom window trigger)? This would help us to better understand your > problem. > > I am also pulling in Klou and Timo who might help with the windowing logic > and the Table to DataStream conversion. > > Cheers, > Till > > On Mon, Sep 28, 2020 at 11:49 PM Austin Cawley-Edwards < > austin.caw...@gmail.com> wrote: > >> Hey all, >> >> I'm not sure if I've missed something in the docs, but I'm having a bit >> of trouble with a streaming SQL job that starts w/ raw SQL queries and then >> transitions to a more traditional streaming job. I'm on Flink 1.10 using >> the Blink planner, running locally with no checkpointing. >> >> The job looks roughly like: >> >> CSV 1 --> >> CSV 2 --> SQL Query to Join --> toRetractStream --> keyed time window w/ >> process func & custom trigger --> some other ops >> CSV 3 --> >> >> >> When I remove the windowing directly after the `toRetractStream`, the >> records make it to the "some other ops" stage, but with the windowing, >> those operations are sometimes not sent any data. I can also get data sent >> to the downstream operators by putting in a no-op map before the window and >> placing some breakpoints in there to manually slow down processing. >> >> >> The logs don't seem to indicate anything went wrong and generally look >> like: >> >> 4819 [Source: Custom File source (1/1)] INFO >> org.apache.flink.runtime.taskmanager.Task - Source: Custom File source >> (1/1) (3578629787c777320d9ab030c004abd4) switched from RUNNING to >> FINISHED.\4819 [Source: Custom File source (1/1)] INFO >> org.apache.flink.runtime.taskmanager.Task - Freeing task resources for >> Source: Custom File source (1/1) (3578629787c777320d9ab030c004abd4). >> 4819 [Source: Custom File source (1/1)] INFO >> org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem >> streams are closed for task Source: Custom File source (1/1) >> (3578629787c777320d9ab030c004abd4) [FINISHED] >> 4820 [flink-akka.actor.default-dispatcher-5] INFO >> org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task >> and sending final execution state FINISHED to JobManager for task Source: >> Custom File source (1/1) 3578629787c777320d9ab030c004abd4. >> ... >> 4996 [Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, >> ProcessWindowFunction$1) (1/1)] INFO >> org.apache.flink.runtime.taskmanager.Task - >> Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, >> ProcessWindowFunction$1) (1/1) (907acf9bfa2f4a9bbd13997b8b34d91f) switched >> from RUNNING to FINISHED. >> 4996 [Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, >> ProcessWindowFunction$1) (1/1)] INFO >> org.apache.flink.runtime.taskmanager.Task - Freeing task resources for >> Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, >> ProcessWindowFunction$1) (1/1) (907acf9bfa2f4a9bbd13997b8b34d91f). >> 4996 [Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, >> ProcessWindowFunction$1) (1/1)] INFO >> org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem >> streams are closed for task Window(TumblingProcessingTimeWindows(60000), >> TimedCountTrigger, ProcessWindowFunction$1) (1/1) >> (907acf9bfa2f4a9bbd13997b8b34d91f) [FINISHED] >> ... >> rest of the shutdown >> ... >> Program execution finished >> Job with JobID 889b161e432c0e69a8d760bbed205d5d has finished. >> Job Runtime: 783 ms >> >> >> Is there something I'm missing in my setup? Could it be my custom window >> trigger? Bug? I'm stumped. >> >> >> Thanks, >> Austin >> >> >>
