Hello, I was looking at source code of https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputDoFnRunner.java ,
It seems the default implementation of OnTimer is to throw (Attempt to deliver a timer to a DoFn, but timers are not supported in Dataflow). Do you know under what circumstances, My code might be throwing this ? Not sure if its some issue in 2.19 which might have been fixed now with 2.22 Thanks and Regards Mohil On Fri, Jul 24, 2020 at 5:21 PM Mohil Khare <mo...@prosimo.io> wrote: > Actually NOT JUST DRAIN, seeing it during regular RUN as well. It's > getting flooded with this exception > > Attempt to deliver a timer to a DoFn, but timers are not supported in > Dataflow. > The changes that I did are the following: > 1. Read one set of logs from one of the kafka topics and create KV(ID, > log1) > 2. Read 2nd set of logs from another kafka topic and create KV(ID, log2) > 3. Put above in session window > 4. CoGroupByKey both logs > > Apart from aforementioned exceptions, I am also seeing the following in > worker logs. What I have noticed is that it happens when the pipeline is > silent for a while. i.e. no new logs to be read from Kafka (Not sure if it > is the actual reason). > > 2020-07-24 17:06:43.532 PDT > Execution of work for P188 for key > cloud@prosimo.ioHe8fc8079-c844-11ea-a6d5-dabe1eb9c630 failed. Will retry > locally. > > <https://console.cloud.google.com/logs/viewer?advancedFilter=insertId%3D%224729149996009968456:27267:0:42419165%22%20resource.type%3D%22dataflow_step%22%20resource.labels.job_id%3D%222020-07-24_16_23_20-4320159532106629306%22%20(logName%3D%22projects%2Fprosimo-test%2Flogs%2Fdataflow.googleapis.com%252Fworker%22)%20severity%3E%3DERROR%0A(timestamp%3E%222020-07-25T00:06:33.224Z%22%20OR%20(insertId%3E%224729149996009968456:27267:0:42264272%22%20AND%20timestamp%3D%222020-07-25T00:06:33.224Z%22))&expandAll=true&interval=JUMP_TO_TIME&dateRangeStart=2020-07-25T00:06:43.532Z&dateRangeEnd=2020-07-25T00:06:43.532Z&project=prosimo-test> > 2020-07-24 17:06:53.863 PDT > Uncaught exception: > > > > On Fri, Jul 24, 2020 at 2:59 PM Mohil Khare <mo...@prosimo.io> wrote: > >> Hello, >> >> I am on java sdk 2.19 and using dataflow for beam job. I use Timers for >> my stateful transformations, but recently I started seeing the following >> exception on DRAINING a job. It used to work fine and not sure what changed. >> >> *java.lang.UnsupportedOperationException:* >> > *Attempt to deliver a timer to a DoFn, but timers are not supported in > Dataflow. * > >> >> 1. >> 1. atorg.apache.beam.runners.dataflow.worker. >> StreamingSideInputDoFnRunner.onTimer ( >> StreamingSideInputDoFnRunner.java:86 >> >> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test> >> ) >> 2. atorg.apache.beam.runners.dataflow.worker. >> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360 >> >> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test> >> ) >> 3. atorg.apache.beam.runners.dataflow.worker. >> SimpleParDoFn.access$600 (SimpleParDoFn.java:73 >> >> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test> >> ) >> 4. atorg.apache.beam.runners.dataflow.worker. >> SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444 >> >> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test> >> ) >> 5. atorg.apache.beam.runners.dataflow.worker. >> SimpleParDoFn.processTimers (SimpleParDoFn.java:473 >> >> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test> >> ) >> 6. atorg.apache.beam.runners.dataflow.worker. >> SimpleParDoFn.processTimers (SimpleParDoFn.java:353 >> >> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test> >> ) >> 7. atorg.apache.beam.runners.dataflow.worker.util.common.worker. >> ParDoOperation.finish (ParDoOperation.java:52 >> >> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test> >> ) >> 8. atorg.apache.beam.runners.dataflow.worker.util.common.worker. >> MapTaskExecutor.execute (MapTaskExecutor.java:85 >> >> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test> >> ) >> 9. atorg.apache.beam.runners.dataflow.worker. >> StreamingDataflowWorker.process (StreamingDataflowWorker.java:1350 >> >> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test> >> ) >> 10. atorg.apache.beam.runners.dataflow.worker. >> StreamingDataflowWorker.access$1100 ( >> StreamingDataflowWorker.java:152 >> >> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test> >> ) >> 2. Sometime back I opened jira for an issue which is related to this >> while doing DRAIN: https://issues.apache.org/jira/browse/BEAM-10053 (Looks >> like no one has taken a stab on this Jira) >> 3. Not sure if the reason is same and due to multiple side inputs >> that i use in my PTransforms. >> >> >> Any help would be appreciated. >> >> Thanks and Regards >> Mohil >> >>
