and it seems be due to TimerType User Thanks Mohil
On Sun, Jul 26, 2020 at 1:42 PM Mohil Khare <mo...@prosimo.io> wrote: > Hello, > > I was looking at source code of > https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputDoFnRunner.java > , > > It seems the default implementation of OnTimer is to throw (Attempt to > deliver a timer to a DoFn, but timers are not supported in Dataflow). > > Do you know under what circumstances, My code might be throwing this ? > Not sure if its some issue in 2.19 which might have been fixed now with > 2.22 > > Thanks and Regards > Mohil > > > On Fri, Jul 24, 2020 at 5:21 PM Mohil Khare <mo...@prosimo.io> wrote: > >> Actually NOT JUST DRAIN, seeing it during regular RUN as well. It's >> getting flooded with this exception >> >> Attempt to deliver a timer to a DoFn, but timers are not supported in >> Dataflow. >> The changes that I did are the following: >> 1. Read one set of logs from one of the kafka topics and create KV(ID, >> log1) >> 2. Read 2nd set of logs from another kafka topic and create KV(ID, log2) >> 3. Put above in session window >> 4. CoGroupByKey both logs >> >> Apart from aforementioned exceptions, I am also seeing the following in >> worker logs. What I have noticed is that it happens when the pipeline is >> silent for a while. i.e. no new logs to be read from Kafka (Not sure if it >> is the actual reason). >> >> 2020-07-24 17:06:43.532 PDT >> Execution of work for P188 for key >> cloud@prosimo.ioHe8fc8079-c844-11ea-a6d5-dabe1eb9c630 failed. Will retry >> locally. >> >> <https://console.cloud.google.com/logs/viewer?advancedFilter=insertId%3D%224729149996009968456:27267:0:42419165%22%20resource.type%3D%22dataflow_step%22%20resource.labels.job_id%3D%222020-07-24_16_23_20-4320159532106629306%22%20(logName%3D%22projects%2Fprosimo-test%2Flogs%2Fdataflow.googleapis.com%252Fworker%22)%20severity%3E%3DERROR%0A(timestamp%3E%222020-07-25T00:06:33.224Z%22%20OR%20(insertId%3E%224729149996009968456:27267:0:42264272%22%20AND%20timestamp%3D%222020-07-25T00:06:33.224Z%22))&expandAll=true&interval=JUMP_TO_TIME&dateRangeStart=2020-07-25T00:06:43.532Z&dateRangeEnd=2020-07-25T00:06:43.532Z&project=prosimo-test> >> 2020-07-24 17:06:53.863 PDT >> Uncaught exception: >> >> >> >> On Fri, Jul 24, 2020 at 2:59 PM Mohil Khare <mo...@prosimo.io> wrote: >> >>> Hello, >>> >>> I am on java sdk 2.19 and using dataflow for beam job. I use Timers for >>> my stateful transformations, but recently I started seeing the following >>> exception on DRAINING a job. It used to work fine and not sure what changed. >>> >>> *java.lang.UnsupportedOperationException:* >>> >> *Attempt to deliver a timer to a DoFn, but timers are not supported in >> Dataflow. * >> >>> >>> 1. >>> 1. atorg.apache.beam.runners.dataflow.worker. >>> StreamingSideInputDoFnRunner.onTimer ( >>> StreamingSideInputDoFnRunner.java:86 >>> >>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test> >>> ) >>> 2. atorg.apache.beam.runners.dataflow.worker. >>> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360 >>> >>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test> >>> ) >>> 3. atorg.apache.beam.runners.dataflow.worker. >>> SimpleParDoFn.access$600 (SimpleParDoFn.java:73 >>> >>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test> >>> ) >>> 4. atorg.apache.beam.runners.dataflow.worker. >>> SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444 >>> >>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test> >>> ) >>> 5. atorg.apache.beam.runners.dataflow.worker. >>> SimpleParDoFn.processTimers (SimpleParDoFn.java:473 >>> >>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test> >>> ) >>> 6. atorg.apache.beam.runners.dataflow.worker. >>> SimpleParDoFn.processTimers (SimpleParDoFn.java:353 >>> >>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test> >>> ) >>> 7. atorg.apache.beam.runners.dataflow.worker.util.common.worker. >>> ParDoOperation.finish (ParDoOperation.java:52 >>> >>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test> >>> ) >>> 8. atorg.apache.beam.runners.dataflow.worker.util.common.worker. >>> MapTaskExecutor.execute (MapTaskExecutor.java:85 >>> >>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test> >>> ) >>> 9. atorg.apache.beam.runners.dataflow.worker. >>> StreamingDataflowWorker.process (StreamingDataflowWorker.java:1350 >>> >>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test> >>> ) >>> 10. atorg.apache.beam.runners.dataflow.worker. >>> StreamingDataflowWorker.access$1100 ( >>> StreamingDataflowWorker.java:152 >>> >>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test> >>> ) >>> 2. Sometime back I opened jira for an issue which is related to this >>> while doing DRAIN: https://issues.apache.org/jira/browse/BEAM-10053 >>> (Looks >>> like no one has taken a stab on this Jira) >>> 3. Not sure if the reason is same and due to multiple side inputs >>> that i use in my PTransforms. >>> >>> >>> Any help would be appreciated. >>> >>> Thanks and Regards >>> Mohil >>> >>>
