- Upstream steps seem to be slowed down by this PTransform (system lag up and elements/sec down) - Unbounded source is PubSubIO
Jacob On Sun, Oct 29, 2017 at 9:22 PM, Jacob Marble <[email protected]> wrote: > - This does not happen when I don't use the reshuffle hack > - HTTP QPS seems to be improved with reshuffling, but also seems to > burst-and-pause > - The "processing lull" log entry occurs 4 times every 5 minutes > - Right now, I'm guessing that "processing lull" means that a map > operation is taking too long > > Jacob > > On Sun, Oct 29, 2017 at 9:15 PM, Jacob Marble <[email protected]> wrote: > >> Good evening- >> >> What should I make of the log warning "processing lull for [instant] in >> state windmill-read" ? >> >> - This happens in a streaming pipeline, in Dataflow. >> - The DoFn that emits the log entry makes HTTP requests to a third-party. >> - This only happens when I added a side input to the PTransform, to >> prevent fusing. >> - The side input is a SingletonView, just an empty string value. >> >> Thanks as usual, >> >> Jacob >> >> Processing lull for PT300.124S in state windmill-read of [step name] >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) >> at com.google.cloud.dataflow.worker.repackaged.com.google.commo >> n.util.concurrent.AbstractFuture.get(AbstractFuture.java:469) >> at com.google.cloud.dataflow.worker.repackaged.com.google.commo >> n.util.concurrent.AbstractFuture$TrustedFuture.get( >> AbstractFuture.java:76) >> at com.google.cloud.dataflow.worker.MetricTrackingWindmillServe >> rStub.getStateData(MetricTrackingWindmillServerStub.java:188) >> at com.google.cloud.dataflow.worker.WindmillStateReader.startBa >> tchAndBlock(WindmillStateReader.java:405) >> at com.google.cloud.dataflow.worker.WindmillStateReader$Wrapped >> Future.get(WindmillStateReader.java:306) >> at com.google.cloud.dataflow.worker.WindmillStateInternals$Wind >> millValue.read(WindmillStateInternals.java:384) >> at com.google.cloud.dataflow.worker.StreamingSideInputFetcher.b >> lockedMap(StreamingSideInputFetcher.java:249) >> at com.google.cloud.dataflow.worker.StreamingSideInputFetcher.s >> toreIfBlocked(StreamingSideInputFetcher.java:186) >> at com.google.cloud.dataflow.worker.StreamingSideInputDoFnRunne >> r.processElement(StreamingSideInputDoFnRunner.java:70) >> at com.google.cloud.dataflow.worker.SimpleParDoFn.processElemen >> t(SimpleParDoFn.java:233) >> at com.google.cloud.dataflow.worker.util.common.worker.ParDoOpe >> ration.process(ParDoOperation.java:48) >> at com.google.cloud.dataflow.worker.util.common.worker.OutputRe >> ceiver.process(OutputReceiver.java:52) >> at com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(Simp >> leParDoFn.java:183) >> > >
