Additionally, if you set the logging to Debug (with the same beam version) there should be two additional log lines printed for each Write operation, which would help significantly.
On Wed, May 25, 2016 at 10:52 AM, Thomas Groh <[email protected]> wrote: > If you inspect the output result (at /tmp/out/result-...-of-00004), once > the pipeline appears to be hung, has the pipeline produced output? The logs > that claim that the write has been finalized suggest that the pipeline is > complete, and the failure is occurring between the underlying executor and > the InProcessPipelineResult, rather than the Pipeline execution. My initial > expectation is that this may be due to an issue between the > InProcessPipelineResult and the underlying executor rather than the > executor proper. > > When the call to `result.awaitCompletion()` is removed, you see the same > behavior, correct? > > On Wed, May 25, 2016 at 8:58 AM, David Olsen <[email protected]> > wrote: > >> I try word count with InProcessPipelineRunner and it basically works. But >> I am not sure if I use the correct way to stop the pipeline running. My >> code is at http://paste.debian.net/702925 >> >> The execution prints the following messages and then seems to hang >> forever with thread not being terminated. >> >> 2016-05-25 22:37:12 INFO Write:183 - Opening writer for write operation >> org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@1b1a53a >> 2016-05-25 22:37:12 INFO Write:183 - Opening writer for write operation >> org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@40b01b >> 2016-05-25 22:37:16 INFO Write:230 - Finalizing write operation >> org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@40b01b >> 2016-05-25 22:37:16 INFO Write:230 - Finalizing write operation >> org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@1e9d9fe >> 2016-05-25 22:37:17 INFO Write:230 - Finalizing write operation >> org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@1b1a53a >> 2016-05-25 22:37:17 INFO Write:230 - Finalizing write operation >> org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@ff69ad >> >> Following code snippet is used to terminate the execution, but I don't >> know whether it's correct usage or not. >> >> final InProcessPipelineResult result = (InProcessPipelineResult) p.run(); >> result.awaitCompletion(); >> >> Also calling 'p.run();' only without obtaining result then >> awaitCompletion() has the same issue. >> >> beam version is git commit 78c8c528e23dba4f41e6ffc98b4c0d78bcea5c08 >> scala 2.11.x >> sbt 0.13.x >> >> I appreciate any suggestions. Thanks. >> > >
