If you inspect the output result (at /tmp/out/result-...-of-00004), once the pipeline appears to be hung, has the pipeline produced output? The logs that claim that the write has been finalized suggest that the pipeline is complete, and the failure is occurring between the underlying executor and the InProcessPipelineResult, rather than the Pipeline execution. My initial expectation is that this may be due to an issue between the InProcessPipelineResult and the underlying executor rather than the executor proper.
When the call to `result.awaitCompletion()` is removed, you see the same behavior, correct? On Wed, May 25, 2016 at 8:58 AM, David Olsen <[email protected]> wrote: > I try word count with InProcessPipelineRunner and it basically works. But > I am not sure if I use the correct way to stop the pipeline running. My > code is at http://paste.debian.net/702925 > > The execution prints the following messages and then seems to hang forever > with thread not being terminated. > > 2016-05-25 22:37:12 INFO Write:183 - Opening writer for write operation > org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@1b1a53a > 2016-05-25 22:37:12 INFO Write:183 - Opening writer for write operation > org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@40b01b > 2016-05-25 22:37:16 INFO Write:230 - Finalizing write operation > org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@40b01b > 2016-05-25 22:37:16 INFO Write:230 - Finalizing write operation > org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@1e9d9fe > 2016-05-25 22:37:17 INFO Write:230 - Finalizing write operation > org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@1b1a53a > 2016-05-25 22:37:17 INFO Write:230 - Finalizing write operation > org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@ff69ad > > Following code snippet is used to terminate the execution, but I don't > know whether it's correct usage or not. > > final InProcessPipelineResult result = (InProcessPipelineResult) p.run(); > result.awaitCompletion(); > > Also calling 'p.run();' only without obtaining result then > awaitCompletion() has the same issue. > > beam version is git commit 78c8c528e23dba4f41e6ffc98b4c0d78bcea5c08 > scala 2.11.x > sbt 0.13.x > > I appreciate any suggestions. Thanks. >
