[ https://issues.apache.org/jira/browse/BEAM-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15948290#comment-15948290 ]
ASF GitHub Bot commented on BEAM-1835: -------------------------------------- GitHub user tgroh opened a pull request: https://github.com/apache/beam/pull/2368 [BEAM-1835] Make Pubsub ackBatch safer Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-<Jira issue #>] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- Checkpoints can be finalized at any point, including while the reader is being closed or after it has been. When ackBatch is called and the reader is closed, it should return. When close is called while a different thread is using the client, it should wait for that thread to complete before closing the reader. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tgroh/beam pubsub_npe Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/2368.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2368 ---- commit f00c850599ec864e5e1f2faea0d02b47d89a2aba Author: Thomas Groh <tg...@google.com> Date: 2017-03-30T02:26:06Z Make Pubsub ackBatch safer Checkpoints can be finalized at any point, including while the reader is being closed or after it has been. When ackBatch is called and the reader is closed, it should return. When close is called while a different thread is using the client, it should wait for that thread to complete before closing the reader. ---- > NPE in DirectRunner PubsubReader.ackBatch > ----------------------------------------- > > Key: BEAM-1835 > URL: https://issues.apache.org/jira/browse/BEAM-1835 > Project: Beam > Issue Type: Bug > Components: runner-direct, sdk-java-core > Reporter: Rafal Wojdyla > Assignee: Rafal Wojdyla > > DirectRunner streaming mode throws null pointer exception: > {noformat} > org.apache.beam.sdk.io.PubsubUnboundedSource$PubsubReader.ackBatch(PubsubUnboundedSource.java:639) > at > org.apache.beam.sdk.io.PubsubUnboundedSource$PubsubCheckpoint.finalizeCheckpoint(PubsubUnboundedSource.java:312) > at > org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.finishRead(UnboundedReadEvaluatorFactory.java:223) > at > org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.processElement(UnboundedReadEvaluatorFactory.java:144) > at > org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:139) > at > org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:107) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {noformat} > This does not happen always, but for large enough number of events it's > pretty reproducible. The problems seems to be the concurrent reuse of a > reader among multiple threads, and a race condition, when one of the threads > "decided" to close the reader, based on: > {code} > private static final double DEFAULT_READER_REUSE_CHANCE = 0.95; > {code} > the close, nulls out pubsub client: > {code} > @Override > public void close() throws IOException { > if (pubsubClient != null) { > pubsubClient.close(); > pubsubClient = null; > } > } > {code} > which if still in use by other thread will result in NPE above. -- This message was sent by Atlassian JIRA (v6.3.15#6346)