On Wed, Oct 19, 2016 at 11:00 AM, Kenneth Knowles <k...@google.com.invalid> wrote:
> I wanted to attempt to explicitly answer Raghu's question by saying that I > think Dan's starting points imply that the recommended behavior for start() > and advance() is to be "non-blocking" in the sense that they return quickly > if in start-up, and a runner can be smart about polling soon. Once the > FooIO tranfsorms are ported to splittable DoFn it will be a non-issue for > them, though if UnboundedReader still exists as a convenience or > compatibility wrapper then the question still needs an answer. > > Is this accurate? Has there been other talk on the PRs? > This sounds great. It would be nice to have this in javadoc for advance() : preferred behavior is non blocking or keep the wait to as small as reasonable (i.e. without incurring much performance penalty, if any). Kafka is now updated to be essentially be non-blocking (max 10millis wait, that too mainly as a safety, in case a runner gets too eager to poll).
