[jira] [Commented] (BEAM-801) SplittableParDo must be a pseudo-primitive, not ParDo(OldDoFn)
[
https://issues.apache.org/jira/browse/BEAM-801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15713305#comment-15713305
]
Eugene Kirpichov commented on BEAM-801:
---
This can be closed.
> SplittableParDo must be a pseudo-primitive, not ParDo(OldDoFn)
> --
>
> Key: BEAM-801
> URL: https://issues.apache.org/jira/browse/BEAM-801
> Project: Beam
> Issue Type: Bug
> Components: runner-core, runner-direct
>Reporter: Kenneth Knowles
>Assignee: Eugene Kirpichov
>
> Today, runners-core contains an expansion of {{ParDo(DoFn)}} that implements
> it via unsupported features of {{ParDo(OldDoFn)}}. This expansion is used by
> the DirectRunner.
> The right approach to provide a ready implementation in runners-core is the
> one taken by {{GroupAlsoByWindowViaWindowSetDoFn}} where the unsupported
> features are provided to the constructor of the {{DoFn}}, rather than
> expected to be passed through the {{ProcessContext}}.
> These features are not guaranteed to be part of the public state & timers API
> of {{DoFn}}, particularly in the case of watermark holds, so it is not
> prudent to plan on waiting for the new state & timers API and porting over.
> These issues create real blockers by causing dependency cycles between
> implementing {{DoFn}} fully (requires no use of unsupported features),
> implementing new state and timers (requires new {{DoFn}}), and keeping the
> hack until new {{DoFn}} has state and timers (requires use of unsupported
> features).
> To break the cycle, the tests that rely on unsupported features are being
> disabled. They can be re-enabled either by following the design suggested
> above (probably the most robust approach) or by waiting and porting to new
> features as they are available.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BEAM-801) SplittableParDo must be a pseudo-primitive, not ParDo(OldDoFn)
[
https://issues.apache.org/jira/browse/BEAM-801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15713249#comment-15713249
]
ASF GitHub Bot commented on BEAM-801:
-
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-beam/pull/1261
> SplittableParDo must be a pseudo-primitive, not ParDo(OldDoFn)
> --
>
> Key: BEAM-801
> URL: https://issues.apache.org/jira/browse/BEAM-801
> Project: Beam
> Issue Type: Bug
> Components: runner-core, runner-direct
>Reporter: Kenneth Knowles
>Assignee: Eugene Kirpichov
>
> Today, runners-core contains an expansion of {{ParDo(DoFn)}} that implements
> it via unsupported features of {{ParDo(OldDoFn)}}. This expansion is used by
> the DirectRunner.
> The right approach to provide a ready implementation in runners-core is the
> one taken by {{GroupAlsoByWindowViaWindowSetDoFn}} where the unsupported
> features are provided to the constructor of the {{DoFn}}, rather than
> expected to be passed through the {{ProcessContext}}.
> These features are not guaranteed to be part of the public state & timers API
> of {{DoFn}}, particularly in the case of watermark holds, so it is not
> prudent to plan on waiting for the new state & timers API and porting over.
> These issues create real blockers by causing dependency cycles between
> implementing {{DoFn}} fully (requires no use of unsupported features),
> implementing new state and timers (requires new {{DoFn}}), and keeping the
> hack until new {{DoFn}} has state and timers (requires use of unsupported
> features).
> To break the cycle, the tests that rely on unsupported features are being
> disabled. They can be re-enabled either by following the design suggested
> above (probably the most robust approach) or by waiting and porting to new
> features as they are available.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BEAM-801) SplittableParDo must be a pseudo-primitive, not ParDo(OldDoFn)
[
https://issues.apache.org/jira/browse/BEAM-801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15629943#comment-15629943
]
Eugene Kirpichov commented on BEAM-801:
---
https://github.com/apache/incubator-beam/pull/1261
> SplittableParDo must be a pseudo-primitive, not ParDo(OldDoFn)
> --
>
> Key: BEAM-801
> URL: https://issues.apache.org/jira/browse/BEAM-801
> Project: Beam
> Issue Type: Bug
> Components: runner-core, runner-direct
>Reporter: Kenneth Knowles
>Assignee: Eugene Kirpichov
>
> Today, runners-core contains an expansion of {{ParDo(DoFn)}} that implements
> it via unsupported features of {{ParDo(OldDoFn)}}. This expansion is used by
> the DirectRunner.
> The right approach to provide a ready implementation in runners-core is the
> one taken by {{GroupAlsoByWindowViaWindowSetDoFn}} where the unsupported
> features are provided to the constructor of the {{DoFn}}, rather than
> expected to be passed through the {{ProcessContext}}.
> These features are not guaranteed to be part of the public state & timers API
> of {{DoFn}}, particularly in the case of watermark holds, so it is not
> prudent to plan on waiting for the new state & timers API and porting over.
> These issues create real blockers by causing dependency cycles between
> implementing {{DoFn}} fully (requires no use of unsupported features),
> implementing new state and timers (requires new {{DoFn}}), and keeping the
> hack until new {{DoFn}} has state and timers (requires use of unsupported
> features).
> To break the cycle, the tests that rely on unsupported features are being
> disabled. They can be re-enabled either by following the design suggested
> above (probably the most robust approach) or by waiting and porting to new
> features as they are available.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
