[
https://issues.apache.org/jira/browse/BEAM-6654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16769386#comment-16769386
]
Ismaël Mejía commented on BEAM-6654:
------------------------------------
Excellent [~tmoulton] I assigned the issue to you, feel free to discuss on the
design via a doc or just in the mailing list. A suggestion is to read [the SDF
post in the Beam
blog|https://beam.apache.org/blog/2017/08/16/splittable-do-fn.html] and
eventually the implementation of
[HBaseIO.ReadAll|https://github.com/apache/beam/blob/1d1319950c6885b8e8be8de8a345aec4bf65ce20/sdks/java/io/hbase/src/main/java/org/apache/beam/sdk/io/hbase/HBaseIO.java#L281]
that uses a SDF implementation not like JdbcIO.ReadAll that is just a classic
DoFn style. (Notice that this is also valid just less powerful).
> Support SDF within ElasticsearchIO
> ----------------------------------
>
> Key: BEAM-6654
> URL: https://issues.apache.org/jira/browse/BEAM-6654
> Project: Beam
> Issue Type: New Feature
> Components: io-java-elasticsearch
> Reporter: Tony Moulton
> Assignee: Tony Moulton
> Priority: Minor
>
> Similar to JdbcIO.ReadAll, it would be very useful to support a SDF based
> transform which can compose queries based off of input elements.
> Implementation would split the work similarly to how the current Source API
> implementation works (leveraging slices within Elasticsearch).
> This work can facilitate a transition of ElasticsearchIO to SDF when more
> runners have implemented support. Until then, I propose leaving the read()
> implementation alone and add a readAll method and associated transform to be
> implemented as an SDF. Once enough runner support is ready, the existing
> Read transform would be updated to use the same SDF as the ReadAll transform
> with a null input element.
> A prototype implementation currently exists (albeit developed in a bubble) at
> [https://github.com/0xdecaf/beam/tree/feature/elasticsearchio.readall]
> however hasn't been synced with master for a few weeks.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
