Tony Moulton created BEAM-6654:
----------------------------------
Summary: Support SDF within ElasticsearchIO
Key: BEAM-6654
URL: https://issues.apache.org/jira/browse/BEAM-6654
Project: Beam
Issue Type: New Feature
Components: io-java-elasticsearch
Reporter: Tony Moulton
Similar to JdbcIO.ReadAll, it would be very useful to support a SDF based
transform which can compose queries based off of input elements.
Implementation would split the work similarly to how the current Source API
implementation works (leveraging slices within Elasticsearch).
This work can facilitate a transition of ElasticsearchIO to SDF when more
runners have implemented support. Until then, I propose leaving the read()
implementation alone and add a readAll method and associated transform to be
implemented as an SDF. Once enough runner support is ready, the existing Read
transform would be updated to use the same SDF as the ReadAll transform with a
null input element.
A prototype implementation currently exists (albeit developed in a bubble) at
[https://github.com/0xdecaf/beam/tree/feature/elasticsearchio.readall] however
hasn't been synced with master for a few weeks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
