sheepdreamofandroids opened a new issue, #22840: URL: https://github.com/apache/beam/issues/22840
### What would you like to happen? ElasticSearchIO.Write accepts a PCollection<String>. This is the exact JSON that will be written to ElasticSearch. Consequently functions like IndexFn have only (the parsed version of) this JSON as input. I have a problem where, among others, the exact same JSON get's written to 2 different indexes in different ways. This means I have to create multiple instances of ElasticSearchIO.Write, one for each permutation of settings. It's not a major problem because there are only 7 permutations in my case but it could be many more. It does lead to a big process graph in dataflow and also it leads many small batches because there is a latency constraint, The reason is that there is no way of adding some context or meta data to that JSON string. I could add an extra field to that json but I have no way of removing it before it gets written to the index. A possible solution would be to have a type parameter for the input and a settable jsonFn that extracts the json string to be written. ### Issue Priority Priority: 3 ### Issue Component Component: io-java-elasticsearch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
