Thank you for the response! We are currently using : <beam.version>2.37.0</beam.version> <beam.es.io.version>2.33.0</beam.es.io.version>
Is there a compatible elasticsearchio version that may work with our beam version? Thank you! Vinod From: Chamikara Jayalath <[email protected]> Date: Thursday, February 2, 2023 at 6:01 PM To: [email protected] <[email protected]>, Kaggal, Vinod C. (Vinod C), M.S. <[email protected]> Subject: [EXTERNAL] Re: ElasticsearchIO write to trigger a percolator On Thu, Feb 2, 2023 at 1:56 PM Kaggal, Vinod C. (Vinod C.), M.S. via user <[email protected]<mailto:[email protected]>> wrote: Hello! Thank you for all the hard work on implementing these useful libraries. Background: We have been using Apache Storm in production for some time (over 8 years) and have recently switched over to Beam. One of the topologies that we had in Storm was to ingest data, index to elastic (write) and right after the write to elastic we would trigger a percolator query for a downstream task to be triggered. The DAGs using Flux allowed us to chain these steps ( [read from source] -> [process] -> [index] -> [percolator] -> [trigger]). Beam: We are able to accomplish much of that using beam. However, using ElasticsearchIO, the index step results in a PDone and hence we cannot chain the remaining tasks ( [percolator] -> [trigger other events]). The question I have is this: is there a way to trigger a percolator query after writing a document to elastic using ElasticsearchIO.Write()? Which version of Beam are you using ? Seems like for recent versions of Beam, we actually return PCollections from ElasticsearchIO sink so this should not be an issue. https://github.com/apache/beam/blob/24f40aab7aa4c4f9eea6dc90c0baa22bb17a962e/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java#L1842<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fbeam%2Fblob%2F24f40aab7aa4c4f9eea6dc90c0baa22bb17a962e%2Fsdks%2Fjava%2Fio%2Felasticsearch%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Fsdk%2Fio%2Felasticsearch%2FElasticsearchIO.java%23L1842&data=05%7C01%7Ckaggal.vinod%40mayo.edu%7C81739ccc43ef401bf06f08db0571604c%7Ca25fff9c3f634fb29a8ad9bdd0321f9a%7C0%7C0%7C638109756723528453%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jzUyYjqB3Cm01rv%2FMP4KGWKGyLO%2BdpiIX%2FNBioUHuJQ%3D&reserved=0> Thanks, Cham Your thoughts would be appreciated! Vinod
