I believe that 2.41.0 is the "oldest" safe version[1] to use as there were initially some bugs introduced when migrating from PDone to outputting the write results.
[1] https://github.com/apache/beam/commit/2cb2ee2ba3b5efb0f08880a9325f092485b3ccf2 On Thu, Feb 2, 2023 at 3:16 PM Kaggal, Vinod C. (Vinod C.), M.S. via user < [email protected]> wrote: > Thank you for the response! > > > > We are currently using : > > <beam.version>2.37.0</beam.version> > <beam.es.io.version>2.33.0</beam.es.io.version> > > > > Is there a compatible elasticsearchio version that may work with our beam > version? > > > > Thank you! > > Vinod > > > > *From: *Chamikara Jayalath <[email protected]> > *Date: *Thursday, February 2, 2023 at 6:01 PM > *To: *[email protected] <[email protected]>, Kaggal, Vinod C. > (Vinod C), M.S. <[email protected]> > *Subject: *[EXTERNAL] Re: ElasticsearchIO write to trigger a percolator > > > > > > On Thu, Feb 2, 2023 at 1:56 PM Kaggal, Vinod C. (Vinod C.), M.S. via user < > [email protected]> wrote: > > Hello! Thank you for all the hard work on implementing these useful > libraries. > > > > *Background:* We have been using Apache Storm in production for some time > (over 8 years) and have recently switched over to Beam. One of the > topologies that we had in Storm was to ingest data, index to elastic > (write) and right after the write to elastic we would trigger a percolator > query for a downstream task to be triggered. The DAGs using Flux allowed us > to chain these steps ( [read from source] -> [process] -> [index] -> > [percolator] -> [trigger]). > > > > *Beam:* > > We are able to accomplish much of that using beam. However, using > ElasticsearchIO, the index step results in a PDone and hence we cannot > chain the remaining tasks ( [percolator] -> [trigger other events]). > > > > The question I have is this: is there a way to trigger a percolator query > after writing a document to elastic using ElasticsearchIO.Write()? > > > > Which version of Beam are you using ? > > Seems like for recent versions of Beam, we actually return > PCollections from ElasticsearchIO sink so this should not be an issue. > > > > > https://github.com/apache/beam/blob/24f40aab7aa4c4f9eea6dc90c0baa22bb17a962e/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java#L1842 > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fbeam%2Fblob%2F24f40aab7aa4c4f9eea6dc90c0baa22bb17a962e%2Fsdks%2Fjava%2Fio%2Felasticsearch%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Fsdk%2Fio%2Felasticsearch%2FElasticsearchIO.java%23L1842&data=05%7C01%7Ckaggal.vinod%40mayo.edu%7C81739ccc43ef401bf06f08db0571604c%7Ca25fff9c3f634fb29a8ad9bdd0321f9a%7C0%7C0%7C638109756723528453%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jzUyYjqB3Cm01rv%2FMP4KGWKGyLO%2BdpiIX%2FNBioUHuJQ%3D&reserved=0> > > > > Thanks, > > Cham > > > > > > Your thoughts would be appreciated! > > Vinod > > > >
