Re: Updating elasticsearch version to 7.9.2 - problem with HadoopFormatIOElasticTest that uses ES emulator
I've already found the assumptions, they were in elastic_test_data.py. The relevant PR is at https://github.com/apache/beam/pull/13085 On 2020/10/22 18:15:11, Tyson Hamilton wrote: > IMO it really comes down to stability & runtime differences. If there are > no significant changes to either of these then keeping it as a > precommit and using test containers is fine. Where are the assumptions in > the IT test, in HadoopFormatIOElasticTest? > > On Mon, Oct 12, 2020 at 10:10 AM Piotr Szuberski < > piotr.szuber...@polidea.com> wrote: > > > I'm trying to update elasticsearch version to 7.9.2 but I've encountered a > > problem with HadoopFormatIOElasticTest that uses ES in-memory emulator that > > is no longer supported: > > https://stackoverflow.com/questions/51316813/elastic-node-on-local-in-6-2 > > > > It's recommended to use testcontainers as proposed here > > https://github.com/allegro/embedded-elasticsearch but it would transform > > the in-memory test to integration test (which has to be done anyway) > > > > There is also Elasticsearch test framework with ESSingleNodeTestCase but > > it causes Jar Hell problem and I don't think it's easily solvable - the > > dependencies in "java core" and "java core test". > > I tried to > > > > Is running the precommit test with testcontainers acceptable? It's the > > easiest fix. > > > > About the integration test: > > I'd like to enable the IT test in Java PostCommit but there are some > > assumptions about the data that is already written to Elasticsearch but I > > can't find anywhere what that data should be (Probably something like > > Item_Price0, Item_Price1 etc but I'm not sure) > > >
Re: Updating elasticsearch version to 7.9.2 - problem with HadoopFormatIOElasticTest that uses ES emulator
IMO it really comes down to stability & runtime differences. If there are no significant changes to either of these then keeping it as a precommit and using test containers is fine. Where are the assumptions in the IT test, in HadoopFormatIOElasticTest? On Mon, Oct 12, 2020 at 10:10 AM Piotr Szuberski < piotr.szuber...@polidea.com> wrote: > I'm trying to update elasticsearch version to 7.9.2 but I've encountered a > problem with HadoopFormatIOElasticTest that uses ES in-memory emulator that > is no longer supported: > https://stackoverflow.com/questions/51316813/elastic-node-on-local-in-6-2 > > It's recommended to use testcontainers as proposed here > https://github.com/allegro/embedded-elasticsearch but it would transform > the in-memory test to integration test (which has to be done anyway) > > There is also Elasticsearch test framework with ESSingleNodeTestCase but > it causes Jar Hell problem and I don't think it's easily solvable - the > dependencies in "java core" and "java core test". > I tried to > > Is running the precommit test with testcontainers acceptable? It's the > easiest fix. > > About the integration test: > I'd like to enable the IT test in Java PostCommit but there are some > assumptions about the data that is already written to Elasticsearch but I > can't find anywhere what that data should be (Probably something like > Item_Price0, Item_Price1 etc but I'm not sure) >
Updating elasticsearch version to 7.9.2 - problem with HadoopFormatIOElasticTest that uses ES emulator
I'm trying to update elasticsearch version to 7.9.2 but I've encountered a problem with HadoopFormatIOElasticTest that uses ES in-memory emulator that is no longer supported: https://stackoverflow.com/questions/51316813/elastic-node-on-local-in-6-2 It's recommended to use testcontainers as proposed here https://github.com/allegro/embedded-elasticsearch but it would transform the in-memory test to integration test (which has to be done anyway) There is also Elasticsearch test framework with ESSingleNodeTestCase but it causes Jar Hell problem and I don't think it's easily solvable - the dependencies in "java core" and "java core test". I tried to Is running the precommit test with testcontainers acceptable? It's the easiest fix. About the integration test: I'd like to enable the IT test in Java PostCommit but there are some assumptions about the data that is already written to Elasticsearch but I can't find anywhere what that data should be (Probably something like Item_Price0, Item_Price1 etc but I'm not sure)
