Hi guys,
I'm working on Elasticsearch 5.x support for Beam IO (it only supports
Elasticsearch 2.x right now). I wanted to have your opinion on some
points related to maintenance.
In this ES case a big part of the code of the IO is common between ES
v2.x and ES v5.x. Still, there are some differences:
- initialization of UT (change in embedded test framework)
- Minor differences in one message format
- New feature that will allow improving the split or new feature that is
worth leveraging (ES pipelines)
=> Question is: what do you think is the best way to architecture the IO
to reduce maintenance
1. We could have an elasticsearchio-common package and two packages that
are specific to each version of the backend. But I find it confusing for
the users to have separate packages and more complex to maintain for us.
2. I'm more in favor of detecting the version at IO initialization time
and then, in the parts that are different do a simple if (version == x).
But it will make code paths more complex. Note that for example project
es-hadoop (ES connectors for big data engines) chose this way.
Another thing related to unit tests: in fact they are more close to
integration tests as they use an embedded backend server. I did it that
way because I wanted to unit test things like split that require a real
instance.
=> What is the recommended way of testing on both supported versions
knowing that both the test code and the test dependencies are different?
For integration tests (they are mainly used as load testing), the test
code and the test dependencies are the same between versions because
there is no embedded ES. So, it will be only needed to run them twice
against 2 versions of the backend.
What do you think?
PS: sorry for the long email :)
Best!
Etienne
