Since we care first about existing compatibility (Hadoop 2.x) we might execute these tests in PostCommit that way the impact is reduced.
We should not be upgrading to 3.3.0 because that is not a stable version yet. It does not make sense to support unstable versions of any of our deps, so we should go probably to version 3.2.1. The problem with automatic dependency reports as we have in Beam is that they miss the little issues of each project. It is the same reason why moving to Hive/HCatalog 3.x makes less sense because the main projects still support at its most recent 2.x On Fri, Oct 30, 2020 at 11:58 AM Piotr Szuberski <[email protected]> wrote: > > I'd like to write tests testing Beam against Hadoop 3.3.0, similiar to what > is done in KafkaIO to verify that users can use Beam with Hadoop3. > > But as I noticed some of these tests take quite a lot of time to accomplish > (HadoopFormatIO locally ~15min). > > Is making such tests desirable or it's too early for Hadoop 3.3.0 and is not > worth it? Maybe there is some other approach to take?
