I know, I don't want to update Hadoop dependencies in BeamModulePlugin to 3.2.1 but to add tests verifying that Beam works with them so users can be assured that they can use this version if they need to. 3.3.0 was a mismatch, I meant the recent 3.x version.
Postcommit sounds reasonable for this. I'll tag you to the PR when I'm done. On 2020/10/30 14:37:56, Ismaël Mejía <[email protected]> wrote: > Since we care first about existing compatibility (Hadoop 2.x) we might execute > these tests in PostCommit that way the impact is reduced. > > We should not be upgrading to 3.3.0 because that is not a stable version yet. > It does not make sense to support unstable versions of any of our deps, so > we should go probably to version 3.2.1. > > The problem with automatic dependency reports as we have in Beam is that they > miss the little issues of each project. It is the same reason why moving to > Hive/HCatalog 3.x makes less sense because the main projects still support at > its most recent 2.x > > > On Fri, Oct 30, 2020 at 11:58 AM Piotr Szuberski > <[email protected]> wrote: > > > > I'd like to write tests testing Beam against Hadoop 3.3.0, similiar to what > > is done in KafkaIO to verify that users can use Beam with Hadoop3. > > > > But as I noticed some of these tests take quite a lot of time to accomplish > > (HadoopFormatIO locally ~15min). > > > > Is making such tests desirable or it's too early for Hadoop 3.3.0 and is > > not worth it? Maybe there is some other approach to take? >
