+1 for doing this in a way that keeps Hive 2 support as that's still our primary Hive version in production and will be for quite some time.
On Mon, 14 Sep 2020 at 09:00, Marton Bod <marton....@gmail.com> wrote: > Hi Ryan, > > Thanks, I absolutely agree with you that we should keep support for Hive2 > as well. I've created a draft PR, which highlights the changes: > https://github.com/apache/iceberg/pull/1455 > We can continue the discussion there. > > Thanks, > > Marton > > On Fri, 11 Sep 2020 at 22:20, Ryan Blue <rb...@netflix.com.invalid> wrote: > >> Hi Marton, could you share a link to your branch with the changes? It >> would be great to see what needs to be done. A quick summary would help as >> well. >> >> Knowing what changes between Hive 2 and 3 in our iceberg-hive-metastore >> project is important because we would ideally use whatever is available at >> runtime. I'm all for upgrading to enable people on the newer versions, but >> I think we want to make sure we maintain support for people that haven't >> migrated yet as well. >> >> rb >> >> On Fri, Sep 11, 2020 at 3:28 AM Marton Bod <marton....@gmail.com> wrote: >> >>> Hi Team, >>> >>> We would like to start a discussion on upgrading Iceberg components to >>> use Hive 3 and Hadoop 3. We have a fork where we have bumped up the hive >>> and hadoop dependency versions and made the necessary changes to get all >>> tests to pass. >>> >>> As some components cannot (e.g. spark2) or might not want to upgrade >>> yet, our solution was to create a separate iceberg-hive2-metastore module, >>> which would keep on using Hive2 and Hadoop2. This would give an option for >>> each component to do the upgrade at their own pace or not at all. >>> >>> At this point, our primary goal is to upgrade iceberg-mr to Hive3. >>> Upgrading iceberg-flink posed no major issues either, but of course it's up >>> to the Flink iceberg community to make this call. As for spark2/spark3, we >>> have left them for now to use Hive2. >>> >>> Any thoughts from the community on this upgrade? >>> >>> Thank you, >>> >>> Marton >>> >> >> >> -- >> Ryan Blue >> Software Engineer >> Netflix >> >
