Hello Kudu developers, With a large majority of the Apache Ranger integration work landing, I wanted to revisit this plan and potentially move forward removing Sentry and upgrading Hive.
Steps 1 through 3 from the plan have been completed. Below are some related commits: 1. commit the Hive 3 preparation patch to simplify upgrading in the future - https://github.com/apache/kudu/commit/76b80e - https://github.com/apache/kudu/commit/ab69837 - https://github.com/apache/kudu/commit/e20487a - https://github.com/apache/kudu/commit/d96f8fc - https://github.com/apache/kudu/commit/8fb170b 2. Verify the feasibility of upgrading with the mentioned POC patches, but do not commit them. - https://gerrit.cloudera.org/#/c/14020/ - https://gerrit.cloudera.org/#/c/13256/ 3. Start work on an Apache Ranger integration for Kudu. - https://github.com/apache/kudu/commit/f392503 - https://github.com/apache/kudu/commit/2ea8478 - https://github.com/apache/kudu/commit/e13fd4a - https://github.com/apache/kudu/commit/0d29977 Since the writing of the original email, Sentry has not added support for Hive 3. This means that step 4 is not an option at this time. As a result I propose to move forward with step 5 and start removing Sentry 3 support and upgrading to Hive 3. The planned steps are as follows: 1. Rebase and commit the patch to disable Sentry tests 2. Rebase and commit the patch to upgrade to upgrade to Hive 3 3. Document the removal of Sentry support in the release notes. 4. Remove Sentry tests and code. - This can be done after the next release. - Though the tests are removed and we use Hive 3 we still technically work with Hive 2 and Sentry. We are no longer testing/validating it though. Please let me know if you have any thoughts or feedback on the above plan. Thank you, Grant On Tue, Aug 13, 2019 at 3:06 AM Adar Lieber-Dembo <a...@cloudera.com.invalid> wrote: > +1, thanks for all of the details. > > On Fri, Aug 9, 2019 at 3:21 PM Grant Henke <ghe...@cloudera.com> wrote: > > > > Hello Kudu developers, > > > > Recently I have started work on upgrading Kudu to use Apache Hive 3.x. > > Given this is a major upgrade it does come with some challenges. As of > Kudu > > 1.10.0 we use Hive in the HMS synchronization feature. This feature > > includes a Kudu server side notification listener and HMS client. It also > > includes a Java side HMS plugin to enforce Kudu imperatives within the > HMS. > > That feature is useful on its own in many ways, but is also required for > > fine grained authorization via Apache Sentry. > > > > The primary challenge is that Apache Sentry currently does not support > Hive > > 3 and it will likely take a large effort to enable support. It is also > > unclear if there is anyone in the Sentry community that want's to > > contribute and release such support. > > > > I have started preliminary efforts to support Hive 3 in Kudu and the HMS > > synchronization feature. This includes 3 patches. The first patch > > <https://gerrit.cloudera.org/#/c/14018/> is changes that work in both > Hive > > 2 and Hive 3 that minimize the work needed when we upgrade in the future. > > This can be committed to master when reviewed and ready. The second patch > > <https://gerrit.cloudera.org/#/c/14006> disables the sentry integration > so > > I can test the changes required to support HMS synchronization on its > own. > > Those changes and testing are the third patch > > <https://gerrit.cloudera.org/#/c/13256/>. > > > > Given fine grained authorization is a critical feature for many users, we > > can't remove Sentry support without providing an alternative > authorization > > implementation. At the same time we have started work on authorization > via > > Apache Ranger. Once that implementation exists and has been > > contributed/released we can make a decision about how to move forward. > > > > Given what we know today and the current situation here is my suggested > > plan: > > > > 1. Commit the Hive 3 preparation patch to simplify upgrading in the > > future > > 2. Verify the feasibility of upgrading with the mentioned POC patches, > > but do not commit them. > > - This means we will remain on Hive 2 until step 4 or 5 below. > > 3. Start work on an Apache Ranger integration for Kudu. > > 4. If Hive 3 support is added in Sentry, consider upgrading to Hive 3 > > then. > > 5. When Ranger support is complete, consider removing Sentry support > in > > favor of Ranger and upgrade to Hive 3. > > - This may require a migration path from Sentry to Ranger. > > > > Please let me know if you have any thoughts or feedback on the above > plan. > > > > Thank you, > > Grant > > -- > > Grant Henke > > Software Engineer | Cloudera > > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke > -- Grant Henke Software Engineer | Cloudera gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke