I find a problem of having different profiles. Let's say we want to support Hive2, Hive3 and Hive4 once it is released; but we also want to support Hadoop3 and Hadoop2 (for compatibility with Hive2); what versions are we going to use for building the jars and releasing Sentry? With profiles we can only choose 1 Hive and 1 Hadoop (what about other components, such as Kafka and Solr?)
The above will become a nightmare if we want to follow the approach of keeping Sentry compatible with several versions. I was looking into how SLF4J works and it looks like a good idea to replicate on Sentry. SLF4J is just a facade and abstract API for logging that uses any logging binding implementation that is available in the classpath, such as Log4j, JDK Logging, etc. It initially uses the desired logging implementation at compile-time, but you can replace any implementation jars at runtime. For Sentry, we could decide to use Hive3 (latest release) and Hadoop3 at compile-time and work with the abstracted API to interact with them. But if a user wants to use Hive2 and Hadoop2, then they can replace the Hive3/Hadoop3 jars with the 2.x without recompile anything and restart Sentry. This would work pretty nice. We should do some compatibility testing between those versions, though, to make sure we don't break compatibility (some automated tests should be in place for this). It sounds like a good amount of work to make this, but what do you think about it? We could use profiles just for the sake of compiling a jar with a specific version, but what about for releasing binaries? - Sergio On Tue, Aug 28, 2018 at 11:12 AM Stephen Moist <mo...@cloudera.com.invalid> wrote: > I don’t think we should do multiple profiles. I think we should just > support Hive through hive-binding-v2, hive-binding-v3, and hive-binding-v4 > and have them be their own maven modules. Customers can decide which > version they want. We can add Thrift calls that are specific for a binding > or amend them with optional parameters. With this, do we need to support > previous minor versions or just the latest version? > > > On Aug 27, 2018, at 1:34 PM, Brian Towles <btow...@cloudera.com.INVALID> > wrote: > > > > It would seem that there shouldn't be an issue doing a different binding > > for Hive 3 as well as hive 4 if there is that much difference. It would > > allow the Hive 2 one to be maintained and could eventually be EOL'd once > it > > becomes not useful. This sort of pattern can be used across multiple > > version of almost anything there is a Sentry binding for. > > > > On Mon, Aug 27, 2018 at 11:09 AM Kalyan Kumar Kalvagadda > > <kkal...@cloudera.com.invalid> wrote: > > > >> Sergio, > >> > >> We could add profiles for to multiple version of Hive till the point we > >> want to support multiple versions. We should also consider supporting > >> Hadoop-3 as well. > >> We could take similar approach to support multiple versions of Hadoop as > >> well. > >> > >> > >> On Mon, Aug 27, 2018 at 10:22 AM Sergio Pena > >> <sergio.p...@cloudera.com.invalid> wrote: > >> > >>> Hi All, > >>> > >>> I'd like to discuss how we can start integrating Apache Hive 3.x and > >> future > >>> 4.x into our Sentry versions. I've noticed that Hive has released two > >>> versions of 3.x this year (3.0 on May/18, 3.1 on Jul/18), and the > current > >>> version development is 4.0. > >>> > >>> Currently, Sentry supports only the Hive 2.x, and there is no news of > new > >>> minor versions on that line to be released in the community. However, > >> there > >>> are still companies using the Hive 2.x version due to its stability and > >> API > >>> compatibility. Also, I see a lot of incompatibilities problems once > Hive > >>> 4.0 is released too. > >>> > >>> So, the question, how can we keep with the most updated Hive releases > but > >>> being compatible with Hive versions that are still active by the Sentry > >>> community? Btw, Hive 3.x has features that allow Sentry support ALTER > >> TABLE > >>> SET OWNER commands to transfer object ownership. > >>> > >>> Hive had a shims interface for Hadoop to support Hadoop 1.x and Hadoop > >> 2.x > >>> in the past. Should we use a similar thing? > >>> > >>> Any other ideas? > >>> > >>> - Sergio > >>> > >> > > -- > > *Brian Towles* | Software Engineer > > t. (512) 415- <0000000000>8105 e. btow...@cloudera.com < > j...@cloudera.com> > > cloudera.com <http://www.cloudera.com/> > > > > [image: Cloudera] <http://www.cloudera.com/> > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > Cloudera > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > ------------------------------ > >
