Hi Stephan, Good idea. Just like hadoop, we can have flink-shaded-hive-uber. Then the startup of hive integration will be very simple with one or two pre-bundled, user just add these dependencies: - flink-connector-hive.jar - flink-shaded-hive-uber-<version>.jar
Some changes are needed, but I think it should work. Another thing is can we put flink-connector-hive.jar into flink/lib, it should clean and no dependencies. Best, Jingsong Lee On Thu, Feb 6, 2020 at 7:13 PM Stephan Ewen <se...@apache.org> wrote: > Hi Jingsong! > > This sounds that with two pre-bundled versions (hive 1.2.1 and hive 2.3.6) > you can cover a lot of versions. > > Would it make sense to add these to flink-shaded (with proper dependency > exclusions of unnecessary dependencies) and offer them as a download, > similar as we offer pre-shaded Hadoop downloads? > > Best, > Stephan > > > On Thu, Feb 6, 2020 at 10:26 AM Jingsong Li <jingsongl...@gmail.com> > wrote: > >> Hi Stephan, >> >> The hive/lib/ has many jars, this lib is for execution, metastore, hive >> client and all things. >> What we really depend on is hive-exec.jar. (hive-metastore.jar is also >> required in the low version hive) >> And hive-exec.jar is a uber jar. We just want half classes of it. These >> half classes are not so clean, but it is OK to have them. >> >> Our solution now: >> - exclude hive jars from build >> - provide 8 versions dependencies way, user choose by his hive version.[1] >> >> Spark's solution: >> - build-in hive 1.2.1 dependencies to support hive 0.12.0 through 2.3.3. >> [2] >> - hive-exec.jar is hive-exec.spark.jar, Spark has modified the >> hive-exec build pom to exclude unnecessary classes including Orc and >> parquet. >> - build-in orc and parquet dependencies to optimizer performance. >> - support hive version 2.3.3 upper by "mvn install -Phive-2.3", to >> built-in hive-exec-2.3.6.jar. It seems that since this version, hive's API >> has been seriously incompatible. >> Most of the versions used by users are hive 0.12.0 through 2.3.3. So the >> default build of Spark is good to most of users. >> >> Presto's solution: >> - Built-in presto's hive.[3] Shade hive classes instead of thrift classes. >> - Rewrite some client related code to solve kinds of issues. >> This approach is the heaviest, but also the cleanest. It can support all >> kinds of hive versions with one build. >> >> So I think we can do: >> >> - The eight versions we now maintain are too many. I think we can move >> forward in the direction of Presto/Spark and try to reduce dependencies >> versions. >> >> - As your said, about provide fat/uber jars or helper script, I prefer >> uber jars, user can download one jar to their startup. Just like Kafka. >> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/hive/#dependencies >> [2] >> https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html#interacting-with-different-versions-of-hive-metastore >> [3] https://github.com/prestodb/presto-hive-apache >> >> Best, >> Jingsong Lee >> >> On Wed, Feb 5, 2020 at 10:15 PM Stephan Ewen <se...@apache.org> wrote: >> >>> Some thoughts about other options we have: >>> >>> - Put fat/shaded jars for the common versions into "flink-shaded" and >>> offer them for download on the website, similar to pre-bundles Hadoop >>> versions. >>> >>> - Look at the Presto code (Metastore protocol) and see if we can reuse >>> that >>> >>> - Have a setup helper script that takes the versions and pulls the >>> required dependencies. >>> >>> Can you share how can a "built-in" dependency could work, if there are >>> so many different conflicting versions? >>> >>> Thanks, >>> Stephan >>> >>> >>> On Tue, Feb 4, 2020 at 12:59 PM Rui Li <li...@apache.org> wrote: >>> >>>> Hi Stephan, >>>> >>>> As Jingsong stated, in our documentation the recommended way to add Hive >>>> deps is to use exactly what users have installed. It's just we ask >>>> users to >>>> manually add those jars, instead of automatically find them based on env >>>> variables. I prefer to keep it this way for a while, and see if there're >>>> real concerns/complaints from user feedbacks. >>>> >>>> Please also note the Hive jars are not the only ones needed to integrate >>>> with Hive, users have to make sure flink-connector-hive and Hadoop jars >>>> are >>>> in classpath too. So I'm afraid a single "HIVE" env variable wouldn't >>>> save >>>> all the manual work for our users. >>>> >>>> On Tue, Feb 4, 2020 at 5:54 PM Jingsong Li <jingsongl...@gmail.com> >>>> wrote: >>>> >>>> > Hi all, >>>> > >>>> > For your information, we have document the dependencies detailed >>>> > information [1]. I think it's a lot clearer than before, but it's >>>> worse >>>> > than presto and spark (they avoid or have built-in hive dependency). >>>> > >>>> > I thought about Stephan's suggestion: >>>> > - The hive/lib has 200+ jars, but we only need hive-exec.jar or plus >>>> two >>>> > or three jars, if so many jars are introduced, maybe will there be a >>>> big >>>> > conflict. >>>> > - And hive/lib is not available on every machine. We need to upload so >>>> > many jars. >>>> > - A separate classloader maybe hard to work too, our >>>> flink-connector-hive >>>> > need hive jars, we may need to deal with flink-connector-hive jar >>>> spacial >>>> > too. >>>> > CC: Rui Li >>>> > >>>> > I think the best system to integrate with hive is presto, which only >>>> > connects hive metastore through thrift protocol. But I understand >>>> that it >>>> > costs a lot to rewrite the code. >>>> > >>>> > [1] >>>> > >>>> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/hive/#dependencies >>>> > >>>> > Best, >>>> > Jingsong Lee >>>> > >>>> > On Tue, Feb 4, 2020 at 1:44 AM Stephan Ewen <se...@apache.org> wrote: >>>> > >>>> >> We have had much trouble in the past from "too deep too custom" >>>> >> integrations that everyone got out of the box, i.e., Hadoop. >>>> >> Flink has has such a broad spectrum of use cases, if we have custom >>>> build >>>> >> for every other framework in that spectrum, we'll be in trouble. >>>> >> >>>> >> So I would also be -1 for custom builds. >>>> >> >>>> >> Couldn't we do something similar as we started doing for Hadoop? >>>> Moving >>>> >> away from convenience downloads to allowing users to "export" their >>>> setup >>>> >> for Flink? >>>> >> >>>> >> - We can have a "hive module (loader)" in flink/lib by default >>>> >> - The module loader would look for an environment variable like >>>> >> "HIVE_CLASSPATH" and load these classes (ideally in a separate >>>> >> classloader). >>>> >> - The loader can search for certain classes and instantiate >>>> catalog / >>>> >> functions / etc. when finding them instantiates the hive module >>>> >> referencing >>>> >> them >>>> >> - That way, we use exactly what users have installed, without >>>> needing to >>>> >> build our own bundles. >>>> >> >>>> >> Could that work? >>>> >> >>>> >> Best, >>>> >> Stephan >>>> >> >>>> >> >>>> >> On Wed, Dec 18, 2019 at 9:43 AM Till Rohrmann <trohrm...@apache.org> >>>> >> wrote: >>>> >> >>>> >> > Couldn't it simply be documented which jars are in the convenience >>>> jars >>>> >> > which are pre built and can be downloaded from the website? Then >>>> people >>>> >> who >>>> >> > need a custom version know which jars they need to provide to >>>> Flink? >>>> >> > >>>> >> > Cheers, >>>> >> > Till >>>> >> > >>>> >> > On Tue, Dec 17, 2019 at 6:49 PM Bowen Li <bowenl...@gmail.com> >>>> wrote: >>>> >> > >>>> >> > > I'm not sure providing an uber jar would be possible. >>>> >> > > >>>> >> > > Different from kafka and elasticsearch connector who have >>>> dependencies >>>> >> > for >>>> >> > > a specific kafka/elastic version, or the kafka universal >>>> connector >>>> >> that >>>> >> > > provides good compatibilities, hive connector needs to deal with >>>> hive >>>> >> > jars >>>> >> > > in all 1.x, 2.x, 3.x versions (let alone all the HDP/CDH >>>> >> distributions) >>>> >> > > with incompatibility even between minor versions, different >>>> versioned >>>> >> > > hadoop and other extra dependency jars for each hive version. >>>> >> > > >>>> >> > > Besides, users usually need to be able to easily see which >>>> individual >>>> >> > jars >>>> >> > > are required, which is invisible from an uber jar. Hive users >>>> already >>>> >> > have >>>> >> > > their hive deployments. They usually have to use their own hive >>>> jars >>>> >> > > because, unlike hive jars on mvn, their own jars contain changes >>>> >> in-house >>>> >> > > or from vendors. They need to easily tell which jars Flink >>>> requires >>>> >> for >>>> >> > > corresponding open sourced hive version to their own hive >>>> deployment, >>>> >> and >>>> >> > > copy in-hosue jars over from hive deployments as replacements. >>>> >> > > >>>> >> > > Providing a script to download all the individual jars for a >>>> specified >>>> >> > hive >>>> >> > > version can be an alternative. >>>> >> > > >>>> >> > > The goal is we need to provide a *product*, not a technology, to >>>> make >>>> >> it >>>> >> > > less hassle for Hive users. Afterall, it's Flink embracing Hive >>>> >> community >>>> >> > > and ecosystem, not the other way around. I'd argue Hive >>>> connector can >>>> >> be >>>> >> > > treat differently because its community/ecosystem/userbase is >>>> much >>>> >> larger >>>> >> > > than the other connectors, and it's way more important than other >>>> >> > > connectors to Flink on the mission of becoming a batch/streaming >>>> >> unified >>>> >> > > engine and get Flink more widely adopted. >>>> >> > > >>>> >> > > >>>> >> > > On Sun, Dec 15, 2019 at 10:03 PM Danny Chan < >>>> yuzhao....@gmail.com> >>>> >> > wrote: >>>> >> > > >>>> >> > > > Also -1 on separate builds. >>>> >> > > > >>>> >> > > > After referencing some other BigData engines for >>>> distribution[1], i >>>> >> > > didn't >>>> >> > > > find strong needs to publish a separate build >>>> >> > > > for just a separate Hive version, indeed there are builds for >>>> >> different >>>> >> > > > Hadoop version. >>>> >> > > > >>>> >> > > > Just like Seth and Aljoscha said, we could push a >>>> >> > > > flink-hive-version-uber.jar to use as a lib of SQL-CLI or >>>> other use >>>> >> > > cases. >>>> >> > > > >>>> >> > > > [1] https://spark.apache.org/downloads.html >>>> >> > > > [2] >>>> >> > > >>>> >> >>>> https://www.elastic.co/guide/en/elasticsearch/hadoop/current/hive.html >>>> >> > > > >>>> >> > > > Best, >>>> >> > > > Danny Chan >>>> >> > > > 在 2019年12月14日 +0800 AM3:03，dev@flink.apache.org，写道： >>>> >> > > > > >>>> >> > > > > >>>> >> > > > >>>> >> > > >>>> >> > >>>> >> >>>> https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/connect.html#dependencies >>>> >> > > > >>>> >> > > >>>> >> > >>>> >> >>>> > >>>> > >>>> > -- >>>> > Best, Jingsong Lee >>>> > >>>> >>> >> >> -- >> Best, Jingsong Lee >> > -- Best, Jingsong Lee
