Here’s how it was done before:
https://github.com/apache/incubator-iceberg/blob/867ec79a5c2f7619cb10546b5cc7f7bbc7d61621/build.gradle#L225-L244
That defines a set of projects called baselineProjects and applies baseline
like this:
configure(baselineProjects) {
apply plugin: 'com.palantir.baseline-checkstyle'
...
}
The baseline config has since been moved into baseline.gradle
<https://github.com/apache/incubator-iceberg/blob/master/baseline.gradle>
so changes should probably go into that file. Thanks for looking into this!
On Thu, Mar 26, 2020 at 6:23 AM Mass Dosage <massdos...@gmail.com> wrote:
> We'd like to know how to do this too. We're working on the Hive
> integration and Hive requires older versions of many of the libraries that
> Iceberg uses (Guava, Calcite and Avro are being the most problematic).
> We're going to need to shade some of these in the iceberg modules we depend
> on but it would also be very useful to be able to override the versions in
> the iceberg-hive and iceberg-mr modules so that they aren't locked to the
> same versions as the rest of the projects.
>
> On Thu, 26 Mar 2020 at 01:53, Saisai Shao <sai.sai.s...@gmail.com> wrote:
>
>> Hi Ryan,
>>
>> As mentioned in the meeting, would you please point me out the way to
>> make some submodules excluded from consistent-versions plugin.
>>
>> Thanks
>> Saisai
>>
>> Anton Okolnychyi <aokolnyc...@apple.com.invalid> 于2020年3月18日周三 上午4:14写道：
>>
>>> I am +1 on having spark-2 and spark-3 modules as well.
>>>
>>> On 7 Mar 2020, at 15:03, RD <rdsr...@gmail.com> wrote:
>>>
>>> I'm +1 to separate modules for spark-2 and spark-3, after the 0.8
>>> release.
>>> I think it would be a big change in organizations to adopt Spark-3 since
>>> that brings in Scala-2.12 which is binary incompatible to previous Scala
>>> versions. Hence this adoption could take a lot of time. I know in our
>>> company we have no near term plans to move to Spark 3.
>>>
>>> -Best,
>>> R.
>>>
>>> On Thu, Mar 5, 2020 at 6:33 PM Saisai Shao <sai.sai.s...@gmail.com>
>>> wrote:
>>>
>>>> I was thinking that if it is possible to limit version lock plugin to
>>>> only iceberg core related subprojects., seems like current
>>>> consistent-versions plugin doesn't allow to do so. So not sure if there're
>>>> some other plugins which could provide similar functionality with more
>>>> flexibility?
>>>>
>>>> Any suggestions on this?
>>>>
>>>> Best regards,
>>>> Saisai
>>>>
>>>> Saisai Shao <sai.sai.s...@gmail.com> 于2020年3月5日周四 下午3:12写道：
>>>>
>>>>> I think the requirement of supporting different version should be
>>>>> quite common. As Iceberg is a table format which should be adapted to
>>>>> different engines like Hive, Flink, Spark. To support different versions
>>>>> is
>>>>> a real problem, Spark is just one case, Hive, Flink could also be the case
>>>>> if the interface is changed across major versions. Also version lock may
>>>>> have problems when several engines coexisted in the same build, as they
>>>>> will transiently introduce lots of dependencies which may be conflicted,
>>>>> it
>>>>> may be hard to figure out one version which could satisfy all, and usually
>>>>> they only confined to a single module.
>>>>>
>>>>> So I think we should figure out a way to support such scenario, not
>>>>> just maintaining branches one by one.
>>>>>
>>>>> Ryan Blue <rb...@netflix.com> 于2020年3月5日周四 上午2:53写道：
>>>>>
>>>>>> I think the key is that this wouldn't be using the same published
>>>>>> artifacts. This work would create a spark-2.4 artifact and a spark-3.0
>>>>>> artifact. (And possibly a spark-common artifact.)
>>>>>>
>>>>>> It seems reasonable to me to have those in the same build instead of
>>>>>> in separate branches, as long as the Spark dependencies are not leaked
>>>>>> outside of the modules. That said, I'd rather have the additional checks
>>>>>> that baseline provides in general since this is a short-term problem. It
>>>>>> would just be nice if we could have versions that are confined to a
>>>>>> single
>>>>>> module. The Nebula plugin that baseline uses claims to support that, but
>>>>>> I
>>>>>> couldn't get it to work.
>>>>>>
>>>>>> On Wed, Mar 4, 2020 at 6:38 AM Saisai Shao <sai.sai.s...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Just think a bit on this. I agree that generally introducing
>>>>>>> different versions of same dependencies could be error prone. But I
>>>>>>> think
>>>>>>> the case here should not lead to issue:
>>>>>>>
>>>>>>> 1. These two sub-modules spark-2 and spark-3 are isolated, they're
>>>>>>> not dependent on either.
>>>>>>> 2. They can be differentiated by names when generating jars, also
>>>>>>> they will not be relied by other modules in Iceberg.
>>>>>>>
>>>>>>> So this dependency issue should not be the case here. And in Maven
>>>>>>> it could be achieved easily. Please correct me if wrong.
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Saisai
>>>>>>>
>>>>>>> Saisai Shao <sai.sai.s...@gmail.com> 于2020年3月4日周三 上午10:01写道：
>>>>>>>
>>>>>>>> Thanks Matt,
>>>>>>>>
>>>>>>>> If branching is the only choice, then we would potentially have two
>>>>>>>> *master* branches until spark-3 is vastly adopted. That will somehow
>>>>>>>> increase the maintenance burden and lead to inconsistency. IMO I'm OK
>>>>>>>> with
>>>>>>>> the branching way, just think that we should have a clear way to keep
>>>>>>>> tracking of two branches.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Saisai
>>>>>>>>
>>>>>>>> Matt Cheah <mch...@palantir.com.invalid> 于2020年3月4日周三 上午9:50写道：
>>>>>>>>
>>>>>>>>> I think it’s generally dangerous and error-prone to try to support
>>>>>>>>> two versions of the same library in the same build, in the same
>>>>>>>>> published
>>>>>>>>> artifacts. This is the stance that Baseline
>>>>>>>>> <https://github.com/palantir/gradle-baseline> + Gradle Consistent
>>>>>>>>> Versions <https://github.com/palantir/gradle-consistent-versions>
>>>>>>>>> takes. Gradle Consistent Versions is specifically opinionated towards
>>>>>>>>> building against one version of a library across all modules in the
>>>>>>>>> build.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I would think that branching would be the best way to build and
>>>>>>>>> publish against multiple versions of a dependency.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -Matt Cheah
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *From: *Saisai Shao <sai.sai.s...@gmail.com>
>>>>>>>>> *Reply-To: *"dev@iceberg.apache.org" <dev@iceberg.apache.org>
>>>>>>>>> *Date: *Tuesday, March 3, 2020 at 5:45 PM
>>>>>>>>> *To: *Iceberg Dev List <dev@iceberg.apache.org>
>>>>>>>>> *Cc: *Ryan Blue <rb...@netflix.com>
>>>>>>>>> *Subject: *Re: [Discuss] Merge spark-3 branch into master
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I didn't realized that Gradle cannot support two different
>>>>>>>>> versions in one build. I think I did such things for Livy to build
>>>>>>>>> scala
>>>>>>>>> 2.10 and 2.11 jars simultaneously with Maven. I'm not so familiar with
>>>>>>>>> Gradle thing, I can take a shot to see if there's some hacky ways to
>>>>>>>>> make it work.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Besides, are we saying that we will move to spark-3 support after
>>>>>>>>> 0.8 release in the master branch to replace Spark-2, or we maintain
>>>>>>>>> two
>>>>>>>>> branches for both spark-2 and spark-3 and make two releases? From
>>>>>>>>> my understanding, the adoption of spark-3 may not be so fast, and
>>>>>>>>> there
>>>>>>>>> still has lots users who stick on spark-2. Ideally, it might be
>>>>>>>>> better to
>>>>>>>>> support two versions in a near future.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> Saisai
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Mass Dosage <massdos...@gmail.com> 于2020年3月4日周三 上午1:33写道：
>>>>>>>>>
>>>>>>>>> +1 for a 0.8.0 release with Spark 2.4 and then move on for Spark
>>>>>>>>> 3.0 when it's ready.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, 3 Mar 2020 at 16:32, Ryan Blue <rb...@netflix.com.invalid>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Thanks for bringing this up, Saisai. I tried to do this a couple
>>>>>>>>> of months ago, but ran into a problem with dependency locks. I
>>>>>>>>> couldn't get
>>>>>>>>> two different versions of Spark packages in the build with baseline,
>>>>>>>>> but
>>>>>>>>> maybe I was missing something. If you can get it working, I think
>>>>>>>>> it's a
>>>>>>>>> great idea to get this into master.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Otherwise, I was thinking about proposing an 0.8.0 release in the
>>>>>>>>> next month or so based on Spark 2.4. Then we could merge the branch
>>>>>>>>> into
>>>>>>>>> master and do another release for Spark 3.0 when it's ready.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> rb
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Mar 3, 2020 at 6:07 AM Saisai Shao <sai.sai.s...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi team,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I was thinking of merging spark-3 branch into master, also per the
>>>>>>>>> discussion before we could make spark-2 and spark-3 coexisted into 2
>>>>>>>>> different sub-modules. With this, one build could generate both
>>>>>>>>> spark-2 and
>>>>>>>>> spark-3 runtime jars, user could pick either at preference.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> One concern is that they share lots of common code in read/write
>>>>>>>>> path, this will increase the maintenance overhead to keep consistency
>>>>>>>>> of
>>>>>>>>> two copies.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> So I'd like to hear your thoughts, any suggestions on it?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> Saisai
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Ryan Blue
>>>>>>>>>
>>>>>>>>> Software Engineer
>>>>>>>>>
>>>>>>>>> Netflix
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ryan Blue
>>>>>> Software Engineer
>>>>>> Netflix
>>>>>>
>>>>>
>>>
--
Ryan Blue
Software Engineer
Netflix
