I'd support the option that Jack suggests if we can set a few expectations for keeping it clean.
First, I'd like to avoid refactoring code to share it across Spark versions -- that introduces risk because we're relying on compiling against one version and running in another and both Spark and Scala change rapidly. A big benefit of options 1 and 2 is that we mostly focus on only one Spark version. I think we should duplicate code rather than spend time refactoring to rely on binary compatibility. I propose we start each new Spark version by copying the last one and updating it. And we should build just the latest supported version by default. The drawback to having everything in a single repo is that we wouldn't be able to cherry-pick changes across Spark versions/branches, but I think Jack is right that having a single build is better. Second, we should make CI faster by running the Spark builds in parallel. It sounds like this is what would happen anyway, with a property that selects the Spark version that you want to build against. Overall, this new suggestion sounds like a promising way forward. Ryan On Wed, Sep 15, 2021 at 11:46 PM Jack Ye <yezhao...@gmail.com> wrote: > I think in Ryan's proposal we will create a ton of modules anyway, as Wing > listed we are just using git branch as an additional dimension, but my > understanding is that you will still have 1 core, 1 extension, 1 runtime > artifact published for each Spark version in either approach. > > In that case, this is just brainstorming, I wonder if we can explore a > modified option 3 that flattens all the versions in each Spark branch in > option 2 into master. The repository structure would look something like: > > iceberg/api/... > /bundled-guava/... > /core/... > ... > /spark/2.4/core/... > /extension/... > /runtime/... > /3.1/core/... > /extension/... > /runtime/... > > The gradle build script in the root is configured to build against the > latest version of Spark by default, unless otherwise specified by the user. > > Intellij can also be configured to only index files of specific versions > based on the same config used in build. > > In this way, I imagine the CI setup to be much easier to do things like > testing version compatibility for a feature or running only a > specific subset of Spark version builds based on the Spark version > directories touched. > > And the biggest benefit is that we don't have the same difficulty as > option 2 of developing a feature when it's both in core and Spark. > > We can then develop a mechanism to vote to stop support of certain > versions, and archive the corresponding directory to avoid accumulating too > many versions in the long term. > > -Jack Ye > > > On Wed, Sep 15, 2021 at 4:17 PM Ryan Blue <b...@tabular.io> wrote: > >> Sorry, I was thinking about CI integration between Iceberg Java and >> Iceberg Spark, I just didn't mention it and I see how that's a big thing to >> leave out! >> >> I would definitely want to test the projects together. One thing we could >> do is have a nightly build like Russell suggests. I'm also wondering if we >> could have some tighter integration where the Iceberg Spark build can be >> included in the Iceberg Java build using properties. Maybe the github >> action could checkout Iceberg, then checkout the Spark integration's latest >> branch, and then run the gradle build with a property that makes Spark a >> subproject in the build. That way we can continue to have Spark CI run >> regularly. >> >> On Wed, Sep 15, 2021 at 3:08 PM Russell Spitzer < >> russell.spit...@gmail.com> wrote: >> >>> I agree that Option 2 is considerably more difficult for development >>> when core API changes need to be picked up by the external Spark module. I >>> also think a monthly release would probably still be prohibitive to >>> actually implementing new features that appear in the API, I would hope we >>> have a much faster process or maybe just have snapshot artifacts published >>> nightly? >>> >>> On Sep 15, 2021, at 4:46 PM, Wing Yew Poon <wyp...@cloudera.com.INVALID> >>> wrote: >>> >>> IIUC, Option 2 is to move the Spark support for Iceberg into a separate >>> repo (subproject of Iceberg). Would we have branches such as 0.13-2.4, >>> 0.13-3.0, 0.13-3.1, and 0.13-3.2? For features that can be supported in all >>> versions or all Spark 3 versions, then we would need to commit the changes >>> to all applicable branches. Basically we are trading more work to commit to >>> multiple branches for simplified build and CI time per branch, which might >>> be an acceptable trade-off. However, the biggest downside is that changes >>> may need to be made in core Iceberg as well as in the engine (in this case >>> Spark) support, and we need to wait for a release of core Iceberg to >>> consume the changes in the subproject. In this case, maybe we should have a >>> monthly release of core Iceberg (no matter how many changes go in, as long >>> as it is non-zero) so that the subproject can consume changes fairly >>> quickly? >>> >>> >>> On Wed, Sep 15, 2021 at 2:09 PM Ryan Blue <b...@tabular.io> wrote: >>> >>>> Thanks for bringing this up, Anton. I’m glad that we have the set of >>>> potential solutions well defined. >>>> >>>> Looks like the next step is to decide whether we want to require people >>>> to update Spark versions to pick up newer versions of Iceberg. If we choose >>>> to make people upgrade, then option 1 is clearly the best choice. >>>> >>>> I don’t think that we should make updating Spark a requirement. Many of >>>> the things that we’re working on are orthogonal to Spark versions, like >>>> table maintenance actions, secondary indexes, the 1.0 API, views, ORC >>>> delete files, new storage implementations, etc. Upgrading Spark is time >>>> consuming and untrusted in my experience, so I think we would be setting up >>>> an unnecessary trade-off between spending lots of time to upgrade Spark and >>>> picking up new Iceberg features. >>>> >>>> Another way of thinking about this is that if we went with option 1, >>>> then we could port bug fixes into 0.12.x. But there are many things that >>>> wouldn’t fit this model, like adding a FileIO implementation for ADLS. So >>>> some people in the community would have to maintain branches of newer >>>> Iceberg versions with older versions of Spark outside of the main Iceberg >>>> project — that defeats the purpose of simplifying things with option 1 >>>> because we would then have more people maintaining the same 0.13.x with >>>> Spark 3.1 branch. (This reminds me of the Spark community, where we wanted >>>> to release a 2.5 line with DSv2 backported, but the community decided not >>>> to so we built similar 2.4+DSv2 branches at Netflix, Tencent, Apple, etc.) >>>> >>>> If the community is going to do the work anyway — and I think some of >>>> us would — we should make it possible to share that work. That’s why I >>>> don’t think that we should go with option 1. >>>> >>>> If we don’t go with option 1, then the choice is how to maintain >>>> multiple Spark versions. I think that the way we’re doing it right now is >>>> not something we want to continue. >>>> >>>> Using multiple modules (option 3) is concerning to me because of the >>>> changes in Spark. We currently structure the library to share as much code >>>> as possible. But that means compiling against different Spark versions and >>>> relying on binary compatibility and reflection in some cases. To me, this >>>> seems unmaintainable in the long run because it requires refactoring common >>>> classes and spending a lot of time deduplicating code. It also creates a >>>> ton of modules, at least one common module, then a module per version, then >>>> an extensions module per version, and finally a runtime module per version. >>>> That’s 3 modules per Spark version, plus any new common modules. And each >>>> module needs to be tested, which is making our CI take a really long time. >>>> We also don’t support multiple Scala versions, which is another gap that >>>> will require even more modules and tests. >>>> >>>> I like option 2 because it would allow us to compile against a single >>>> version of Spark (which will be much more reliable). It would give us an >>>> opportunity to support different Scala versions. It avoids the need to >>>> refactor to share code and allows people to focus on a single version of >>>> Spark, while also creating a way for people to maintain and update the >>>> older versions with newer Iceberg releases. I don’t think that this would >>>> slow down development. I think it would actually speed it up because we’d >>>> be spending less time trying to make multiple versions work in the same >>>> build. And anyone in favor of option 1 would basically get option 1: you >>>> don’t have to care about branches for older Spark versions. >>>> >>>> Jack makes a good point about wanting to keep code in a single >>>> repository, but I think that the need to manage more version combinations >>>> overrides this concern. It’s easier to make this decision in python because >>>> we’re not trying to depend on two projects that change relatively quickly. >>>> We’re just trying to build a library. >>>> >>>> Ryan >>>> >>>> On Wed, Sep 15, 2021 at 2:58 AM OpenInx <open...@gmail.com> wrote: >>>> >>>>> Thanks for bringing this up, Anton. >>>>> >>>>> Everyone has great pros/cons to support their preferences. Before >>>>> giving my preference, let me raise one question: what's the top >>>>> priority >>>>> thing for apache iceberg project at this point in time ? This question >>>>> will help us to answer the following question: Should we support more >>>>> engine versions more robustly or be a bit more aggressive and concentrate >>>>> on getting the new features that users need most in order to keep the >>>>> project more competitive ? >>>>> >>>>> If people watch the apache iceberg project and check the issues & >>>>> PR frequently, I guess more than 90% people will answer the priority >>>>> question: There is no doubt for making the whole v2 story to be >>>>> production-ready. The current roadmap discussion also proofs the thing : >>>>> https://lists.apache.org/x/thread.html/r84e80216c259c81f824c6971504c321cd8c785774c489d52d4fc123f@%3Cdev.iceberg.apache.org%3E >>>>> . >>>>> >>>>> In order to ensure the highest priority at this point in time, I will >>>>> prefer option-1 to reduce the cost of engine maintenance, so as to free up >>>>> resources to make v2 production-ready. >>>>> >>>>> On Wed, Sep 15, 2021 at 3:00 PM Saisai Shao <sai.sai.s...@gmail.com> >>>>> wrote: >>>>> >>>>>> From Dev's point, it has less burden to always support the latest >>>>>> version of Spark (for example). But from user's point, especially for us >>>>>> who maintain Spark internally, it is not easy to upgrade the Spark >>>>>> version >>>>>> for the first time (since we have many customizations internally), and >>>>>> we're still promoting to upgrade to 3.1.2. If the community ditches the >>>>>> support of old version of Spark3, users have to maintain it themselves >>>>>> unavoidably. >>>>>> >>>>>> So I'm inclined to make this support in community, not by users >>>>>> themselves, as for Option 2 or 3, I'm fine with either. And to relieve >>>>>> the >>>>>> burden, we could support limited versions of Spark (for example 2 >>>>>> versions). >>>>>> >>>>>> Just my two cents. >>>>>> >>>>>> -Saisai >>>>>> >>>>>> >>>>>> Jack Ye <yezhao...@gmail.com> 于2021年9月15日周三 下午1:35写道： >>>>>> >>>>>>> Hi Wing Yew, >>>>>>> >>>>>>> I think 2.4 is a different story, we will continue to support Spark >>>>>>> 2.4, but as you can see it will continue to have very limited >>>>>>> functionalities comparing to Spark 3. I believe we discussed about >>>>>>> option 3 >>>>>>> when we were doing Spark 3.0 to 3.1 upgrade. Recently we are seeing the >>>>>>> same issue for Flink 1.11, 1.12 and 1.13 as well. I feel we need a >>>>>>> consistent strategy around this, let's take this chance to make a good >>>>>>> community guideline for all future engine versions, especially for >>>>>>> Spark, >>>>>>> Flink and Hive that are in the same repository. >>>>>>> >>>>>>> I can totally understand your point of view Wing, in fact, speaking >>>>>>> from the perspective of AWS EMR, we have to support over 40 versions of >>>>>>> the >>>>>>> software because there are people who are still using Spark 1.4, >>>>>>> believe it >>>>>>> or not. After all, keep backporting changes will become a liability not >>>>>>> only on the user side, but also on the service provider side, so I >>>>>>> believe >>>>>>> it's not a bad practice to push for user upgrade, as it will make the >>>>>>> life >>>>>>> of both parties easier in the end. New feature is definitely one of the >>>>>>> best incentives to promote an upgrade on user side. >>>>>>> >>>>>>> I think the biggest issue of option 3 is about its scalability, >>>>>>> because we will have an unbounded list of packages to add and compile in >>>>>>> the future, and we probably cannot drop support of that package once >>>>>>> created. If we go with option 1, I think we can still publish a few >>>>>>> patch >>>>>>> versions for old Iceberg releases, and committers can control the >>>>>>> amount of >>>>>>> patch versions to guard people from abusing the power of patching. I see >>>>>>> this as a consistent strategy also for Flink and Hive. With this >>>>>>> strategy, >>>>>>> we can truly have a compatibility matrix for engine versions against >>>>>>> Iceberg versions. >>>>>>> >>>>>>> -Jack >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Sep 14, 2021 at 10:00 PM Wing Yew Poon < >>>>>>> wyp...@cloudera.com.invalid> wrote: >>>>>>> >>>>>>>> I understand and sympathize with the desire to use new DSv2 >>>>>>>> features in Spark 3.2. I agree that Option 1 is the easiest for >>>>>>>> developers, >>>>>>>> but I don't think it considers the interests of users. I do not think >>>>>>>> that >>>>>>>> most users will upgrade to Spark 3.2 as soon as it is released. It is a >>>>>>>> "minor version" upgrade in name from 3.1 (or from 3.0), but I think we >>>>>>>> all >>>>>>>> know that it is not a minor upgrade. There are a lot of changes from >>>>>>>> 3.0 to >>>>>>>> 3.1 and from 3.1 to 3.2. I think there are even a lot of users running >>>>>>>> Spark 2.4 and not even on Spark 3 yet. Do we also plan to stop >>>>>>>> supporting >>>>>>>> Spark 2.4? >>>>>>>> >>>>>>>> Please correct me if I'm mistaken, but the folks who have spoken >>>>>>>> out in favor of Option 1 all work for the same organization, don't >>>>>>>> they? >>>>>>>> And they don't have a problem with making their users, all internal, >>>>>>>> simply >>>>>>>> upgrade to Spark 3.2, do they? (Or they are already running an internal >>>>>>>> fork that is close to 3.2.) >>>>>>>> >>>>>>>> I work for an organization with customers running different >>>>>>>> versions of Spark. It is true that we can backport new features to >>>>>>>> older >>>>>>>> versions if we wanted to. I suppose the people contributing to Iceberg >>>>>>>> work >>>>>>>> for some organization or other that either use Iceberg in-house, or >>>>>>>> provide >>>>>>>> software (possibly in the form of a service) to customers, and either >>>>>>>> way, >>>>>>>> the organizations have the ability to backport features and fixes to >>>>>>>> internal versions. Are there any users out there who simply use Apache >>>>>>>> Iceberg and depend on the community version? >>>>>>>> >>>>>>>> There may be features that are broadly useful that do not depend on >>>>>>>> Spark 3.2. Is it worth supporting them on Spark 3.0/3.1 (and even 2.4)? >>>>>>>> >>>>>>>> I am not in favor of Option 2. I do not oppose Option 1, but I >>>>>>>> would consider Option 3 too. Anton, you said 5 modules are required; >>>>>>>> what >>>>>>>> are the modules you're thinking of? >>>>>>>> >>>>>>>> - Wing Yew >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Sep 14, 2021 at 5:38 PM Yufei Gu <flyrain...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Option 1 sounds good to me. Here are my reasons: >>>>>>>>> >>>>>>>>> 1. Both 2 and 3 will slow down the development. Considering the >>>>>>>>> limited resources in the open source community, the upsides of option >>>>>>>>> 2 and >>>>>>>>> 3 are probably not worthy. >>>>>>>>> 2. Both 2 and 3 assume the use cases may not exist. It's hard to >>>>>>>>> predict anything, but even if these use cases are legit, users can >>>>>>>>> still >>>>>>>>> get the new feature by backporting it to an older version in case of >>>>>>>>> upgrading to a newer version isn't an option. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> >>>>>>>>> Yufei >>>>>>>>> >>>>>>>>> `This is not a contribution` >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Sep 14, 2021 at 4:54 PM Anton Okolnychyi < >>>>>>>>> aokolnyc...@apple.com.invalid> wrote: >>>>>>>>> >>>>>>>>>> To sum up what we have so far: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *Option 1 (support just the most recent minor Spark 3 version)* >>>>>>>>>> >>>>>>>>>> The easiest option for us devs, forces the user to upgrade to the >>>>>>>>>> most recent minor Spark version to consume any new Iceberg >>>>>>>>>> features. >>>>>>>>>> >>>>>>>>>> *Option 2 (a separate project under Iceberg)* >>>>>>>>>> >>>>>>>>>> Can support as many Spark versions as needed and the codebase is >>>>>>>>>> still separate as we can use separate branches. >>>>>>>>>> Impossible to consume any unreleased changes in core, may slow >>>>>>>>>> down the development. >>>>>>>>>> >>>>>>>>>> *Option 3 (separate modules for Spark 3.1/3.2)* >>>>>>>>>> >>>>>>>>>> Introduce more modules in the same project. >>>>>>>>>> Can consume unreleased changes but it will required at least 5 >>>>>>>>>> modules to support 2.4, 3.1 and 3.2, making the build and testing >>>>>>>>>> complicated. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Are there any users for whom upgrading the minor Spark version >>>>>>>>>> (e3.1 to 3.2) to consume new features is a blocker? >>>>>>>>>> We follow Option 1 internally at the moment but I would like to >>>>>>>>>> hear what other people think/need. >>>>>>>>>> >>>>>>>>>> - Anton >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 14 Sep 2021, at 09:44, Russell Spitzer < >>>>>>>>>> russell.spit...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>> I think we should go for option 1. I already am not a big fan of >>>>>>>>>> having runtime errors for unsupported things based on versions and I >>>>>>>>>> don't >>>>>>>>>> think minor version upgrades are a large issue for users. I'm >>>>>>>>>> especially >>>>>>>>>> not looking forward to supporting interfaces that only exist in >>>>>>>>>> Spark 3.2 >>>>>>>>>> in a multiple Spark version support future. >>>>>>>>>> >>>>>>>>>> On Sep 14, 2021, at 11:32 AM, Anton Okolnychyi < >>>>>>>>>> aokolnyc...@apple.com.INVALID> wrote: >>>>>>>>>> >>>>>>>>>> First of all, is option 2 a viable option? We discussed >>>>>>>>>> separating the python module outside of the project a few weeks ago, >>>>>>>>>> and >>>>>>>>>> decided to not do that because it's beneficial for code cross >>>>>>>>>> reference and >>>>>>>>>> more intuitive for new developers to see everything in the same >>>>>>>>>> repository. >>>>>>>>>> I would expect the same argument to also hold here. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> That’s exactly the concern I have about Option 2 at this moment. >>>>>>>>>> >>>>>>>>>> Overall I would personally prefer us to not support all the minor >>>>>>>>>> versions, but instead support maybe just 2-3 latest versions in a >>>>>>>>>> major >>>>>>>>>> version. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> This is when it gets a bit complicated. If we want to support >>>>>>>>>> both Spark 3.1 and Spark 3.2 with a single module, it means we have >>>>>>>>>> to >>>>>>>>>> compile against 3.1. The problem is that we rely on DSv2 that is >>>>>>>>>> being >>>>>>>>>> actively developed. 3.2 and 3.1 have substantial differences. On top >>>>>>>>>> of >>>>>>>>>> that, we have our extensions that are extremely low-level and may >>>>>>>>>> break not >>>>>>>>>> only between minor versions but also between patch releases. >>>>>>>>>> >>>>>>>>>> f there are some features requiring a newer version, it makes >>>>>>>>>> sense to move that newer version in master. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Internally, we don’t deliver new features to older Spark versions >>>>>>>>>> as it requires a lot of effort to port things. Personally, I don’t >>>>>>>>>> think it >>>>>>>>>> is too bad to require users to upgrade if they want new features. At >>>>>>>>>> the >>>>>>>>>> same time, there are valid concerns with this approach too that we >>>>>>>>>> mentioned during the sync. For example, certain new features would >>>>>>>>>> also >>>>>>>>>> work fine with older Spark versions. I generally agree with that and >>>>>>>>>> that >>>>>>>>>> not supporting recent versions is not ideal. However, I want to find >>>>>>>>>> a >>>>>>>>>> balance between the complexity on our side and ease of use for the >>>>>>>>>> users. >>>>>>>>>> Ideally, supporting a few recent versions would be sufficient but >>>>>>>>>> our Spark >>>>>>>>>> integration is too low-level to do that with a single module. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 13 Sep 2021, at 20:53, Jack Ye <yezhao...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>> First of all, is option 2 a viable option? We discussed >>>>>>>>>> separating the python module outside of the project a few weeks ago, >>>>>>>>>> and >>>>>>>>>> decided to not do that because it's beneficial for code cross >>>>>>>>>> reference and >>>>>>>>>> more intuitive for new developers to see everything in the same >>>>>>>>>> repository. >>>>>>>>>> I would expect the same argument to also hold here. >>>>>>>>>> >>>>>>>>>> Overall I would personally prefer us to not support all the minor >>>>>>>>>> versions, but instead support maybe just 2-3 latest versions in a >>>>>>>>>> major >>>>>>>>>> version. This avoids the problem that some users are unwilling to >>>>>>>>>> move to a >>>>>>>>>> newer version and keep patching old Spark version branches. If there >>>>>>>>>> are >>>>>>>>>> some features requiring a newer version, it makes sense to move that >>>>>>>>>> newer >>>>>>>>>> version in master. >>>>>>>>>> >>>>>>>>>> In addition, because currently Spark is considered the most >>>>>>>>>> feature-complete reference implementation compared to all other >>>>>>>>>> engines, I >>>>>>>>>> think we should not add artificial barriers that would slow down its >>>>>>>>>> development speed. >>>>>>>>>> >>>>>>>>>> So my thinking is closer to option 1. >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> Jack Ye >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Sep 13, 2021 at 7:39 PM Anton Okolnychyi < >>>>>>>>>> aokolnyc...@apple.com.invalid> wrote: >>>>>>>>>> >>>>>>>>>>> Hey folks, >>>>>>>>>>> >>>>>>>>>>> I want to discuss our Spark version support strategy. >>>>>>>>>>> >>>>>>>>>>> So far, we have tried to support both 3.0 and 3.1. It is great >>>>>>>>>>> to support older versions but because we compile against 3.0, we >>>>>>>>>>> cannot use >>>>>>>>>>> any Spark features that are offered in newer versions. >>>>>>>>>>> Spark 3.2 is just around the corner and it brings a lot of >>>>>>>>>>> important features such dynamic filtering for v2 tables, required >>>>>>>>>>> distribution and ordering for writes, etc. These features are too >>>>>>>>>>> important >>>>>>>>>>> to ignore them. >>>>>>>>>>> >>>>>>>>>>> Apart from that, I have an end-to-end prototype for >>>>>>>>>>> merge-on-read with Spark that actually leverages some of the 3.2 >>>>>>>>>>> features. >>>>>>>>>>> I’ll be implementing all new Spark DSv2 APIs for us internally and >>>>>>>>>>> would >>>>>>>>>>> love to share that with the rest of the community. >>>>>>>>>>> >>>>>>>>>>> I see two options to move forward: >>>>>>>>>>> >>>>>>>>>>> Option 1 >>>>>>>>>>> >>>>>>>>>>> Migrate to Spark 3.2 in master, maintain 0.12 for a while by >>>>>>>>>>> releasing minor versions with bug fixes. >>>>>>>>>>> >>>>>>>>>>> Pros: almost no changes to the build configuration, no extra >>>>>>>>>>> work on our side as just a single Spark version is actively >>>>>>>>>>> maintained. >>>>>>>>>>> Cons: some new features that we will be adding to master could >>>>>>>>>>> also work with older Spark versions but all 0.12 releases will only >>>>>>>>>>> contain >>>>>>>>>>> bug fixes. Therefore, users will be forced to migrate to Spark 3.2 >>>>>>>>>>> to >>>>>>>>>>> consume any new Spark or format features. >>>>>>>>>>> >>>>>>>>>>> Option 2 >>>>>>>>>>> >>>>>>>>>>> Move our Spark integration into a separate project and introduce >>>>>>>>>>> branches for 3.0, 3.1 and 3.2. >>>>>>>>>>> >>>>>>>>>>> Pros: decouples the format version from Spark, we can support as >>>>>>>>>>> many Spark versions as needed. >>>>>>>>>>> Cons: more work initially to set everything up, more work to >>>>>>>>>>> release, will need a new release of the core format to consume any >>>>>>>>>>> changes >>>>>>>>>>> in the Spark integration. >>>>>>>>>>> >>>>>>>>>>> Overall, I think option 2 seems better for the user but my main >>>>>>>>>>> worry is that we will have to release the format more frequently >>>>>>>>>>> (which is >>>>>>>>>>> a good thing but requires more work and time) and the overall Spark >>>>>>>>>>> development may be slower. >>>>>>>>>>> >>>>>>>>>>> I’d love to hear what everybody thinks about this matter. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Anton >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>> >>>> -- >>>> Ryan Blue >>>> Tabular >>>> >>> >>> >> >> -- >> Ryan Blue >> Tabular >> > -- Ryan Blue Tabular
