The Arrow data format is not yet stable, meaning there are no guarantees on
backwards/forwards compatibility. Once version 1.0 is released, it will
have those guarantees but it's hard to say when that will be. The remaining
work to get there can be seen at
Okay, that makes sense, but is the Arrow data format stable? If not, we risk
breakage when Arrow changes in the future and some libraries using this feature
are begin to use the new Arrow code.
Matei
> On Apr 20, 2019, at 1:39 PM, Bobby Evans wrote:
>
> I want to be clear that this SPIP is
I want to be clear that this SPIP is not proposing exposing Arrow
APIs/Classes through any Spark APIs. SPARK-24579 is doing that, and
because of the overlap between the two SPIPs I scaled this one back to
concentrate just on the columnar processing aspects. Sorry for the
confusion as I didn't
+1 from me too.
It seems like there is support for merging the Jackson change into
2.4.x (and, I think, a few more minor dependency updates) but this
doesn't have to go into 2.4.2. That said, if there is another RC for
any reason, I think we could include it. Otherwise can wait for 2.4.3.
On
FYI, I’d also be concerned about exposing the Arrow API or format as a public
API if it’s not yet stable. Is stabilization of the API and format coming soon
on the roadmap there? Maybe someone can work with the Arrow community to make
that happen.
We’ve been bitten lots of times by API changes
I think you misunderstood the point of this SPIP. I responded to your
comments in the SPIP JIRA.
On Sat, Apr 20, 2019 at 12:52 AM Xiangrui Meng wrote:
> I posted my comment in the JIRA
>
