Hi again,
Does anyone have thoughts on either the idea or the implementation?
Thanks,
Andrew
On Thu, Apr 9, 2020 at 11:32 PM Andrew Melo wrote:
>
> Hi all,
>
> I've opened a WIP PR here https://github.com/apache/spark/pull/28159
> I'm a novice at Scala, so I'm sure the code isn't idiomatic,
Hi all,
I've opened a WIP PR here https://github.com/apache/spark/pull/28159
I'm a novice at Scala, so I'm sure the code isn't idiomatic, but it
behaves functionally how I'd expect. I've added unit tests to the PR,
but if you would like to verify the intended functionality, I've
uploaded a fat
On Wed, Apr 8, 2020 at 8:35 AM Wenchen Fan wrote:
>
> It would be good to support your use case, but I'm not sure how to accomplish
> that. Can you open a PR so that we can discuss it in detail? How can `public
> Class getImplementation();` be possible in 3.0 as
> there is no `DataSourceV2`?
It would be good to support your use case, but I'm not sure how to
accomplish that. Can you open a PR so that we can discuss it in detail? How
can `public Class getImplementation();` be
possible in 3.0 as there is no `DataSourceV2`?
On Wed, Apr 8, 2020 at 1:12 PM Andrew Melo wrote:
> Hello
>
>
Hello
On Tue, Apr 7, 2020 at 23:16 Wenchen Fan wrote:
> Are you going to provide a single artifact for Spark 2.4 and 3.0? I'm not
> sure this is possible as the DS V2 API is very different in 3.0, e.g. there
> is no `DataSourceV2` anymore, and you should implement `TableProvider` (if
> you
Are you going to provide a single artifact for Spark 2.4 and 3.0? I'm not
sure this is possible as the DS V2 API is very different in 3.0, e.g. there
is no `DataSourceV2` anymore, and you should implement `TableProvider` (if
you don't have database/table).
On Wed, Apr 8, 2020 at 6:58 AM Andrew
Hi Ryan,
On Tue, Apr 7, 2020 at 5:21 PM Ryan Blue wrote:
>
> Hi Andrew,
>
> With DataSourceV2, I recommend plugging in a catalog instead of using
> DataSource. As you've noticed, the way that you plug in data sources isn't
> very flexible. That's one of the reasons why we changed the plugin
Hi Andrew,
With DataSourceV2, I recommend plugging in a catalog instead of using
DataSource. As you've noticed, the way that you plug in data sources isn't
very flexible. That's one of the reasons why we changed the plugin system
and made it possible to use named catalogs that load
Hi all,
I posted an improvement ticket in JIRA and Hyukjin Kwon requested I
send an email to the dev list for discussion.
As the DSv2 API evolves, some breaking changes are occasionally made
to the API. It's possible to split a plugin into a "common" part and
multiple version-specific parts and
