Re: DSv2 & DataSourceRegister

2020-04-08 Thread Andrew Melo
On Wed, Apr 8, 2020 at 8:35 AM Wenchen Fan wrote: > > It would be good to support your use case, but I'm not sure how to accomplish > that. Can you open a PR so that we can discuss it in detail? How can `public > Class getImplementation();` be possible in 3.0 as > there is no `DataSourceV2`?

Re: DSv2 & DataSourceRegister

2020-04-08 Thread Wenchen Fan
It would be good to support your use case, but I'm not sure how to accomplish that. Can you open a PR so that we can discuss it in detail? How can `public Class getImplementation();` be possible in 3.0 as there is no `DataSourceV2`? On Wed, Apr 8, 2020 at 1:12 PM Andrew Melo wrote: > Hello > >

sortWithinPartitions in Structured Streaming

2020-04-08 Thread Kimahriman
Currently, all sorting is disallowed with structured streaming queries. Not allowing global sorting makes sense, as you can't sort an infinite list, but could non-global sorting (i.e. sortWithinPartitions) be allowed? I'm running into this with an external source I'm using, but not sure if this