I would argue that its confusing enough to a user for options from DataFrameWriter to be silently dropped when instantiating the data source to consider this a bug. They asked for partitioning to occur, and we are doing nothing (not even telling them we can't). I was certainly surprised by this behavior. Do you have a different proposal about how this should be handled?
On Tue, Apr 16, 2019 at 4:23 PM Ryan Blue <rb...@netflix.com> wrote: > Is this a bug fix? It looks like a new feature to me. > > On Tue, Apr 16, 2019 at 4:13 PM Michael Armbrust <mich...@databricks.com> > wrote: > >> Hello All, >> >> I know we just released Spark 2.4.1, but in light of fixing SPARK-27453 >> <https://issues.apache.org/jira/browse/SPARK-27453> I was wondering if >> it might make sense to follow up quickly with 2.4.2. Without this fix its >> very hard to build a datasource that correctly handles partitioning without >> using unstable APIs. There are also a few other fixes that have trickled >> in since 2.4.1. >> >> If there are no objections, I'd like to start the process shortly. >> >> Michael >> > > > -- > Ryan Blue > Software Engineer > Netflix >
