[
https://issues.apache.org/jira/browse/SPARK-29748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16975054#comment-16975054
]
Maciej Szymkiewicz commented on SPARK-29748:
--------------------------------------------
[~jhereth]
{quote}With simply removing sorting we change the semantics, e.g. `Row(a=1,
b=2) != Row(b=2, a=1)` (opposed to what we currently have.{quote}
It is even more messy. At the moment we adhere to {{tuple}} semantics so
{{Row(a=1, b=2) == Row(y=1, z=2)}}. That might be acceptable (namedtuples use
the same approach, but I think we should state that explicitly).
{quote}I think Maciej Szymkiewicz was thinking about changes for the upcoming
3.0?{quote}
Indeed.
[~bryanc]
Let me clarify things - I am not suggesting that any of these changes should be
implemented here. Instead I think we should have clear picture what {{Row}}
suppose to be (not only in terms of API, but also intended applications) before
we decide on a concrete solution. That's particularly important because we
already have special cases that were introduced specifically to target
{{**kwargs}} and sorting behavior.
That being said, if we want to discuss this case in isolation
* Introducing {{LegacyRow}} seems to make little sense if implementation of
{{Row}} stays the same otherwise. Sorting or not, depending on the config,
should be enough.
* {quote} Users with Python < 3.6 will have to create Rows with an OrderedDict
or by using the Row class as a factory (explained in the pydoc). {quote} I
don't think we should introduce such behavior now, when 3.5 is deprecated.
Having yet another way to initialize {{Row}} will be confusing at best (and
introduce new problems when using complex structures). Furthermore we already
have one mechanism that provides ordered behavior independent of version.
Instead I'd suggest we:
* Make legacy behavior the only option for Python < 3.6.
* For Python 3.6 let's introduce legacy sorting mechanism (keeping only single
{{Row}}) class, enabled by default and deprecated.
> Remove sorting of fields in PySpark SQL Row creation
> ----------------------------------------------------
>
> Key: SPARK-29748
> URL: https://issues.apache.org/jira/browse/SPARK-29748
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
> Affects Versions: 3.0.0
> Reporter: Bryan Cutler
> Priority: Major
>
> Currently, when a PySpark Row is created with keyword arguments, the fields
> are sorted alphabetically. This has created a lot of confusion with users
> because it is not obvious (although it is stated in the pydocs) that they
> will be sorted alphabetically, and then an error can occur later when
> applying a schema and the field order does not match.
> The original reason for sorting fields is because kwargs in python < 3.6 are
> not guaranteed to be in the same order that they were entered. Sorting
> alphabetically would ensure a consistent order. Matters are further
> complicated with the flag {{__from_dict__}} that allows the {{Row}} fields to
> to be referenced by name when made by kwargs, but this flag is not serialized
> with the Row and leads to inconsistent behavior.
> This JIRA proposes that any sorting of the Fields is removed. Users with
> Python 3.6+ creating Rows with kwargs can continue to do so since Python will
> ensure the order is the same as entered. Users with Python < 3.6 will have to
> create Rows with an OrderedDict or by using the Row class as a factory
> (explained in the pydoc). If kwargs are used, an error will be raised or
> based on a conf setting it can fall back to a LegacyRow that will sort the
> fields as before. This LegacyRow will be immediately deprecated and removed
> once support for Python < 3.6 is dropped.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
