[ https://issues.apache.org/jira/browse/SPARK-29748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16975054#comment-16975054 ]
Maciej Szymkiewicz commented on SPARK-29748: -------------------------------------------- [~jhereth] {quote}With simply removing sorting we change the semantics, e.g. `Row(a=1, b=2) != Row(b=2, a=1)` (opposed to what we currently have.{quote} It is even more messy. At the moment we adhere to {{tuple}} semantics so {{Row(a=1, b=2) == Row(y=1, z=2)}}. That might be acceptable (namedtuples use the same approach, but I think we should state that explicitly). {quote}I think Maciej Szymkiewicz was thinking about changes for the upcoming 3.0?{quote} Indeed. [~bryanc] Let me clarify things - I am not suggesting that any of these changes should be implemented here. Instead I think we should have clear picture what {{Row}} suppose to be (not only in terms of API, but also intended applications) before we decide on a concrete solution. That's particularly important because we already have special cases that were introduced specifically to target {{**kwargs}} and sorting behavior. That being said, if we want to discuss this case in isolation * Introducing {{LegacyRow}} seems to make little sense if implementation of {{Row}} stays the same otherwise. Sorting or not, depending on the config, should be enough. * {quote} Users with Python < 3.6 will have to create Rows with an OrderedDict or by using the Row class as a factory (explained in the pydoc). {quote} I don't think we should introduce such behavior now, when 3.5 is deprecated. Having yet another way to initialize {{Row}} will be confusing at best (and introduce new problems when using complex structures). Furthermore we already have one mechanism that provides ordered behavior independent of version. Instead I'd suggest we: * Make legacy behavior the only option for Python < 3.6. * For Python 3.6 let's introduce legacy sorting mechanism (keeping only single {{Row}}) class, enabled by default and deprecated. > Remove sorting of fields in PySpark SQL Row creation > ---------------------------------------------------- > > Key: SPARK-29748 > URL: https://issues.apache.org/jira/browse/SPARK-29748 > Project: Spark > Issue Type: Bug > Components: PySpark, SQL > Affects Versions: 3.0.0 > Reporter: Bryan Cutler > Priority: Major > > Currently, when a PySpark Row is created with keyword arguments, the fields > are sorted alphabetically. This has created a lot of confusion with users > because it is not obvious (although it is stated in the pydocs) that they > will be sorted alphabetically, and then an error can occur later when > applying a schema and the field order does not match. > The original reason for sorting fields is because kwargs in python < 3.6 are > not guaranteed to be in the same order that they were entered. Sorting > alphabetically would ensure a consistent order. Matters are further > complicated with the flag {{__from_dict__}} that allows the {{Row}} fields to > to be referenced by name when made by kwargs, but this flag is not serialized > with the Row and leads to inconsistent behavior. > This JIRA proposes that any sorting of the Fields is removed. Users with > Python 3.6+ creating Rows with kwargs can continue to do so since Python will > ensure the order is the same as entered. Users with Python < 3.6 will have to > create Rows with an OrderedDict or by using the Row class as a factory > (explained in the pydoc). If kwargs are used, an error will be raised or > based on a conf setting it can fall back to a LegacyRow that will sort the > fields as before. This LegacyRow will be immediately deprecated and removed > once support for Python < 3.6 is dropped. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org