[ https://issues.apache.org/jira/browse/SPARK-13802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Szymon Matejczyk updated SPARK-13802: ------------------------------------- Description: When using Row constructor from kwargs, fields in the tuple underneath are sorted by name. When Schema is reading the row, it is not using the fields in this order. {code} from pyspark.sql import Row from pyspark.sql.types import * schema = StructType([ StructField("id", StringType()), StructField("first_name", StringType())]) row = Row(id="39", first_name="Szymon") schema.toInternal(row) Out[5]: ('Szymon', '39') {code} {code} df = sqlContext.createDataFrame([row], schema) df.show(1) +------+----------+ | id|first_name| +------+----------+ |Szymon| 39| +------+----------+ {code} was: When using Row constructor from kwargs, fields in the tuple underneath are sorted by name. When Schema is reading the row, it is not using the fields in this order. {code} from pyspark.sql import Row from pyspark.sql.types import * schema = StructType([ StructField("id", StringType()), StructField("first_name", StringType())]) row = Row(id="39", first_name="Szymon") schema.toInternal(row) Out[5]: ('Szymon', '39') {code} {code} df = sqlContext.createDataFrame([row], schema) df.show(1) +----------+----------+ | id |first_name| +----------+----------+ |Szymon | 39| +----------+----------+ {code} > Fields order in Row(**kwargs) is not consistent with Schema.toInternal method > ----------------------------------------------------------------------------- > > Key: SPARK-13802 > URL: https://issues.apache.org/jira/browse/SPARK-13802 > Project: Spark > Issue Type: Bug > Components: PySpark > Affects Versions: 1.6.0 > Reporter: Szymon Matejczyk > > When using Row constructor from kwargs, fields in the tuple underneath are > sorted by name. When Schema is reading the row, it is not using the fields in > this order. > {code} > from pyspark.sql import Row > from pyspark.sql.types import * > schema = StructType([ > StructField("id", StringType()), > StructField("first_name", StringType())]) > row = Row(id="39", first_name="Szymon") > schema.toInternal(row) > Out[5]: ('Szymon', '39') > {code} > {code} > df = sqlContext.createDataFrame([row], schema) > df.show(1) > +------+----------+ > | id|first_name| > +------+----------+ > |Szymon| 39| > +------+----------+ > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org