[ https://issues.apache.org/jira/browse/SPARK-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994833#comment-15994833 ]
John Berryman commented on SPARK-12467: --------------------------------------- I believe there is still something here that needs to be fixed. Consider the following code: {code} from pyspark.sql import Row from datetime import datetime rows = [ dict(number=1, letters='real1', some_date=datetime(2017,12,1,3,15)), dict(number=2, letters='real2', some_date=datetime(2017,12,2,3,15)), dict(number=3, letters='real3', some_date=datetime(2017,12,3,3,15)), ] rows_rdd = spark.sparkContext.parallelize(rows).map(lambda r: Row(**r)) df = spark.createDataFrame(rows_rdd) spark.sql('CREATE DATABASE test_trash') df.write.mode(saveMode='overwrite').saveAsTable('test_trash.thingy') schema = spark.sql('SELECT letters, number, some_date FROM test_trash.thingy').schema # A-works # schema = spark.sql('SELECT some_date, number, letters FROM test_trash.thingy').schema # B-fails schema = spark.sql('SELECT number, letters, some_date FROM test_trash.thingy').schema # C-works rows_rdd = spark.sparkContext.parallelize(rows).map(lambda r: Row(**r)) df = spark.createDataFrame(rows_rdd, schema) df.count() {code} If I uncomment line #A it works, line #B fails, and line #C works. The only difference is the ordering of the named fields. The behavior is inconsistent. Also, ``Row`` objects have named fields so why should there be any dependence upon ordering at all? Also, the errors don't really convey the problem ``AttributeError: 'int' object has no attribute 'tzinfo'`` - the error should be about some explicit schema mismatch (though I contend that this isn't really a mismatch at all; the above lines should all work). > Get rid of sorting in Row's constructor in pyspark > -------------------------------------------------- > > Key: SPARK-12467 > URL: https://issues.apache.org/jira/browse/SPARK-12467 > Project: Spark > Issue Type: Bug > Components: PySpark, SQL > Affects Versions: 1.5.2 > Reporter: Irakli Machabeli > Priority: Minor > > Current implementation of Row's __new__ sorts columns by name > First of all there is no obvious reason to sort, second, if one converts > dataframe to rdd and than back to dataframe, order of column changes. While > this is not a bug, nevetheless it makes looking at the data really > inconvenient. > def __new__(self, *args, **kwargs): > if args and kwargs: > raise ValueError("Can not use both args " > "and kwargs to create Row") > if args: > # create row class or objects > return tuple.__new__(self, args) > elif kwargs: > # create row objects > names = sorted(kwargs.keys()) # just get rid of sorting here!!! > row = tuple.__new__(self, [kwargs[n] for n in names]) > row.__fields__ = names > return row > else: > raise ValueError("No args or kwargs") -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org