[ https://issues.apache.org/jira/browse/SPARK-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Armbrust updated SPARK-6677: ------------------------------------ Assignee: Davies Liu > pyspark.sql nondeterministic issue with row fields > -------------------------------------------------- > > Key: SPARK-6677 > URL: https://issues.apache.org/jira/browse/SPARK-6677 > Project: Spark > Issue Type: Bug > Components: PySpark > Affects Versions: 1.3.0 > Environment: spark version: spark-1.3.0-bin-hadoop2.4 > python version: Python 2.7.6 > operating system: MacOS, x86_64 x86_64 x86_64 GNU/Linux > Reporter: Stefano Parmesan > Assignee: Davies Liu > Labels: pyspark, row, sql > > The following issue happens only when running pyspark in the python > interpreter, it works correctly with spark-submit. > Reading two json files containing objects with a different structure leads > sometimes to the definition of wrong Rows, where the fields of a file are > used for the other one. > I was able to write a sample code that reproduce this issue one out of three > times; the code snippet is available at the following link, together with > some (very simple) data samples: > https://gist.github.com/armisael/e08bb4567d0a11efe2db -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org