Stefano Parmesan created SPARK-6677:
---------------------------------------
Summary: pyspark.sql nondeterministic issue with row fields
Key: SPARK-6677
URL: https://issues.apache.org/jira/browse/SPARK-6677
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 1.3.0
Environment: spark version: spark-1.3.0-bin-hadoop2.4
python version: Python 2.7.6
operating system: MacOS, x86_64 x86_64 x86_64 GNU/Linux
Reporter: Stefano Parmesan
The following issue happens only when running pyspark in the python
interpreter, it works correctly with spark-submit.
Reading two json files containing objects with a different structure leads
sometimes to the definition of wrong Rows, where the fields of a file are used
for the other one.
I was able to write a sample code that reproduce this issue one out of three
times; the code snippet is available at the following link, together with some
(very simple) data samples:
https://gist.github.com/armisael/e08bb4567d0a11efe2db
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
