[ https://issues.apache.org/jira/browse/SPARK-12980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christopher Bourez updated SPARK-12980: --------------------------------------- Description: I installed spark 1.6 on many different computers. On Windows, PySpark textfile method, followed by take(1), does not work on a file of 13M. If I set numpartitions to 2000 or take a smaller file, the method works well. The Pyspark is set with all RAM memory of the computer thanks to the command --conf spark.driver.memory=5g in local mode. On Mac OS, I'm able to launch the exact same program with Pyspark with 16G RAM for a file of much bigger in comparison, of 5G. Memory is correctly allocated, removed etc On Ubuntu, no trouble, I can also launch a cluster http://christopher5106.github.io/big/data/2016/01/19/computation-power-as-you-need-with-EMR-auto-termination-cluster-example-random-forest-python.html The error message on Windows is : java.net.SocketException: Connection reset by peer: socket write error What could be the reason to have the windows spark textfile method fail ? was: I installed spark 1.6 on many different computers. On Windows, PySpark textfile method, followed by take(1), does not work on a file of 13M. If I set numpartitions to 2000 or take a smaller file, the method works well. The Pyspark is set with all RAM memory of the computer thanks to the command --conf spark.driver.memory=5g in local mode. On Mac OS, I'm able to launch the exact same program with Pyspark with 16G RAM for a file of much bigger in comparison, of 5G. Memory is correctly allocated, removed etc On Ubuntu, no trouble, I can also launch a cluster http://christopher5106.github.io/big/data/2016/01/19/computation-power-as-you-need-with-EMR-auto-termination-cluster-example-random-forest-python.html What could be the reason to have the windows spark textfile method fail ? > pyspark crash for large dataset - clone > --------------------------------------- > > Key: SPARK-12980 > URL: https://issues.apache.org/jira/browse/SPARK-12980 > Project: Spark > Issue Type: Bug > Affects Versions: 1.5.2 > Environment: windows > Reporter: Christopher Bourez > > I installed spark 1.6 on many different computers. > On Windows, PySpark textfile method, followed by take(1), does not work on a > file of 13M. > If I set numpartitions to 2000 or take a smaller file, the method works well. > The Pyspark is set with all RAM memory of the computer thanks to the command > --conf spark.driver.memory=5g in local mode. > On Mac OS, I'm able to launch the exact same program with Pyspark with 16G > RAM for a file of much bigger in comparison, of 5G. Memory is correctly > allocated, removed etc > On Ubuntu, no trouble, I can also launch a cluster > http://christopher5106.github.io/big/data/2016/01/19/computation-power-as-you-need-with-EMR-auto-termination-cluster-example-random-forest-python.html > The error message on Windows is : java.net.SocketException: Connection reset > by peer: socket write error > What could be the reason to have the windows spark textfile method fail ? -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org