I am using spark-csv to save files in s3, it shown Size exceeds.
Please let me know how to fix it. Thanks.
df.write()
.format("com.databricks.spark.csv")
.option("header", "true")
.save("s3://newcars.csv");
java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE
at
big it would be, I assume it’s over 2 GB
From: Zhang, Jingyu [mailto:jingyu.zh...@news.com.au]
Sent: 16 November 2015 10:17
To: user <user@spark.apache.org>
Subject: Size exceeds Integer.MAX_VALUE on EMR 4.0.0 Spark 1.4.1
I am using spark-csv to save files in s3, it shown Size exceeds. Plea
