after joining
two files with one partition each. It would either keep it at one or expand
it to two.
Why do DataFrames expand out the partitions so much?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-DataFrames-uses-too-many-partition-tp24214.html
Sent
the partitions so much?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-DataFrames-uses-too-many-partition-tp24214.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
[mailto:alasdair.mcbr...@gmail.com]
Sent: Tuesday, August 11, 2015 11:31 PM
To: user@spark.apache.org
Subject: Spark DataFrames uses too many partition
I am using DataFrames with Spark 1.4.1. I really like DataFrames but the
partitioning makes no sense to me.
I am loading lots of very small files
this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-DataFrames-uses-too-many-partition-tp24214p24223.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user
-DataFrames-uses-too-many-partition-tp24214.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h
/browse/SPARK-4630)
Hao
-Original Message-
From: Al M [mailto:alasdair.mcbr...@gmail.com]
Sent: Tuesday, August 11, 2015 11:31 PM
To: user@spark.apache.org
Subject: Spark DataFrames uses too many partition
I am using DataFrames with Spark 1.4.1. I really like DataFrames
