Ankush Kankariya created SPARK-33161:
----------------------------------------
Summary: Spark 3: Partition count changing on dataframe select cols
Key: SPARK-33161
URL: https://issues.apache.org/jira/browse/SPARK-33161
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.0.1, 3.0.0
Reporter: Ankush Kankariya
I am noticing a difference in behaviour on upgrading to spark 3 where the
NumPartitions are changing on df.select which causing my zip operations to fail
on mismatch. With spark 2.4.4 it works fine. This does not happen with filter
but only with select cols
{code:java}
spark = SparkSession.builder.appName("pytest-pyspark-local-testing"). \
master("local[5]"). \ config("spark.executor.memory", "2g"). \
config("spark.driver.memory", "2g"). \ config("spark.ui.showConsoleProgress",
"false"). \ config("spark.sql.shuffle.partitions",10). \
config("spark.sql.optimizer.dynamicPartitionPruning.enabled","false").getOrCreate()
{code}
With Spark 2.4.4:
df = spark.table("tableA")
print(df.rdd.getNumPartitions()) #10
new_df = df.filter("id is not null")
print(new_df.rdd.getNumPartitions()) #10
new_2_df = df.select("id")
print(new_2_df.rdd.getNumPartitions()) #10
With Spark 3.0.0:
df = spark.table("tableA")
print(df.rdd.getNumPartitions()) #10
new_df = df.filter("id is not null")
print(new_df.rdd.getNumPartitions()) #10
new_2_df = df.select("id")
print(new_2_df.rdd.getNumPartitions()) #1
See the last line where it changes to 1 partition from initial 10. Any thoughts?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
