John Engelhart created SPARK-38653:
--------------------------------------
Summary: Repartition by Column that is Int not working properly
only particular numbers. (11, 33)
Key: SPARK-38653
URL: https://issues.apache.org/jira/browse/SPARK-38653
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.1.2
Environment: This was running on EMR 6.4.0 using Spark 3.1.2 in an EMR
Notebook writing to S3
Reporter: John Engelhart
My understanding is when you call .repartition(a column). For each unique key
in that field. The data will go to that partition. There should never be two
keys repartitioned the same part. That behavior is true with a String column.
That behavior is also true with an Int column except on certain numbers. In my
use case. The magic numbers 11 and 33.
{code:java}
//Int based column repartition
spark.sparkContext.parallelize(Seq(1, 11, 33)).toDF("collectionIndex").
repartition($"collectionIndex").write.mode("overwrite").parquet("path")
//Produces two part files
//String based column repartition
spark.sparkContext.parallelize(Seq("1", "11", "33")).toDF("collectionIndex").
repartition($"collectionIndex").write.mode("overwrite").parquet("path1")
//Produces three part files {code}
{code:java}
//Not working as expected
spark.read.parquet("path/part-00000...").distinct.show
spark.read.parquet("path/part-00001...").distinct.show
//Working as expected
spark.read.parquet("path1/part-00000...").distinct.show
spark.read.parquet("path1/part-00001...").distinct.show
spark.read.parquet("path1/part-00002...").distinct.show {code}
!image-2022-03-24-22-09-26-917.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
