cloud-fan edited a comment on pull request #28032: URL: https://github.com/apache/spark/pull/28032#issuecomment-849765244
AFAIK this is how Hive inserts partitioned tables. I don't think it's superior to Spark's behavior, but it does help to avoid small files. I'm +1 to adding this as an option for Spark users. A few points: 1. there are so many write commands (v1, v2, append, overwrite, etc.), shall we handle all of them in one rule? 2. even if the table is not partitioned, small files can still happen if the input RDD has many small partitions. Shall we add repartition to normal insertion as well so that AQE can coalesce partitions? cc @yaooqinn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
