ulysses-you commented on issue #1299: URL: https://github.com/apache/incubator-kyuubi/issues/1299#issuecomment-952682819
thank you @melin . Kyuubi has already supported merge small file during writing by adding an extra shuffle, can see [here](https://github.com/apache/incubator-kyuubi/blob/master/dev/kyuubi-extension-spark-common/src/main/scala/org/apache/kyuubi/sql/RepartitionBeforeWritingBase.scala). So I believe it's easy to do the similar things using Kyuubi by: ```sql INSERT OVERWRITE TABLE $table AS SELECT * FROM $table; ``` For this case, we probably can enhance the existed rule to support `fileCount` and `maxRecordsPerFile` (which is this issue want to do) by using `parquet` or `orc` footer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
