taozhiyi13 opened a new issue, #12737: URL: https://github.com/apache/seatunnel/issues/12737
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened `LanceSinkWriter.flushBatch()` buffers rows, then calls `FragmentConverter.reconvert()` once per row. Each call builds a one-row `VectorSchemaRoot` and runs `Fragment.create()`, and Lance writes one `.lance` file per `Fragment.create()`. The following `Append.commit()` only publishes fragment metadata and does not merge the files. Buffering therefore batches the commit, not the data files, so a flush of N rows produces N fragments. Writing 10,000 rows on current `dev` produces 10,000 fragments and 10,000 data files. ### SeaTunnel Version `dev` (`c7304ace6`) ### SeaTunnel Config ```conf sink { Lance { dataset_path = "/tmp/lance_sink_table" namespace_type = "dir" namespace_id = "root" table = "lance_sink_table" lance.write.mode = "APPEND" } } `dev` has no `lance.write.batch-size`. The writer hard-codes a buffer of 1000 rows, and each of those rows still becomes its own fragment. ``` ### Running Command ```shell Run any batch job that writes rows to the Lance sink, then count `data/*.lance` files or `Dataset.getFragments()` on the output dataset. ``` ### Error Exception ```log No exception. The job succeeds and writes one fragment per row. ``` ### Zeta or Flink or Spark Version Zeta ### Java or Scala Version Java 17 ### Screenshots _No response_ ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
