andygrove commented on issue #3882: URL: https://github.com/apache/datafusion-comet/issues/3882#issuecomment-5539820624
Closing this in favour of #5678. The framing here has been overtaken by what we measured afterwards. Of the two fixes I proposed in this thread, the second one — encoding the schema once per partition instead of once per block — turned out not to be worth doing: #2928 measured a ~25% size reduction for a single-column schema but only ~1% at 100 columns of mixed types, and I closed it. The per-block schema is a real cost, but it is not what makes these files big, except possibly on deeply nested schemas, which are tracked in #5355. What #5006 did land is pre-encoding the schema once per writer, which saves the CPU of re-serializing it but leaves the bytes on the wire. That leaves the first proposal — sizing shuffle blocks by bytes rather than rows — as the live hypothesis, and it has never actually been measured. #5678 restates the current behaviour accurately, records what is already known (including that the repro is very data-dependent, and that doubling the batch size took one workload from 50% larger / 10% slower to 8% larger / 15% faster), and lists what needs measuring before anyone changes the format. The note about re-enabling `WriteDistributionAndOrderingSuite` is carried over to the new issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
