iemejia commented on PR #55932: URL: https://github.com/apache/spark/pull/55932#issuecomment-5462451756
You're right on both counts - thanks for pinning this down. I have updated the PR to lead with the fix rather than the speedup: - **User-facing change is now "Yes"** with the before/after spelled out: on a page whose decoded lengths exceed the data region, `master`'s per-value loop does not fail - `in.skip` returns `-1` once the stream is dry, `remaining` grows by one per iteration, and it exits only when the `int` wraps at 2^31 (~2^31 non-interruptible iterations per short value), so the query used to complete with correct rows after a long unkillable stall. With `skipFully` the same page throws `ParquetDecodingException`; under `ignoreCorruptFiles=true`, `FileScanRDD` skips the rest of the file and the query returns fewer rows. - **Reframed the spin** so it no longer reads as though `master` were fine: the description now states plainly that `master` already spins today and only exits via the `int` wrap, and files only the negative-length drift and the summed-`int` overflow under "issues a naive `long` bulk form would introduce". The title now leads with the fix. On classification: agreed this should reach the maintenance branches, so I intend to reclassify SPARK-56907 from a sub-task of the SPARK-57415 performance umbrella to a standalone **Bug**. The buggy `skipBinary` loop is present as far back as `branch-3.4`, so it is broadly backportable. Do you have a preference on the target branches (e.g. 3.5 / 4.0 and the active 4.x lines), or should I set the affected versions and let the backport follow the usual policy? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
