JingsongLi commented on PR #10429: URL: https://github.com/apache/paimon/pull/10429#issuecomment-6054916847
**[P2] Preserve block order when dropping columns after the global score sort** In [`search_result.py:74–77`](https://github.com/apache/paimon/blob/e5e81fda6f9dc47d3b55cc4f3d46c69b1620a472/paimon-python/pypaimon/ray/search_result.py#L74-L77), the sorted Dataset is followed by `drop_columns` calls. With Ray 2.59.0, `preserve_order` defaults to `False`, and `Sort` does not automatically enable it. Since `finish_batch` leaves the inferred output schema unknown, these calls run as parallel `MapBatches` stages and emit blocks in task completion order. A slow earlier partition can therefore place lower-scoring rows before higher-scoring rows, violating `order_by_score()` and potentially causing `take(1)` to return a candidate other than the highest-scoring one. This also affects batch queries using the same helper. I reproduced this using a real table with 24 rows across four files, with row `i` containing vector `[float(i), 0.0]`: ```python ds = (table.search([0.0, 0.0]).select(["id"]) .with_score().order_by_score().limit(24) .to_ray(execution="local", override_num_blocks=4)) ``` Adding a one-second delay to the first sorted partition's column-dropping task, without changing its data or projection, produced IDs `[5, ..., 23, 0, ..., 4]` instead of `[0, ..., 23]`. The same reproduction with `preserve_order=True` returned the expected order. The unmodified execution plan also confirms that both column-dropping stages occur after `Sort`. Please explicitly preserve order for this Dataset when score ordering is requested, without changing the global DataContext, and add a regression test with multiple blocks and a slow first partition. The 11 new tests pass with the default configuration, so their current cases do not expose this scheduling-dependent failure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
