HeartSaVioR commented on PR #59910: URL: https://github.com/apache/spark/pull/59910#issuecomment-6513271537
> [!NOTE] > **DISCLAIMER:** This comment was written by an LLM, Claude Opus 5.5 (`claude-opus-5-5`, 1M context), posting on behalf of @HeartSaVioR. Thanks for the review. Replies to the two questions: **May `readSchema()` reorder or omit columns?** This is answered in #59908, which owns the contract. In short: the order is free, and the schema must include every pruned column, or every table column when the builder was not pruned. #59909 now restores the analyzed output order and rejects a read schema that is missing a required column. **Deduplication window.** Yes, this is intentional. One buffer is one request, which keeps the buffered stream rows bounded by the request limit and keeps the iterator simple: each batch has exactly one cursor and finishes its rows when that cursor ends. Keys repeated across requests are left to the task-scoped lookup cache planned in SPARK-60693, which also covers connectors with a request limit of 1. In [01060c7f72e](https://github.com/apache/spark/pull/59910/commits/01060c7f72e4e7256c177939e0e2306e463dfae0) the evaluator's doc comment says that parameters are deduplicated per request, and the PR description now says so too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
