cloud-fan commented on code in PR #58530:
URL: https://github.com/apache/spark/pull/58530#discussion_r3937515176
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlStatementSplitter.scala:
##########
@@ -58,6 +58,26 @@ case class SqlStatementSplitResult(
def isEmpty: Boolean = completeStatements.isEmpty && partialStatement.isEmpty
}
+/** A split SQL statement together with its 0-based start in the original
input. */
+private[sql] case class PositionedSqlStatement(
+ statement: String,
+ terminator: String,
+ start: Int) {
+ def length: Int = statement.length
Review Comment:
**Blocking (P1):** `Token.getStartIndex`/`getStopIndex` come from ANTLR's
`CodePointCharStream`, so they count Unicode code points, while both
`String.length` here and `sqlText.substring` in `tryParseRegion` use UTF-16
code units. For example, with `SELECT '😀😀'; SELECT 2;`, the first delimiter's
token offset makes `tryParseRegion` cut the candidate before the closing quote;
the splitter can then pass the whole batch to `parse_sql` as one partial
segment and return one parse error instead of two successful entries. Even one
supplementary character makes a later statement's `start` incompatible with
`length`. Please convert token start/stop offsets with `offsetByCodePoints`
before slicing or storing them, keep `start` and `length` in that same UTF-16
coordinate system, and add a regression that recovers both statements from the
reported spans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
