cloud-fan commented on code in PR #58530:
URL: https://github.com/apache/spark/pull/58530#discussion_r3940933503
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlStatementSplitter.scala:
##########
@@ -158,6 +190,36 @@ object SqlStatementSplitter {
// interpretation (e.g. `double_quoted_identifiers`).
val conf = SqlApiConf.get
+ def appendToken(token: Token): Unit = {
+ if (buffer.isEmpty) {
+ // CodePointCharStream token offsets count Unicode code points, while
+ // String offsets and lengths count UTF-16 code units.
+ bufferStart = sqlText.offsetByCodePoints(0, token.getStartIndex)
Review Comment:
**Non-blocking (P2):** This conversion starts at index 0 for every
statement, and `tryParseRegion` performs the same from-zero conversion for
candidate boundaries. A batch of many short statements therefore rescans
progressively larger prefixes, making ordinary splitting quadratic despite the
documented O(n) contract. Please build the code-point-to-UTF-16 boundary
mapping once, or maintain equivalent incremental offsets, so token-boundary
lookup is constant time.
##########
sql/core/src/main/scala/org/apache/spark/sql/catalyst/expressions/ParseSql.scala:
##########
@@ -27,34 +27,37 @@ import org.apache.spark.sql.types.{AbstractDataType,
DataType, StringType}
import org.apache.spark.unsafe.types.UTF8String
/**
- * Parses a SQL statement string and returns a compact JSON description of the
- * unresolved statement (identifier/code, lineage references, select-list
names,
- * parameters), or a STANDARD-format error object when the statement does not
- * parse.
+ * Parses a SQL batch string and returns a compact JSON array describing its
+ * unresolved statements (source position, identifier/code, lineage references,
+ * select-list names, parameters). A statement that does not parse is
represented
+ * by a STANDARD-format error object at its position in the array.
*
* Behind [[SQLConf.PARSE_SQL_ENABLED]] while the JSON contract is still
* evolving. Designed for batch evaluation over DataFrames of SQL text.
* User-facing parse errors become JSON; unexpected internal failures
propagate.
*/
// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = """_FUNC_(sqlStmt) - Parses `sqlStmt` with the stock Spark SQL
parser and
- returns a JSON string describing the statement (parse success, Table 39
statement
- identifier/code, target and source table references for lineage,
select-list column
- names, and parameter markers). Session parser extensions are not applied.
+ usage = """_FUNC_(sqlStmt) - Splits `sqlStmt` into SQL statements, parses
each with
+ the stock Spark SQL parser, and returns a JSON array describing them
(1-based start
Review Comment:
**Non-blocking (P2):** The returned `start` and `length` values are UTF-16
code-unit coordinates, but this public description does not name that unit. For
`SELECT '😀😀'; SELECT 2;`, the API reports the first length as 13 and the second
start as 16, while code-point-indexed clients reasonably expect 11 and 14 and
will slice the wrong text. Please state explicitly that `start` is a 1-based
UTF-16 code-unit offset and `length` is a UTF-16 code-unit count, and include a
non-BMP example so clients can interpret the fields reliably.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlStatementSplitter.scala:
##########
@@ -158,6 +190,36 @@ object SqlStatementSplitter {
// interpretation (e.g. `double_quoted_identifiers`).
val conf = SqlApiConf.get
+ def appendToken(token: Token): Unit = {
+ if (buffer.isEmpty) {
+ // CodePointCharStream token offsets count Unicode code points, while
+ // String offsets and lengths count UTF-16 code units.
+ bufferStart = sqlText.offsetByCodePoints(0, token.getStartIndex)
+ }
+ buffer.append(token.getText)
+ }
+
+ def resetBuffer(): Unit = {
+ buffer.setLength(0)
+ bufferStart = -1
+ bufferHasContent = false
+ }
+
+ def positionedStatement(terminator: String):
Option[PositionedSqlStatement] = {
+ val raw = buffer.toString
+ val statement = raw.trim
Review Comment:
**Non-blocking (P2):** `String.trim` only removes characters up to U+0020,
but the Spark SQL lexer also treats Unicode spaces such as U+00A0 as
whitespace. Consequently, `parse_sql("\u00a0SELECT 1\u00a0;")` retains both
NBSPs and reports start 1/length 10 instead of the promised trimmed span at
start 2/length 8. Please trim with the lexer's whitespace semantics and derive
the leading UTF-16 offset from the same boundary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
