andygrove opened a new issue, #197:
URL: https://github.com/apache/datafusion-spark/issues/197
Part of #177 and apache/datafusion-comet#7300.
### What the function does
`lpad(str, len[, pad])` returns `str` left-padded with `pad` to a length of
`len`. If `str` is longer than `len`, it is truncated to its first `len`
characters (or bytes). Available since Spark 1.5.0.
`lpad` is registered through `LPadExpressionBuilder`, which picks one of two
implementations at analysis time:
- **String form** (`StringLPad`): `str` STRING, `len` INT, `pad` STRING
(default `' '`). These are `ImplicitCastInputTypes`, so numeric or other
`str`/`pad` values are cast to STRING and `len` is cast to INT. The return type
is `str`'s type, collation included. Length is counted in characters, where a
character is one UTF-8 code point, so a surrogate pair (4-byte UTF-8) counts as
one character.
- **Binary form** (`BinaryPad`, a `StaticInvoke` of `ByteArray.lpad`): used
when `str` is BINARY and either there is no `pad` or `pad` is also BINARY. The
default `pad` is `X'00'`. Length is counted in bytes and the result is BINARY.
This form is used only when `spark.sql.legacy.lpadRpadAlwaysReturnString` is
`false` (internal, default `false`, added in 3.3.0). When it is `true`, binary
input goes through the string form.
The function is null-intolerant: a NULL in any argument returns NULL.
### Examples
```sql
SELECT lpad('hi', 5, '??');
-- ???hi
SELECT lpad('hi', 1, '??');
-- h
SELECT lpad('hi', 5);
-- hi
SELECT lpad('hi', 5, 'xy');
-- xyxhi
SELECT lpad('hi', -5, 'xy');
-- (empty string)
SELECT lpad('hi', 5, '');
-- hi
SELECT lpad('hello', 2);
-- he
SELECT hex(lpad(unhex('aabb'), 5));
-- 000000AABB
SELECT hex(lpad(unhex('aabb'), 5, unhex('1122')));
-- 112211AABB
SELECT hex(lpad(unhex('aabbcc'), 2));
-- AABB
SELECT hex(lpad(unhex('aa'), 6, unhex('1f2e')));
-- 1F2E1F2E1FAA
SELECT hex(lpad(unhex('aabbcc'), 6, unhex('')));
-- AABBCC
SELECT lpad('abc', 5, x'57'); -- mixed types use the string form
-- WWabc
SELECT lpad(x'57', 5, 'abc');
-- abcaW
```
Unicode cases from `UTF8StringSuite` (calls to `UTF8String.lpad`):
```text
lpad('数据砖头', 3, '????') -> 数据砖
lpad('数据砖头', 6, '孙行者') -> 孙行数据砖头
lpad('数据砖头', 12, '孙行者') -> 孙行者孙行者孙行数据砖头
lpad('数据砖头', -10, '') -> ''
lpad('', 7, '?????') -> ???????
```
### Edge cases to test
- NULL `str`, NULL `len`, or NULL `pad` returns NULL, for both the string
and the binary form.
- Strings with `len <= 0` return `''`. When `len` is less than the input
length, the input is truncated to its first `len` characters (truncation keeps
the left end, even though this is *left* padding).
- An empty `pad` never pads. The result is `substring(str, 0, len)`: the
input unchanged if it is shorter than `len`, otherwise truncated.
- A multi-character `pad` is repeated and the last copy is cut short,
counting characters (`'xy'` padding to 5 gives `xyxhi`).
- Multi-byte and supplementary characters: `len` and the pad slicing count
code points, not bytes or graphemes. Combining sequences (e.g. `e` + U+0301)
count as 2.
- Binary form:
- With `len = 0` it returns empty bytes.
- With `len < 0` Spark's `ByteArray.lpad` does `new byte[len]` and throws
a raw `java.lang.NegativeArraySizeException`. This applies with an empty pad
too (`padWithEmptyPattern` uses `min(bytes.length, len)`). There is no error
condition for this; the implementation needs to decide whether to mirror it
with an execution error. This was derived from reading the code; Spark has no
test for it.
- With an empty pad it returns the first `min(len, bytes.length)` bytes.
- The default pad is the zero byte.
- Type dispatch: `lpad(binary, n, string)` and `lpad(string, n, binary)`
both use the string form and return STRING. Only (binary, n) and (binary, n,
binary) return BINARY.
- `len` comes in as INT after implicit cast. `lpad('hi', 'invalid_length')`
fails with `CAST_INVALID_INPUT` under ANSI mode (the default since 4.0) and
returns NULL with `spark.sql.ansi.enabled=false`. The cast itself is handled by
the analyzer, but the kernel must accept Int32.
- Very large results: the string form computes the result size with
`Math.toIntExact` and throws `ArithmeticException` if it is over 2^31-1 bytes.
- Scalar and array combinations for each argument (column `str` with literal
`pad`, literal `str` with column `pad`, column `len`).
- Collations (4.0+): any collation is accepted
(`StringTypeWithCollation(supportsTrimCollation = true)`), but padding ignores
collation. The result keeps the collation of `str`. Mixing two different
explicit collations is an analysis error.
### Differences across Spark versions
- 3.4 / 3.5: semantics as above. Binary-aware `lpad` already exists (added
in 3.3.0).
- 4.0: collation support added (`StringTypeWithCollation`). ANSI mode is on
by default, which affects how a non-numeric `len` is cast. Padding semantics
are unchanged for `UTF8_BINARY`.
- 4.1 / 4.2: none found.
### Implementation notes
- Spark source:
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala`
(`LPadExpressionBuilder`, `PadExpressionBuilderBase`, `StringLPad`,
`BinaryPad`). Kernels are in
`common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java`
(`lpad`) and
`common/unsafe/src/main/java/org/apache/spark/unsafe/types/ByteArray.java`
(`lpad`, `padWithEmptyPattern`, `fillWithPattern`).
- DataFusion core has `lpad` (`datafusion/functions/src/unicode/lpad.rs`).
Since apache/datafusion#21405 (DataFusion 54) it counts code points, not
graphemes. Its string semantics are close to Spark's:
- negative length becomes 0;
- it truncates to the first `len` characters;
- an empty fill returns the input unchanged.
Remaining differences:
- It has no BINARY support at all.
- `len` is Int64, and it errors when `len > i32::MAX`. That can't happen
in Spark because `len` is INT.
A Spark wrapper could coerce `len` to Int32 semantics and delegate the
string path to core, but it needs its own binary kernel.
- Belongs in `crates/spark/src/function/string/` (e.g. a shared `pad.rs`
with `rpad`). The binary kernel should share code with `rpad`.
- Tests: there is already a placeholder
`crates/sqllogictest/test_files/string/lpad.slt` with commented-out queries.
Enable those and add the binary, negative-length, empty-pad and Unicode cases
above. Document with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]