voonhous commented on code in PR #20638:
URL: https://github.com/apache/hudi/pull/20638#discussion_r4572870597
##########
hudi-trino/src/main/java/io/trino/plugin/hudi/util/HudiAvroSerializer.java:
##########
@@ -458,7 +459,7 @@ else if (value instanceof SqlVarbinary sqlVarbinary) {
format("Unhandled type for %s: %s | value type: %s",
type.getJavaType().getSimpleName(), type, value.getClass().getName()));
}
}
- else if (type instanceof CharType) {
+ else if (type instanceof CharType charType) {
Review Comment:
**minor:** The VARCHAR branch above (lines 439-448) has the same gap for
bounded `varchar(n)`: it writes the value without truncating, while Trino's
Parquet reader truncates (`BoundedVarcharPlainValueDecoder` calls
`Varchars.byteCount`). Native Parquet log files are already truncated by that
decoder, so as far as I can tell this only shows with block-format (`#HUDI#`)
logs on older table versions. Not blocking; could we also apply
`Varchars.truncateToLength(slice, varcharType)` there when
`!varcharType.isUnbounded()`?
##########
hudi-trino/src/test/java/io/trino/plugin/hudi/util/TestHudiAvroCharSerialization.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.trino.plugin.hudi.util;
+
+import io.trino.spi.block.BlockBuilder;
+import io.trino.spi.type.CharType;
+import org.apache.avro.util.Utf8;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.List;
+import java.util.stream.Stream;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+public class TestHudiAvroCharSerialization
Review Comment:
**minor:** `TestHudiAvroSerializer` in this package already tests
`HudiAvroSerializer.appendTo` with the same
`@ParameterizedTest`/`@MethodSource` setup
(`testAppendShortDecimalFromAvroFixed`). Not blocking; could these cases move
there as a package-private `testAppendChar`? That would also fix the name,
since `testCharPreservesLeadingSpaces` covers truncation and tabs too.
##########
hudi-trino/src/main/java/io/trino/plugin/hudi/util/HudiAvroSerializer.java:
##########
@@ -472,7 +473,7 @@ else if (value instanceof String) {
}
// IMPORTANT: Char types may be padded with trailing "space"
characters to make up for length if the contents are lesser than defined length.
// Need to trim out trailing spaces as Slice representing Char
should not have trailing spaces
Review Comment:
**nit:** The comment still only mentions trimming trailing spaces; the code
now also truncates to the declared length and strips only U+0020. Feel free to
ignore.
```suggestion
// Match Trino's Parquet CHAR reader: truncate to the declared
length in code points, then strip
// trailing U+0020 padding only, keeping leading whitespace and
trailing control characters
```
##########
hudi-trino/src/test/java/io/trino/plugin/hudi/util/TestHudiAvroCharSerialization.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.trino.plugin.hudi.util;
+
+import io.trino.spi.block.BlockBuilder;
+import io.trino.spi.type.CharType;
+import org.apache.avro.util.Utf8;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.List;
+import java.util.stream.Stream;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+public class TestHudiAvroCharSerialization
+{
+ @ParameterizedTest
+ @MethodSource("charValues")
+ public void testCharPreservesLeadingSpaces(String input, String expected)
+ {
+ CharType type = CharType.createCharType(5);
+ for (Object value : List.of(input, new Utf8(input))) {
+ BlockBuilder builder = type.createBlockBuilder(null, 1);
+ HudiAvroSerializer.appendTo(type, value, builder);
Review Comment:
**minor:** This calls `appendTo` directly, so nothing checks
base-vs-snapshot parity, which is what the fix is about. The one existing CHAR
MOR check (`TestHudiSmokeTest.java:1196`, `'charval1'`) passes on both old and
new code. #20659 has three tests in `TestHudiMorMergeModeSemantics` on a
`char(5)` column added to `CommitTimeOrderingHudiTablesInitializer`; they fail
on master and pass with this PR. Not blocking; could we add them here?
##########
hudi-trino/src/test/java/io/trino/plugin/hudi/util/TestHudiAvroCharSerialization.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.trino.plugin.hudi.util;
+
+import io.trino.spi.block.BlockBuilder;
+import io.trino.spi.type.CharType;
+import org.apache.avro.util.Utf8;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.List;
+import java.util.stream.Stream;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+public class TestHudiAvroCharSerialization
+{
+ @ParameterizedTest
+ @MethodSource("charValues")
+ public void testCharPreservesLeadingSpaces(String input, String expected)
+ {
+ CharType type = CharType.createCharType(5);
+ for (Object value : List.of(input, new Utf8(input))) {
+ BlockBuilder builder = type.createBlockBuilder(null, 1);
+ HudiAvroSerializer.appendTo(type, value, builder);
+ assertThat(type.getSlice(builder.build(),
0).toStringUtf8()).isEqualTo(expected);
+ }
+ }
+
+ private static Stream<Arguments> charValues()
+ {
+ return Stream.of(
+ Arguments.of(" ab ", " ab"),
+ Arguments.of(" ", ""),
+ Arguments.of("\tab\t ", "\tab\t"),
+ Arguments.of(" a😀 ", " a😀"),
Review Comment:
**nit:** `" "` on line 46 passes on the old `trim()` as well, and this
emoji case fits within length, so code-point truncation is not tested. Feel
free to ignore; an overlength emoji case covers it (passes locally with this
PR):
```suggestion
Arguments.of(" a\uD83D\uDE00\uD83D\uDE00\uD83D\uDE00x", "
a\uD83D\uDE00\uD83D\uDE00\uD83D\uDE00"),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]