Gabriel39 commented on code in PR #67921:
URL: https://github.com/apache/doris/pull/67921#discussion_r4002352714
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceTypeConverter.java:
##########
@@ -135,23 +130,23 @@ private static Type toDorisType(Field field, boolean
allowNull) {
case LargeList:
case FixedSizeList:
Review Comment:
Fixed in 027201d093. Recursive conversion now propagates the explicit
UNSUPPORTED sentinel instead of applying generic isSupported() to successfully
converted composites. Added FE coverage for lists, large/fixed lists, structs,
and maps containing list<null> or map<string,null>, plus deeper BE schema
coverage. The new FE tests failed before the fix; all 12 focused FE tests now
pass.
##########
be/src/format_v2/lance/lance_reader_helper.cpp:
##########
@@ -237,16 +234,16 @@ Status arrow_field_to_doris_type(const
std::shared_ptr<arrow::Field>& field,
case arrow::Type::FIXED_SIZE_LIST: {
const auto list =
std::static_pointer_cast<arrow::BaseListType>(arrow_type);
DataTypePtr value_type;
- RETURN_IF_ERROR(arrow_field_to_doris_type(list->value_field(),
&value_type, false));
+ RETURN_IF_ERROR(arrow_field_to_doris_type(list->value_field(),
&value_type));
*doris_type =
make_nullable(std::make_shared<DataTypeArray>(value_type));
return Status::OK();
}
case arrow::Type::MAP: {
const auto map = std::static_pointer_cast<arrow::MapType>(arrow_type);
DataTypePtr key_type;
DataTypePtr item_type;
Review Comment:
Fixed in 027201d093. Both FE and BE schema conversion reject non-nullable
Arrow Null leaves before physical SerDe selection. Coverage includes Null map
keys, required Null map values, list variants, structs, and deeper nesting. The
new schema rejection tests failed before the fix. All 10 focused BE tests and
12 focused FE tests pass; affected C++ sources also pass syntax checks against
current headers.
##########
docker/thirdparties/docker-compose/iceberg/scripts/lance_build_nested_null.py:
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Generate the nested Null regression fixture with pylance 7.0.0 and PyArrow
21.0.0."""
+
+import argparse
+from pathlib import Path
+
+import lance
+import pyarrow as pa
+
+
+def build(output: Path) -> None:
+ null_struct = pa.struct([("empty", pa.null()), ("value", pa.int32())])
+ schema = pa.schema([
+ ("id", pa.int32()),
+ ("null_list", pa.list_(pa.null())),
+ ("null_large_list", pa.large_list(pa.null())),
+ ("null_fixed_list", pa.list_(pa.null(), 2)),
+ ("null_struct", null_struct),
+ ("nested_list", pa.list_(null_struct)),
+ ("null_map", pa.map_(pa.string(), pa.null())),
+ ])
+ rows = [
+ {"id": 1, "null_list": [None, None], "null_large_list": [None],
+ "null_fixed_list": [None, None], "null_struct": {"empty": None,
"value": 10},
+ "nested_list": [{"empty": None, "value": 11}, {"empty": None,
"value": 12}],
+ "null_map": [("a", None), ("b", None)]},
+ {"id": 2, "null_list": None, "null_large_list": None,
"null_fixed_list": None,
+ "null_struct": {"empty": None, "value": 20}, "nested_list": None,
"null_map": None},
+ {"id": 3, "null_list": [], "null_large_list": [],
+ "null_fixed_list": [None, None], "null_struct": {"empty": None,
"value": 30},
+ "nested_list": [], "null_map": []},
+ {"id": 4, "null_list": [None], "null_large_list": [None, None, None],
+ "null_fixed_list": [None, None], "null_struct": {"empty": None,
"value": 40},
+ "nested_list": [{"empty": None, "value": 41}], "null_map": [("c",
None)]},
+ ]
+ # pylance 7 cannot encode a null parent struct with a Null child; BE tests
cover that shape.
+ table = pa.Table.from_pylist(rows, schema=schema)
+ dataset = lance.write_dataset(table, str(output),
data_storage_version="2.2")
Review Comment:
Fixed in 027201d093. The canonical builder now regenerates nested_null.lance
in staging, and check_catalog validates its persisted schema and values before
and after promotion. Also corrected a pre-existing missing-argument call in
build_multi_frag that prevented the canonical rebuild from running. Verified a
full temporary rebuild with --repin, the committed catalog with --check, and
rejection when the nested-null fixture is removed. No committed binary fixtures
were regenerated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
