OIiveirra commented on code in PR #69032:
URL: https://github.com/apache/doris/pull/69032#discussion_r4310471187
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/Ndv.java:
##########
@@ -63,6 +63,10 @@ private Ndv(AggregateFunctionParams functionParams) {
@Override
public void checkLegalityBeforeTypeCoercion() {
+ // The BE kernel has no spatial value-state dispatch; fail before
casts hide the input type.
+ if (getArgumentType(0).isSpatialType()) {
+ throw new AnalysisException("ndv does not support spatial
arguments");
Review Comment:
Fixed in
[6aad808131b](https://github.com/apache/doris/commit/6aad808131b466212d0e3318efe216b00e3c19aa).
Kept the explicit user NDV rejection. The shared ColumnStatistic policy now
rejects both spatial primitive kinds independently of CRS/algorithm, and
StatisticsUtil plus literal estimation use that policy. Existing
manual/automatic collection entry points therefore exclude these columns
without changing their SQL templates or ordinary-type behavior.
The three new FE reproductions fail before repair; all 44 focused FE tests
pass afterward, covering implicit/explicit ANALYZE selection, external
auto-analysis column pairs, parameterized types and literal estimation. The
complete real Iceberg suite passes generation and non-generation verification:
default, projected and property-referenced mixed tables complete FULL/SYNC
ANALYZE and show statistics only for id. Direct spatial NDV still fails
explicitly.
##########
be/src/format/transformer/viceberg_parquet_writer.cpp:
##########
@@ -65,6 +193,28 @@ VIcebergParquetWriter::VIcebergParquetWriter(RuntimeState*
state, io::FileWriter
_iceberg_schema_json(iceberg_schema_json == nullptr ? "" :
*iceberg_schema_json),
_nan_count_field_ids(nan_count_field_ids) {}
+arrow::Status VIcebergParquetWriter::_open_file_writer() {
+ bool has_spatial_column = false;
+ for (const auto& column : _iceberg_schema.columns()) {
+ has_spatial_column = has_spatial_column ||
contains_spatial_type(column.field_type());
+ }
+ if (!has_spatial_column) {
+ return VParquetWriter::_open_file_writer();
+ }
+
+ ARROW_ASSIGN_OR_RAISE(
+ auto parquet_schema,
+ make_iceberg_parquet_schema(_arrow_convertor()->arrow_schema(),
_iceberg_schema,
Review Comment:
Fixed in
[424f6edc8c7](https://github.com/apache/doris/commit/424f6edc8c7e1b463689197dfa10a68154456275).
Failed open now destroys the transformer/Parquet stream, synchronously
finishes a FileWriter that is not CLOSED, and destroys that writer before the
final object deletion. This also waits for pending uploads after a close error.
The original open error is retained; successful writes and generic S3/Parquet
interfaces are unchanged.
The old isolated binaries reproduce two physical zero-byte files under
id=1/id=2. Both new BE reproductions fail before repair; all 20 related ASAN BE
tests pass afterward. The complete Iceberg suite passes generation and
non-generation verification, comparing fully paginated physical key/size
snapshots for both spatial kinds, unpartitioned/dynamic targets and repeated
failures. No new objects remain, target rows/committed files stay zero, and
ordinary dynamic writes succeed. The 33 previous golden sections are unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
