suxiaogang223 opened a new pull request, #69524:
URL: https://github.com/apache/doris/pull/69524
### What problem does this PR solve?
Issue Number: None
Related PR: #65548
Problem Summary:
`COUNT(*)` / `COUNT(1)` can return zero for a nonempty Hive JsonSerDe
TEXTFILE table when column pruning retains only a partition column. An
all-STRING data schema with a STRING partition key triggers this shape because
the partition key is exposed as a narrower VARCHAR slot.
The two scanner implementations lose cardinality at different boundaries:
- V1 JSON still parses real records under a COUNT plan, but the inherited
post-read hook skips partition/default filling for COUNT. No file column
carries the parsed rows into the output block. A zero-file-column scan also
uses the unchanged block row count as its batch limit.
- V2 maps the partition slot to a constant and requests no physical file
columns. JSON derives its returned row count from the zero-column file block,
so TableReader sees zero rows and never materializes the partition constant.
The fix makes V1 JSON run its normal partition/default fill for parsed
records, including COUNT plans, and bounds zero-file-column batches by parsed
records. V2 keeps valid record cardinality independently of the physical block,
including array elements and ignore-malformed null rows, and rolls that
cardinality back together with column data after a parse error. TableReader
then uses its existing partition/default materialization path. No planner,
metadata-count shortcut, or wire contract is changed.
### Release note
Fix incorrect zero counts and missing rows when querying Hive JSON tables
with only partition columns projected, including external COUNT pushdown with
file scanners V1 and V2.
### Check List (For Author)
- Test: Unit Test / Regression test added; **not run locally**, as
requested. No local compilation, BE/FE unit execution, or regression execution
was performed.
- Four parameterized BE cases exercise V1/V2 with COUNT enabled/disabled,
an empty physical projection, partition filling, source NULL, blank records, a
final record without a newline, two-row batches, and EOF. The V2 cases run
through TableReader.
- Additional JSON V2 coverage compares empty and nonempty projections for
arrays, NULL, empty arrays, strict malformed input, and ignore-malformed row
cardinality.
- `test_hive_json_partition_count` prepares Hive 2/3 JsonSerDe partitioned
tables and compares Doris against live Hive results across scanner V1/V2 and
COUNT pushdown on/off. It covers COUNT star/one/nullable data/nullable
partition, partition and file predicates, empty results, grouping,
partition-only projection, and multiple batches. The test uses a runtime Hive
oracle; no result snapshot was handwritten or generated locally.
- Modified C++ files formatted with the repository clang-format v16 script
scoped to those files. Build header hygiene and `git diff --check` passed;
these are static checks, not test execution.
- Behavior changed: Yes. Nonempty partition-only JSON scans preserve their
row cardinality rather than silently emitting no rows.
- Does this need documentation: No.
Review notes:
- Correctness and scope: both scanner paths are covered; real record
parsing, NULL handling, filtering, and strict/ignore-malformed error contracts
are retained. Execution validation remains pending CI.
- Concurrency and lifecycle: no shared state, locks, threads, or new owned
resources. The V2 cardinality is reader-local and resets per block; error
rollback restores it with the columns.
- Memory and errors: one scalar counter is added, with no new untracked
buffers; existing Status propagation and cleanup paths are retained.
- Compatibility and persistence: no thrift/protobuf, metadata persistence,
transaction, network, configuration, or public SQL syntax changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
