Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality ...................................................................... Patch Set 10: (2 comments) Before making the changes which might be large, let me reply the discussion first. We can discuss more until we make a consensus. http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java File fe/src/main/java/org/apache/impala/planner/UnionNode.java: http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@391 PS8, Line 391: public void appendScanInputStats(TPlanNodeRun execStats) { > That's a good example! Initially I thought the IGNORE_PARTITION_CONSTANTS s I'm thinking a strategy-independent order that can be used here and in UnionNode.generateHboKeyString(). Then we don't need to generate a list of TScanInputStats for each strategy and send them to BE. If we do so, we need more lists in the future when HBO supports more stats types - they could also impact the order of the HBO key string since they would add more info in the key. Here is one approach for soring operands that I found from Presto: - First, sort by the scan table name string of each operand. If the operand has a single scan in its subtree, use the table name as the string. If the operand has multiple scans, use a comma seperated list of the sorted table names as the string. - If there are duplicate strings, abort sorting and use the original order of the query. Note that such cases are rare in practise so might not worth efforts in optimization. Note that Presto adds a flag, use_perfectly_consistent_histories, to sort by the HBO key strings. But it's disabled by default. http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@415 PS8, Line 415: msg.union_node = new TUnionNode( > Thanks, the PS10 guard looks good. Could you also add the regression test f The query won't fail since the HBO write path is independent and async to the query execution. There are just INFO level logs like this: I20260805 20:38:00.834429 135178 impala-server.cc:1797] Storing history stats for query a9431bb921856083:684eef8300000000 I20260805 20:38:00.834708 135178 JniFrontend.java:1001] execution stats from BE: THistoricalStatsUpdate(plan_node_runs:[TPlanNodeRunWithKeys(run:TPlanNodeRun(num_rows:2), hash_keys:{EXPR_REWRITE=a41799e17d0931a61bd649d4fdb57037}, stats_type:CARDINALITY)]) I20260805 20:38:00.835738 135178 jni-util.cc:335] java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:496) at org.apache.impala.service.HistoricalStats.getSimilarRunIndex(HistoricalStats.java:116) at org.apache.impala.service.HistoricalStats.writePlanNodeStats(HistoricalStats.java:147) at org.apache.impala.service.HistoricalStats.writeStats(HistoricalStats.java:126) at org.apache.impala.service.JniFrontend.storeExecStats(JniFrontend.java:1002) I20260805 20:38:00.835767 135178 status.cc:123] IllegalStateException: null @ 0x137d14d impala::Status::Status() @ 0x1d56081 impala::JniUtil::GetJniExceptionMsg() @ 0x12c87d9 impala::JniCall::Call<>() @ 0x198fd5d impala::Frontend::StoreExecStats() @ 0x1a7427d impala::ImpalaServer::StoreExecutionStats() @ 0x1a7461e impala::ImpalaServer::CloseClientRequestState() @ 0x1a758d7 impala::ImpalaServer::FinishUnregisterQuery() @ 0x1a94e27 boost::detail::function::void_function_obj_invoker2<>::invoke() @ 0x1ab09e6 impala::ThreadPool<>::WorkerThread() @ 0x1a94ed1 boost::detail::function::void_function_obj_invoker0<>::invoke() @ 0x1df61a4 boost::function0<>::operator()() @ 0x1df5080 impala::Thread::SuperviseThread() @ 0x1df5ae1 boost::detail::thread_data<>::run() @ 0x296e827 thread_proxy @ 0x70db8ee94ac3 (unknown) @ 0x70db8ef268c0 (unknown) @ (nil) (unknown) Maybe I should change the log level and test on not seeing such logs. Do you have better ideas about adding the test? -- To view, visit http://gerrit.cloudera.org:8080/24556 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8 Gerrit-Change-Number: 24556 Gerrit-PatchSet: 10 Gerrit-Owner: Quanlong Huang <[email protected]> Gerrit-Reviewer: Aleksandr Efimov <[email protected]> Gerrit-Reviewer: Aman Sinha <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Steve Carlin <[email protected]> Gerrit-Comment-Date: Mon, 10 Aug 2026 16:17:33 +0000 Gerrit-HasComments: Yes
