Aggarwal-Raghav commented on code in PR #6579:
URL: https://github.com/apache/hive/pull/6579#discussion_r3528704331
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java:
##########
@@ -476,39 +489,27 @@ private void initBufferedColumns() {
}
private void initExpressionColumns() {
- for (int i = 0; i < evaluators.length; i++) {
- VectorPTFEvaluatorBase evaluator = evaluators[i];
+ for (VectorPTFEvaluatorBase evaluator : evaluators) {
/*
- * Non-streaming evaluators work on buffered batches, we need to adapt
them. Before PTF
- * bounded start vectorization (HIVE-24761),
VectorExpression.outputColumnNum was closed and
- * VectorExpression.inputColumnNum didn't even exist (even though the
vast majority of
- * VectorExpression subclasses use at least 1 input column). Since
VectorPTFOperator and
- * VectorPTFGroupBatches work on modified batches (by not storing all
the columns, and having
- * ordering columns first), the expressions planned in compile-time
won't work with the
- * original config (column layout). It would make sense to move this
logic to compile time,
- * because here in runtime, a very simple mapping (bufferedColumnMap) is
used, so it might be
- * used. However, vectorized expression compilation affects many layers
of code (having
- * VectorizationContext as the common scope), and moving the calculation
of bufferedColumnMap
- * and this override logic to compile-time would create much more
complicated behavior there
- * (probably involving hacking most of the time, or maybe a great
re-design) just because of
- * the optimized column layout of the PTF vectorization.
+ * Non-streaming window function evaluators (like SUM or LEAD) work on
packed, buffered batches.
+ * VectorPTFOperator and VectorPTFGroupBatches significantly modify the
batch structure to save memory
+ * (e.g. by dropping unused columns and moving ordering columns to the
front).
+ *
+ * Because of this dense packing, the absolute column indices assigned
by the compiler (e.g., Index 15)
+ * are no longer valid for the buffer (where it might now be Index 2).
+ * Therefore, we must dynamically map the evaluator's inputColumnNum to
its new location in the packed array.
+ *
+ * NOTE: We ONLY patch the evaluator's input index here. We specifically
DO NOT patch
+ * evaluator.inputVecExpr (the math expressions like A+B) because those
are now evaluated eagerly
Review Comment:
**I'm not an expert on vectorization** but from the understanding while
debugging this:
There is **no performance degradation** because the PTF operator does not
filter rows; every row evaluated eagerly would have been lazily evaluated
anyway. **Infact, it provides benefits,**
- Instead of writing raw inputs to a memory/disk buffer and fetching them
back later to do the math
- By doing the math eagerly on the original batch, the buffering logic can
drop the intermediate child scratch column (example Index 14) columns before
buffering even starts. This means our `BufferedVectorizedRowBatch` takes up
less RAM and disk footprint!"
Please let me know If I'm missing something.
```mermaid
flowchart TD
subgraph Lazy ["😬 BEFORE (Lazy Evaluation) - High Memory & Evaluation
Overhead"]
direction TB
L1["1. Raw Data Arrives"]
L2["2. Pack into Buffer\n(Writes Raw Inputs to Memory / Disk)"]
L3["3. Partition Finishes"]
L4["4. Fetch rows back from Buffer"]
L5["5. Evaluate Math (A+B+C)\n(Creates intermediate Index 14 in
RAM)"]
L6["6. Run SUM()"]
L1 --> L2 --> L3 --> L4 --> L5 --> L6
end
subgraph Eager ["😀 AFTER (Eager Evaluation) - Max Memory Efficiency"]
direction TB
E1["1. Raw Data Arrives"]
E2["2. Evaluate Math IMMEDIATELY\n(Index 1 âž” Index 14 âž” Index 15)"]
E3["3. Pack Final Answer into Buffer\n(Drops Index 14 to save RAM!
Writes Index 15)"]
E4["4. Partition Finishes"]
E5["5. Run SUM()\n(Reads final answer directly)"]
E1 --> E2 --> E3 --> E4 --> E5
end
classDef bad fill:#f8d7da,stroke:#dc3545,stroke-width:2px;
classDef good fill:#d4edda,stroke:#28a745,stroke-width:2px;
classDef mem_win fill:#cce5ff,stroke:#004085,stroke-width:2px;
class L4,L5 bad;
class E1,E2 good;
class E3 mem_win;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
