Hi IoTDB community, Following the previous requirements discussion, I have prepared an initial technical design for adding the Prometheus-like rate(), irate(), increase(), and delta() aggregate functions to the IoTDB table model.
Related issue: https://github.com/apache/iotdb/issues/17976 The main design is summarized below. Integration with the existing aggregation framework The four functions will be implemented entirely within the existing table-model aggregation framework. No new SQL execution layer or third-party dependency will be introduced. Both execution paths will be supported: TableAccumulator for global aggregation and streaming aggregation when groups are fully ordered. GroupedAccumulator for HashAggregationOperator and StreamingHashAggregationOperator. Each function will provide both TableAccumulator and GroupedAccumulator implementations. The signatures are: rate(value_col, time_col, window_start, window_end) increase(value_col, time_col, window_start, window_end) irate(value_col, time_col) delta(value_col, time_col, window_start, window_end) The value column supports INT32, INT64, FLOAT, and DOUBLE. Values are converted to double internally, and all four functions return DOUBLE. Time arguments support TIMESTAMP and INT64. Ordered and unordered implementations Each accumulator will support two internal modes rather than introducing separate Ordered and Naive accumulator classes. When the aggregation step is SINGLE and the planner can prove that samples within each group are ordered by time_col, the accumulator will use a streaming implementation with fixed-size state: Time complexity: O(n) Additional space per group: O(1) Otherwise, the accumulator will buffer all valid samples and sort them before the final calculation: Time complexity: O(n log n) Additional space per group: O(n) The two modes will have identical function semantics, validation rules, and results. A new inputOrderedByTimeAscending property will be added to AggregationNode.Aggregation. The ordered implementation will be selected only when: step == SINGLE && inputOrderedByTimeAscending PARTIAL, INTERMEDIATE, and FINAL aggregation stages will always use the buffered implementation because they need to generate, merge, or consume intermediate states. The ordered mode will still perform runtime checks for duplicate timestamps and unexpected out-of-order input. Accumulator organization rate() and increase() will share common base classes because they use the same counter-reset correction and boundary-extrapolation logic. The main TableAccumulator classes will include: AbstractRateIncreaseAccumulator RateAccumulator IncreaseAccumulator IrateAccumulator DeltaAccumulator Equivalent GroupedAccumulator classes will maintain state independently for each groupId. In ordered mode, the accumulators retain only fixed state such as the first, previous, and last samples, sample count, corrected increase, and window boundaries. In unordered mode, samples will be stored in a new TimeValueBuffer. Grouped accumulators will use TimeValueBufferBigArray to maintain one buffer per groupId. Intermediate-state handling For distributed aggregation, PARTIAL, INTERMEDIATE, and FINAL stages must preserve the complete sample sequence because counter-reset detection, duplicate-timestamp validation, and the final result depend on global time ordering. The intermediate state will contain: A state version window_start and window_end, except for irate() The number of valid samples All timestamp/value pairs PARTIAL stages collect and serialize their samples without calculating a final result. INTERMEDIATE stages deserialize and merge sample buffers without sorting them. The FINAL stage merges all states, sorts the complete sample set by timestamp, checks for duplicate timestamps, and then performs the final calculation. Deserialization will validate the sample count and serialized length to reject corrupted intermediate states and prevent invalid memory allocation. Memory management The unordered implementation will integrate with the existing query memory-management mechanism, following a design similar to the exact percentile() accumulator. TimeValueBuffer and TimeValueBufferBigArray will report their retained memory. Accumulators will use MemoryReservationManager to reserve or release the difference whenever buffers grow, merge, or shrink. reset() will clear the calculation state, shrink oversized internal arrays to their initial capacity, and release the corresponding query memory. Shared extrapolation logic A shared ExtrapolationUtil will implement the boundary-extrapolation algorithm used by rate(), increase(), and delta(). It will be responsible for: Converting timestamp differences to seconds according to the current timestamp_precision Calculating the average sample interval Applying the 1.1-times extrapolation threshold Limiting extrapolation to half an average interval when a boundary is too far away Applying counter zero-point protection for rate() and increase() Calculating and validating the final extrapolation factor Counter-reset detection and sample traversal will remain in the accumulators rather than in ExtrapolationUtil. rate() will divide the extrapolated increase by the complete window duration. increase() will return the extrapolated increase directly. delta() will extrapolate the raw difference between the last and first values without counter zero-point protection. irate() will use only the final two samples and will not perform boundary extrapolation. All time conversions will respect the configured ms, us, or ns timestamp precision, while rate() and irate() will always return values per second. Please review this initial design, especially the ordered-input property, the complete-sample intermediate state, and the memory-management approach. Any comments or alternative suggestions are welcome. Best regards, Xinqi Zhao
