Re: [I] [Feature Request]: Moving Average (Last N Events) in (Memory) Group By Step (hop)
mattcasters closed issue #7023: [Feature Request]: Moving Average (Last N Events) in (Memory) Group By Step URL: https://github.com/apache/hop/issues/7023 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Moving Average (Last N Events) in (Memory) Group By Step (hop)
dmainou commented on issue #7023: URL: https://github.com/apache/hop/issues/7023#issuecomment-4308829281 @hansva but the group fields would include everything needed to calculate. e.g. Share | Date | Price | 3dMA -- | -- | -- | -- BHP | 2026-04-01 | 44.20 | BHP | 2026-04-02 | 44.60 | BHP | 2026-04-03 | 45.10 | 44.63 BHP | 2026-04-06 | 44.90 | 44.87 BHP | 2026-04-07 | 45.40 | 45.13 BHP | 2026-04-08 | 45.80 | 45.37 BHP | 2026-04-09 | 45.30 | 45.50 BHP | 2026-04-10 | 46.00 | 45.70 BHP | 2026-04-13 | 46.20 | 45.83 BHP | 2026-04-14 | 46.50 | 46.23 CBA | 2026-04-01 | 118.40 | CBA | 2026-04-02 | 119.10 | CBA | 2026-04-03 | 118.80 | 118.77 CBA | 2026-04-06 | 120.00 | 119.30 CBA | 2026-04-07 | 120.50 | 119.77 CBA | 2026-04-08 | 121.20 | 120.57 CBA | 2026-04-09 | 120.90 | 120.87 CBA | 2026-04-10 | 121.60 | 121.23 CBA | 2026-04-13 | 122.10 | 121.53 CBA | 2026-04-14 | 122.40 | 122.03 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Moving Average (Last N Events) in (Memory) Group By Step (hop)
hansva commented on issue #7023: URL: https://github.com/apache/hop/issues/7023#issuecomment-4303488017 It does sort, but only on the group fields, in this case you need to sort on an arbitrary extra column, which would also be possible just needs some more work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Moving Average (Last N Events) in (Memory) Group By Step (hop)
dmainou commented on issue #7023: URL: https://github.com/apache/hop/issues/7023#issuecomment-4302538620 I'll amend as soon as I get home. I had the group by in mind and just included the other because I see them as analogous just one requires a sort and the other doesn't. Me thinking the other sorts as part of the step -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Moving Average (Last N Events) in (Memory) Group By Step (hop)
mattcasters commented on issue #7023: URL: https://github.com/apache/hop/issues/7023#issuecomment-4302518329 Exactly, in the Group By transform we have an order and we can use the "value' field to specify the minimal number of rows to start calculating the moving average/sum/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Moving Average (Last N Events) in (Memory) Group By Step (hop)
dmainou commented on issue #7023: URL: https://github.com/apache/hop/issues/7023#issuecomment-4302505155 Ah, now I understand. Then yes. The ticket needs to be changed to only group by -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Moving Average (Last N Events) in (Memory) Group By Step (hop)
mattcasters commented on issue #7023: URL: https://github.com/apache/hop/issues/7023#issuecomment-4302489098 Diego, the Memory Group By transform does not have the requirement of input data being sorted on the group fields. That's the whole point. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Moving Average (Last N Events) in (Memory) Group By Step (hop)
dmainou commented on issue #7023: URL: https://github.com/apache/hop/issues/7023#issuecomment-4302176053 Hi @mattcasters you are correct in that the expectation is that the data has been pre-sorted (as required anyway by the step) when using the group by. Memory group by would need to do more heavy lifting. Would be great to enable calculating a moving average for the last X events https://en.wikipedia.org/wiki/Moving_average time, value, 2 "day" moving average" T0,10,null T1,10,10 T2,12,11 T3,10,11 T4,11,10.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Moving Average (Last N Events) in (Memory) Group By Step (hop)
mattcasters commented on issue #7023: URL: https://github.com/apache/hop/issues/7023#issuecomment-4298798749 Is this request logically correct? The Memory Group By transform outputs records after processing all incoming rows, avoiding the need for the input data to be sorted. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
[I] [Feature Request]: Moving Average (Last N Events) in (Memory) Group By Step (hop)
dmainou opened a new issue, #7023: URL: https://github.com/apache/hop/issues/7023 ### What would you like to happen? ### Summary Add support in the **Group By** and ** Memory Group By** transforms to calculate a moving average over the last **N events** within each group. ### Problem Current aggregation functions operate on the full group only (e.g. sum, avg, min, max). There is no native way to compute a rolling/moving average within a group without additional steps or custom logic. ### Proposed Solution Introduce a new aggregation type: - **Moving Average (Last N Events)** With configuration options: - Field to aggregate - Window size (**N**) - Order field (to define event sequence within the group) ### Example For a group `customer_id`, calculate: - Moving average of `transaction_amount` - Over last `N = 5` events - Ordered by `transaction_timestamp` ### Expected Behaviour - Resets per group - Respects defined sort/order field - Outputs a new field with the rolling average ### Benefits - Simplifies common analytical patterns - Reduces need for multiple transforms or scripting - Improves readability and maintainability of pipelines ### Additional Notes - Should handle edge cases where fewer than N events exist (e.g. partial window) - Performance considerations for large datasets should be taken into account ### Issue Priority Priority: 3 ### Issue Component Component: Hop Gui -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
