walterddr opened a new issue #7448: URL: https://github.com/apache/pinot/issues/7448
Some of the `@ScalarFunction` we built and potentially other 3rd party customized function can benefit from a stateful optimization. E.g. if we know a scalarfunction is used with literal arguments. those literal arguments can be preprocessed (e.g. regexp pattern compilation) However since ScalarFunction are used not only during the query code path but also the ingestion codepath. we need to be support both the stateful and stateless version of the function to ensure performance optimization and compatibility. https://github.com/apache/pinot/pull/7443 created an example of how this could benefit. But it doesn't work on the ingestion path and there are some other issues to consider: - [ ] need to guarantee that all the stateful version of the FunctionInvoker internal instances are properly initialized, in both query and ingestion code path - [ ] support type matching with Literal detection, there's no point of optimizing scalar function if there's no prior knowledge of the arguments, might as well just do the computation during actual call. - [ ] should not incur any additional branch condition in critical path. - [ ] should ensure that multiple instances of the FunctionInvoker (and its underlying instances) doesn't share any states. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
