+1 for the idea of exposing more controls of the event processor. There are
other scenarios that admin would like to control/update the event processor:
1. When the event processor is lagging behind and users have used some
external invalidate/refresh commands to flush out the stale metadata, admin
can manually update the last-synced-event-id so the event processor can
jump to the latest events and catch up the lag immediately.
2. In the future, we can add back the light-weight mode of the event
processor that just invalidates tables (not refresh). Event-processor used
to work in this mode which can now be used to catch up the lag faster.
Admin can decide when to turn on this mode and when to turn off.
For how to expose the control, I think both SQL commands and WebUI will be
helpful.
- SQL commands will be easier for authorization. Only users with server
level ALL privilege can execute such commands. We do have similar commands
like REFRESH AUTHORIZATION, :SHUTDOWN(). BTW, we use SET commands for query
options. So "SET EVENT ACTIVE" might be misleading. We can discuss more
about the syntax, e.g.
- RESTART EVENTPROCESSOR
- RESTART EVENTPROCESSOR FROM EVENT 12345
- RESTART EVENTPROCESSOR IN INVALIDATE MODE
- RESTART EVENTPROCESSOR IN RELOAD MODE
- WebUI is easier to implement since we just need changes in the
catalogd side. But all users that can view the WebUI can perform the
operations. We currently have the same issue for canceling queries or
adjusting the log level in WebUI. Maybe it's ok to expose the
event-processor operations in webUI as well, and improve the privilege
control of the WebUI separately in the future.
Looking for more ideas. Feel free to file some JIRAs.
Thanks,
Quanlong
On Thu, Sep 28, 2023 at 11:27 AM guo Maxwell <[email protected]> wrote:
> *background : *
> Sometimes when the EventProcessorStatus of our impala changes to
> NEED_INVALIDATE, and we manually perform invalidate metadata operations on
> certain tables(as we know the changed schema).But the EventProcessorStatus
> may not be changed, as we can see the description in IMPALA-12468
> <https://issues.apache.org/jira/browse/IMPALA-12468> .
>
> So what about support some impala operation and maintenance behavior
> through SQL. As I know some database has already exposed some operation and
> maintenance operations through sql.
>
> *Action :*
> To fix this problem , I recommend providing Impala with a new SQL named :
> SET EVENT ACTIVE.
> Initially we can only support set ACTIVE status when we found the status is
> all right . Other states can be supported on demand.
>
> So what do you think? Thanks.
>
