*Hi AllI have a question on best practices on building scheduling /
throttling mechanisms on Flight data services in a multi-tenant
environment.My high level understandings are- The Flight data service uses
a thread-pool-based model. i.e. each data server normally runs with a
fixed-size thread pool. During processing, each request will occupy the
entire thread in its lifecycle.- The size of requests is heterogeneous,
meaning some requests process a few MBs of data and may just take a few
hundred milliseconds, while others may need to process hundreds of GBs of
data taking hours.- For simplicity, let's use thread as the unit of
resources shared among multiple users across data servers to facilitate
discussionOne natural way to start is to only allow a user to use a share
of the thread pool per server. For example, each user is allowed to use up
to 5% of threads in the thread pool on a server.- This mechanism, however,
has a defect on fairness when there are many whale users (users who send
much more concurrent requests than the total number threads allowed for
that user from all servers). Using the example above, if there are 20 such
users (each taking 5% of the thread pool) at all time, they will use up all
threads in the fleet very quickly.- Adding more servers doesn't solve this
issue as each whale user will take threads from the new servers quickly as
well.In other words, how to ensure fairness to not starve regular users
when there are many whale users?My question is: is there any best practice
in Flight data services to handle this with local scheduling/throttling? Or
this can be only solved with global throttling: e.g., track concurrent
requests from a user in a centralized place, and then each Flight metadata
or data service fetches the user concurrencies periodically?Thanks in
advance.*
--
Chen Song
