Implemented this way, options like max_restarts max_children and so on
would occur per partition. I take it the plan would be to simply note that
in the docs? I don't see any easy way to enforce those values across all
partitions, which I think is just fine.
which_children/1 and so on won't work as expected either, so maybe we want
a helper function inside of DynamicSupervisor that flat_maps that over the
child supervisors?
Overall though, +1 from me.
On Thursday, September 2, 2021 at 11:28:19 AM UTC-4 José Valim wrote:
> Given supervisors are also processes, they may also become bottlenecks.
> While this is unlikely to happen to a Supervisor, since it is mostly
> static, it can happen to a DynamicSupervisor.
>
> We can address this by partitioning the dynamic supervisor. Imagine the
> following dynamic supervisor:
>
> defmodule MyApp.DynamicSupervisor do
> use DynamicSupervisor
>
> def start_link(opts) do
> DynamicSupervisor.start_link(__MODULE__, arg, opts)
> end
>
> def init(_arg) do
> DynamicSupervisor.init(strategy: :one_for_one)
> end
> end
>
> In order to partition it, we can start 8 instances of said supervisor
> inside a regular Supervisor, and then pick one partition at random when
> starting a child. For example:
>
> defmodule MyApp.Supervisor do
> use Supervisor
>
> @partitions 8
> @name __MODULE__
>
> def start_child(module, arg) do
> i = :erlang.phash2(self(), @partitions) + 1
> DynamicSupervisor.start_child(:"#{__MODULE__}#{i}", {module, arg})
> end
>
> def start_link do
> Supervisor.start_link(__MODULE__, arg, name: @name)
> end
>
> def init(arg) do
> children =
> for i <- 1..@partitions do
> name = :"#{__MODULE__}#{i}"
> Supervisor.child_spec({MyApp.DynamicSupervisor, name: name}, id:
> name)
> end
>
> Supervisor.init(children, strategy: :one_for_one)
> end
> end
>
> I would like to make the above more convenient by introducing a
> :partitions option to DynamicSupervisor.start_link. When given, the new
> option will automatically start N dynamic supervisors under a supervisor,
> like above:
>
> DynamicSupervisor.start_link(__MODULE__, :ok, partitions: 8, name: @name)
>
> For now, the :name option will be required.
>
> Now, when spawning child processes, you will use via tuples:
>
> DynamicSupervisor.start_child({:via, DynamicSupervisor, {@name, self()}},
> {module, arg})
>
> The via tuple has the format {:via, DynamicSupervisor, {supervisor_name,
> value_to_partition_on}}. Once invoked, it will take care of partitioning
> based on the current process and dispatching it.
>
> Overall, encapsulating the partitioning of DynamicSupervisor (and
> consequently of Task.Supervisor) into an easy to use API can help to
> vertically scale up applications that use those constructs.
>
> ## Open questions
>
> One of the confusing aspects of the above is that the :name option no
> longer reflects the name of the DynamicSupervisor but of the parent
> Supervisor. One alternative is to *not* accept the :name option when
> :partitions is given, instead we could have a :root_name option instead (or
> something more appropriately named).
>
> Implementation wise, we will store the available processes in ETS or using
> a Registry (to be started alongside the Elixir application).
>
> Feedback?
>
--
You received this message because you are subscribed to the Google Groups
"elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elixir-lang-core+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elixir-lang-core/bae8be24-a2e5-4a07-89db-7758ccfaf0d4n%40googlegroups.com.
