Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/967#issuecomment-127236312
We are thinking about more general tooling for parameter servers (with
implementations different from Ignite), because the request for that has come
up. Whether we want to ship a parameter server with Flink, or simply add
interfaces that help connecting against PS, is undecided, yet.
For that reason, we should probably not overdesign this now. Let's add SSP
to the runtime, week Ignite out, and make a nice self-contained example where
Ignite is purely used in the user-code.
I think we can make a self-contained example in the following way:
The example MapFunction that communicates with the parameter server
instantiates it's Ignite in the `open()` function, if it is in the first
superstep. It then proceeds as in your code.
To make this real-world applicable, I assume one would need dedicated
Ignite processes anyways. In the example, the parameters are lost after the
server is torn down, but so are they with an "in taskmanager parameter server",
for all users of YARN as well (where TMs are shut down after a job).
The example here serves the purpose of demonstrating the capabilities and
use of SSP, and to give people a blueprint how to connect this to a parameter
server.
What do you think about this?
Her are some other comments:
- Using Longs as parameter IDs is probably quite a bit cheaper than
Strings.
- The pull request also still contains markup from merging.
- Some constants are hard-wired into the code. Would be good to pull them
into static final fields at least, so it is very visible that these are
predefined constants.
- You seem to re-initialize the caches in every superstep. In general,
initialization in `open()` in iterations needs to be done carefully, because
`open()` is called at the beginning of each superstep. You can always check in
which superstep you are using
`getIterationRuntimeContext().getCurrentSuperstep()`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
