Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/967#issuecomment-127236312 We are thinking about more general tooling for parameter servers (with implementations different from Ignite), because the request for that has come up. Whether we want to ship a parameter server with Flink, or simply add interfaces that help connecting against PS, is undecided, yet. For that reason, we should probably not overdesign this now. Let's add SSP to the runtime, week Ignite out, and make a nice self-contained example where Ignite is purely used in the user-code. I think we can make a self-contained example in the following way: The example MapFunction that communicates with the parameter server instantiates it's Ignite in the `open()` function, if it is in the first superstep. It then proceeds as in your code. To make this real-world applicable, I assume one would need dedicated Ignite processes anyways. In the example, the parameters are lost after the server is torn down, but so are they with an "in taskmanager parameter server", for all users of YARN as well (where TMs are shut down after a job). The example here serves the purpose of demonstrating the capabilities and use of SSP, and to give people a blueprint how to connect this to a parameter server. What do you think about this? Her are some other comments: - Using Longs as parameter IDs is probably quite a bit cheaper than Strings. - The pull request also still contains markup from merging. - Some constants are hard-wired into the code. Would be good to pull them into static final fields at least, so it is very visible that these are predefined constants. - You seem to re-initialize the caches in every superstep. In general, initialization in `open()` in iterations needs to be done carefully, because `open()` is called at the beginning of each superstep. You can always check in which superstep you are using `getIterationRuntimeContext().getCurrentSuperstep()`
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---