You can put generic objects in the closure as well, just as you can put a configuration in the closure.
You can also distribute your objects into the cluster and then use them as a broadcast variable: ------------------------ DataSet<MyType> aux = env.fromElements(new MyType()); someOtherData.map(new MyMapper()).withBroadcastSet(aux, "my broadcast data"); ------------------------ See http://flink.incubator.apache.org/docs/0.7-incubating/programming_guide.html#broadcast-variables Stephan On Mon, Nov 10, 2014 at 7:32 PM, Flavio Pompermaier <[email protected]> wrote: > I think Stefano was asking for a different way to pass a generic > Configuration object, not just a subclass of it. > For example, in our use case, it would be helpful to broadcast around > generic objects/pojos. > Is that possible? > On Nov 10, 2014 6:46 PM, "Stephan Ewen" <[email protected]> wrote: > >> Hi! >> >> I hope I understand correctly what you are trying to do: To have a config >> file available in the functions, you can simply do wither of the following: >> >> ----------------------- >> Closure >> ----------------------- >> >> Configuration conf = ... >> >> data.map(new RichMapFunction<String, Integer>() { >> >> public void open (Conficuration c) { >> // access the conf object here >> conf.getString(...); >> } >> >> public Integer map(String value) { >> // whatever >> } >> }); >> ----------------------- >> >> >> ----------------------- >> Config Parameters >> ----------------------- >> >> Configuration conf = ... >> >> data.map(new RichMapFunction<String, Integer>() { >> >> public void open (Conficuration c) { >> // access the c - it will will have all elements of the conf - see >> withParameters() below >> c.getString(...); >> } >> >> public Integer map(String value) { >> // whatever >> } >> }) >> .withParameters(conf); >> ----------------------- >> >> >> Stephan >> >> >> On Mon, Nov 10, 2014 at 5:36 PM, Stefano Bortoli <[email protected]> >> wrote: >> >>> Hi, >>> >>> trying to run some legacy code as part of Flink Job, I had to replicate >>> configurations files across my cluster. Not a big deal with a small >>> cluster, but it would be nice to have these configuration objects >>> broadcast-able. Namely, it would be nice to reuse the old "read from conf >>> file" logic to build objects that then could be serialized and used along >>> the processing through the broadcast mechanism. >>> >>> Do you think it will be possible? with the new Kryo serialization it >>> should not be extremely complicated. >>> >>> saluti, >>> Stefano >>> >>> >>> >>
