Any type Flink supports as a data type. Am 10.11.2014 20:19 schrieb "Flavio Pompermaier" <[email protected]>:
> But should MyType be serializable or can be of any type? > On Nov 10, 2014 8:06 PM, "Stephan Ewen" <[email protected]> wrote: > >> You can put generic objects in the closure as well, just as you can put a >> configuration in the closure. >> >> You can also distribute your objects into the cluster and then use them >> as a broadcast variable: >> >> ------------------------ >> DataSet<MyType> aux = env.fromElements(new MyType()); >> >> >> someOtherData.map(new MyMapper()).withBroadcastSet(aux, "my broadcast >> data"); >> >> ------------------------ >> >> See >> http://flink.incubator.apache.org/docs/0.7-incubating/programming_guide.html#broadcast-variables >> >> >> >> Stephan >> >> >> On Mon, Nov 10, 2014 at 7:32 PM, Flavio Pompermaier <[email protected] >> > wrote: >> >>> I think Stefano was asking for a different way to pass a generic >>> Configuration object, not just a subclass of it. >>> For example, in our use case, it would be helpful to broadcast around >>> generic objects/pojos. >>> Is that possible? >>> On Nov 10, 2014 6:46 PM, "Stephan Ewen" <[email protected]> wrote: >>> >>>> Hi! >>>> >>>> I hope I understand correctly what you are trying to do: To have a >>>> config file available in the functions, you can simply do wither of the >>>> following: >>>> >>>> ----------------------- >>>> Closure >>>> ----------------------- >>>> >>>> Configuration conf = ... >>>> >>>> data.map(new RichMapFunction<String, Integer>() { >>>> >>>> public void open (Conficuration c) { >>>> // access the conf object here >>>> conf.getString(...); >>>> } >>>> >>>> public Integer map(String value) { >>>> // whatever >>>> } >>>> }); >>>> ----------------------- >>>> >>>> >>>> ----------------------- >>>> Config Parameters >>>> ----------------------- >>>> >>>> Configuration conf = ... >>>> >>>> data.map(new RichMapFunction<String, Integer>() { >>>> >>>> public void open (Conficuration c) { >>>> // access the c - it will will have all elements of the conf - see >>>> withParameters() below >>>> c.getString(...); >>>> } >>>> >>>> public Integer map(String value) { >>>> // whatever >>>> } >>>> }) >>>> .withParameters(conf); >>>> ----------------------- >>>> >>>> >>>> Stephan >>>> >>>> >>>> On Mon, Nov 10, 2014 at 5:36 PM, Stefano Bortoli <[email protected]> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> trying to run some legacy code as part of Flink Job, I had to >>>>> replicate configurations files across my cluster. Not a big deal with a >>>>> small cluster, but it would be nice to have these configuration objects >>>>> broadcast-able. Namely, it would be nice to reuse the old "read from conf >>>>> file" logic to build objects that then could be serialized and used along >>>>> the processing through the broadcast mechanism. >>>>> >>>>> Do you think it will be possible? with the new Kryo serialization it >>>>> should not be extremely complicated. >>>>> >>>>> saluti, >>>>> Stefano >>>>> >>>>> >>>>> >>>> >>
