But should MyType be serializable or can be of any type? On Nov 10, 2014 8:06 PM, "Stephan Ewen" <[email protected]> wrote:
> You can put generic objects in the closure as well, just as you can put a > configuration in the closure. > > You can also distribute your objects into the cluster and then use them as > a broadcast variable: > > ------------------------ > DataSet<MyType> aux = env.fromElements(new MyType()); > > > someOtherData.map(new MyMapper()).withBroadcastSet(aux, "my broadcast > data"); > > ------------------------ > > See > http://flink.incubator.apache.org/docs/0.7-incubating/programming_guide.html#broadcast-variables > > > > Stephan > > > On Mon, Nov 10, 2014 at 7:32 PM, Flavio Pompermaier <[email protected]> > wrote: > >> I think Stefano was asking for a different way to pass a generic >> Configuration object, not just a subclass of it. >> For example, in our use case, it would be helpful to broadcast around >> generic objects/pojos. >> Is that possible? >> On Nov 10, 2014 6:46 PM, "Stephan Ewen" <[email protected]> wrote: >> >>> Hi! >>> >>> I hope I understand correctly what you are trying to do: To have a >>> config file available in the functions, you can simply do wither of the >>> following: >>> >>> ----------------------- >>> Closure >>> ----------------------- >>> >>> Configuration conf = ... >>> >>> data.map(new RichMapFunction<String, Integer>() { >>> >>> public void open (Conficuration c) { >>> // access the conf object here >>> conf.getString(...); >>> } >>> >>> public Integer map(String value) { >>> // whatever >>> } >>> }); >>> ----------------------- >>> >>> >>> ----------------------- >>> Config Parameters >>> ----------------------- >>> >>> Configuration conf = ... >>> >>> data.map(new RichMapFunction<String, Integer>() { >>> >>> public void open (Conficuration c) { >>> // access the c - it will will have all elements of the conf - see >>> withParameters() below >>> c.getString(...); >>> } >>> >>> public Integer map(String value) { >>> // whatever >>> } >>> }) >>> .withParameters(conf); >>> ----------------------- >>> >>> >>> Stephan >>> >>> >>> On Mon, Nov 10, 2014 at 5:36 PM, Stefano Bortoli <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> trying to run some legacy code as part of Flink Job, I had to replicate >>>> configurations files across my cluster. Not a big deal with a small >>>> cluster, but it would be nice to have these configuration objects >>>> broadcast-able. Namely, it would be nice to reuse the old "read from conf >>>> file" logic to build objects that then could be serialized and used along >>>> the processing through the broadcast mechanism. >>>> >>>> Do you think it will be possible? with the new Kryo serialization it >>>> should not be extremely complicated. >>>> >>>> saluti, >>>> Stefano >>>> >>>> >>>> >>> >
