Yes, Flink is obviously the next target. I'm not expecting too many issues there beyond getting a cluster set up to test on. I read you can run the Flink Runner locally so that will help a lot in testing.
However, I'm first trying to solve the complicated issue of grouping records together in Beam in a safe way so that they can batched up. Batching up is really important for fast loading into a lot of output targets. I'll probably use some group by behind the scenes or something like that, need to think about that. Having the ability to re-use the existing Kettle steps without having to write new code is really key. Once that is done (in a few weeks) I'll give Flink a shot. Cheers, Matt Op di 12 feb. 2019 om 12:02 schreef Maximilian Michels <m...@apache.org>: > @Dan: Thanks for sharing the presentation. Kettle is a great way to make > Beam more accessible. > > @Matt: Thanks for the plug. It's good to hear you enjoyed it. I think > the link to your slides got messed up: http://beam.kettle.be > > Are you planning to add execution via the Flink Runner to Kettle? Saw in > the presentation that you already support Direct, Spark, and Dataflow. > > On 11.02.19 20:50, Matt Casters wrote: > > By the way, Maximilian, I linked and plugged your wonderful FOSDEM > > presentation in my slides http://beam kettle.be <http://kettle.be> > slide > > 19. If you mind, let me know and I'll get it out of the slides. In any > > case, great content worth promoting I thought. > > > > Op wo 6 feb. 2019 18:03 schreef Maximilian Michels <m...@apache.org > > <mailto:m...@apache.org>: > > > > Hi Dan, > > > > Thanks for the info. Would be great to share a video of the > > presentation. > > > > Cheers, > > Max > > > > On 30.01.19 10:00, Dan wrote: > > > Hi, in just over a week you're all welcome to come and see the > very > > > first public reveal of Kettle running on beam! (Including spark, > > > dataflow and flink support) > > > > > > > https://www.meetup.com/Pentaho-London-User-Group/events/256773962/ > > > > > > So this ingenious integration combines the power of visual > > development, > > > with the platform agnostic benefits of beam - impressive stuff. No > > > vendor lock-in here! > > > > > > > > > See you there! > > > Dan > > >
