Hello Denis, This is really gret news, I think Ignite can be integrated on Beam as an IO in that case Beam developers will read/write their data from/to Ignite from their data processing pipelines.
You can take a look at some of the existing IOs for ideas and follow the Ptransform guide for style https://github.com/apache/beam/tree/master/sdks/java/io https://beam.apache.org/contribute/ptransform-style-guide/ Notice that there is an open JIRA to support a JCache based connector so a good idea would be to implement it and use Ignite as the reference example (of course you can go the Ignite native route but community wise this would be neat). https://issues.apache.org/jira/browse/BEAM-2584 >From a quicklook at the Compute Grid documentation in the website it seems also that it could make sense to integrate Ignite into Beam as a runner. This requires translating the Beam model into the appropriate Ignite API. For this the best reference to start is : https://beam.apache.org/contribute/runner-guide/ Also I saw you that you guys have a Filesystem Ignite’s (IGFS) with support for HDFS so a first quick contribution would be to validate that it works with Beam and add some documentation on how to use it. Don’t hesitate to ask questions, create JIRAs, or contact us here or in the slack channel if needed. Best, Ismaël On Fri, Dec 8, 2017 at 6:54 AM, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > Hi > > This sounds awesome to have an Ignite runner which could compete with > hazelcast-jet. > > The entry point would be https://beam.apache.org/contribute/runner-guide/ > IMHO. > > Being on Ignite cluster also opens a lot of doors - reusing the filesystem > or distributed structures. Very exiting. > > Le 8 déc. 2017 05:46, "Denis Magda" <dma...@apache.org> a écrit : >> >> Hello Apache Beam fellows! >> >> We at Apache Ignite community came across your project and would be happy >> to integrate with it. >> >> In short, Ignite is a distributed database and computational platform that >> has its own map-reduce like component: >> https://apacheignite.readme.io/docs/compute-grid >> >> The integration will give Beam users an ability to use Ignite as a >> distributed processing back-end system and database. >> >> How should we proceed? Please share any relevant information. >> >> — >> Denis >> Ignite PMC
