Dear Ignite community, Apache Apex (http://apex.incubator.apache.org/) is an in-memory stream processing and analytics platform, built for scalability, low-latency processing, high availability and operability. It has a pipeline processing architecture can be used for real-time and batch processing. Apex runs natively on YARN and HDFS.
Typical Apex applications integrate with external systems: ingestion, state saving, lookup data for processing, land compute results for downstream consumers etc. The Malhar library has a wide range of connectors and compute functions readily available to assemble Apex applications. I wanted to see if there is interest to develop Ignite integrations as part of Malhar. Some ideas: - Checkpointing ( https://www.datatorrent.com/blog/blog-introduction-to-checkpoint/) could use the IGFS (possibly with Ignite native integration of the storage agent interface). - Compute results could be stored in Ignite (example dimension computation https://www.datatorrent.com/blog/blog-dimensions-computation-aggregate-navigator-part-1-intro/ ) - Enrichment/lookup could be implemented with Ignite as storage layer. Thanks, Thomas
