a. No. I have not executed any tests. I am doing a theoretical study first to understand the memory footprint and data movement between the spark & ignite nodes
c. So basically there is no use case when working with spark (for data processing) and ignite (for in-memory data storage) that can benefit from ignite transactions. Since spark is non-transactional, I am trying to understand how can use the ACID transactional support feature of ignite when performing data processing in spark. Let me know if you think otherwise. > To be honest, I'm not quite sure you should use IgniteRDD because it > sounds like your use case is bigger than that (indexed SQL, transactions, > etc.). Did you consider using pure Ignite API without integrating it with > Spark? Yes, I have performed tests with pure Ignite API but the performance didn't turn out to be well. Additionally we do have complex requirements in the analytics space that could be served well with Spark. Are you suggesting that IgniteRDD (in other words - spark integration) should be used when the data set is small? From various blogs/articles I understand that the main benefit of using spark with Ignite is to easily share data across spark jobs where ignite is used for in-memory data storage and spark for its high speed data processing capabilities. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-Spark-Ignite-Integration-tp8556p9082.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
