Hi, You can look at my example prepared for Ignite and Spark streaming use cases: https://github.com/dmagda/IgniteSparkIoT
The data is being streamed into Ignite through a shared RDD and being queried using Ignite SQL. If you know the keys in advance then you can use Spark native RDDs APIs. That example is in Java but it’s easy convertible to Scala. Also, there is a simple Scala example available in every Ignite distribution that should be useful: https://github.com/apache/ignite/blob/master/examples/src/main/scala/org/apache/ignite/scalar/examples/spark/ScalarSharedRDDExample.scala <https://github.com/apache/ignite/blob/master/examples/src/main/scala/org/apache/ignite/scalar/examples/spark/ScalarSharedRDDExample.scala> — Denis > On Nov 1, 2017, at 1:22 PM, roshan joe <[email protected]> wrote: > > I am very new to Ignite and went through the documentation and samples on the > web. Below is the use case I am hoping to resolve using shared spark RDDs. > • Spark Streaming App-1 writes incremental records, which does not > already exist (by using a hash value generated inside the App) to a shared > RDD. > • Spark Streaming App-2 looks up the data in this shared RDD using key > columns and gets additional column values from the shared RDD populated by > the App-1 above. > • Spark Streaming App-3, Spark Streaming App-4 etc do the same lookup > against shared RDD, same as the above. > I am sure this use-case has been implemented before and am trying to avoid > spending time to build this from scratch. Is there some sample code to do > this using scala that can be shared? Thank you very much! > > >
