Re: Apache Spark & Ignite Integration

2016-12-14 Thread vkulichenko
e the first one. b. Ignite works in concurrent fashion without locking the world. While you save new data, you can still read it and run queries. -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-Spark-Ignite-Integration-tp8556p9544.html Sent from th

Re: Apache Spark & Ignite Integration

2016-12-12 Thread pragmaticbigdata
going to be updated and lock only those? Thanks. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-Spark-Ignite-Integration-tp8556p9502.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Apache Spark & Ignite Integration

2016-12-12 Thread vkulichenko
Hi, To be honest, I don't quite understand these diagrams :) Can you give some comments? -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-Spark-Ignite-Integration-tp8556p9494.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Apache Spark & Ignite Integration

2016-12-05 Thread pragmaticbigdata
ers.70518.x6.nabble.com/file/n9393/q7e83SI.png> -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-Spark-Ignite-Integration-tp8556p9393.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Apache Spark & Ignite Integration

2016-11-22 Thread vkulichenko
lose the whole purpose of the shared RDD. -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-Spark-Ignite-Integration-tp8556p9137.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Apache Spark & Ignite Integration

2016-11-21 Thread pragmaticbigdata
. Kindly let me know. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-Spark-Ignite-Integration-tp8556p9121.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Apache Spark & Ignite Integration

2016-11-21 Thread vkulichenko
in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-Spark-Ignite-Integration-tp8556p9114.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Apache Spark & Ignite Integration

2016-11-21 Thread pragmaticbigdata
e runs in standalone mode, right? i.e When spark fetches the ignite cache through the IgniteRDD api (val igniteRDD = igniteContext.fromCache("") a copy of data is created in the spark worker's memory. Thanks. -- View this message in context: http://apache-ignite-users.70518.x6.na

Re: Apache Spark & Ignite Integration

2016-11-18 Thread vkulichenko
APIs, you can always combine - use Ignite to update the data in transactional manner and Spark for analytics. -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-Spark-Ignite-Integration-tp8556p9093.html Sent from the Apache Ignite Users mailing list

Re: Apache Spark & Ignite Integration

2016-11-18 Thread pragmaticbigdata
te is to easily share data across spark jobs where ignite is used for in-memory data storage and spark for its high speed data processing capabilities. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-Spark-Ignite-Integration-tp8556p9082.html Sent from the Apache

Re: Apache Spark & Ignite Integration

2016-11-17 Thread vkulichenko
with Spark? -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-Spark-Ignite-Integration-tp8556p9062.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Apache Spark & Ignite Integration

2016-11-17 Thread pragmaticbigdata
e flow correct? If so, how do I maintain transaction isolation when other spark jobs try to read the data from ignite in parallel? Thanks. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-Spark-Ignite-Integration-tp8556p9047.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Apache Spark & Ignite Integration

2016-11-16 Thread vkulichenko
/query/annotations/QuerySqlFunction.html -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-Spark-Ignite-Integration-tp8556p9027.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Apache Spark & Ignite Integration

2016-11-16 Thread pragmaticbigdata
8.x6.nabble.com/Apache-Spark-Ignite-Integration-tp8556p9019.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Apache Spark & Ignite Integration

2016-11-15 Thread vkulichenko
executor). This should minimize data movement. 2. For this you will have to use Ignite API directly. You can use IgniteContext.ignite() to get the local client on the driver. -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-Spark-Ignite-Integration

Re: Apache Spark & Ignite Integration

2016-11-15 Thread pragmaticbigdata
e.com/Apache-Spark-Ignite-Integration-tp8556p8984.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Apache Spark & Ignite Integration

2016-11-14 Thread Denis Magda
any. > > 2. Since spark does not have transaction support, how I can use the ACID > transaction support that Ignite provides when updating RDD's? A code example > would be helpful if possible. > > Thanks. > > > > -- > View this message in context: > http://a

Re: Apache Spark & Ignite Integration

2016-11-14 Thread pragmaticbigdata
ction support that Ignite provides when updating RDD's? A code example would be helpful if possible. Thanks. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-Spark-Ignite-Integration-tp8556p8951.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Apache Spark & Ignite Integration

2016-10-27 Thread vkulichenko
on IgniteRDD. -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-Spark-Ignite-Integration-tp8556p8568.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Apache Spark & Ignite Integration

2016-10-27 Thread pragmaticbigdata
spark dataset take the advantage of these indexes when queries are executed on it? Do I need to do anything explicit for that other than enabling the indexes while defining IgniteConfiguration? Thanks! -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-Spark