Hello, Valentin. Did you have a chance to look at my changes?
Now I think I have done almost all required features. I want to make some performance test to ensure my implementation work properly with a significant amount of data. And I definitely need some feedback for my changes. 2017-10-09 18:45 GMT+03:00 Николай Ижиков <nizhikov....@gmail.com>: > Hello, guys. > > Which version of Spark do we want to use? > > 1. Currently, Ignite depends on Spark 2.1.0. > > * Can be run on JDK 7. > * Still supported: 2.1.2 will be released soon. > > 2. Latest Spark version is 2.2.0. > > * Can be run only on JDK 8+ > * Released Jul 11, 2017. > * Already supported by huge vendors(Amazon for example). > > Note that in IGNITE-3084 I implement some internal Spark API. > So It will take some effort to switch between Spark 2.1 and 2.2 > > > 2017-09-27 2:20 GMT+03:00 Valentin Kulichenko < > valentin.kuliche...@gmail.com>: > >> I will review in the next few days. >> >> -Val >> >> On Tue, Sep 26, 2017 at 2:23 PM, Denis Magda <dma...@apache.org> wrote: >> >> > Hello Nikolay, >> > >> > This is good news. Finally this capability is coming to Ignite. >> > >> > Val, Vladimir, could you do a preliminary review? >> > >> > Answering on your questions. >> > >> > 1. Yardstick should be enough for performance measurements. As a Spark >> > user, I will be curious to know what’s the point of this integration. >> > Probably we need to compare Spark + Ignite and Spark + Hive or Spark + >> > RDBMS cases. >> > >> > 2. If Spark community is reluctant let’s include the module in >> > ignite-spark integration. >> > >> > — >> > Denis >> > >> > > On Sep 25, 2017, at 11:14 AM, Николай Ижиков <nizhikov....@gmail.com> >> > wrote: >> > > >> > > Hello, guys. >> > > >> > > Currently, I’m working on integration between Spark and Ignite [1]. >> > > >> > > For now, I implement following: >> > > * Ignite DataSource implementation(IgniteRelationProvider) >> > > * DataFrame support for Ignite SQL table. >> > > * IgniteCatalog implementation for a transparent resolving of >> ignites >> > > SQL tables. >> > > >> > > Implementation of it can be found in PR [2] >> > > It would be great if someone provides feedback for a prototype. >> > > >> > > I made some examples in PR so you can see how API suppose to be used >> [3]. >> > > [4]. >> > > >> > > I need some advice. Can you help me? >> > > >> > > 1. How should this PR be tested? >> > > >> > > Of course, I need to provide some unit tests. But what about >> scalability >> > > tests, etc. >> > > Maybe we need some Yardstick benchmark or similar? >> > > What are your thoughts? >> > > Which scenarios should I consider in the first place? >> > > >> > > 2. Should we provide Spark Catalog implementation inside Ignite >> codebase? >> > > >> > > A current implementation of Spark Catalog based on *internal Spark >> API*. >> > > Spark community seems not interested in making Catalog API public or >> > > including Ignite Catalog in Spark code base [5], [6]. >> > > >> > > *Should we include Spark internal API implementation inside Ignite >> code >> > > base?* >> > > >> > > Or should we consider to include Catalog implementation in some >> external >> > > module? >> > > That will be created and released outside Ignite?(we still can support >> > and >> > > develop it inside Ignite community). >> > > >> > > [1] https://issues.apache.org/jira/browse/IGNITE-3084 >> > > [2] https://github.com/apache/ignite/pull/2742 >> > > [3] https://github.com/apache/ignite/pull/2742/files#diff- >> > > f4ff509cef3018e221394474775e0905 >> > > [4] https://github.com/apache/ignite/pull/2742/files#diff- >> > > f2b670497d81e780dfd5098c5dd8a89c >> > > [5] http://apache-spark-developers-list.1001551.n3. >> > > nabble.com/Spark-Core-Custom-Catalog-Integration-between- >> > > Apache-Ignite-and-Apache-Spark-td22452.html >> > > [6] https://issues.apache.org/jira/browse/SPARK-17767 >> > > >> > > -- >> > > Nikolay Izhikov >> > > nizhikov....@gmail.com >> > >> > >> > > > > -- > Nikolay Izhikov > nizhikov....@gmail.com > -- Nikolay Izhikov nizhikov....@gmail.com
