Thanks Val for the info on indexes with DF. Do you know if adding index /
affinitykeys on the cache help with the join, when the IgniteRDD is joined with
a spark DF? The below from docs say that
“IgniteRDD also provides affinity information to Spark via
getPrefferredLocations method so that RDD computations use data locality.”
I was wondering, if the affinitykey on the cache can be utilized in the spark
join?
On 1/3/18, 12:27 PM, "vkulichenko" <valentin.kuliche...@gmail.com> wrote:
Indexes would not be used during joins, at least in current implementation.
Current integration is implemented as a regular Spark data source which
provides each relation separately. Spark then performs join by itself, so
Ignite indexes do not help.
The easiest way to get binaries would be to use a nightly build [1] , but it
seems to be broken for some reason (latest is from May 31). I guess the only
option at the moment is to build from source.
[1]
https://builds.apache.org/view/H-L/view/Ignite/job/Ignite-nightly/lastSuccessfulBuild/
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
