Great! Let me know if you need any assistance and/or intermediate review.
-Val
On Thu, Nov 30, 2017 at 12:05 AM, Николай Ижиков
wrote:
> Valentin,
>
> > Can you please create a separate ticket for the strategy implementation
> then?
>
> Done.
>
> https://issues.apache.org/jira/browse/IGNITE-707
Valentin,
> Can you please create a separate ticket for the strategy implementation then?
Done.
https://issues.apache.org/jira/browse/IGNITE-7077
> Any idea on how long will it take?
I think it will take 2-4 weeks to implement such a strategy.
I try my best to make a ready to review PR before
Nikolay,
Can you please create a separate ticket for the strategy implementation
then? Any idea on how long will it take?
As for querying a partition, both SqlQuery and SqlFieldQuery allow to
specify set of partitions to work with (see setPartitions method). I think
that should be enough.
-Val
Hi Nikolay,
No, it is not possible to get this info from public API, neither we planned
to expose it. See IGNITE-4509 and commit *fbf0e353* to get better
understanding on how this was implemented.
Vladimir.
On Wed, Nov 29, 2017 at 2:01 PM, Николай Ижиков
wrote:
> Hello, Vladimir.
>
> > partiti
Valentin,
> process the AST generated by Spark and convert it to Ignite SQL...
> Does it make sense to you?
Yes.
I think it is a great approach.
Let's implement such feature as the second step of Data Frame integration.
2017-11-29 3:23 GMT+03:00 Valentin Kulichenko :
> Nikolay,
>
> Custom stra
Hello, Vladimir.
> partition pruning is already implemented in Ignite, so there is no need
to do this on your own.
Spark work with partitioned data set.
It is required to provide data partition information to Spark from custom
Data Source(Ignite).
Can I get information about pruned partitions th
Nikolay,
Regarding p3. - partition pruning is already implemented in Ignite, so
there is no need to do this on your own.
On Wed, Nov 29, 2017 at 3:23 AM, Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:
> Nikolay,
>
> Custom strategy allows to fully process the AST generated by Spark
Nikolay,
Custom strategy allows to fully process the AST generated by Spark and
convert it to Ignite SQL, so there will be no execution on Spark side at
all. This is what we are trying to achieve here. Basically, one will be
able to use DataFrame API to execute queries directly on Ignite. Does it
Hello, guys.
I have implemented basic support of Spark Data Frame API [1], [2] for Ignite.
Spark provides API for a custom strategy to optimize queries from spark to
underlying data source(Ignite).
The goal of optimization(obvious, just to be on the same page):
Minimize data transfer between Sp