Re: Apache Ignite 3 Alpha 2 webinar follow up questions

[Courtney Robinson]

Hi Ivan,
Atri's description of the query plan being cached is what I was thinking of
with my description.

I lack the knowledge on how the statistics are maintained to really comment
constructively Atri but my first question about the problem you raise with
statistics would be:

How/where are the stats maintained and if a query plan is cached based on
some stats, is it not possible to invalidate the cached plan periodically
or based on statistics changes?

Regards,
Courtney Robinson
Founder and CEO, Hypi
Tel: ++44 208 123 2413 (GMT+0) 


https://hypi.io


On Sat, Jul 31, 2021 at 8:54 AM Atri Sharma  wrote:

> Query caching works on three levels - caching results, caching blocks and
> caching query plans.
>
> Prepared queries work by caching a plan for a query and reusing that plan
> by changing the parameters for the incoming query. So the query remains the
> same, but input values keep changing.
>
> The problem with prepared queries is that query execution can go bad very
> fast if the underlying data distribution changes and the cached plan is no
> longer optimal for the given statistics.
>
> On Sat, 31 Jul 2021, 12:54 Ivan Pavlukhin,  wrote:
>
> > Hi Courtney,
> >
> > Please clarify what do you mean by prepared queries and query caching?
> > Do you mean caching query results? If so, in my mind material views
> > are the best approach here (Ignite 2 does not support them). Do you
> > have other good approaches in your mind? E.g. implemented in other
> > databases.
> >
> > 2021-07-26 21:27 GMT+03:00, Valentin Kulichenko <
> > valentin.kuliche...@gmail.com>:
> > > Hi Courtney,
> > >
> > > Generally speaking, query caching certainly makes sense. As far as I
> > know,
> > > Ignite 2.x actually does that, but most likely there might be room for
> > > improvement as well. We will look into this.
> > >
> > > As for the SQL API - the answer is yes. The requirement for a dummy
> cache
> > > is an artifact of the current architecture. This is 100% wrong and will
> > be
> > > changed in 3.0.
> > >
> > > -Val
> > >
> > > On Sun, Jul 25, 2021 at 2:51 PM Courtney Robinson
> > > 
> > > wrote:
> > >
> > >> Something else came to mind, are there plans to support prepared
> > queries?
> > >>
> > >> I recall someone saying before that Ignite does internally cache
> queries
> > >> but it's not at all clear if or how it does do that. I assume a simple
> > >> hash
> > >> of the query isn't enough.
> > >>
> > >> We generate SQL queries based on user runtime settings and they can
> get
> > >> to
> > >> hundreds of lines long, I imagine this means most of our queries are
> not
> > >> being cached but there are patterns so we could generate and manage
> > >> prepared queries ourselves.
> > >>
> > >> Also, will there be a dedicated API for doing SQL queries rather than
> > >> having to pass a SqlFieldsQuery to a cache that has nothing to do with
> > >> the
> > >> cache being queried? When I first started with Ignite years ago, this
> > was
> > >> beyond confusing for me. I'm trying to run select x from B but I pass
> > >> this
> > >> to a cache called DUMMY or whatever arbitrary name...
> > >>
> > >> On Fri, Jul 23, 2021 at 4:05 PM Courtney Robinson <
> > >> courtney.robin...@hypi.io>
> > >> wrote:
> > >>
> > >> > Andrey,
> > >> > Thanks for the response - see my comments inline.
> > >> >
> > >> >
> > >> >> I've gone through the questions and have no the whole picture of
> your
> > >> use
> > >> >> case.
> > >> >
> > >> > Would you please clarify how you exactly use the Ignite? what are
> the
> > >> >> integration points?
> > >> >>
> > >> >
> > >> > I'll try to clarify - we have a low/no code platform. A user
> designs a
> > >> > model for their application and we map this model to Ignite tables
> and
> > >> > other data sources. The model I'll describe is what we're building
> now
> > >> and
> > >> > expected to be in alpha some time in Q4 21. Our current production
> > >> > architecture is different and isn't as generic, it is heavily tied
> to
> > >> > Ignite and we've redesigned to get some flexibility where Ignite
> > >> > doesn't
> > >> > provide what we want. Things like window functions and other SQL-99
> > >> limits.
> > >> >
> > >> > In the next gen version we're working on you can create a model for
> a
> > >> > Tweet(content, to) and we will create an Ignite table with content
> and
> > >> > to
> > >> > columns using the type the user selects. This is the simplest case.
> > >> > We are adding generic support for sources and sinks and using
> Calcite
> > >> > as
> > >> a
> > >> > data virtualisation layer. Ignite is one of the available
> > source/sinks.
> > >> >
> > >> > When a user creates a model for Tweet, we also allow them to specify
> > >> > how
> > >> > they want to index the data. We have a copy of the calcite
> > >> > Elasticsearch
> > >> > adapter modified for Solr.
> > >> >
> > >> > When a source is queried (Ignite or any other that we support), we
> > >> > generate SQL 

[MTCGA]: new failures in builds [6106850] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 *Test with high flaky rate in master 
IgniteClusterSnapshotSelfTest.testClusterSnapshotWithRebalancing 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-321293767845540479=%3Cdefault%3E=testDetails
 No changes in the build

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 10:59:26 31-07-2021 

Re: Apache Ignite 3 Alpha 2 webinar follow up questions

[Atri Sharma]

Query caching works on three levels - caching results, caching blocks and
caching query plans.

Prepared queries work by caching a plan for a query and reusing that plan
by changing the parameters for the incoming query. So the query remains the
same, but input values keep changing.

The problem with prepared queries is that query execution can go bad very
fast if the underlying data distribution changes and the cached plan is no
longer optimal for the given statistics.

On Sat, 31 Jul 2021, 12:54 Ivan Pavlukhin,  wrote:

> Hi Courtney,
>
> Please clarify what do you mean by prepared queries and query caching?
> Do you mean caching query results? If so, in my mind material views
> are the best approach here (Ignite 2 does not support them). Do you
> have other good approaches in your mind? E.g. implemented in other
> databases.
>
> 2021-07-26 21:27 GMT+03:00, Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
> > Hi Courtney,
> >
> > Generally speaking, query caching certainly makes sense. As far as I
> know,
> > Ignite 2.x actually does that, but most likely there might be room for
> > improvement as well. We will look into this.
> >
> > As for the SQL API - the answer is yes. The requirement for a dummy cache
> > is an artifact of the current architecture. This is 100% wrong and will
> be
> > changed in 3.0.
> >
> > -Val
> >
> > On Sun, Jul 25, 2021 at 2:51 PM Courtney Robinson
> > 
> > wrote:
> >
> >> Something else came to mind, are there plans to support prepared
> queries?
> >>
> >> I recall someone saying before that Ignite does internally cache queries
> >> but it's not at all clear if or how it does do that. I assume a simple
> >> hash
> >> of the query isn't enough.
> >>
> >> We generate SQL queries based on user runtime settings and they can get
> >> to
> >> hundreds of lines long, I imagine this means most of our queries are not
> >> being cached but there are patterns so we could generate and manage
> >> prepared queries ourselves.
> >>
> >> Also, will there be a dedicated API for doing SQL queries rather than
> >> having to pass a SqlFieldsQuery to a cache that has nothing to do with
> >> the
> >> cache being queried? When I first started with Ignite years ago, this
> was
> >> beyond confusing for me. I'm trying to run select x from B but I pass
> >> this
> >> to a cache called DUMMY or whatever arbitrary name...
> >>
> >> On Fri, Jul 23, 2021 at 4:05 PM Courtney Robinson <
> >> courtney.robin...@hypi.io>
> >> wrote:
> >>
> >> > Andrey,
> >> > Thanks for the response - see my comments inline.
> >> >
> >> >
> >> >> I've gone through the questions and have no the whole picture of your
> >> use
> >> >> case.
> >> >
> >> > Would you please clarify how you exactly use the Ignite? what are the
> >> >> integration points?
> >> >>
> >> >
> >> > I'll try to clarify - we have a low/no code platform. A user designs a
> >> > model for their application and we map this model to Ignite tables and
> >> > other data sources. The model I'll describe is what we're building now
> >> and
> >> > expected to be in alpha some time in Q4 21. Our current production
> >> > architecture is different and isn't as generic, it is heavily tied to
> >> > Ignite and we've redesigned to get some flexibility where Ignite
> >> > doesn't
> >> > provide what we want. Things like window functions and other SQL-99
> >> limits.
> >> >
> >> > In the next gen version we're working on you can create a model for a
> >> > Tweet(content, to) and we will create an Ignite table with content and
> >> > to
> >> > columns using the type the user selects. This is the simplest case.
> >> > We are adding generic support for sources and sinks and using Calcite
> >> > as
> >> a
> >> > data virtualisation layer. Ignite is one of the available
> source/sinks.
> >> >
> >> > When a user creates a model for Tweet, we also allow them to specify
> >> > how
> >> > they want to index the data. We have a copy of the calcite
> >> > Elasticsearch
> >> > adapter modified for Solr.
> >> >
> >> > When a source is queried (Ignite or any other that we support), we
> >> > generate SQL that Calcite executes. Calcite will push down the
> >> > generated
> >> > queries to Solr and Solr produces a list of IDs (in case of Ignite)
> and
> >> we
> >> > do a multi-get from Ignite to produce the actual results.
> >> >
> >> > Obviously there's a lot more to this but that should give you a
> general
> >> > idea.
> >> >
> >> > and maybe share some experience with using Ignite SPIs?
> >> >>
> >> > Our evolution with Ignite started from the key value + compute APIs.
> We
> >> > used the SPIs then but have since moved to using only the Ignite SQL
> >> > API
> >> > (we gave up transactions for this).
> >> >
> >> > We originally used the indexing SPI to keep our own lucene index of
> >> > data
> >> > in a cache. We did not use the Ignite FTS as it is very limited
> >> > compared
> >> to
> >> > what we allow customers to do. If I remember correctly, we were using
> >> > an
> >> 

Re: MTCGA site is down

[Ivan Pavlukhin]

Folks,

Am I getting it right that MTCGA bot IP addresses are changed from
time to time? If so, can we use something like CNAME [1] to alias
mtcga.ignite.apache.org to mtcga.gridgain.com?

What do you think?

[1] https://en.wikipedia.org/wiki/CNAME_record

2021-07-28 11:13 GMT+03:00, Alex Plehanov :
> Petr, thank you!
>
> It's working now.
>
> ср, 28 июл. 2021 г. в 11:03, Petr Ivanov :
>
>> I've created issue in INFRA project about this matter [1].
>>
>> Please, stand by,
>>
>>
>>
>> [1] https://issues.apache.org/jira/browse/INFRA-22150
>>
>> > On 28 Jul 2021, at 09:42, Alex Plehanov 
>> > wrote:
>> >
>> > Hello, Igniters!
>> >
>> > Looks like the MTCGA site hosted on Apache Ignite domain [1] is
>> > currently
>> > down (at least since yesterday), however, this site hosted on GridGain
>> > domain [2] is working.
>> > Can anyone have a look what's happening?
>> >
>> > [1] : mtcga.ignite.apache.org
>> > [2] : mtcga.gridgain.com
>>
>>
>


-- 

Best regards,
Ivan Pavlukhin

Re: Apache Ignite 3 Alpha 2 webinar follow up questions

[Ivan Pavlukhin]

Hi Courtney,

Please clarify what do you mean by prepared queries and query caching?
Do you mean caching query results? If so, in my mind material views
are the best approach here (Ignite 2 does not support them). Do you
have other good approaches in your mind? E.g. implemented in other
databases.

2021-07-26 21:27 GMT+03:00, Valentin Kulichenko :
> Hi Courtney,
>
> Generally speaking, query caching certainly makes sense. As far as I know,
> Ignite 2.x actually does that, but most likely there might be room for
> improvement as well. We will look into this.
>
> As for the SQL API - the answer is yes. The requirement for a dummy cache
> is an artifact of the current architecture. This is 100% wrong and will be
> changed in 3.0.
>
> -Val
>
> On Sun, Jul 25, 2021 at 2:51 PM Courtney Robinson
> 
> wrote:
>
>> Something else came to mind, are there plans to support prepared queries?
>>
>> I recall someone saying before that Ignite does internally cache queries
>> but it's not at all clear if or how it does do that. I assume a simple
>> hash
>> of the query isn't enough.
>>
>> We generate SQL queries based on user runtime settings and they can get
>> to
>> hundreds of lines long, I imagine this means most of our queries are not
>> being cached but there are patterns so we could generate and manage
>> prepared queries ourselves.
>>
>> Also, will there be a dedicated API for doing SQL queries rather than
>> having to pass a SqlFieldsQuery to a cache that has nothing to do with
>> the
>> cache being queried? When I first started with Ignite years ago, this was
>> beyond confusing for me. I'm trying to run select x from B but I pass
>> this
>> to a cache called DUMMY or whatever arbitrary name...
>>
>> On Fri, Jul 23, 2021 at 4:05 PM Courtney Robinson <
>> courtney.robin...@hypi.io>
>> wrote:
>>
>> > Andrey,
>> > Thanks for the response - see my comments inline.
>> >
>> >
>> >> I've gone through the questions and have no the whole picture of your
>> use
>> >> case.
>> >
>> > Would you please clarify how you exactly use the Ignite? what are the
>> >> integration points?
>> >>
>> >
>> > I'll try to clarify - we have a low/no code platform. A user designs a
>> > model for their application and we map this model to Ignite tables and
>> > other data sources. The model I'll describe is what we're building now
>> and
>> > expected to be in alpha some time in Q4 21. Our current production
>> > architecture is different and isn't as generic, it is heavily tied to
>> > Ignite and we've redesigned to get some flexibility where Ignite
>> > doesn't
>> > provide what we want. Things like window functions and other SQL-99
>> limits.
>> >
>> > In the next gen version we're working on you can create a model for a
>> > Tweet(content, to) and we will create an Ignite table with content and
>> > to
>> > columns using the type the user selects. This is the simplest case.
>> > We are adding generic support for sources and sinks and using Calcite
>> > as
>> a
>> > data virtualisation layer. Ignite is one of the available source/sinks.
>> >
>> > When a user creates a model for Tweet, we also allow them to specify
>> > how
>> > they want to index the data. We have a copy of the calcite
>> > Elasticsearch
>> > adapter modified for Solr.
>> >
>> > When a source is queried (Ignite or any other that we support), we
>> > generate SQL that Calcite executes. Calcite will push down the
>> > generated
>> > queries to Solr and Solr produces a list of IDs (in case of Ignite) and
>> we
>> > do a multi-get from Ignite to produce the actual results.
>> >
>> > Obviously there's a lot more to this but that should give you a general
>> > idea.
>> >
>> > and maybe share some experience with using Ignite SPIs?
>> >>
>> > Our evolution with Ignite started from the key value + compute APIs. We
>> > used the SPIs then but have since moved to using only the Ignite SQL
>> > API
>> > (we gave up transactions for this).
>> >
>> > We originally used the indexing SPI to keep our own lucene index of
>> > data
>> > in a cache. We did not use the Ignite FTS as it is very limited
>> > compared
>> to
>> > what we allow customers to do. If I remember correctly, we were using
>> > an
>> > affinity compute job to send queries to the right Ignite node and
>> > then doing a multi-get to pull the data from caches.
>> > I think we used one or two other SPIs and we found them very useful to
>> > be
>> > able to extend and customise Ignite without having to fork/change
>> upstream
>> > classes. We only stopped using them because we eventually concluded
>> > that
>> > using the SQL only API was better for numerous reasons.
>> >
>> >
>> >> We'll keep the information in mind while developing the Ignite,
>> >> because this may help us to make a better product.
>> >>
>> >> By the way, I'll try to answer the questions.
>> >>
>> >> >   1. Schema change - does that include the ability to change the
>> >> > types
>> >> of
>> >> >   fields/columns?
>> >> Yes, we plan to support transparent