Re: Entries get deleted from Ignite Cache

[Denis Magda]

It sounds like your persistence volumes are reset upon restarts. Please
double check that persistence is mapped to a permanent location:
https://apacheignite.readme.io/docs/stateful-deployment#section-separate-disk-for-wal

-
Denis


On Wed, Dec 11, 2019 at 10:33 AM naidusm  wrote:

> We have a three-node ignite cluster running inside Kubernetes with native
> persistence enabled.
>
> We have an application which uses ignite cache in client mode. And the
> application as well running in Kubernetes scaled to 3 pods.
>
> Recently we have started to observe that when we scale down the application
> to 0 pods at once and scale up again to 3, then we observe that some of the
> entries from the cache are lost.
>
> We see the cache entries are present in the cache after scaling down and
> before scaling up the application again to 3. But after scaling up, entries
> are getting deleted.
>
> It would be very helpful if you can suggest what needs to be done here to
> fix this issue.
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

Entries get deleted from Ignite Cache

[naidusm]

We have a three-node ignite cluster running inside Kubernetes with native
persistence enabled.

We have an application which uses ignite cache in client mode. And the
application as well running in Kubernetes scaled to 3 pods.

Recently we have started to observe that when we scale down the application
to 0 pods at once and scale up again to 3, then we observe that some of the
entries from the cache are lost.

We see the cache entries are present in the cache after scaling down and
before scaling up the application again to 3. But after scaling up, entries
are getting deleted.

It would be very helpful if you can suggest what needs to be done here to
fix this issue.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: [DISCUSS] dependencies and release process for Ignite Extensions

[Saikat Maitra]

Hi Ivan,

Thank you for sharing your feedback. I will try to install all modules in
local agent's repository and run the build.

I will share updates.

Regards,
Saikat

On Tue, Dec 10, 2019 at 9:56 AM Ivan Pavlukhin  wrote:

> Saikat,
>
> Let me share my understanding how do we run the majority of our jobs
> executing test suites. Hope it will help.
> 1. ~Build Apache Ignite~ in a separate run and publish everything as a
> single artifact ignite.zip (it contains ignite working directory with
> compiled modules, e.g. modules/core/target and others).
> 2. Download and extract ignite.zip and run particular test suites
> against extracted files.
>
> In a such setup test suites run the same as you do it on your local
> machine and it always works with a single multi-module maven project
> (apache-ignite). So, all dependencies are resolved in the working
> directory.
>
> For a separate IgniteExtensions maven project you need to install all
> dependencies from downloaded ignite.zip into .m2/repository local to
> the agent. I suppose we can do it as follows:
> 1. Download ignite.zip and extract it into "ignite" subfolder.
> 2. Run a maven command installing all modules into local maven
> repository. Something like "mvn install" but here we need to make sure
> that nothing is compiled once again.
> 3. Build IgniteExtensions.
>
> There might be a better way to do the same, I just tried to describe a
> schema how it can be done.
>
> вт, 10 дек. 2019 г. в 06:30, Saikat Maitra :
> >
> > Hi Ivan, Ilya
> >
> > Thank you for your reply. I have added you as dev in IgniteExtensions
> > project and you should be able to check the build configurations.
> >
> > Yes, when I set artifact dependencies for ~Build Apache Ignite~ similar
> to
> > other projects then I receive below error
> >
> > [ERROR] [ERROR] Could not find the selected project in the reactor:
> > :ignite-flink-ext @
> >
> > I do not see the same error when I am running the build in local and
> > maven package command can still get 2.9.0-SNAPSHOT dependencies from my
> > local .m2 repository but it is failing in teamcity.
> >
> > Regards,
> > Saikat
> >
> >
> >
> >
> > On Mon, Dec 9, 2019 at 5:18 AM Ilya Kasnacheev <
> ilya.kasnach...@gmail.com>
> > wrote:
> >
> > > Hello!
> > >
> > > I think that your build should depend on an Apache Ignite build(s) or
> just
> > > use an already released version.
> > >
> > > In the same fashion as all our teamcity depend on "Build Apache
> Ignite" and
> > > use its artifacts.
> > >
> > > Here you want Ignite snapshot but Teamcity does not know where to take
> it
> > > from.
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > пн, 9 дек. 2019 г. в 06:40, Saikat Maitra :
> > >
> > > > Hello,
> > > >
> > > > I am running into a problem specific to teamcity build for Ignite
> > > > Extensions project. When I set the dependencies to 2.9.0-SNAPSHOT I
> am
> > > > getting an error message during build as below
> > > >
> > > > [06:24:12][Step 4/5] Failed to execute goal on project
> ignite-flink-ext:
> > > > Could not resolve dependencies for project
> > > > org.apache.ignite.ext:ignite-flink-ext:jar:1.0.0-SNAPSHOT: The
> following
> > > > artifacts could not be resolved:
> > > > org.apache.ignite:ignite-core:jar:2.9.0-SNAPSHOT,
> > > > org.apache.ignite:ignite-core:jar:tests:2.9.0-SNAPSHOT,
> > > > org.apache.ignite:ignite-log4j:jar:2.9.0-SNAPSHOT,
> > > > org.apache.ignite:ignite-spring:jar:2.9.0-SNAPSHOT: Could not find
> > > artifact
> > > > org.apache.ignite:ignite-core:jar:2.9.0-SNAPSHOT in h2database.com (
> > > > https://h2database.com/m2-repo)
> > > >
> > > >
> > > > and if set artifact dependencies for ~Build Apache Ignite~ then I
> receive
> > > > below error
> > > >
> > > > [ERROR] [ERROR] Could not find the selected project in the reactor:
> > > > :ignite-flink-ext @
> > > >
> > > > Build url
> > > >
> > > >
> > >
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteExtensions_Build&branch_IgniteExtensions=pull%2F1%2Fhead&tab=buildTypeStatusDiv
> > > >
> > > > Can you please let me know if you faced similar problem with teamcity
> > > > build?
> > > >
> > > > I can set Ignite Extensions dependencies to released ignite-core
> > > artifacts
> > > > version like 2.7.6 and build works fine.
> > > >
> > > > Regards,
> > > > Saikat
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Sat, Nov 30, 2019 at 1:50 PM Saikat Maitra <
> saikat.mai...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello Denis,
> > > > >
> > > > > Thank you for your email and sharing your thoughts on the release
> > > > process.
> > > > > I will update the artifact id and dependencies for
> ignite-extensions
> > > > > accordingly.
> > > > >
> > > > > I had created Ignite-Extensions project as a root level project
> and in
> > > > > teamcity I was facing issues pulling dependencies for
> 2.8.0-SNAPSHOT
> > > > > whereas I was able to pull dependencies for ignite-core 2.7.6 from
> > > maven
> > > > > central. I will look into it furthe

[jira] [Created] (IGNITE-12431) Allow to set inline size for implicit indexes per table

[Mikhail Cherkasov (Jira)]

Mikhail Cherkasov created IGNITE-12431:
--

 Summary: Allow to set inline size for implicit indexes per table
 Key: IGNITE-12431
 URL: https://issues.apache.org/jira/browse/IGNITE-12431
 Project: Ignite
  Issue Type: Bug
  Components: sql
Reporter: Mikhail Cherkasov


Right now you can specify inline size only for explicit indexes, but there's no 
way you can set it for implicit caches, except global flag: 
IGNITE_MAX_INDEX_PAYLOAD_SIZE or per cache:
[https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/CacheConfiguration.html#setSqlIndexMaxInlineSize-int-]
 however, if there's only one table which requires a big inline for implicit 
indices is too big overhead to specify it for the whole cache.
Let's introduce some way to do this per table.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: BinaryObject API confuses users

[Ilya Kasnacheev]

Hello!

I think we should check that setField happily accepts both BinaryObject and
BinaryObjectBuilder, and then deprecate this method.

Regards,
-- 
Ilya Kasnacheev


пт, 6 дек. 2019 г. в 16:02, Alexei Scherbakov :

> Looks like it's already possible to pass binary object as value using
> BinaryObjectBuilder setField(String name, Object val);
>
> If it works we can remove a signature with BinaryObjectBuilder.
>
> The API is truly confusing.
>
> пт, 6 дек. 2019 г. в 15:55, Николай Ижиков :
>
> > I’m too, Alex.
> >
> > But, this signature leads to the same error as I mentioned.
> >
> >
> >
> > > 6 дек. 2019 г., в 15:53, Alexei Scherbakov <
> alexey.scherbak...@gmail.com>
> > написал(а):
> > >
> > > I wonder why signature is not
> > >
> > > setField(String name, BinaryObject obj)
> > >
> > > пт, 6 дек. 2019 г. в 15:00, Ilya Kasnacheev  >:
> > >
> > >> Hello!
> > >>
> > >> I think that repeating argument type name in method name is a code
> > smell.
> > >> Rather, we should describe how this method is different from other
> > methods
> > >> of its bunch.
> > >>
> > >> In this case, it is different since it allows you to create nested
> > binary
> > >> objects, i.e., ones with non-flat structure.
> > >>
> > >> Regards,
> > >> --
> > >> Ilya Kasnacheev
> > >>
> > >>
> > >> пт, 6 дек. 2019 г. в 13:45, Николай Ижиков :
> > >>
> > >>> Hello, Ilya.
> > >>>
> > >>> I don’t get your point
> > >>>
> >  We don’t passing … binary, just a … BinaryObjeсtBuilder.
> > >>>
> > >>> May be we should go with `setBinaryObjectBuilderField` or
> > >>> `setBinaryObjectField`?
> > >>>
> > >>>
> >  6 дек. 2019 г., в 13:38, Ilya Kasnacheev  >
> > >>> написал(а):
> > 
> >  Hello!
> > 
> >  I can see where you are getting at, can we call it "setFieldNested"
> > >>> instead
> >  or something like that?
> > 
> >  setBinaryField is confusing because we're not passing anything
> > >> especially
> >  binary, just a nested BinaryObjectBuilder.
> > 
> >  Regards,
> >  --
> >  Ilya Kasnacheev
> > 
> > 
> >  пт, 6 дек. 2019 г. в 13:17, Николай Ижиков :
> > 
> > > Hello, Igniters.
> > >
> > > We have confusing API in `BinaryObjectBuilder` class.
> > >
> > > The code below leads to the `ClassCastException`
> > > The cause is java method resolution rules [1]
> > >
> > >> There may be more than one such method, in which case the most
> > >> specific
> > > one is chosen
> > >
> > > I suggest to deprecate `setField(String name, @Nullable
> > > BinaryObjectBuilder builder);` method and introduce
> > > `setBinaryField(String name, @Nullable BinaryObjectBuilder
> builder);`
> > > instead of it.
> > >
> > > What do you think?
> > >
> > > ```
> > > public class BugTest extends GridCommonAbstractTest {
> > >   @Test public void testBinaryObject() throws Exception {
> > >   try (Ignite ignite = startGrid(0)) {
> > >   BinaryObjectBuilder builder =
> > > ignite.binary().builder("testVal")
> > >   .setField("name", "John Doe", String.class);
> > >
> > >   builder.setField("name", builder.getField("name"));
> > >   }
> > >   }
> > > }
> > > ```
> > >
> > > [1]
> > >
> > >>>
> > >>
> >
> https://docs.oracle.com/javase/specs/jls/se11/html/jls-15.html#jls-15.12.2
> > >>>
> > >>>
> > >>
> > >
> > >
> > > --
> > >
> > > Best regards,
> > > Alexei Scherbakov
> >
> >
>
> --
>
> Best regards,
> Alexei Scherbakov
>

Re: Adding support for Ignite secondary indexes to Apache Calcite planner

[Roman Kondakov]

Alexey,

from my point of view Drill's approach looks like somewhat a hack:
sortedness and index lookups added to a removed from the query plan by
the special rules (which look very messy and complicated). Compare it to
the Phoneix approach where index is added to optimizer as a sorted view
of a table.


-- 
Kind Regards
Roman Kondakov


On 10.12.2019 17:44, Alexey Zinoviev wrote:
> I'd like Drill approach, worked and debugged with something similar, it's
> more easy to support
> 
> 
> Buuut, you have an implemented prototype (it votes for Phoenix in my mind)
> 
> вт, 10 дек. 2019 г. в 17:19, Vladimir Ozerov :
> 
>> Hi Roman,
>>
>> Why do you think that Drill-style will not let you exploit collation?
>> Collation should be propagated from the index scan in the same way as in
>> other sorted operators, such as merge join or streaming aggregate. Provided
>> that you use converter-hack (or any alternative solution to trigger parent
>> re-analysis).
>> In other words, propagation of collation from Drill-style indexes should be
>> no different from other sorted operators.
>>
>> Regards,
>> Vladimir.
>>
>> вт, 10 дек. 2019 г. в 16:40, Zhenya Stanilovsky >> :
>>
>>>
>>> Roman just as fast remark, Phoenix builds their approach on
>>> already existing monolith HBase architecture, most cases it`s just a stub
>>> for someone who wants use secondary indexes with a base with no
>>> native support of it. Don`t think it`s good idea here.
>>>


 --- Forwarded message ---
 From: "Roman Kondakov" < kondako...@mail.ru.invalid >
 To:  dev@ignite.apache.org
 Cc:
 Subject: Adding support for Ignite secondary indexes to Apache Calcite
 planner
 Date: Tue, 10 Dec 2019 15:55:52 +0300

 Hi all!

 As you may know there is an activity on integration of Apache Calcite
 query optimizer into Ignite codebase is being carried out [1],[2].

 One of a bunch of problems in this integration is the absence of
 out-of-the-box support for secondary indexes in Apache Calcite. After
 some research I came to conclusion that this problem has a couple of
 workarounds. Let's name them
 1. Phoenix-style approach - representing secondary indexes as
 materialized views which are natively supported by Calcite engine [3]
 2. Drill-style approach - pushing filters into the table scans and
 choose appropriate index for lookups when possible [4]

 Both these approaches have advantages and disadvantages:

 Phoenix style pros:
 - natural way of adding indexes as an alternative source of rows: index
 can be considered as a kind of sorted materialized view.
 - possibility of using index sortedness for stream aggregates,
 deduplication (DISTINCT operator), merge joins, etc.
 - ability to support other types of indexes (i.e. functional indexes).

 Phoenix style cons:
 - polluting optimizer's search space extra table scans hence increasing
 the planning time.

 Drill style pros:
 - easier to implement (although it's questionable).
 - search space is not inflated.

 Drill style cons:
 - missed opportunity to exploit sortedness.

 There is a good discussion about using both approaches can be found in
>>> [5].

 I made a small sketch [6] in order to demonstrate the applicability of
 the Phoenix approach to Ignite. Key design concepts are:
 1. On creating indexes are registered as tables in Calcite schema. This
 step is needed for internal Calcite's routines.
 2. On planner initialization we register these indexes as materialized
 views in Calcite's optimizer using VolcanoPlanner#addMaterialization
 method.
 3. Right before the query execution Calcite selects all materialized
 views (indexes) which can be potentially used in query.
 4. During the query optimization indexes are registered by planner as
 usual TableScans and hence can be chosen by optimizer if they have lower
 cost.

 This sketch shows the ability to exploit index sortedness only. So the
 future work in this direction should be focused on using indexes for
 fast index lookups. At first glance FilterableTable and
 FilterTableScanRule are good points to start. We can push Filter into
 the TableScan and then use FilterableTable for fast index lookups
 avoiding reading the whole index on TableScan step and then filtering
 its output on the Filter step.

 What do you think?



 [1]

>>>
>> http://apache-ignite-developers.2346864.n4.nabble.com/New-SQL-execution-engine-tt43724.html#none
 [2]

>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-37%3A+New+query+execution+engine
 [3]  https://issues.apache.org/jira/browse/PHOENIX-2047
 [4]  https://issues.apache.org/jira/browse/DRILL-6381
 [5]  https://issues.apache.org/jira/browse/DRILL-3929
 [6]  https://github.com/apache/ignite/pull/7

Re: Adding support for Ignite secondary indexes to Apache Calcite planner

[Roman Kondakov]

Hi Vladimir,

from what I understand, Drill does not exploit collation of indexes. To
be precise it does not exploit index collation in "natural" way where,
say, we a have sorted TableScan and hence we do not create a new Sort.
Instead of it Drill always create a Sort operator, but if TableScan can
be replaced with an IndexScan, this Sort operator is removed by the
dedicated rule.

Lets consider initial an operator tree:

Project
  Sort
TableScan

after applying rule DbScanToIndexScanPrule this tree will be converted to:

Project
  Sort
IndexScan

and finally, after applying DbScanSortRemovalRule we have:

Project
  IndexScan

while for Phoenix approach we would have two equivalent subsets in our
planner:

Project
  Sort
TableScan

and

Project
  IndexScan

and most likely the last plan  will be chosen as the best one.

-- 
Kind Regards
Roman Kondakov


On 10.12.2019 17:19, Vladimir Ozerov wrote:
> Hi Roman,
> 
> Why do you think that Drill-style will not let you exploit collation?
> Collation should be propagated from the index scan in the same way as in
> other sorted operators, such as merge join or streaming aggregate. Provided
> that you use converter-hack (or any alternative solution to trigger parent
> re-analysis).
> In other words, propagation of collation from Drill-style indexes should be
> no different from other sorted operators.
> 
> Regards,
> Vladimir.
> 
> вт, 10 дек. 2019 г. в 16:40, Zhenya Stanilovsky > :
> 
>>
>> Roman just as fast remark, Phoenix builds their approach on
>> already existing monolith HBase architecture, most cases it`s just a stub
>> for someone who wants use secondary indexes with a base with no
>> native support of it. Don`t think it`s good idea here.
>>
>>>
>>>
>>> --- Forwarded message ---
>>> From: "Roman Kondakov" < kondako...@mail.ru.invalid >
>>> To:  dev@ignite.apache.org
>>> Cc:
>>> Subject: Adding support for Ignite secondary indexes to Apache Calcite
>>> planner
>>> Date: Tue, 10 Dec 2019 15:55:52 +0300
>>>
>>> Hi all!
>>>
>>> As you may know there is an activity on integration of Apache Calcite
>>> query optimizer into Ignite codebase is being carried out [1],[2].
>>>
>>> One of a bunch of problems in this integration is the absence of
>>> out-of-the-box support for secondary indexes in Apache Calcite. After
>>> some research I came to conclusion that this problem has a couple of
>>> workarounds. Let's name them
>>> 1. Phoenix-style approach - representing secondary indexes as
>>> materialized views which are natively supported by Calcite engine [3]
>>> 2. Drill-style approach - pushing filters into the table scans and
>>> choose appropriate index for lookups when possible [4]
>>>
>>> Both these approaches have advantages and disadvantages:
>>>
>>> Phoenix style pros:
>>> - natural way of adding indexes as an alternative source of rows: index
>>> can be considered as a kind of sorted materialized view.
>>> - possibility of using index sortedness for stream aggregates,
>>> deduplication (DISTINCT operator), merge joins, etc.
>>> - ability to support other types of indexes (i.e. functional indexes).
>>>
>>> Phoenix style cons:
>>> - polluting optimizer's search space extra table scans hence increasing
>>> the planning time.
>>>
>>> Drill style pros:
>>> - easier to implement (although it's questionable).
>>> - search space is not inflated.
>>>
>>> Drill style cons:
>>> - missed opportunity to exploit sortedness.
>>>
>>> There is a good discussion about using both approaches can be found in
>> [5].
>>>
>>> I made a small sketch [6] in order to demonstrate the applicability of
>>> the Phoenix approach to Ignite. Key design concepts are:
>>> 1. On creating indexes are registered as tables in Calcite schema. This
>>> step is needed for internal Calcite's routines.
>>> 2. On planner initialization we register these indexes as materialized
>>> views in Calcite's optimizer using VolcanoPlanner#addMaterialization
>>> method.
>>> 3. Right before the query execution Calcite selects all materialized
>>> views (indexes) which can be potentially used in query.
>>> 4. During the query optimization indexes are registered by planner as
>>> usual TableScans and hence can be chosen by optimizer if they have lower
>>> cost.
>>>
>>> This sketch shows the ability to exploit index sortedness only. So the
>>> future work in this direction should be focused on using indexes for
>>> fast index lookups. At first glance FilterableTable and
>>> FilterTableScanRule are good points to start. We can push Filter into
>>> the TableScan and then use FilterableTable for fast index lookups
>>> avoiding reading the whole index on TableScan step and then filtering
>>> its output on the Filter step.
>>>
>>> What do you think?
>>>
>>>
>>>
>>> [1]
>>>
>> http://apache-ignite-developers.2346864.n4.nabble.com/New-SQL-execution-engine-tt43724.html#none
>>> [2]
>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-37%3A+New+query+execution+engine
>>> [3]  https://issues.apach

Re: Getting errors while setup new cluster using existing cluster data

[Ilya Kasnacheev]

Hello!

I don't think Apache Ignite supports it. We even clear caches explicitly if
a node was reset when WAL was disabled, and your scenario is just like this
one but worse.

Regards,
-- 
Ilya Kasnacheev


ср, 4 дек. 2019 г. в 20:43, Gangaiah Gundeboina :

> Thanks Denis.
>
> It's ok to loose delta data(which are writing data to cluster while
> coping),
> but want to up the cluster whatever data is valid.
>
> Is there any way.
>
> I don't have idea about Kafka-like approach, will check.
>
> Thanks and Regards,
> Gangaiah
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

Re: Full-text queries in Thin Client protocol

[Ilya Kasnacheev]

Hello!

Let's fix MOVING partitions first, think about thin client support later.

If nobody wants to fix that, then I'm pretty sure nobody wants thin client
support also.

Regards,
-- 
Ilya Kasnacheev


ср, 4 дек. 2019 г. в 11:35, Ivan Pavlukhin :

> Pavel mentioned a thread [1] discussion further development of text
> queries in Ignite. And currently there is some work in progress. And
> there is a lot to be improved. As for me, I think that we might think
> about thin clients support when user demand appear.
>
> [1]
> http://apache-ignite-developers.2346864.n4.nabble.com/Text-queries-indexes-GridLuceneIndex-QueryTextFiled-td43335.html
>
> вт, 3 дек. 2019 г. в 18:33, Igor Sapego :
> >
> > Pavel,
> >
> > I'm against adding this feature, as there were talks recently, that we
> > should stop supporting TextQuery altogether. No sense in adding
> > something, that we will need to depreciate and remove soon.
> >
> > Best Regards,
> > Igor
> >
> >
> > On Tue, Dec 3, 2019 at 5:46 PM Pavel Tupitsyn 
> wrote:
> >
> > > Oh wow, the next thread actually discusses this very issue about
> > > discontinuation:
> > >
> > >
> http://apache-ignite-developers.2346864.n4.nabble.com/Text-queries-indexes-GridLuceneIndex-QueryTextFiled-td43335.html
> > >
> > > But the question stands - add to Thin Clients or not?
> > >
> > > On Tue, Dec 3, 2019 at 5:43 PM Pavel Tupitsyn 
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > I'd like to discuss adding full-text queries (TextQuery, our
> Lucene-based
> > > > engine) to the Thin Client protocol.
> > > >
> > > > In my opinion, this is a good addition, and rather easy to implement.
> > > > However, I vaguely remember in-person discussions with some community
> > > > members along the lines of
> > > > "Ignite full-text search capabilities may be deprecated or reworked
> soon,
> > > > let's not add to thin clients".
> > > >
> > > > Any insights on this? Thoughts, objections?
> > > >
> > > > Thanks,
> > > > Pavel
> > > >
> > > >
> > > >
> > > >
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

Re: Adding support for Ignite secondary indexes to Apache Calcite planner

[Roman Kondakov]

Zhenya,

there is nothing in common in implementation of Ignite indexes and
Phoenix indexes. I just borrowed the idea how Phoenix supplies the index
metadata (index name, columns, sorting, etc.) to Calcite optimizer. It's
not about index implementation, it's about metadata handling.


-- 
Kind Regards
Roman Kondakov


On 10.12.2019 16:40, Zhenya Stanilovsky wrote:
> 
> Roman just as fast remark, Phoenix builds their approach on already existing 
> monolith HBase architecture, most cases it`s just a stub for someone who 
> wants use secondary indexes with a base with no native support of it. Don`t 
> think it`s good idea here.
>    
>>
>>
>> --- Forwarded message ---
>> From: "Roman Kondakov" < kondako...@mail.ru.invalid >
>> To:  dev@ignite.apache.org
>> Cc:
>> Subject: Adding support for Ignite secondary indexes to Apache Calcite
>> planner
>> Date: Tue, 10 Dec 2019 15:55:52 +0300
>>
>> Hi all!
>>
>> As you may know there is an activity on integration of Apache Calcite
>> query optimizer into Ignite codebase is being carried out [1],[2].
>>
>> One of a bunch of problems in this integration is the absence of
>> out-of-the-box support for secondary indexes in Apache Calcite. After
>> some research I came to conclusion that this problem has a couple of
>> workarounds. Let's name them
>> 1. Phoenix-style approach - representing secondary indexes as
>> materialized views which are natively supported by Calcite engine [3]
>> 2. Drill-style approach - pushing filters into the table scans and
>> choose appropriate index for lookups when possible [4]
>>
>> Both these approaches have advantages and disadvantages:
>>
>> Phoenix style pros:
>> - natural way of adding indexes as an alternative source of rows: index
>> can be considered as a kind of sorted materialized view.
>> - possibility of using index sortedness for stream aggregates,
>> deduplication (DISTINCT operator), merge joins, etc.
>> - ability to support other types of indexes (i.e. functional indexes).
>>
>> Phoenix style cons:
>> - polluting optimizer's search space extra table scans hence increasing
>> the planning time.
>>
>> Drill style pros:
>> - easier to implement (although it's questionable).
>> - search space is not inflated.
>>
>> Drill style cons:
>> - missed opportunity to exploit sortedness.
>>
>> There is a good discussion about using both approaches can be found in [5].
>>
>> I made a small sketch [6] in order to demonstrate the applicability of
>> the Phoenix approach to Ignite. Key design concepts are:
>> 1. On creating indexes are registered as tables in Calcite schema. This
>> step is needed for internal Calcite's routines.
>> 2. On planner initialization we register these indexes as materialized
>> views in Calcite's optimizer using VolcanoPlanner#addMaterialization
>> method.
>> 3. Right before the query execution Calcite selects all materialized
>> views (indexes) which can be potentially used in query.
>> 4. During the query optimization indexes are registered by planner as
>> usual TableScans and hence can be chosen by optimizer if they have lower
>> cost.
>>
>> This sketch shows the ability to exploit index sortedness only. So the
>> future work in this direction should be focused on using indexes for
>> fast index lookups. At first glance FilterableTable and
>> FilterTableScanRule are good points to start. We can push Filter into
>> the TableScan and then use FilterableTable for fast index lookups
>> avoiding reading the whole index on TableScan step and then filtering
>> its output on the Filter step.
>>
>> What do you think?
>>
>>
>>
>> [1]
>> http://apache-ignite-developers.2346864.n4.nabble.com/New-SQL-execution-engine-tt43724.html#none
>> [2]
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-37%3A+New+query+execution+engine
>> [3]  https://issues.apache.org/jira/browse/PHOENIX-2047
>> [4]  https://issues.apache.org/jira/browse/DRILL-6381
>> [5]  https://issues.apache.org/jira/browse/DRILL-3929
>> [6]  https://github.com/apache/ignite/pull/7115 
>  
>  
>  
>  
> 

Re: Re[4]: Text queries/indexes (GridLuceneIndex, @QueryTextFiled)

[Ilya Kasnacheev]

Hello!

Yes, I guess you are right :(

I can surely fix the range issue, It's just that it was so broken that I
could not figure the correct behavior for this case.

Regards,
-- 
Ilya Kasnacheev


пн, 2 дек. 2019 г. в 15:01, Ivan Pavlukhin :

> Ilya,
>
> I checked your test on a revision before "limit" and it fails there as
> well. Could you please double check?
>
> пн, 2 дек. 2019 г. в 13:21, Ilya Kasnacheev :
> >
> > Hello!
> >
> > The problem is NOT specific to range queries. Range queries were broken
> > previously and they are broken now, but now even a simple "token in field
> > with limit" returns duplicates.
> >
> > Before limits were introduced, any tested use cases were unaffected by
> > duplicates, but now they are.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пн, 2 дек. 2019 г. в 12:23, Ivan Pavlukhin :
> >
> > > And is the problem specific to range queries or not?
> > >
> > > пн, 2 дек. 2019 г. в 11:12, Ivan Pavlukhin :
> > > >
> > > > Yuriy,
> > > >
> > > > Thank you for investigating the problem [1]. Still cannot realize how
> > > > the problem relates to introduced "limit"? Is it right that there
> were
> > > > no duplicates before "limit" support? After that support is
> introduced
> > > > are only limited queries contain duplicates, or unlimited, or both?
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-12401
> > > >
> > > > чт, 28 нояб. 2019 г. в 18:30, Ilya Kasnacheev <
> ilya.kasnach...@gmail.com
> > > >:
> > > > >
> > > > > Hello!
> > > > >
> > > > > I have just found what I consider a major regression in Text
> Queries:
> > > it
> > > > > seems to me that text queries with limits will return same
> key-value
> > > > > entries multiple times.
> > > > >
> > > > > Please check the issue
> > > https://issues.apache.org/jira/browse/IGNITE-12401
> > > > > and corresponding build
> > > > > https://ci.ignite.apache.org/viewQueued.html?itemId=4799634
> > > > >
> > > > > Regards,
> > > > > --
> > > > > Ilya Kasnacheev
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

Re: [DISCUSS] dependencies and release process for Ignite Extensions

[Ivan Pavlukhin]

Saikat,

Let me share my understanding how do we run the majority of our jobs
executing test suites. Hope it will help.
1. ~Build Apache Ignite~ in a separate run and publish everything as a
single artifact ignite.zip (it contains ignite working directory with
compiled modules, e.g. modules/core/target and others).
2. Download and extract ignite.zip and run particular test suites
against extracted files.

In a such setup test suites run the same as you do it on your local
machine and it always works with a single multi-module maven project
(apache-ignite). So, all dependencies are resolved in the working
directory.

For a separate IgniteExtensions maven project you need to install all
dependencies from downloaded ignite.zip into .m2/repository local to
the agent. I suppose we can do it as follows:
1. Download ignite.zip and extract it into "ignite" subfolder.
2. Run a maven command installing all modules into local maven
repository. Something like "mvn install" but here we need to make sure
that nothing is compiled once again.
3. Build IgniteExtensions.

There might be a better way to do the same, I just tried to describe a
schema how it can be done.

вт, 10 дек. 2019 г. в 06:30, Saikat Maitra :
>
> Hi Ivan, Ilya
>
> Thank you for your reply. I have added you as dev in IgniteExtensions
> project and you should be able to check the build configurations.
>
> Yes, when I set artifact dependencies for ~Build Apache Ignite~ similar to
> other projects then I receive below error
>
> [ERROR] [ERROR] Could not find the selected project in the reactor:
> :ignite-flink-ext @
>
> I do not see the same error when I am running the build in local and
> maven package command can still get 2.9.0-SNAPSHOT dependencies from my
> local .m2 repository but it is failing in teamcity.
>
> Regards,
> Saikat
>
>
>
>
> On Mon, Dec 9, 2019 at 5:18 AM Ilya Kasnacheev 
> wrote:
>
> > Hello!
> >
> > I think that your build should depend on an Apache Ignite build(s) or just
> > use an already released version.
> >
> > In the same fashion as all our teamcity depend on "Build Apache Ignite" and
> > use its artifacts.
> >
> > Here you want Ignite snapshot but Teamcity does not know where to take it
> > from.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пн, 9 дек. 2019 г. в 06:40, Saikat Maitra :
> >
> > > Hello,
> > >
> > > I am running into a problem specific to teamcity build for Ignite
> > > Extensions project. When I set the dependencies to 2.9.0-SNAPSHOT I am
> > > getting an error message during build as below
> > >
> > > [06:24:12][Step 4/5] Failed to execute goal on project ignite-flink-ext:
> > > Could not resolve dependencies for project
> > > org.apache.ignite.ext:ignite-flink-ext:jar:1.0.0-SNAPSHOT: The following
> > > artifacts could not be resolved:
> > > org.apache.ignite:ignite-core:jar:2.9.0-SNAPSHOT,
> > > org.apache.ignite:ignite-core:jar:tests:2.9.0-SNAPSHOT,
> > > org.apache.ignite:ignite-log4j:jar:2.9.0-SNAPSHOT,
> > > org.apache.ignite:ignite-spring:jar:2.9.0-SNAPSHOT: Could not find
> > artifact
> > > org.apache.ignite:ignite-core:jar:2.9.0-SNAPSHOT in h2database.com (
> > > https://h2database.com/m2-repo)
> > >
> > >
> > > and if set artifact dependencies for ~Build Apache Ignite~ then I receive
> > > below error
> > >
> > > [ERROR] [ERROR] Could not find the selected project in the reactor:
> > > :ignite-flink-ext @
> > >
> > > Build url
> > >
> > >
> > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteExtensions_Build&branch_IgniteExtensions=pull%2F1%2Fhead&tab=buildTypeStatusDiv
> > >
> > > Can you please let me know if you faced similar problem with teamcity
> > > build?
> > >
> > > I can set Ignite Extensions dependencies to released ignite-core
> > artifacts
> > > version like 2.7.6 and build works fine.
> > >
> > > Regards,
> > > Saikat
> > >
> > >
> > >
> > >
> > >
> > > On Sat, Nov 30, 2019 at 1:50 PM Saikat Maitra 
> > > wrote:
> > >
> > > > Hello Denis,
> > > >
> > > > Thank you for your email and sharing your thoughts on the release
> > > process.
> > > > I will update the artifact id and dependencies for ignite-extensions
> > > > accordingly.
> > > >
> > > > I had created Ignite-Extensions project as a root level project and in
> > > > teamcity I was facing issues pulling dependencies for 2.8.0-SNAPSHOT
> > > > whereas I was able to pull dependencies for ignite-core 2.7.6 from
> > maven
> > > > central. I will look into it further why teamcity build was not able to
> > > > pull snapshot dependencies.
> > > >
> > > > I will also create "ignite-core-2.9+" branch for the upcoming release
> > > > process.
> > > >
> > > > Thank you,
> > > > Saikat
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Nov 27, 2019 at 1:05 PM Denis Magda  wrote:
> > > >
> > > >> Hi Saikat,
> > > >>
> > > >> Thanks for driving this activity forward and raising the question. Let
> > > me
> > > >> share my thoughts below and let's see what the broader community
> > thinks.
> > > >>
> > > >> Each extension needs t

Re: Adding support for Ignite secondary indexes to Apache Calcite planner

[Alexey Zinoviev]

I'd like Drill approach, worked and debugged with something similar, it's
more easy to support


Buuut, you have an implemented prototype (it votes for Phoenix in my mind)

вт, 10 дек. 2019 г. в 17:19, Vladimir Ozerov :

> Hi Roman,
>
> Why do you think that Drill-style will not let you exploit collation?
> Collation should be propagated from the index scan in the same way as in
> other sorted operators, such as merge join or streaming aggregate. Provided
> that you use converter-hack (or any alternative solution to trigger parent
> re-analysis).
> In other words, propagation of collation from Drill-style indexes should be
> no different from other sorted operators.
>
> Regards,
> Vladimir.
>
> вт, 10 дек. 2019 г. в 16:40, Zhenya Stanilovsky  >:
>
> >
> > Roman just as fast remark, Phoenix builds their approach on
> > already existing monolith HBase architecture, most cases it`s just a stub
> > for someone who wants use secondary indexes with a base with no
> > native support of it. Don`t think it`s good idea here.
> >
> > >
> > >
> > >--- Forwarded message ---
> > >From: "Roman Kondakov" < kondako...@mail.ru.invalid >
> > >To:  dev@ignite.apache.org
> > >Cc:
> > >Subject: Adding support for Ignite secondary indexes to Apache Calcite
> > >planner
> > >Date: Tue, 10 Dec 2019 15:55:52 +0300
> > >
> > >Hi all!
> > >
> > >As you may know there is an activity on integration of Apache Calcite
> > >query optimizer into Ignite codebase is being carried out [1],[2].
> > >
> > >One of a bunch of problems in this integration is the absence of
> > >out-of-the-box support for secondary indexes in Apache Calcite. After
> > >some research I came to conclusion that this problem has a couple of
> > >workarounds. Let's name them
> > >1. Phoenix-style approach - representing secondary indexes as
> > >materialized views which are natively supported by Calcite engine [3]
> > >2. Drill-style approach - pushing filters into the table scans and
> > >choose appropriate index for lookups when possible [4]
> > >
> > >Both these approaches have advantages and disadvantages:
> > >
> > >Phoenix style pros:
> > >- natural way of adding indexes as an alternative source of rows: index
> > >can be considered as a kind of sorted materialized view.
> > >- possibility of using index sortedness for stream aggregates,
> > >deduplication (DISTINCT operator), merge joins, etc.
> > >- ability to support other types of indexes (i.e. functional indexes).
> > >
> > >Phoenix style cons:
> > >- polluting optimizer's search space extra table scans hence increasing
> > >the planning time.
> > >
> > >Drill style pros:
> > >- easier to implement (although it's questionable).
> > >- search space is not inflated.
> > >
> > >Drill style cons:
> > >- missed opportunity to exploit sortedness.
> > >
> > >There is a good discussion about using both approaches can be found in
> > [5].
> > >
> > >I made a small sketch [6] in order to demonstrate the applicability of
> > >the Phoenix approach to Ignite. Key design concepts are:
> > >1. On creating indexes are registered as tables in Calcite schema. This
> > >step is needed for internal Calcite's routines.
> > >2. On planner initialization we register these indexes as materialized
> > >views in Calcite's optimizer using VolcanoPlanner#addMaterialization
> > >method.
> > >3. Right before the query execution Calcite selects all materialized
> > >views (indexes) which can be potentially used in query.
> > >4. During the query optimization indexes are registered by planner as
> > >usual TableScans and hence can be chosen by optimizer if they have lower
> > >cost.
> > >
> > >This sketch shows the ability to exploit index sortedness only. So the
> > >future work in this direction should be focused on using indexes for
> > >fast index lookups. At first glance FilterableTable and
> > >FilterTableScanRule are good points to start. We can push Filter into
> > >the TableScan and then use FilterableTable for fast index lookups
> > >avoiding reading the whole index on TableScan step and then filtering
> > >its output on the Filter step.
> > >
> > >What do you think?
> > >
> > >
> > >
> > >[1]
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/New-SQL-execution-engine-tt43724.html#none
> > >[2]
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-37%3A+New+query+execution+engine
> > >[3]  https://issues.apache.org/jira/browse/PHOENIX-2047
> > >[4]  https://issues.apache.org/jira/browse/DRILL-6381
> > >[5]  https://issues.apache.org/jira/browse/DRILL-3929
> > >[6]  https://github.com/apache/ignite/pull/7115
> >
> >
> >
> >
>

Re: Adding support for Ignite secondary indexes to Apache Calcite planner

[Vladimir Ozerov]

Hi Roman,

Why do you think that Drill-style will not let you exploit collation?
Collation should be propagated from the index scan in the same way as in
other sorted operators, such as merge join or streaming aggregate. Provided
that you use converter-hack (or any alternative solution to trigger parent
re-analysis).
In other words, propagation of collation from Drill-style indexes should be
no different from other sorted operators.

Regards,
Vladimir.

вт, 10 дек. 2019 г. в 16:40, Zhenya Stanilovsky :

>
> Roman just as fast remark, Phoenix builds their approach on
> already existing monolith HBase architecture, most cases it`s just a stub
> for someone who wants use secondary indexes with a base with no
> native support of it. Don`t think it`s good idea here.
>
> >
> >
> >--- Forwarded message ---
> >From: "Roman Kondakov" < kondako...@mail.ru.invalid >
> >To:  dev@ignite.apache.org
> >Cc:
> >Subject: Adding support for Ignite secondary indexes to Apache Calcite
> >planner
> >Date: Tue, 10 Dec 2019 15:55:52 +0300
> >
> >Hi all!
> >
> >As you may know there is an activity on integration of Apache Calcite
> >query optimizer into Ignite codebase is being carried out [1],[2].
> >
> >One of a bunch of problems in this integration is the absence of
> >out-of-the-box support for secondary indexes in Apache Calcite. After
> >some research I came to conclusion that this problem has a couple of
> >workarounds. Let's name them
> >1. Phoenix-style approach - representing secondary indexes as
> >materialized views which are natively supported by Calcite engine [3]
> >2. Drill-style approach - pushing filters into the table scans and
> >choose appropriate index for lookups when possible [4]
> >
> >Both these approaches have advantages and disadvantages:
> >
> >Phoenix style pros:
> >- natural way of adding indexes as an alternative source of rows: index
> >can be considered as a kind of sorted materialized view.
> >- possibility of using index sortedness for stream aggregates,
> >deduplication (DISTINCT operator), merge joins, etc.
> >- ability to support other types of indexes (i.e. functional indexes).
> >
> >Phoenix style cons:
> >- polluting optimizer's search space extra table scans hence increasing
> >the planning time.
> >
> >Drill style pros:
> >- easier to implement (although it's questionable).
> >- search space is not inflated.
> >
> >Drill style cons:
> >- missed opportunity to exploit sortedness.
> >
> >There is a good discussion about using both approaches can be found in
> [5].
> >
> >I made a small sketch [6] in order to demonstrate the applicability of
> >the Phoenix approach to Ignite. Key design concepts are:
> >1. On creating indexes are registered as tables in Calcite schema. This
> >step is needed for internal Calcite's routines.
> >2. On planner initialization we register these indexes as materialized
> >views in Calcite's optimizer using VolcanoPlanner#addMaterialization
> >method.
> >3. Right before the query execution Calcite selects all materialized
> >views (indexes) which can be potentially used in query.
> >4. During the query optimization indexes are registered by planner as
> >usual TableScans and hence can be chosen by optimizer if they have lower
> >cost.
> >
> >This sketch shows the ability to exploit index sortedness only. So the
> >future work in this direction should be focused on using indexes for
> >fast index lookups. At first glance FilterableTable and
> >FilterTableScanRule are good points to start. We can push Filter into
> >the TableScan and then use FilterableTable for fast index lookups
> >avoiding reading the whole index on TableScan step and then filtering
> >its output on the Filter step.
> >
> >What do you think?
> >
> >
> >
> >[1]
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/New-SQL-execution-engine-tt43724.html#none
> >[2]
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-37%3A+New+query+execution+engine
> >[3]  https://issues.apache.org/jira/browse/PHOENIX-2047
> >[4]  https://issues.apache.org/jira/browse/DRILL-6381
> >[5]  https://issues.apache.org/jira/browse/DRILL-3929
> >[6]  https://github.com/apache/ignite/pull/7115
>
>
>
>

Re: Adding support for Ignite secondary indexes to Apache Calcite planner

[Zhenya Stanilovsky]

Roman just as fast remark, Phoenix builds their approach on already existing 
monolith HBase architecture, most cases it`s just a stub for someone who wants 
use secondary indexes with a base with no native support of it. Don`t think 
it`s good idea here.
   
>
>
>--- Forwarded message ---
>From: "Roman Kondakov" < kondako...@mail.ru.invalid >
>To:  dev@ignite.apache.org
>Cc:
>Subject: Adding support for Ignite secondary indexes to Apache Calcite
>planner
>Date: Tue, 10 Dec 2019 15:55:52 +0300
>
>Hi all!
>
>As you may know there is an activity on integration of Apache Calcite
>query optimizer into Ignite codebase is being carried out [1],[2].
>
>One of a bunch of problems in this integration is the absence of
>out-of-the-box support for secondary indexes in Apache Calcite. After
>some research I came to conclusion that this problem has a couple of
>workarounds. Let's name them
>1. Phoenix-style approach - representing secondary indexes as
>materialized views which are natively supported by Calcite engine [3]
>2. Drill-style approach - pushing filters into the table scans and
>choose appropriate index for lookups when possible [4]
>
>Both these approaches have advantages and disadvantages:
>
>Phoenix style pros:
>- natural way of adding indexes as an alternative source of rows: index
>can be considered as a kind of sorted materialized view.
>- possibility of using index sortedness for stream aggregates,
>deduplication (DISTINCT operator), merge joins, etc.
>- ability to support other types of indexes (i.e. functional indexes).
>
>Phoenix style cons:
>- polluting optimizer's search space extra table scans hence increasing
>the planning time.
>
>Drill style pros:
>- easier to implement (although it's questionable).
>- search space is not inflated.
>
>Drill style cons:
>- missed opportunity to exploit sortedness.
>
>There is a good discussion about using both approaches can be found in [5].
>
>I made a small sketch [6] in order to demonstrate the applicability of
>the Phoenix approach to Ignite. Key design concepts are:
>1. On creating indexes are registered as tables in Calcite schema. This
>step is needed for internal Calcite's routines.
>2. On planner initialization we register these indexes as materialized
>views in Calcite's optimizer using VolcanoPlanner#addMaterialization
>method.
>3. Right before the query execution Calcite selects all materialized
>views (indexes) which can be potentially used in query.
>4. During the query optimization indexes are registered by planner as
>usual TableScans and hence can be chosen by optimizer if they have lower
>cost.
>
>This sketch shows the ability to exploit index sortedness only. So the
>future work in this direction should be focused on using indexes for
>fast index lookups. At first glance FilterableTable and
>FilterTableScanRule are good points to start. We can push Filter into
>the TableScan and then use FilterableTable for fast index lookups
>avoiding reading the whole index on TableScan step and then filtering
>its output on the Filter step.
>
>What do you think?
>
>
>
>[1]
>http://apache-ignite-developers.2346864.n4.nabble.com/New-SQL-execution-engine-tt43724.html#none
>[2]
>https://cwiki.apache.org/confluence/display/IGNITE/IEP-37%3A+New+query+execution+engine
>[3]  https://issues.apache.org/jira/browse/PHOENIX-2047
>[4]  https://issues.apache.org/jira/browse/DRILL-6381
>[5]  https://issues.apache.org/jira/browse/DRILL-3929
>[6]  https://github.com/apache/ignite/pull/7115 
 
 
 
 

Adding support for Ignite secondary indexes to Apache Calcite planner

[Roman Kondakov]

Hi all!

As you may know there is an activity on integration of Apache Calcite
query optimizer into Ignite codebase is being carried out [1],[2].

One of a bunch of problems in this integration is the absence of
out-of-the-box support for secondary indexes in Apache Calcite. After
some research I came to conclusion that this problem has a couple of
workarounds. Let's name them
1. Phoenix-style approach - representing secondary indexes as
materialized views which are natively supported by Calcite engine [3]
2. Drill-style approach - pushing filters into the table scans and
choose appropriate index for lookups when possible [4]

Both these approaches have advantages and disadvantages:

Phoenix style pros:
- natural way of adding indexes as an alternative source of rows: index
can be considered as a kind of sorted materialized view.
- possibility of using index sortedness for stream aggregates,
deduplication (DISTINCT operator), merge joins, etc.
- ability to support other types of indexes (i.e. functional indexes).

Phoenix style cons:
- polluting optimizer's search space extra table scans hence increasing
the planning time.

Drill style pros:
- easier to implement (although it's questionable).
- search space is not inflated.

Drill style cons:
- missed opportunity to exploit sortedness.

There is a good discussion about using both approaches can be found in [5].

I made a small sketch [6] in order to demonstrate the applicability of
the Phoenix approach to Ignite. Key design concepts are:
1. On creating indexes are registered as tables in Calcite schema. This
step is needed for internal Calcite's routines.
2. On planner initialization we register these indexes as materialized
views in Calcite's optimizer using VolcanoPlanner#addMaterialization method.
3. Right before the query execution Calcite selects all materialized
views (indexes) which can be potentially used in query.
4. During the query optimization indexes are registered by planner as
usual TableScans and hence can be chosen by optimizer if they have lower
cost.

This sketch shows the ability to exploit index sortedness only. So the
future work in this direction should be focused on using indexes for
fast index lookups. At first glance FilterableTable and
FilterTableScanRule are good points to start. We can push Filter into
the TableScan and then use FilterableTable for fast index lookups
avoiding reading the whole index on TableScan step and then filtering
its output on the Filter step.

What do you think?



[1]
http://apache-ignite-developers.2346864.n4.nabble.com/New-SQL-execution-engine-tt43724.html#none
[2]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-37%3A+New+query+execution+engine
[3] https://issues.apache.org/jira/browse/PHOENIX-2047
[4] https://issues.apache.org/jira/browse/DRILL-6381
[5] https://issues.apache.org/jira/browse/DRILL-3929
[6] https://github.com/apache/ignite/pull/7115


-- 
Kind Regards
Roman Kondakov

Re: GridGain Web Console is available free of charge for Apache Ignite

[Prasad Bhalerao]

Hi,

We found 3 vulnerabilities while scanning Grid Gain Web console application.

We are using HTTP and not HTTPS due to some issues on our side. Although
vulnerabilities are of lower severity, but thought of reporting it here.

1) HTTP TRACE / TRACK Methods Enabled. (CVE-2004-2320
, CVE-2010-0386
, CVE-2003-1567
)
2) Session Cookie Does Not Contain the "Secure" Attribute.
3) Web Server HTTP Trace/Track Method Support Cross-Site Tracing
Vulnerability. (CVE-2004-2320
, CVE-2007-3008
)

Can these be fixed?

Thanks,
Prasad


On Tue, Dec 10, 2019 at 4:39 PM Denis Magda  wrote:

> It's free software without limitations. Just download and use it.
>
> -
> Denis
>
>
> On Tue, Dec 10, 2019 at 1:21 PM Prasad Bhalerao <
> prasadbhalerao1...@gmail.com> wrote:
>
>> Hi,
>>
>> Can apache ignite users use it for free in their production environments?
>> What license does it fall under?
>>
>> Thanks,
>> Prasad
>>
>> On Fri, Oct 4, 2019 at 5:33 AM Denis Magda  wrote:
>>
>>> Igniters,
>>>
>>> There is good news. GridGain made its distribution of Web Console
>>> completely free. It goes with advanced monitoring and management
>>> dashboard
>>> and other handy screens. More details are here:
>>>
>>> https://www.gridgain.com/resources/blog/gridgain-road-simplicity-new-docs-and-free-tools-apache-ignite
>>>
>>> -
>>> Denis
>>>
>>

Re: GridGain Web Console is available free of charge for Apache Ignite

[Denis Magda]

It's free software without limitations. Just download and use it.

-
Denis


On Tue, Dec 10, 2019 at 1:21 PM Prasad Bhalerao <
prasadbhalerao1...@gmail.com> wrote:

> Hi,
>
> Can apache ignite users use it for free in their production environments?
> What license does it fall under?
>
> Thanks,
> Prasad
>
> On Fri, Oct 4, 2019 at 5:33 AM Denis Magda  wrote:
>
>> Igniters,
>>
>> There is good news. GridGain made its distribution of Web Console
>> completely free. It goes with advanced monitoring and management dashboard
>> and other handy screens. More details are here:
>>
>> https://www.gridgain.com/resources/blog/gridgain-road-simplicity-new-docs-and-free-tools-apache-ignite
>>
>> -
>> Denis
>>
>

Re: GridGain Web Console is available free of charge for Apache Ignite

[Prasad Bhalerao]

Hi,

Can apache ignite users use it for free in their production environments?
What license does it fall under?

Thanks,
Prasad

On Fri, Oct 4, 2019 at 5:33 AM Denis Magda  wrote:

> Igniters,
>
> There is good news. GridGain made its distribution of Web Console
> completely free. It goes with advanced monitoring and management dashboard
> and other handy screens. More details are here:
>
> https://www.gridgain.com/resources/blog/gridgain-road-simplicity-new-docs-and-free-tools-apache-ignite
>
> -
> Denis
>

[jira] [Created] (IGNITE-12430) Move PagePool to a separate class

[Alexey Goncharuk (Jira)]

Alexey Goncharuk created IGNITE-12430:
-

 Summary: Move PagePool to a separate class
 Key: IGNITE-12430
 URL: https://issues.apache.org/jira/browse/IGNITE-12430
 Project: Ignite
  Issue Type: Improvement
Reporter: Alexey Goncharuk
Assignee: Alexey Goncharuk


This is a refactoring required for IGNITE-12412 in order to create a separate 
test.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)