Re: Is Model class OR a value object mandatory for inserting the data thru non SQL API

[Naveen]

This way if we insert 

gnite.binary().builder("ANY_STRING_HERE");
builder.setField("id", 1);
builder.setField("name", "John");

We wont be able to retrieve the data thru SQL API. I did the same, but I
dont see the records which I inserted thru Binary API by using the above. 

We get this error while inserting it

[2020-12-29 12:04:47,862][WARN ][sys-stripe-7-#8][GridQueryProcessor]
Key-value pair is not inserted into any SQL table
[cacheName=TEST_STATS_NAVEEN_NO_VAL,
expValType=SQL_PUBLIC_TEST_STATS_NAVEEN_NO_VAL_532f9920_b7a2_43de_b6a2_2f45ad8d5c2a,
actualValType=TestStatsNaveen]

It does expects the same value_type which Ignite has assigned when the table
was created.
However ScanQuery returns all the rows, guess it does the table scan. 

Am i missing something or its fair understanding on how this work

Thanks
Naveen



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Is Model class OR a value object mandatory for inserting the data thru non SQL API

[Pavel Tupitsyn]

How do you configure the cache?
QueryEntity.valueType should be the same string as you use in
ignite.binary().builder()

On Tue, Dec 29, 2020 at 11:08 AM Naveen  wrote:

> This way if we insert
>
> gnite.binary().builder("ANY_STRING_HERE");
> builder.setField("id", 1);
> builder.setField("name", "John");
>
> We wont be able to retrieve the data thru SQL API. I did the same, but I
> dont see the records which I inserted thru Binary API by using the above.
>
> We get this error while inserting it
>
> [2020-12-29 12:04:47,862][WARN ][sys-stripe-7-#8][GridQueryProcessor]
> Key-value pair is not inserted into any SQL table
> [cacheName=TEST_STATS_NAVEEN_NO_VAL,
>
> expValType=SQL_PUBLIC_TEST_STATS_NAVEEN_NO_VAL_532f9920_b7a2_43de_b6a2_2f45ad8d5c2a,
> actualValType=TestStatsNaveen]
>
> It does expects the same value_type which Ignite has assigned when the
> table
> was created.
> However ScanQuery returns all the rows, guess it does the table scan.
>
> Am i missing something or its fair understanding on how this work
>
> Thanks
> Naveen
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Is Model class OR a value object mandatory for inserting the data thru non SQL API

[Naveen]

I just created the cache by running the below SQL from SQLLine console

CREATE TABLE IF NOT EXISTS TEST_STATS_NAVEEN_NO_VAL (
  stats_id VARCHAR, testname VARCHAR, testcityId VARCHAR, PRIMARY KEY
(stats_id))
  WITH "template=myCacheTemplate,
DATA_REGION=persistwarm,CACHE_NAME=TEST_STATS_NAVEEN_NO_VAL";  

Since I have not given any value_type, Ignite has created a value for
value_type like this
"SQL_PUBLIC_TEST_STATS_NAVEEN_NO_VAL_532f9920_b7a2_43de_b6a2_2f45ad8d5c2a" 

So, when we are doing ignite.binary().builder(), we should exactly pass the
same value
SQL_PUBLIC_TEST_STATS_NAVEEN_NO_VAL_532f9920_b7a2_43de_b6a2_2f45ad8d5c2a, 
then only we should be able to retrieve the data inserted thru BinaryAPI
from SQL.
Thanks





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[ANNOUNCE] Apache Ignite 2.9.1 Released

[Yaroslav Molochkov]

The Apache Ignite Community is pleased to announce the release of
Apache Ignite 2.9.1.

Apache Ignite [1] is a memory-centric distributed database, caching
and processing platform for transactional, analytical, and streaming
workloads delivering in-memory speeds at petabyte scale.

This release fixes several critical issues, please, see the release notes:
https://ignite.apache.org/releases/2.9.1/release_notes.html

Download the latest Ignite version here:
https://ignite.apache.org/download.cgi

Please let us know [2] if you encounter any problems.

Regards,
Yaroslav Molochkov

[1] https://ignite.apache.org
[2] https://ignite.apache.org/community/resources.html#ask

Re: Ignite thread pools

[Никита Сафонов]

Hi  Raymond,

Thanks for the feedback!

I'll extend the documentation.
I've already created a ticket for this:
https://issues.apache.org/jira/browse/IGNITE-13932

Regards,
Nikita

пн, 28 дек. 2020 г. в 08:33, Raymond Wilson :

> Hi,
>
> I have been drilling into some of the Ignite thread pool support and have
> found references in the code to three thread pools that don't appear to be
> referenced in the Ignite thread pool documentation at
> https://apacheignite.readme.io/docs/thread-pools.
>
> These are:
>
> asyncCallbackThreadPoolSize
> managementThreadPoolSize
> utilityCacheThreadPoolSize
>
> By default these pools will be set to Max(8, Environment.ProcessorCount)
> []we use the IA C# client]
>
> Note: These configurations are referenced in the Ignite Javadoc pages, but
> the descriptions are just single sentences mirroring the names of the
> thread pools themselves.
>
> Can the thread pool documentation be extended to include these additional
> pools, particularly with respect to their purpose and roles, and the
> circumstances when you would change them from the default value.
>
> Thanks,
> Raymond.
>
> --
> 
> Raymond Wilson
> Solution Architect, Civil Construction Software Systems (CCSS)
> raymond_wil...@trimble.com
>
>
> 
>

Re: Ignite thread pools

[Raymond Wilson]

Thanks Nikita! :)

Sent from my iPhone

On 30/12/2020, at 2:41 AM, Никита Сафонов  wrote:


Hi  Raymond,

Thanks for the feedback!

I'll extend the documentation.
I've already created a ticket for this:
https://issues.apache.org/jira/browse/IGNITE-13932

Regards,
Nikita

пн, 28 дек. 2020 г. в 08:33, Raymond Wilson :

> Hi,
>
> I have been drilling into some of the Ignite thread pool support and have
> found references in the code to three thread pools that don't appear to be
> referenced in the Ignite thread pool documentation at
> https://apacheignite.readme.io/docs/thread-pools.
>
> These are:
>
> asyncCallbackThreadPoolSize
> managementThreadPoolSize
> utilityCacheThreadPoolSize
>
> By default these pools will be set to Max(8, Environment.ProcessorCount)
> []we use the IA C# client]
>
> Note: These configurations are referenced in the Ignite Javadoc pages, but
> the descriptions are just single sentences mirroring the names of the
> thread pools themselves.
>
> Can the thread pool documentation be extended to include these additional
> pools, particularly with respect to their purpose and roles, and the
> circumstances when you would change them from the default value.
>
> Thanks,
> Raymond.
>
> --
> 
> Raymond Wilson
> Solution Architect, Civil Construction Software Systems (CCSS)
> raymond_wil...@trimble.com
>
>
> 
>

Re: Ignite in-memory + other SQL store without fully loading all data into Ignite

[Denis Magda]

Hi Courtney,

Glad to hear from you! It's been a while since we met last time. It's truly
disappointing seeing you struggle with Ignite that much. Thanks for being
open and kicking the discussion off.

How about three of us (you, Val, and I) meet the second week of January and
talk out the issues? Then you can share a talk summary here with a broader
community, if you wish. Between us, I've been personally championing the
column-type-change feature for a while, and with this IEP of Ignite 3.0

it should be doable.

In the meantime, some items for our agenda and pointers:

   - How does your upgrade procedure look like? Ignite doesn't have a
   rolling-upgrade feature; thus, to upgrade from version A to B in a
   *consistent way* the one should stop the cluster.
   - re: the frustration about spending too much time on the administration
   and infrastructure-related activities. Are you considering any managed
   service options? As a developer, I understand this frustration. I was just
   lucky to stay away from a need to administer Postgres, Oracle, MySQL, or
   Ignite just because we either had an IT administrator or made use of a
   managed-service option. Not selling anything here, just know that all those
   administration routines are unavoidable. The thing with Ignite is that
   distributed nature makes things more complicated.
   - Ignite ML and SQL: in fact, you can already call Ignite ML models from
   a SQL query. I need to search for pointers. @Alexey Zinoviev
   , you can probably share some examples quicker.
   - Calcite and Ignite: in Q2 we're planning to release the
   Calcite-powered SQL engine in Ignite. That would be a dramatic improvement
   in our SQL capabilities. It should be possible to enable push-downs much
   more easily.


-
Denis


On Mon, Dec 28, 2020 at 11:50 PM Courtney Robinson <
courtney.robin...@hypi.io> wrote:

> Hi Val,
> Thanks. You're not missing anything and we have been using Ignite
> persistence for years.
> Among the reasons we want to switch to in-memory only is how easily we
> seem to get corrupt nodes. I mentioned in the first email. We haven't had a
> situation where upgrading corrupts more than 1 of our 3 replicas but we
> genuinely fear upgrading sometimes as a result and is why we put so much
> effort into our backup/restore solution.
>
> One of the arguments for disabling persistence is that other DBs seems to
> have had longer to solve the issue. We've operated Postgres for longer than
> Ignite persistence and have never had it corrupt its data on disk
> (admittedly it's not distributed so that could play a role) but we've also
> been running Cassandra for longer as well without ever having any data
> corruption. All 3 in Kubernetes. With Ignite we've tried many things and a
> few settings between Ignite rebalanceDelay (I forget which other ones) and
> k8s readiness/liveness probes seems to have landed in a sweet spot that's
> reduced how often it happens but if we happen to have any issues with the
> k8s control plane, scheduling delays or network issues then the chances of
> it skyrockets. It then requires manual intervention I believe the most
> frequent issue is that a node will start and the topology diverged so the
> cluster doesn't think it's a part of it, the most effective thing we've
> found is deleting the data on that node and having it join as a new node
> for re-balancing to kicking and push data back to it. I think we have a few
> tickets somewhere with details.
>
> This is the primary motivator for wanting to replace Ignite persistence.
>
>  Here's our use case:
> We provide a low-code app development platform. As a developer you define
> tables in a visual interface or via GraphQL. We map these on to Ignite
> tables. It is a multi-tenant system complete with
> authentication/authorisation, serverless functions, API Gateway and
> more...all the bells and whistles for building modern applications in
> record time. We make it easy and convenient to get the scale and
> performance you expect from a large company with as little as the effort of
> an intern.
>
> The thing about convenience is that there are expectations set by others
> (the industry in this case) that the convenient things you provide include
> a baseline that matches industry or whoever has set the expectations.
> It is a sizeable task to implement/enforce referential integrity and
> delete cascade application side and I don't know of any distributed
> database which has this built in but the lack of transactions in SQL means
> we can't really implement it without pretty much implementing what the DB
> should be doing to keep track and rollback on failure.
>
> In an earlier version of the platform we implemented this using the key
> value APIs but we were not using SQL at all. We moved to SQL 1.5yrs ago or
> so because we needed greater and greater query capability and we were
> implementing a c

Re: Questions related to check pointing

[Raymond Wilson]

I noticed an entry in the Ignite 2.9.1 changelog:

   - Improved checkpoint concurrent behaviour

I am having trouble finding the relevant Jira ticket for this in the 2.9.1
Jira area at
https://issues.apache.org/jira/browse/IGNITE-13876?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%202.9.1%20and%20status%20%3D%20Resolved

Perhaps this change may improve the checkpointing issue we are seeing?

Raymond.


On Tue, Dec 29, 2020 at 8:35 PM Raymond Wilson 
wrote:

> Hi Zhenya,
>
> 1. We currently use AWS EFS for primary storage, with provisioned IOPS to
> provide sufficient IO. Our Ignite cluster currently tops out at ~10% usage
> (with at least 5 nodes writing to it, including WAL and WAL archive), so we
> are not saturating the EFS interface. We use the default page size
> (experiments with larger page sizes showed instability when checkpointing
> due to free page starvation, so we reverted to the default size).
>
> 2. Thanks for the detail, we will look for that in thread dumps when we
> can create them.
>
> 3. We are using the default CP buffer size, which is max(256Mb,
> DataRagionSize / 4) according to the Ignite documentation, so this should
> have more than enough checkpoint buffer space to cope with writes. As
> additional information, the cache which is displaying very slow writes is
> in a data region with relatively slow write traffic. There is a primary
> (default) data region with large write traffic, and the vast majority of
> pages being written in a checkpoint will be for that default data region.
>
> 4. Yes, this is very surprising. Anecdotally from our logs it appears
> write traffic into the low write traffic cache is blocked during
> checkpoints.
>
> Thanks,
> Raymond.
>
>
>
> On Tue, Dec 29, 2020 at 7:31 PM Zhenya Stanilovsky 
> wrote:
>
>>
>>1. Additionally to Ilya reply you can check vendors page for
>>additional info, all in this page are applicable for ignite too [1].
>>Increasing threads number leads to concurrent io usage, thus if your have
>>something like nvme — it`s up to you but in case of sas possibly better
>>would be to reduce this param.
>>2. Log will shows you something like :
>>
>>Parking thread=%Thread name% for timeout(ms)= %time%
>>
>>and appropriate :
>>
>>Unparking thread=
>>
>>3. No additional looging with cp buffer usage are provided. cp buffer
>>need to be more than 10% of overall persistent  DataRegions size.
>>4. 90 seconds or longer —  Seems like problems in io or system
>>tuning, it`s very bad score i hope.
>>
>> [1]
>> https://www.gridgain.com/docs/latest/perf-troubleshooting-guide/persistence-tuning
>>
>>
>>
>>
>>
>> Hi,
>>
>> We have been investigating some issues which appear to be related to
>> checkpointing. We currently use the IA 2.8.1 with the C# client.
>>
>> I have been trying to gain clarity on how certain aspects of the Ignite
>> configuration relate to the checkpointing process:
>>
>> 1. Number of check pointing threads. This defaults to 4, but I don't
>> understand how it applies to the checkpointing process. Are more threads
>> generally better (eg: because it makes the disk IO parallel across the
>> threads), or does it only have a positive effect if you have many data
>> storage regions? Or something else? If this could be clarified in the
>> documentation (or a pointer to it which Google has not yet found), that
>> would be good.
>>
>> 2. Checkpoint frequency. This is defaulted to 180 seconds. I was thinking
>> that reducing this time would result in smaller less disruptive check
>> points. Setting it to 60 seconds seems pretty safe, but is there a
>> practical lower limit that should be used for use cases with new data
>> constantly being added, eg: 5 seconds, 10 seconds?
>>
>> 3. Write exclusivity constraints during checkpointing. I understand that
>> while a checkpoint is occurring ongoing writes will be supported into the
>> caches being check pointed, and if those are writes to existing pages then
>> those will be duplicated into the checkpoint buffer. If this buffer becomes
>> full or stressed then Ignite will throttle, and perhaps block, writes until
>> the checkpoint is complete. If this is the case then Ignite will emit
>> logging (warning or informational?) that writes are being throttled.
>>
>> We have cases where simple puts to caches (a few requests per second) are
>> taking up to 90 seconds to execute when there is an active check point
>> occurring, where the check point has been triggered by the checkpoint
>> timer. When a checkpoint is not occurring the time to do this is usually in
>> the milliseconds. The checkpoints themselves can take 90 seconds or longer,
>> and are updating up to 30,000-40,000 pages, across a pair of data storage
>> regions, one with 4Gb in-memory space allocated (which should be 1,000,000
>> pages at the standard 4kb page size), and one small region with 128Mb.
>> There is no 'throttling' logging being emitted that we can tell, so the
>> checkpoin

Re: Ignite in-memory + other SQL store without fully loading all data into Ignite

[Courtney Robinson]

Hey Denis,

It's been a while. Hope you've been keeping well!
A meet the second week of Jan will be great. The agenda looks good too.

We've been watching the calcite SQL engine

wiki
and task list

but
I got the impression it wouldn't be that soon. May be worth getting
involved there to keep some of the calcite APIs exposed for us to be able
to tailor it because my ideal design would have Ignite in front of anything
we end up doing.

We'll sort out the call details closer to the time. Thanks Denis, speak to
you and Val in two weeks.

Regards,
Courtney Robinson
Founder and CEO, Hypi
Tel: ++44 208 123 2413 (GMT+0) 


https://hypi.io


On Wed, Dec 30, 2020 at 12:02 AM Denis Magda  wrote:

> Hi Courtney,
>
> Glad to hear from you! It's been a while since we met last time. It's
> truly disappointing seeing you struggle with Ignite that much. Thanks for
> being open and kicking the discussion off.
>
> How about three of us (you, Val, and I) meet the second week of January
> and talk out the issues? Then you can share a talk summary here with a
> broader community, if you wish. Between us, I've been personally
> championing the column-type-change feature for a while, and with this IEP
> of Ignite 3.0
> 
> it should be doable.
>
> In the meantime, some items for our agenda and pointers:
>
>- How does your upgrade procedure look like? Ignite doesn't have a
>rolling-upgrade feature; thus, to upgrade from version A to B in a
>*consistent way* the one should stop the cluster.
>- re: the frustration about spending too much time on the
>administration and infrastructure-related activities. Are you considering
>any managed service options? As a developer, I understand this frustration.
>I was just lucky to stay away from a need to administer Postgres, Oracle,
>MySQL, or Ignite just because we either had an IT administrator or made use
>of a managed-service option. Not selling anything here, just know that all
>those administration routines are unavoidable. The thing with Ignite is
>that distributed nature makes things more complicated.
>- Ignite ML and SQL: in fact, you can already call Ignite ML models
>from a SQL query. I need to search for pointers. @Alexey Zinoviev
>, you can probably share some examples quicker.
>- Calcite and Ignite: in Q2 we're planning to release the
>Calcite-powered SQL engine in Ignite. That would be a dramatic improvement
>in our SQL capabilities. It should be possible to enable push-downs much
>more easily.
>
>
> -
> Denis
>
>
> On Mon, Dec 28, 2020 at 11:50 PM Courtney Robinson <
> courtney.robin...@hypi.io> wrote:
>
>> Hi Val,
>> Thanks. You're not missing anything and we have been using Ignite
>> persistence for years.
>> Among the reasons we want to switch to in-memory only is how easily we
>> seem to get corrupt nodes. I mentioned in the first email. We haven't had a
>> situation where upgrading corrupts more than 1 of our 3 replicas but we
>> genuinely fear upgrading sometimes as a result and is why we put so much
>> effort into our backup/restore solution.
>>
>> One of the arguments for disabling persistence is that other DBs seems to
>> have had longer to solve the issue. We've operated Postgres for longer than
>> Ignite persistence and have never had it corrupt its data on disk
>> (admittedly it's not distributed so that could play a role) but we've also
>> been running Cassandra for longer as well without ever having any data
>> corruption. All 3 in Kubernetes. With Ignite we've tried many things and a
>> few settings between Ignite rebalanceDelay (I forget which other ones) and
>> k8s readiness/liveness probes seems to have landed in a sweet spot that's
>> reduced how often it happens but if we happen to have any issues with the
>> k8s control plane, scheduling delays or network issues then the chances of
>> it skyrockets. It then requires manual intervention I believe the most
>> frequent issue is that a node will start and the topology diverged so the
>> cluster doesn't think it's a part of it, the most effective thing we've
>> found is deleting the data on that node and having it join as a new node
>> for re-balancing to kicking and push data back to it. I think we have a few
>> tickets somewhere with details.
>>
>> This is the primary motivator for wanting to replace Ignite persistence.
>>
>>  Here's our use case:
>> We provide a low-code app development platform. As a developer you define
>> tables in a visual interface or via GraphQL. We map these on to Ignite
>> tables. It is a multi-tenant system complete with
>> authentication/authorisation, serverless functions, API Gateway and
>> more...all the bells and

Re[2]: Questions related to check pointing

[Zhenya Stanilovsky]

Don`t think so, checkpointing work perfectly well already before this fix.
Need additional info for start digging your problem, can you share ignite logs 
somewhere?
 
>I noticed an entry in the Ignite 2.9.1 changelog:
>*  Improved checkpoint concurrent behaviour
>I am having trouble finding the relevant Jira ticket for this in the 2.9.1 
>Jira area at  
>https://issues.apache.org/jira/browse/IGNITE-13876?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%202.9.1%20and%20status%20%3D%20Resolved
> 
>Perhaps this change may improve the checkpointing issue we are seeing?
> 
>Raymond.
>   
>On Tue, Dec 29, 2020 at 8:35 PM Raymond Wilson < raymond_wil...@trimble.com > 
>wrote:
>>Hi Zhenya,
>> 
>>1. We currently use AWS EFS for primary storage, with provisioned IOPS to 
>>provide sufficient IO. Our Ignite cluster currently tops out at ~10% usage 
>>(with at least 5 nodes writing to it, including WAL and WAL archive), so we 
>>are not saturating the EFS interface. We use the default page size 
>>(experiments with larger page sizes showed instability when checkpointing due 
>>to free page starvation, so we reverted to the default size). 
>> 
>>2. Thanks for the detail, we will look for that in thread dumps when we can 
>>create them.
>> 
>>3. We are using the default CP buffer size, which is max(256Mb, 
>>DataRagionSize / 4) according to the Ignite documentation, so this should 
>>have more than enough checkpoint buffer space to cope with writes. As 
>>additional information, the cache which is displaying very slow writes is in 
>>a data region with relatively slow write traffic. There is a primary 
>>(default) data region with large write traffic, and the vast majority of 
>>pages being written in a checkpoint will be for that default data region.
>> 
>>4. Yes, this is very surprising. Anecdotally from our logs it appears write 
>>traffic into the low write traffic cache is blocked during checkpoints.
>> 
>>Thanks,
>>Raymond.
>>    
>>   
>>On Tue, Dec 29, 2020 at 7:31 PM Zhenya Stanilovsky < arzamas...@mail.ru > 
>>wrote:
>>>*  
>>>Additionally to Ilya reply you can check vendors page for additional info, 
>>>all in this page are applicable for ignite too [1]. Increasing threads 
>>>number leads to concurrent io usage, thus if your have something like nvme — 
>>>it`s up to you but in case of sas possibly better would be to reduce this 
>>>param.
>>>*  Log will shows you something like :
>>>Parking thread=%Thread name% for timeout(ms)= %time% and appropriate :
>>>Unparking thread=
>>>*  No additional looging with cp buffer usage are provided. cp buffer need 
>>>to be more than 10% of overall persistent  DataRegions size.
>>>*  90 seconds or longer  —    Seems like problems in io or system tuning, 
>>>it`s very bad score i hope. 
>>>[1]  
>>>https://www.gridgain.com/docs/latest/perf-troubleshooting-guide/persistence-tuning
>>>
>>>
>>> 
Hi,
 
We have been investigating some issues which appear to be related to 
checkpointing. We currently use the IA 2.8.1 with the C# client.
 
I have been trying to gain clarity on how certain aspects of the Ignite 
configuration relate to the checkpointing process:
 
1. Number of check pointing threads. This defaults to 4, but I don't 
understand how it applies to the checkpointing process. Are more threads 
generally better (eg: because it makes the disk IO parallel across the 
threads), or does it only have a positive effect if you have many data 
storage regions? Or something else? If this could be clarified in the 
documentation (or a pointer to it which Google has not yet found), that 
would be good.
 
2. Checkpoint frequency. This is defaulted to 180 seconds. I was thinking 
that reducing this time would result in smaller less disruptive check 
points. Setting it to 60 seconds seems pretty safe, but is there a 
practical lower limit that should be used for use cases with new data 
constantly being added, eg: 5 seconds, 10 seconds?
 
3. Write exclusivity constraints during checkpointing. I understand that 
while a checkpoint is occurring ongoing writes will be supported into the 
caches being check pointed, and if those are writes to existing pages then 
those will be duplicated into the checkpoint buffer. If this buffer becomes 
full or stressed then Ignite will throttle, and perhaps block, writes until 
the checkpoint is complete. If this is the case then Ignite will emit 
logging (warning or informational?) that writes are being throttled.
 
We have cases where simple puts to caches (a few requests per second) are 
taking up to 90 seconds to execute when there is an active check point 
occurring, where the check point has been triggered by the checkpoint 
timer. When a checkpoint is not occurring the time to do this is usually in 
the milliseconds. The checkpoints themselves can take 90 seconds or longer, 
and are updating up to