Re: Kubernetes liveness and readiness probes

[Surinder Mehra]

Yeah I faced the same issue, ended up disabling it. Better to query
cluster.isActive() if you need to apply wait for readiness anywhere in
native persistence mode.
Otherwise cluster size in non persistence mode.

On Thu, 17 Aug 2023, 03:01 Humphrey Lopez,  wrote:

> Has anyone deploy ignite in kubernetes and supply a readiness probe? If I
> try to start up like 10 nodes at the same time with the readiness probe
> they all start up as separate cluster, whereas if I don’t supply the probes
> they are finding each other and I’ll have one cluster.
> I’m using the kubernetes IpFinder.
> Used version 2.15 of ignite (Java/Kotlin) and spring boot.
>
>
> Humphrey

Re: Create a Blog with Example code

[Surinder Mehra]

Hello,
Hope this helps you with some more examples. This is my GitHub repo which
has examples on various features of ignite.

https://github.com/Rednirus/apache-ignite-masterclass

Cheers !


On Mon, 6 Mar 2023, 13:47 Humphrey Lopez,  wrote:

> What I need is a nice example where I can demonstrate Affinity
> Collocation. Would like to populate data (maybe stream) from a Rest Api
> outside kubernetes, what also would be nice is to fetch some data from the
> internet using a CacheStoreAdapter.
> Primarily using ignite only in Memory Mode (with backups 1 or 2). And
> maybe later I could dive further into using persistent with a StateFullSet
> deployment. I think it would be a series of several blogs which we will
> build step by step from scratch. But for that I would like to have the
> final project first worked out and then create the blog. Any input or
> reference to other examples that covers the first part (in memory grid,
> affinity collocation). I need an example of a use case (Company with
> Employee, or Cars) or something like that.
>
> Humphrey
>
> Op za 4 mrt 2023 om 18:02 schreef Kseniya Romanova  >:
>
>> Humphrey as an inspiration you can also check Stack Overflow questions
>> tagged "ignite". Many of them are about working with Ignite and Spring.
>> Step by step tutorial following your experience sounds good too.
>>
>> On Sat, Mar 4, 2023 at 8:16 AM  wrote:
>>
>>> Hi Humphrey,
>>>
>>> Here is my own contribution in sharing experience of Apache Ignite. Back
>>> in those times I was starting a new project and spent a few weeks in
>>> solving data load perfomance problems. I was inspired by the post [2], it
>>> was very helpful that I can load, compile and debug complete project code
>>> from github. So I spent another few weeks and wrote my own post [1], also
>>> with compilable project inside.
>>>
>>> As for your question, I think a post about monitoring the cashes with
>>> grafana will be usefull
>>>
>>> PS
>>> Many thanks to Kseniya Romanova, Denis Magda and Susan Ledford for their
>>> help in my work
>>>
>>> Vladimir
>>>
>>> [1]
>>> https://www.gridgain.com/resources/blog/how-fast-load-large-datasets-apache-ignite-using-key-value-api
>>> [2]
>>> https://www.gridgain.com/resources/blog/implementing-microservices-apache-ignite-service-apis-part-i-0
>>> 00:41, 4 марта 2023 г., Humphrey Lopez :
>>>
>>> I’ve no idea yet. Lately working with spring boot and ignite again think
>>> maybe more people could use some help getting started. Would also like to
>>> setup local K8s cluster with microk8s and deploy the service and some nodes
>>> (client and server) and use rest endpoint to populate and monitor the
>>> cashes with grafana.
>>>
>>> Humphrey
>>>
>>> On 3 Mar 2023, at 10:57, Kseniya Romanova  wrote:
>>>
>>> 
>>> Hi Humphrey! Good idea! If you need a review before publishing,
>>> please let me know and I'll find a comitter who could help with this.
>>> Where do you plan to publish your blog? I know that many igniters prefer
>>> dzone or dev.to, but personal github or medium blogs work as well.
>>>
>>> Cheers,
>>> Kseniya
>>>
>>> On Fri, Mar 3, 2023 at 10:21 AM Humphrey  wrote:
>>>
>>> Hi,
>>>
>>> I would like to create a blog with code and share experience of Apache
>>> Ignite. What’s the best way to do that?
>>>
>>> Greetings Humphrey
>>>
>>>
>>>
>>> --
>>> Отправлено из мобильного приложения Яндекс Почты
>>
>>

Re: Ignite integration with MongoDB

[Surinder Mehra]

Given link has all the details needed. You either write your db queries in
custom cache store or move them to a separate db DAO class and have it's
instance in cache store. Then it will be similar to how you connect your
app to DB. Next you need to have these custom classes in ignite class path
and it should work. I have tried this with Java app and Oracle.

On Fri, 24 Feb 2023, 19:16 satyajit.mandal.barclays.com via user, <
user@ignite.apache.org> wrote:

> Hi  Team,
>
>
>
> We are using  .NET Api  for  accessing Ignite cache.  We want  to
> understand  how can we  implement  the custom  cache store  and  extend
> it  to  save data  in  MongoDB from  Ignite. Can  someone  share  detail
> steps  how  to  extend  the cachestore and  how  it  will  integrate  with
> Ignite  libraries? We are referring  this  documentation
> https://ignite.apache.org/docs/latest/persistence/custom-cache-store
> but  it  does not  mentions  what  to  do  after  we have written  custom
> cache  store and what  are  the  next  steps?
>
>
>
> Thanks
>
> Satyajit
>
>
>
>
>
> Barclays Execution Services Limited registered in England. Registered No.
> 1767980. Registered office: 1 Churchill Place, London, E14 5HP
>
> Barclays Execution Services Limited provides support and administrative
> services across Barclays group. Barclays Execution Services Limited is an
> appointed representative of Barclays Bank UK plc, Barclays Bank plc and
> Clydesdale Financial Services Limited. Barclays Bank UK plc and Barclays
> Bank plc are authorised by the Prudential Regulation Authority and
> regulated by the Financial Conduct Authority and the Prudential Regulation
> Authority. Clydesdale Financial Services Limited is authorised and
> regulated by the Financial Conduct Authority.
>
> This email and any attachments are confidential and intended solely for
> the addressee and may also be privileged or exempt from disclosure under
> applicable law. If you are not the addressee, or have received this email
> in error, please notify the sender and immediately delete it and any
> attachments from your system. Do not copy, use, disclose or otherwise act
> on any part of this email or its attachments.
>
> Internet communications are not guaranteed to be secure or virus-free. The
> Barclays group does not accept responsibility for any loss arising from
> unauthorised access to, or interference with, any internet communications
> by any third party, or from the transmission of any viruses. Replies to
> this email may be monitored by the Barclays group for operational or
> business reasons.
>
> Any opinion or other information in this email or its attachments that
> does not relate to the business of the Barclays group is personal to the
> sender and is not given or endorsed by the Barclays group.
>
> Unless specifically indicated, this e-mail is not an offer to buy or sell
> or a solicitation to buy or sell any securities, investment products or
> other financial product or service, an official confirmation of any
> transaction, or an official statement of Barclays.
>

Re: Get TotalServerNodes using controlscript/Rest Api

[Surinder Mehra]

You amazed me sir. Thanks a lot, I will update the linked thread on
stackoverflow. Command looks like below now

 kubectl --kubeconfig  config.conf exec  -n {client_name} --
/opt/ignite/apache-ignite/bin/./control.sh --metric
 cluster.TotalServerNodes | grep -v  "metric" | grep
cluster.TotalServerNodes | cut -d " " -f5 | sed 's/^ *//g'

On Wed, Feb 1, 2023 at 4:34 PM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> Cool. You learn something new every day!
>
> On 1 Feb 2023, at 10:55, Николай Ижиков  wrote:
>
> Hello.
>
> You can query any metric value or system view content via control script
> [1], [2]
>
> > control.sh --system-view nodes
>
> or [3]
>
> > control.sh —metric cluster.TotalBaselineNodes
> > control.sh —metric cluster.TotalServerNodes
> > control.sh —metric cluster.TotalClientNodes
>
> [1]
> https://ignite.apache.org/docs/latest/tools/control-script#metric-command
> [2]
> https://ignite.apache.org/docs/latest/tools/control-script#system-view-command
> [3]
> https://ignite.apache.org/docs/2.11.1/monitoring-metrics/new-metrics#cluster
>
>
> 1 февр. 2023 г., в 13:28, Stephen Darlington <
> stephen.darling...@gridgain.com> написал(а):
>
> I’m not aware of any single command to do it, but there are a few options.
> I’ll list a few as they’re all imperfect.
>
> There are some command-line JMX reader commands you could use to extract
> the information from Ignite’s metrics.
>
> You could extract the information from the REST API. I needed to use the
> jq command to parse it into just a count:
>
> http 'http://localhost:8080/ignite?cmd=top' | jq '[ .response[] ] |
> length'
>
> You could search the logs for “Topology snapshot” lines.
>
> There’s also ignitevisorcmd that you can script.
>
> Regards,
> Stephen
>
> On 1 Feb 2023, at 09:32, Surinder Mehra  wrote:
>
> Thanks, I am aware of it. But I needed this in a script. So either curl or
> control options would be good.
>
> On Wed, 1 Feb 2023, 14:51 Aleksandr Pakhomov,  wrote:
>
>> Hi,
>>
>> Do you have a chance to use a java client?
>>
>> https://ignite.apache.org/docs/latest/thin-clients/java-thin-client
>>
>> It seems this could help
>> igniteClient.cluster().forServers().nodes().count();
>>
>> Best regards,
>> Aleksandr
>>
>> On 1 Feb 2023, at 10:21, Surinder Mehra  wrote:
>>
>> Hi,
>> I am trying to find a way to fetch the count of server nodes in ignite
>> cluster but don't see any control script option or in Rest api. Could you
>> please suggest if it is possible. Below link shows this is exposed as a
>> cluster metric but not accessible through above two options.
>>
>>
>> https://ignite.apache.org/docs/latest/monitoring-metrics/new-metrics#cluster
>>
>> Related stackoverflow post:
>>
>> https://stackoverflow.com/questions/75301565/ignite-cluster-size-using-control-script/
>>
>> Regards,
>> Surinder
>>
>>
>>
>
>
>

Re: Get TotalServerNodes using controlscript/Rest Api

[Surinder Mehra]

Thanks, I am aware of it. But I needed this in a script. So either curl or
control options would be good.

On Wed, 1 Feb 2023, 14:51 Aleksandr Pakhomov,  wrote:

> Hi,
>
> Do you have a chance to use a java client?
>
> https://ignite.apache.org/docs/latest/thin-clients/java-thin-client
>
> It seems this could help
> igniteClient.cluster().forServers().nodes().count();
>
> Best regards,
> Aleksandr
>
> On 1 Feb 2023, at 10:21, Surinder Mehra  wrote:
>
> Hi,
> I am trying to find a way to fetch the count of server nodes in ignite
> cluster but don't see any control script option or in Rest api. Could you
> please suggest if it is possible. Below link shows this is exposed as a
> cluster metric but not accessible through above two options.
>
>
> https://ignite.apache.org/docs/latest/monitoring-metrics/new-metrics#cluster
>
> Related stackoverflow post:
>
> https://stackoverflow.com/questions/75301565/ignite-cluster-size-using-control-script/
>
> Regards,
> Surinder
>
>
>

Get TotalServerNodes using controlscript/Rest Api

[Surinder Mehra]

Hi,
I am trying to find a way to fetch the count of server nodes in ignite
cluster but don't see any control script option or in Rest api. Could you
please suggest if it is possible. Below link shows this is exposed as a
cluster metric but not accessible through above two options.

https://ignite.apache.org/docs/latest/monitoring-metrics/new-metrics#cluster

Related stackoverflow post:
https://stackoverflow.com/questions/75301565/ignite-cluster-size-using-control-script/

Regards,
Surinder

Re: Read from backup with PRIMARY_SYNC

[Surinder Mehra]

Thanks Stephen

On Wed, Dec 21, 2022 at 11:54 PM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> Please note that volunteers answer questions here. If you need SLAs for
> answers to your questions, there are companies that provide support.
>
> Full sync would work. You could also use transactions.
>
> On 21 Dec 2022, at 17:58, Surinder Mehra  wrote:
>
> Gentle reminder.
>
> On Tue, 20 Dec 2022, 12:53 Surinder Mehra,  wrote:
>
>> Hi,
>> If we have cache synchronization mode as Primary Sync and read from
>> backup as true(default), would this cause consistency issues if primary
>> goes down without flushing all data to disk.
>> I understand WAL is used to replay unflushed data when a node comes back
>> online again. But if say out of 10 records written to cache, only 5 of them
>> were written to disk by checkpointing thread and data was not replicated to
>> the backup node yet(PRIMARY-SYNC) when primary node went node, the client
>> would assume backup node would have all 10 records . But it might not be
>> the case.
>>
>> So am I correct in assuming we need to either turn off read from backup
>> or change cache synchronization mode to FULL-SYNC
>>
>> Answer to stackoverflow question below suggests disabling read from
>> backup. Do we need to disable it even in FULL-SYNC mode ?
>>
>>
>> https://stackoverflow.com/questions/67598175/ignite-read-stale-data-from-backup-node
>>
>
>

Re: Read from backup with PRIMARY_SYNC

[Surinder Mehra]

Gentle reminder.

On Tue, 20 Dec 2022, 12:53 Surinder Mehra,  wrote:

> Hi,
> If we have cache synchronization mode as Primary Sync and read from backup
> as true(default), would this cause consistency issues if primary goes down
> without flushing all data to disk.
> I understand WAL is used to replay unflushed data when a node comes back
> online again. But if say out of 10 records written to cache, only 5 of them
> were written to disk by checkpointing thread and data was not replicated to
> the backup node yet(PRIMARY-SYNC) when primary node went node, the client
> would assume backup node would have all 10 records . But it might not be
> the case.
>
> So am I correct in assuming we need to either turn off read from backup or
> change cache synchronization mode to FULL-SYNC
>
> Answer to stackoverflow question below suggests disabling read from
> backup. Do we need to disable it even in FULL-SYNC mode ?
>
>
> https://stackoverflow.com/questions/67598175/ignite-read-stale-data-from-backup-node
>

Read from backup with PRIMARY_SYNC

[Surinder Mehra]

Hi,
If we have cache synchronization mode as Primary Sync and read from backup
as true(default), would this cause consistency issues if primary goes down
without flushing all data to disk.
I understand WAL is used to replay unflushed data when a node comes back
online again. But if say out of 10 records written to cache, only 5 of them
were written to disk by checkpointing thread and data was not replicated to
the backup node yet(PRIMARY-SYNC) when primary node went node, the client
would assume backup node would have all 10 records . But it might not be
the case.

So am I correct in assuming we need to either turn off read from backup or
change cache synchronization mode to FULL-SYNC

Answer to stackoverflow question below suggests disabling read from backup.
Do we need to disable it even in FULL-SYNC mode ?

https://stackoverflow.com/questions/67598175/ignite-read-stale-data-from-backup-node

Re: Backup filter in ignite [Multi AZ deployment]

[Surinder Mehra]

Understood thanks !

On Sun, 6 Nov 2022, 21:33 Jeremy McMillan, 
wrote:

> Think of each AZ as being a massive piece of server hardware running VMs
> or workloads for you. When hardware (or infrastructure maintenance process)
> fails, assume everything on one AZ is lost at the same time.
>
> On Sun, Nov 6, 2022, 09:58 Surinder Mehra  wrote:
>
>> That's partially true. Whole excercise of configuring AZ as backup filter
>> is because we want to handle AZ level failure.
>>
>> Anyway, thanks for inputs. Will figure out further steps
>>
>> On Sun, 6 Nov 2022, 20:55 Jeremy McMillan, 
>> wrote:
>>
>>> Don't configure 2 backups when you only have two failure domains.
>>>
>>> You're worried about node level failure, but you're telling Ignite to
>>> worry about AZ level failure.
>>>
>>>
>>> On Sat, Nov 5, 2022, 21:57 Surinder Mehra  wrote:
>>>
>>>> Yeah I think there is a misunderstanding. Although I figured out my
>>>> answers from our discussion, I will try one final attempt to clarify my
>>>> point on 2X space for node3
>>>>
>>>> Node setup:
>>>> Node1 and node 2 placed in AZ1
>>>> Node 3 placed in AZ2
>>>>
>>>>  Since I am using AZ as backup filter as I mentioned in my first
>>>> message. Back up if node 1 cannot be placed on node2 and back up of node 2
>>>> cannot be placed on node1 as they are in same AZ. This simply means their
>>>> backups would go to node3 which in another AZ. Hence node 3 space =(node3
>>>> primary partitions+node 1 back up partitions+node2 backup partitions)
>>>>
>>>> Wouldn't this mean node 3 need 2X space as compared to node 1 and node2.
>>>> Assuming backup partitions of node 3 would be equally distributed among
>>>> other two nodes. They would need almost same space.
>>>>
>>>>
>>>> On Tue, 1 Nov 2022, 23:30 Jeremy McMillan, <
>>>> jeremy.mcmil...@gridgain.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Tue, Nov 1, 2022 at 10:02 AM Surinder Mehra 
>>>>> wrote:
>>>>>
>>>>>> Even if we have 2 copies of data and primary and backup copy would be
>>>>>> stored in different AZs. My question remains valid in this case as well.
>>>>>>
>>>>>
>>>>> I think additional backup copies in the same AZ are superfluous if we
>>>>> start with the assumption that multiple concurrent failures are most 
>>>>> likely
>>>>> to affect resources in the concurrent AZ. A second node failure, if that's
>>>>> your failure budget, is likely to corrupt all the backup copies in the
>>>>> second AZ.
>>>>>
>>>>> If you only have two AZs available in some data centers/deployments,
>>>>> but you need 3-way redundancy on certain caches/tables, then using AZ node
>>>>> attribute for backup filtering is too coarse grained. Using AZ is a 
>>>>> general
>>>>> case best practice which gives your cluster the best chance of surviving
>>>>> multiple hardware failures in AWS because they pool hardware resources in
>>>>> AZs. Maybe you just need three AZs? Maybe AZ isn't the correct failure
>>>>> domain for your use case?
>>>>>
>>>>>
>>>>>> Do we have to ensure nodes in two AZs are always present or does
>>>>>> ignite have a way to indicate it couldn't create backups. Silently 
>>>>>> killing
>>>>>> backups is not desirable state.
>>>>>>
>>>>>
>>>>> Do you use synchronous or asynchronous backups?
>>>>>
>>>>> https://ignite.apache.org/docs/2.11.1/configuring-caches/configuring-backups#synchronous-and-asynchronous-backups
>>>>>
>>>>> You can periodically poll caches' configurations or hook a cluster
>>>>> state event, and re-compare the cache backup configuration against the
>>>>> enumerated available AZs, and raise an exception or log a message or
>>>>> whatever to detect the issue as soon as AZ count drops below minimum. This
>>>>> way might also be good for fuzzy warning condition detection point for
>>>>> proactive infrastructure operations. If you count all of the nodes in each
>>>>> AZ, you can detect and track AZ load imbalances as 

Re: Backup filter in ignite [Multi AZ deployment]

[Surinder Mehra]

That's partially true. Whole excercise of configuring AZ as backup filter
is because we want to handle AZ level failure.

Anyway, thanks for inputs. Will figure out further steps

On Sun, 6 Nov 2022, 20:55 Jeremy McMillan, 
wrote:

> Don't configure 2 backups when you only have two failure domains.
>
> You're worried about node level failure, but you're telling Ignite to
> worry about AZ level failure.
>
>
> On Sat, Nov 5, 2022, 21:57 Surinder Mehra  wrote:
>
>> Yeah I think there is a misunderstanding. Although I figured out my
>> answers from our discussion, I will try one final attempt to clarify my
>> point on 2X space for node3
>>
>> Node setup:
>> Node1 and node 2 placed in AZ1
>> Node 3 placed in AZ2
>>
>>  Since I am using AZ as backup filter as I mentioned in my first message.
>> Back up if node 1 cannot be placed on node2 and back up of node 2 cannot be
>> placed on node1 as they are in same AZ. This simply means their backups
>> would go to node3 which in another AZ. Hence node 3 space =(node3 primary
>> partitions+node 1 back up partitions+node2 backup partitions)
>>
>> Wouldn't this mean node 3 need 2X space as compared to node 1 and node2.
>> Assuming backup partitions of node 3 would be equally distributed among
>> other two nodes. They would need almost same space.
>>
>>
>> On Tue, 1 Nov 2022, 23:30 Jeremy McMillan, 
>> wrote:
>>
>>>
>>>
>>> On Tue, Nov 1, 2022 at 10:02 AM Surinder Mehra 
>>> wrote:
>>>
>>>> Even if we have 2 copies of data and primary and backup copy would be
>>>> stored in different AZs. My question remains valid in this case as well.
>>>>
>>>
>>> I think additional backup copies in the same AZ are superfluous if we
>>> start with the assumption that multiple concurrent failures are most likely
>>> to affect resources in the concurrent AZ. A second node failure, if that's
>>> your failure budget, is likely to corrupt all the backup copies in the
>>> second AZ.
>>>
>>> If you only have two AZs available in some data centers/deployments, but
>>> you need 3-way redundancy on certain caches/tables, then using AZ node
>>> attribute for backup filtering is too coarse grained. Using AZ is a general
>>> case best practice which gives your cluster the best chance of surviving
>>> multiple hardware failures in AWS because they pool hardware resources in
>>> AZs. Maybe you just need three AZs? Maybe AZ isn't the correct failure
>>> domain for your use case?
>>>
>>>
>>>> Do we have to ensure nodes in two AZs are always present or does ignite
>>>> have a way to indicate it couldn't create backups. Silently killing backups
>>>> is not desirable state.
>>>>
>>>
>>> Do you use synchronous or asynchronous backups?
>>>
>>> https://ignite.apache.org/docs/2.11.1/configuring-caches/configuring-backups#synchronous-and-asynchronous-backups
>>>
>>> You can periodically poll caches' configurations or hook a cluster state
>>> event, and re-compare the cache backup configuration against the enumerated
>>> available AZs, and raise an exception or log a message or whatever to
>>> detect the issue as soon as AZ count drops below minimum. This way might
>>> also be good for fuzzy warning condition detection point for proactive
>>> infrastructure operations. If you count all of the nodes in each AZ, you
>>> can detect and track AZ load imbalances as the ratio between the smallest
>>> AZ node count and the average AZ node count.
>>>
>>>
>>>> 2. In my original message with 2 nodes(node1 and node2) in AZ1, and
>>>> 3rdnode in second AZ, backups of node1 and node2 would be placed one node 3
>>>> in AZ2. It would mean it need to have 2X space to store backups.
>>>> Just trying to ensure my understanding is correct.
>>>>
>>>
>>> If you have three nodes, you divide your total footprint by three to get
>>> the minimum node capacity.
>>>
>>> If you have 2 backups, that is one primary copy plus two more backup
>>> copies, so you multiply your total footprint by 3.
>>>
>>> If you multiply, say 32GB by three for redundancy, that would be 96GB
>>> total space needed for the sum of all nodes' footprint.
>>>
>>> If you divide the 96GB storage commitment among three nodes, then each
>>> node must have a minimum of 32GB. That's what we started

Re: Backup filter in ignite [Multi AZ deployment]

[Surinder Mehra]

Yeah I think there is a misunderstanding. Although I figured out my answers
from our discussion, I will try one final attempt to clarify my point on 2X
space for node3

Node setup:
Node1 and node 2 placed in AZ1
Node 3 placed in AZ2

 Since I am using AZ as backup filter as I mentioned in my first message.
Back up if node 1 cannot be placed on node2 and back up of node 2 cannot be
placed on node1 as they are in same AZ. This simply means their backups
would go to node3 which in another AZ. Hence node 3 space =(node3 primary
partitions+node 1 back up partitions+node2 backup partitions)

Wouldn't this mean node 3 need 2X space as compared to node 1 and node2.
Assuming backup partitions of node 3 would be equally distributed among
other two nodes. They would need almost same space.


On Tue, 1 Nov 2022, 23:30 Jeremy McMillan, 
wrote:

>
>
> On Tue, Nov 1, 2022 at 10:02 AM Surinder Mehra  wrote:
>
>> Even if we have 2 copies of data and primary and backup copy would be
>> stored in different AZs. My question remains valid in this case as well.
>>
>
> I think additional backup copies in the same AZ are superfluous if we
> start with the assumption that multiple concurrent failures are most likely
> to affect resources in the concurrent AZ. A second node failure, if that's
> your failure budget, is likely to corrupt all the backup copies in the
> second AZ.
>
> If you only have two AZs available in some data centers/deployments, but
> you need 3-way redundancy on certain caches/tables, then using AZ node
> attribute for backup filtering is too coarse grained. Using AZ is a general
> case best practice which gives your cluster the best chance of surviving
> multiple hardware failures in AWS because they pool hardware resources in
> AZs. Maybe you just need three AZs? Maybe AZ isn't the correct failure
> domain for your use case?
>
>
>> Do we have to ensure nodes in two AZs are always present or does ignite
>> have a way to indicate it couldn't create backups. Silently killing backups
>> is not desirable state.
>>
>
> Do you use synchronous or asynchronous backups?
>
> https://ignite.apache.org/docs/2.11.1/configuring-caches/configuring-backups#synchronous-and-asynchronous-backups
>
> You can periodically poll caches' configurations or hook a cluster state
> event, and re-compare the cache backup configuration against the enumerated
> available AZs, and raise an exception or log a message or whatever to
> detect the issue as soon as AZ count drops below minimum. This way might
> also be good for fuzzy warning condition detection point for proactive
> infrastructure operations. If you count all of the nodes in each AZ, you
> can detect and track AZ load imbalances as the ratio between the smallest
> AZ node count and the average AZ node count.
>
>
>> 2. In my original message with 2 nodes(node1 and node2) in AZ1, and
>> 3rdnode in second AZ, backups of node1 and node2 would be placed one node 3
>> in AZ2. It would mean it need to have 2X space to store backups.
>> Just trying to ensure my understanding is correct.
>>
>
> If you have three nodes, you divide your total footprint by three to get
> the minimum node capacity.
>
> If you have 2 backups, that is one primary copy plus two more backup
> copies, so you multiply your total footprint by 3.
>
> If you multiply, say 32GB by three for redundancy, that would be 96GB
> total space needed for the sum of all nodes' footprint.
>
> If you divide the 96GB storage commitment among three nodes, then each
> node must have a minimum of 32GB. That's what we started with as a nominal
> data footprint, so 1x not 2x. Node 1 will need to accommodate backups from
> node 2 and node 3. Node 2 will need to accommodate backups from node 1 and
> node 3. Each node has one primary and two backup partition copies for each
> partition of each cache with two backups.
>
>
>> Hope my queries are clear to you now
>>
>
> I still don't understand your operational goals, so I feel like we may be
> dancing around a misunderstanding.
>
>
>> On Tue, 1 Nov 2022, 20:19 Surinder Mehra,  wrote:
>>
>>> Thanks for your reply. Let me try to answer your 2 questions below.
>>> 1. I understand that it sacrifices the backups incase it can't place
>>> backups appropriately. Question is, is it possible to fail the deployment
>>> rather than risking single copy of data present in cluster. If this only
>>> copy goes down, we will have downtime as data won't be present in cluster.
>>> We should rather throw error if enough hardware is not present than risking
>>> data unavailability issue during business activity
>>>

Re: Backup filter in ignite [Multi AZ deployment]

[Surinder Mehra]

Thanks for suggestions. Will try to ensure infra as suggested and will
explore topology validator if this can be used.

On Tue, 1 Nov 2022, 21:51 Jeremy McMillan, 
wrote:

> Can you tell two stories which start out all nodes in the intended cluster
> configuration are down, one story resulting in a successful cluster
> startup, but the other detecting an invalid configuration, and refusing to
> start?
>
> I can anticipate problems understanding what to do when the first node
> attempts to start, but only has its own AZ represented in the topology. How
> can this first node know whether future nodes will be able to fulfill the
> condition backup_replicas + 1 >=  AZ_count? The general case, allowing
> elastic deployment, requires individual Ignite nodes to work in a
> best-effort capacity.
>
> I would approach this from a DevOps perspective, and just validate the
> deployment before starting up any infrastructure. Look at all of the
> relevant config files which would be deployed. Enumerate a projection of
> deployed nodes and their AZs. Compare this against the desired backup
> filter configuration and fail before starting any Ignite nodes with a
> deployment automation tool exception.
>
> On Tue, Nov 1, 2022 at 9:49 AM Surinder Mehra  wrote:
>
>> Thanks for your reply. Let me try to answer your 2 questions below.
>> 1. I understand that it sacrifices the backups incase it can't place
>> backups appropriately. Question is, is it possible to fail the deployment
>> rather than risking single copy of data present in cluster. If this only
>> copy goes down, we will have downtime as data won't be present in cluster.
>> We should rather throw error if enough hardware is not present than risking
>> data unavailability issue during business activity
>>
>> 2. Why we want 3 copies of data. It's a design choice. We want to ensure
>> even if 2 nodes go down, we still have 3rd present to serve the data.
>>
>> Hope I answered your question
>>
>> On Tue, 1 Nov 2022, 19:40 Jeremy McMillan, 
>> wrote:
>>
>>> This question is a design question.
>>>
>>> What kids of fault states do you expect to tolerate? What is your
>>> failure budget?
>>>
>>> Why are you trying to make more than 2 copies of the data distribute
>>> across only two failure domains?
>>>
>>> Also "fail fast" means discover your implementation defects faster than
>>> your release cycle, not how fast you can cause data loss.
>>>
>>> On Tue, Nov 1, 2022, 09:01 Surinder Mehra  wrote:
>>>
>>>> gentle reminder.
>>>> One additional question: We have observed that if available AZs are
>>>> less than backups count, ignite skips creating backups. Is this correct
>>>> understanding? If yes, how can we fail fast if backups can not be placed
>>>> due to AZ limitation?
>>>>
>>>> On Mon, Oct 31, 2022 at 6:30 PM Surinder Mehra 
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> As per link attached, to ensure primary and backup partitions are not
>>>>> stored on same node, We used AWS AZ as backup filter and now I can see if 
>>>>> I
>>>>> start two ignite nodes on the same machine, primary partitions are evenly
>>>>> distributed but backups are always zero which is expected.
>>>>>
>>>>>
>>>>> https://www.gridgain.com/docs/latest/installation-guide/aws/multiple-availability-zone-aws
>>>>>
>>>>> My question is what would happen if AZ-1 has 2 machines and AZ-2 has 1
>>>>> machine and ignite cluster has only 3 nodes, each machine having one 
>>>>> ignite
>>>>> node.
>>>>>
>>>>> Node1[AZ1] - keys 1-100
>>>>> Node2[AZ1] -  keys 101-200
>>>>> Node3[AZ2] - keys  201 -300
>>>>>
>>>>> In the above scenario, if the backup count is 2, how would back up
>>>>> partitions be distributed.
>>>>>
>>>>> 1. Would it mean node3 will have 2 backup copies of primary partitions
>>>>> of node 1 and 2 ?
>>>>> 2. If we have a 4 node cluster with 2 nodes in each AZ, would backup
>>>>> copies also be placed on different nodes(In other words, does the backup
>>>>> filter also apply to how backup copies are placed on nodes) ?
>>>>>
>>>>>
>>>>>

Re: Backup filter in ignite [Multi AZ deployment]

[Surinder Mehra]

Even if we have 2 copies of data and primary and backup copy would be
stored in different AZs. My question remains valid in this case as well.

Do we have to ensure nodes in two AZs are always present or does ignite
have a way to indicate it couldn't create backups. Silently killing backups
is not desirable state.

2. In my original message with 2 nodes(node1 and node2) in AZ1, and 3rdnode
in second AZ, backups of node1 and node2 would be placed one node 3 in AZ2.
It would mean it need to have 2X space to store backups.
Just trying to ensure my understanding is correct.

Hope my queries are clear to you now

On Tue, 1 Nov 2022, 20:19 Surinder Mehra,  wrote:

> Thanks for your reply. Let me try to answer your 2 questions below.
> 1. I understand that it sacrifices the backups incase it can't place
> backups appropriately. Question is, is it possible to fail the deployment
> rather than risking single copy of data present in cluster. If this only
> copy goes down, we will have downtime as data won't be present in cluster.
> We should rather throw error if enough hardware is not present than risking
> data unavailability issue during business activity
>
> 2. Why we want 3 copies of data. It's a design choice. We want to ensure
> even if 2 nodes go down, we still have 3rd present to serve the data.
>
> Hope I answered your question
>
> On Tue, 1 Nov 2022, 19:40 Jeremy McMillan, 
> wrote:
>
>> This question is a design question.
>>
>> What kids of fault states do you expect to tolerate? What is your failure
>> budget?
>>
>> Why are you trying to make more than 2 copies of the data distribute
>> across only two failure domains?
>>
>> Also "fail fast" means discover your implementation defects faster than
>> your release cycle, not how fast you can cause data loss.
>>
>> On Tue, Nov 1, 2022, 09:01 Surinder Mehra  wrote:
>>
>>> gentle reminder.
>>> One additional question: We have observed that if available AZs are less
>>> than backups count, ignite skips creating backups. Is this correct
>>> understanding? If yes, how can we fail fast if backups can not be placed
>>> due to AZ limitation?
>>>
>>> On Mon, Oct 31, 2022 at 6:30 PM Surinder Mehra 
>>> wrote:
>>>
>>>> Hi,
>>>> As per link attached, to ensure primary and backup partitions are not
>>>> stored on same node, We used AWS AZ as backup filter and now I can see if I
>>>> start two ignite nodes on the same machine, primary partitions are evenly
>>>> distributed but backups are always zero which is expected.
>>>>
>>>>
>>>> https://www.gridgain.com/docs/latest/installation-guide/aws/multiple-availability-zone-aws
>>>>
>>>> My question is what would happen if AZ-1 has 2 machines and AZ-2 has 1
>>>> machine and ignite cluster has only 3 nodes, each machine having one ignite
>>>> node.
>>>>
>>>> Node1[AZ1] - keys 1-100
>>>> Node2[AZ1] -  keys 101-200
>>>> Node3[AZ2] - keys  201 -300
>>>>
>>>> In the above scenario, if the backup count is 2, how would back up
>>>> partitions be distributed.
>>>>
>>>> 1. Would it mean node3 will have 2 backup copies of primary partitions
>>>> of node 1 and 2 ?
>>>> 2. If we have a 4 node cluster with 2 nodes in each AZ, would backup
>>>> copies also be placed on different nodes(In other words, does the backup
>>>> filter also apply to how backup copies are placed on nodes) ?
>>>>
>>>>
>>>>

Re: Backup filter in ignite [Multi AZ deployment]

[Surinder Mehra]

Thanks for your reply. Let me try to answer your 2 questions below.
1. I understand that it sacrifices the backups incase it can't place
backups appropriately. Question is, is it possible to fail the deployment
rather than risking single copy of data present in cluster. If this only
copy goes down, we will have downtime as data won't be present in cluster.
We should rather throw error if enough hardware is not present than risking
data unavailability issue during business activity

2. Why we want 3 copies of data. It's a design choice. We want to ensure
even if 2 nodes go down, we still have 3rd present to serve the data.

Hope I answered your question

On Tue, 1 Nov 2022, 19:40 Jeremy McMillan, 
wrote:

> This question is a design question.
>
> What kids of fault states do you expect to tolerate? What is your failure
> budget?
>
> Why are you trying to make more than 2 copies of the data distribute
> across only two failure domains?
>
> Also "fail fast" means discover your implementation defects faster than
> your release cycle, not how fast you can cause data loss.
>
> On Tue, Nov 1, 2022, 09:01 Surinder Mehra  wrote:
>
>> gentle reminder.
>> One additional question: We have observed that if available AZs are less
>> than backups count, ignite skips creating backups. Is this correct
>> understanding? If yes, how can we fail fast if backups can not be placed
>> due to AZ limitation?
>>
>> On Mon, Oct 31, 2022 at 6:30 PM Surinder Mehra 
>> wrote:
>>
>>> Hi,
>>> As per link attached, to ensure primary and backup partitions are not
>>> stored on same node, We used AWS AZ as backup filter and now I can see if I
>>> start two ignite nodes on the same machine, primary partitions are evenly
>>> distributed but backups are always zero which is expected.
>>>
>>>
>>> https://www.gridgain.com/docs/latest/installation-guide/aws/multiple-availability-zone-aws
>>>
>>> My question is what would happen if AZ-1 has 2 machines and AZ-2 has 1
>>> machine and ignite cluster has only 3 nodes, each machine having one ignite
>>> node.
>>>
>>> Node1[AZ1] - keys 1-100
>>> Node2[AZ1] -  keys 101-200
>>> Node3[AZ2] - keys  201 -300
>>>
>>> In the above scenario, if the backup count is 2, how would back up
>>> partitions be distributed.
>>>
>>> 1. Would it mean node3 will have 2 backup copies of primary partitions
>>> of node 1 and 2 ?
>>> 2. If we have a 4 node cluster with 2 nodes in each AZ, would backup
>>> copies also be placed on different nodes(In other words, does the backup
>>> filter also apply to how backup copies are placed on nodes) ?
>>>
>>>
>>>

Re: Backup filter in ignite [Multi AZ deployment]

[Surinder Mehra]

gentle reminder.
One additional question: We have observed that if available AZs are less
than backups count, ignite skips creating backups. Is this correct
understanding? If yes, how can we fail fast if backups can not be placed
due to AZ limitation?

On Mon, Oct 31, 2022 at 6:30 PM Surinder Mehra  wrote:

> Hi,
> As per link attached, to ensure primary and backup partitions are not
> stored on same node, We used AWS AZ as backup filter and now I can see if I
> start two ignite nodes on the same machine, primary partitions are evenly
> distributed but backups are always zero which is expected.
>
>
> https://www.gridgain.com/docs/latest/installation-guide/aws/multiple-availability-zone-aws
>
> My question is what would happen if AZ-1 has 2 machines and AZ-2 has 1
> machine and ignite cluster has only 3 nodes, each machine having one ignite
> node.
>
> Node1[AZ1] - keys 1-100
> Node2[AZ1] -  keys 101-200
> Node3[AZ2] - keys  201 -300
>
> In the above scenario, if the backup count is 2, how would back up
> partitions be distributed.
>
> 1. Would it mean node3 will have 2 backup copies of primary partitions of
> node 1 and 2 ?
> 2. If we have a 4 node cluster with 2 nodes in each AZ, would backup
> copies also be placed on different nodes(In other words, does the backup
> filter also apply to how backup copies are placed on nodes) ?
>
>
>

Backup filter in ignite [Multi AZ deployment]

[Surinder Mehra]

Hi,
As per link attached, to ensure primary and backup partitions are not
stored on same node, We used AWS AZ as backup filter and now I can see if I
start two ignite nodes on the same machine, primary partitions are evenly
distributed but backups are always zero which is expected.

https://www.gridgain.com/docs/latest/installation-guide/aws/multiple-availability-zone-aws

My question is what would happen if AZ-1 has 2 machines and AZ-2 has 1
machine and ignite cluster has only 3 nodes, each machine having one ignite
node.

Node1[AZ1] - keys 1-100
Node2[AZ1] -  keys 101-200
Node3[AZ2] - keys  201 -300

In the above scenario, if the backup count is 2, how would back up
partitions be distributed.

1. Would it mean node3 will have 2 backup copies of primary partitions of
node 1 and 2 ?
2. If we have a 4 node cluster with 2 nodes in each AZ, would backup copies
also be placed on different nodes(In other words, does the backup filter
also apply to how backup copies are placed on nodes) ?

Re: Re[6]: Checkpointing threads

[Surinder Mehra]

Hi,
Throttling is disabled in ignite config as mentioned in prev reply. What do
you suggest to make ignite catchup with SSD limits on checkpointing.

On Mon, 12 Sept 2022, 11:32 Zhenya Stanilovsky via user, <
user@ignite.apache.org> wrote:

>
>
>
>
>
> We have observed one interesting issue with checkpointing. We are using
> 64G RAM 12 CPU with 3K iops/128mbps SSDs. Our application fills up the WAL
> directory really fast and hence the RAM. We made the following observations
>
> 0. Not so bad news first, it resumes processing after getting stuck for
> several minutes.
>
> 1. WAL and WAL Archive writes are a lot faster than writes to the work
> directory through checkpointing. Very curious to know why this is the case.
> checkpointing writes never exceeds 15 mbps while wal and wal archive go
> really high upto max limits of ssd
>
>
> Very simple example : sequential changing of 1 key, so in wal you obtain
> all changes and in (in your terms — checkpointing) only one key change.
>
>
>
> 2. We observed that when offheap memory usage tend to zero , checkpointing
> takes minutes to complete , sometimes 30+ minutes which stalls the
> application writes completely on all nodes. It means the whole cluster
> freezes.
>
>
> Seems ignite enables throttling in such a case, you need some system and
> cluster tuning.
>
>
>
> 3. Checkpointing thread get stuck at checkpointing page futures.get and
> after several minutes, it logs this error and grid resumes processing
>
> "sys-stripe-0-#1" #19 prio=5 os_prio=0 cpu=86537.69ms elapsed=2166.63s
> tid=0x7fa52a6f1000 nid=0x3b waiting on condition  [0x7fa4c58be000]
>java.lang.Thread.State: WAITING (parking)
> at jdk.internal.misc.Unsafe.park(java.base@11.0.14.1/Native Method)
> at java.util.concurrent.locks.LockSupport.park(java.base@11.0.14.1/Unknown
> Source)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.getUninterruptibly(GridFutureAdapter.java:146)
> at
> org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointTimeoutLock.checkpointReadLock(CheckpointTimeoutLock.java:144)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1613)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3313)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:143)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:322)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:317)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1151)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:592)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:393)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:319)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:110)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:309)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1908)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1529)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$5300(GridIoManager.java:242)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1422)
> at
> org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55)
> at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:569)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(java.base@11.0.14.1/Unknown Source)
> CheckpointProgress pages = checkpointer.scheduleCheckpoint(0, "too many
> dirty pages");
>
> checkpointReadWriteLock.readUnlock();
>
> if (timeout > 0 && U.currentTimeMillis() - start >= timeout)
> failCheckpointReadLock();
>
> try {
> pages
> .futureFor(LOCK_RELEASED)
> .getUninterruptibly();
> }
>
>
> [2022-09-09 18:58:35,148][ERROR][sys-stripe-9-#10][CheckpointTimeoutLock]
> Checkpoint read lock acquisition has been timed out.
> class
> org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointTimeoutLock$CheckpointReadLockTimeoutException:
> Checkpoint read lock acquisition has been timed out.
> at
> org

Re: Re[4]: Checkpointing threads

[Surinder Mehra]

We have observed one interesting issue with checkpointing. We are using 64G
RAM 12 CPU with 3K iops/128mbps SSDs. Our application fills up the WAL
directory really fast and hence the RAM. We made the following observations

0. Not so bad news first, it resumes processing after getting stuck for
several minutes.

1. WAL and WAL Archive writes are a lot faster than writes to the work
directory through checkpointing. Very curious to know why this is the case.
checkpointing writes never exceeds 15 mbps while wal and wal archive go
really high upto max limits of ssd

2. We observed that when offheap memory usage tend to zero , checkpointing
takes minutes to complete , sometimes 30+ minutes which stalls the
application writes completely on all nodes. It means the whole cluster
freezes.

3. Checkpointing thread get stuck at checkpointing page futures.get and
after several minutes, it logs this error and grid resumes processing

"sys-stripe-0-#1" #19 prio=5 os_prio=0 cpu=86537.69ms elapsed=2166.63s
tid=0x7fa52a6f1000 nid=0x3b waiting on condition  [0x7fa4c58be000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.14.1/Native Method)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.14.1/Unknown
Source)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.getUninterruptibly(GridFutureAdapter.java:146)
at
org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointTimeoutLock.checkpointReadLock(CheckpointTimeoutLock.java:144)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1613)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3313)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:143)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:322)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:317)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1151)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:592)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:393)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:319)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:110)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:309)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1908)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1529)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5300(GridIoManager.java:242)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1422)
at
org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:569)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(java.base@11.0.14.1/Unknown Source)
CheckpointProgress pages = checkpointer.scheduleCheckpoint(0, "too many
dirty pages");

checkpointReadWriteLock.readUnlock();

if (timeout > 0 && U.currentTimeMillis() - start >= timeout)
failCheckpointReadLock();

try {
pages
.futureFor(LOCK_RELEASED)
.getUninterruptibly();
}


[2022-09-09 18:58:35,148][ERROR][sys-stripe-9-#10][CheckpointTimeoutLock]
Checkpoint read lock acquisition has been timed out.
class
org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointTimeoutLock$CheckpointReadLockTimeoutException:
Checkpoint read lock acquisition has been timed out.
at
org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointTimeoutLock.failCheckpointReadLock(CheckpointTimeoutLock.java:210)
at
org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointTimeoutLock.checkpointReadLock(CheckpointTimeoutLock.java:108)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1613)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3313)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$6

Re: ClassCastException while using ignite service proxy

[Surinder Mehra]

Hi Stephen,
I see you are deploying service from the same client node where proxy is
obtained.
In my setup, I have deployed service through ignite config on server start
and try to create a client later and hence the proxy. It works when I try
to obtain a proxy on the server node. But when I start a client node and
try to obtain service instance through proxy, it throws this exception
mentioned above

On Wed, Aug 31, 2022 at 6:13 PM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> You’ll need to share more of your code and configuration. As far as I can
> tell, it works. This is my entire code/configuration, using Ignite 2.11.1
> and Java 11.0.16.1+1.
>
> var igniteConfiguration = new IgniteConfiguration()
> .setPeerClassLoadingEnabled(true)
> .setClientMode(true);
> try (var ignite = Ignition.start(igniteConfiguration)) {
> var cfg = new ServiceConfiguration()
> .setName("MyService")
> .setTotalCount(1)
> .setMaxPerNodeCount(1)
> .setNodeFilter(x -> !x.isClient())
> .setService(new MyServiceImpl());
> ignite.services().deploy(cfg);
>
>   var s = ignite.services().serviceProxy("MyService", MyService.class, false);
> s.sayHello();
> }
>
> public interface MyService {
> public void sayHello();
> }
>
> public class MyServiceImpl implements MyService, Service {
> @Override
> public void cancel(ServiceContext serviceContext) {
>
> }
>
> @Override
> public void init(ServiceContext serviceContext) throws Exception {
>
> }
>
> @Override
> public void execute(ServiceContext serviceContext) throws Exception {
>
> }
>
> @Override
> public void sayHello() {
> System.out.println("Hello, world.");
> }
> }
>
> On 31 Aug 2022, at 04:17, Surinder Mehra  wrote:
>
> Please find below
> ignite version: apache-ignite-2.11.1
> VM information: OpenJDK Runtime Environment 11.0.15+9-LTS
>
> On Wed, Aug 31, 2022 at 12:12 AM Stephen Darlington <
> stephen.darling...@gridgain.com> wrote:
>
>> Which version of Ignite? Which version of Java?
>>
>> On 30 Aug 2022, at 13:40, Surinder Mehra  wrote:
>>
>>
>> 
>> Hi Stephen ,
>>  yes that is implemented correctly and it's running on server nodes as
>> well. Somehow it doesn't work when accessed through proxy
>>
>> On Tue, Aug 30, 2022 at 5:45 PM Stephen Darlington <
>> stephen.darling...@gridgain.com> wrote:
>>
>>> Your service needs to implement org.apache.ignite.services.Service.
>>>
>>> > On 30 Aug 2022, at 12:40, Surinder Mehra  wrote:
>>> >
>>> > Hi,
>>> > can you help me find out the reason for this exception in thick client
>>> while getting instance of ignite service:
>>> >
>>> > getIgnite()
>>> > .services()
>>> > .serviceProxy("sampleService", SampleService.class, false)
>>> >
>>> > java.lang.ClassCastException: class com.sun.proxy.$Proxy148 cannot be
>>> cast to class com.test.ignite.stuff.services.SampleServiceImpl
>>> (com.sun.proxy.$Proxy148 and
>>> com.test.ignite.stuff.services.SampleServiceImpl are in unnamed module of
>>> loader 'app')
>>> >
>>> > interface SampleService{
>>> >
>>> > }
>>> >
>>> > class SampleServiceImpl implements SampleService{
>>> >
>>> > }
>>> >
>>> > ignite config:
>>> >
>>> > 
>>> >   
>>> > 
>>> >   
>>> >   
>>> >   
>>> >   
>>> > >> class="com.test.ignite.stuff.services.SampleServiceImpl"/>
>>> >   
>>> >   
>>> > >> class="com.test.ignite.stuff.node.filter.ServerNodeFilter"/>
>>> >   
>>> > 
>>> >   
>>> > 
>>> >
>>> >
>>> >
>>>
>>>
>

Re: ClassCastException while using ignite service proxy

[Surinder Mehra]

Please find below
ignite version: apache-ignite-2.11.1
VM information: OpenJDK Runtime Environment 11.0.15+9-LTS

On Wed, Aug 31, 2022 at 12:12 AM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> Which version of Ignite? Which version of Java?
>
> On 30 Aug 2022, at 13:40, Surinder Mehra  wrote:
>
>
> 
> Hi Stephen ,
>  yes that is implemented correctly and it's running on server nodes as
> well. Somehow it doesn't work when accessed through proxy
>
> On Tue, Aug 30, 2022 at 5:45 PM Stephen Darlington <
> stephen.darling...@gridgain.com> wrote:
>
>> Your service needs to implement org.apache.ignite.services.Service.
>>
>> > On 30 Aug 2022, at 12:40, Surinder Mehra  wrote:
>> >
>> > Hi,
>> > can you help me find out the reason for this exception in thick client
>> while getting instance of ignite service:
>> >
>> > getIgnite()
>> > .services()
>> > .serviceProxy("sampleService", SampleService.class, false)
>> >
>> > java.lang.ClassCastException: class com.sun.proxy.$Proxy148 cannot be
>> cast to class com.test.ignite.stuff.services.SampleServiceImpl
>> (com.sun.proxy.$Proxy148 and
>> com.test.ignite.stuff.services.SampleServiceImpl are in unnamed module of
>> loader 'app')
>> >
>> > interface SampleService{
>> >
>> > }
>> >
>> > class SampleServiceImpl implements SampleService{
>> >
>> > }
>> >
>> > ignite config:
>> >
>> > 
>> >   
>> > 
>> >   
>> >   
>> >   
>> >   
>> > > class="com.test.ignite.stuff.services.SampleServiceImpl"/>
>> >   
>> >   
>> > > class="com.test.ignite.stuff.node.filter.ServerNodeFilter"/>
>> >   
>> > 
>> >   
>> > 
>> >
>> >
>> >
>>
>>

Re: ClassCastException while using ignite service proxy

[Surinder Mehra]

Hi Stephen ,
 yes that is implemented correctly and it's running on server nodes as
well. Somehow it doesn't work when accessed through proxy

On Tue, Aug 30, 2022 at 5:45 PM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> Your service needs to implement org.apache.ignite.services.Service.
>
> > On 30 Aug 2022, at 12:40, Surinder Mehra  wrote:
> >
> > Hi,
> > can you help me find out the reason for this exception in thick client
> while getting instance of ignite service:
> >
> > getIgnite()
> > .services()
> > .serviceProxy("sampleService", SampleService.class, false)
> >
> > java.lang.ClassCastException: class com.sun.proxy.$Proxy148 cannot be
> cast to class com.test.ignite.stuff.services.SampleServiceImpl
> (com.sun.proxy.$Proxy148 and
> com.test.ignite.stuff.services.SampleServiceImpl are in unnamed module of
> loader 'app')
> >
> > interface SampleService{
> >
> > }
> >
> > class SampleServiceImpl implements SampleService{
> >
> > }
> >
> > ignite config:
> >
> > 
> >   
> > 
> >   
> >   
> >   
> >   
> >  class="com.test.ignite.stuff.services.SampleServiceImpl"/>
> >   
> >   
> >  class="com.test.ignite.stuff.node.filter.ServerNodeFilter"/>
> >   
> > 
> >   
> > 
> >
> >
> >
>
>

ClassCastException while using ignite service proxy

[Surinder Mehra]

Hi,
can you help me find out the reason for this exception in thick client
while getting instance of ignite service:

getIgnite()
.services()
.serviceProxy("sampleService", SampleService.class, false)

java.lang.ClassCastException: class com.sun.proxy.$Proxy148 cannot be cast
to class com.test.ignite.stuff.services.SampleServiceImpl
(com.sun.proxy.$Proxy148 and
com.test.ignite.stuff.services.SampleServiceImpl are in unnamed module of
loader 'app')

interface SampleService{

}

class SampleServiceImpl implements SampleService{

}

ignite config:


  

  
  
  
  

  
  

  

  

Re: Ignite-schedule

[Surinder Mehra]

I need to schedule a task in ignite cluster every day same time. Since
ignite is java application, spring cron job is not useful here. Can you
please suggest an alternative

On Fri, 26 Aug 2022, 23:01 Gianluca Bonetti, 
wrote:

> Hello
>
> I don't think that piece of software is being used anywhere.
> You can see it's an old jar from year 2015, so definitely ancient and not
> up to date.
> I never had to use that ignite-schedule anywhere in my projects.
> Any particular reason to resurrect this old component?
>
> Cheers
> Gianluca
>
> On Fri, 26 Aug 2022 at 18:08, Surinder Mehra  wrote:
>
>> Hi,
>> Could you please suggest if it is safe to use below version of
>> ignite-schedule
>> It has warnings about CVE
>>
>>
>> https://mvnrepository.com/artifact/org.apache.ignite/ignite-schedule/1.2.0-incubating
>>
>> Vulnerabilities from dependencies:
>> CVE-2020-1963
>> <https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-1963>
>> CVE-2018-8018
>> <https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-8018>
>> CVE-2018-1295
>> <https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1295>
>> CVE-2017-7686
>> <https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-7686>
>>
>

Ignite-schedule

[Surinder Mehra]

Hi,
Could you please suggest if it is safe to use below version of
ignite-schedule
It has warnings about CVE

https://mvnrepository.com/artifact/org.apache.ignite/ignite-schedule/1.2.0-incubating

Vulnerabilities from dependencies:
CVE-2020-1963 
CVE-2018-8018 
CVE-2018-1295 
CVE-2017-7686 

Re: Query default Index in Ignite Sql cache

[Surinder Mehra]

Thanks Maksim !

On Thu, Jun 23, 2022 at 11:01 AM Maksim Timonin 
wrote:

> Hi,
>
> > 1. Department object has an id and affinity key on the name(it
> doesn't make sense in the real world). Does ignite create an index for
> affinity keys as well ? I don't see the affinity key index in the list
> below.
>
> You need to configure affinity key with annotations a little bit
> differently, please check docs [1]: AffinityKeyMapped must be part of cache
> key, not value object. AffinityKeyMapped doesn't affect if it is used for
> field in cache value.
> So, it should look like
>
> DepartmentKey {
> @QuerySqlField
> @AffinityKeyMapped
> private final String deptName;
> ...
> }
>
> and then cache will be like IgniteCache.
>
> > 2. When I use the default index explicitly in IndexQuery, it throws an
> exception that the index name doesn't exist.
>
> Which index name did you use?
>
> > So by default it uses primary key index or hash index when* IndexQuery* is
> executed with _KEY in criteria* ?*
>
> They PrimaryKey index executes if criteria consist of single _KEY criteria.
>
> [1]
> https://ignite.apache.org/docs/2.11.1/data-modeling/affinity-collocation#configuring-affinity-key
>
> On Wed, Jun 22, 2022 at 9:49 PM Surinder Mehra  wrote:
>
>> Thanks for the reply. I have another question on the affinity key
>> index(if any). When we enable sql on ignite, it by default creates an index
>> on the primary key(highlighted in red below). We have created a custom
>> index on deptId(highlighted in red).
>>
>> 1. Department object has an id and affinity key on the name(it
>> doesn't make sense in the real world). Does ignite create an index for
>> affinity keys as well ? I don't see the affinity key index in the list
>> below.
>> 2. When I use the default index explicitly in IndexQuery, it throws an
>> exception that the index name doesn't exist. So by default it uses primary
>> key index or hash index when* IndexQuery* is executed with _KEY in
>> criteria* ?*
>>
>> [image: image.png]
>> [image: image.png]
>>
>> On Wed, Jun 22, 2022 at 9:35 PM Николай Ижиков 
>> wrote:
>>
>>> SELECT * FROM SYS.INDEXES
>>>
>>> 22 июня 2022 г., в 18:38, Surinder Mehra 
>>> написал(а):
>>>
>>> Hi,
>>> We have defined indexes on sql enabled ignite cache and are able to see
>>> indexes being used while querying.
>>> sqline *!indexes* also shows those indexes in output. But we cant see
>>> default index created by ignite on *primary key *and *affinity key*.
>>> We would like to use index on key and affinity key to get results sorted
>>> accordingly.
>>>
>>> For example, in official docs link below, Is it possible to use the
>>> above mentioned default indexes instead of ORG_SALARY_IDX ?
>>> How can we get more details about these indexes which are not shown in
>>> sqlline commands.
>>>
>>>
>>>
>>> https://ignite.apache.org/docs/latest/key-value-api/using-cache-queries#executing-index-queries
>>>
>>>
>>> QueryCursor> cursor = cache.query(
>>> new IndexQuery(Person.class, "ORG_SALARY_IDX")
>>> .setCriteria(eq("orgId", 1), gt("salary", 1000)));
>>>
>>>
>>>

Re: Query default Index in Ignite Sql cache

[Surinder Mehra]

Thanks for the reply. I have another question on the affinity key index(if
any). When we enable sql on ignite, it by default creates an index on the
primary key(highlighted in red below). We have created a custom index on
deptId(highlighted in red).

1. Department object has an id and affinity key on the name(it doesn't make
sense in the real world). Does ignite create an index for affinity keys as
well ? I don't see the affinity key index in the list below.
2. When I use the default index explicitly in IndexQuery, it throws an
exception that the index name doesn't exist. So by default it uses primary
key index or hash index when* IndexQuery* is executed with _KEY in criteria*
?*

[image: image.png]
[image: image.png]

On Wed, Jun 22, 2022 at 9:35 PM Николай Ижиков  wrote:

> SELECT * FROM SYS.INDEXES
>
> 22 июня 2022 г., в 18:38, Surinder Mehra  написал(а):
>
> Hi,
> We have defined indexes on sql enabled ignite cache and are able to see
> indexes being used while querying.
> sqline *!indexes* also shows those indexes in output. But we cant see
> default index created by ignite on *primary key *and *affinity key*.
> We would like to use index on key and affinity key to get results sorted
> accordingly.
>
> For example, in official docs link below, Is it possible to use the above
> mentioned default indexes instead of ORG_SALARY_IDX ?
> How can we get more details about these indexes which are not shown in
> sqlline commands.
>
>
>
> https://ignite.apache.org/docs/latest/key-value-api/using-cache-queries#executing-index-queries
>
>
> QueryCursor> cursor = cache.query(
> new IndexQuery(Person.class, "ORG_SALARY_IDX")
> .setCriteria(eq("orgId", 1), gt("salary", 1000)));
>
>
>

Query default Index in Ignite Sql cache

[Surinder Mehra]

Hi,
We have defined indexes on sql enabled ignite cache and are able to see
indexes being used while querying.
sqline *!indexes* also shows those indexes in output. But we cant see
default index created by ignite on *primary key *and *affinity key*.
We would like to use index on key and affinity key to get results sorted
accordingly.

For example, in official docs link below, Is it possible to use the above
mentioned default indexes instead of ORG_SALARY_IDX ?
How can we get more details about these indexes which are not shown in
sqlline commands.


https://ignite.apache.org/docs/latest/key-value-api/using-cache-queries#executing-index-queries


QueryCursor> cursor = cache.query(
new IndexQuery(Person.class, "ORG_SALARY_IDX")
.setCriteria(eq("orgId", 1), gt("salary", 1000)));

Re: Get SqlFieldQueryResults by name

[Surinder Mehra]

Created: https://issues.apache.org/jira/browse/IGNITE-17158

Thanks

On Tue, Jun 14, 2022 at 2:10 PM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> You can file a ticket on Ignite’s Jira:
> https://issues.apache.org/jira/browse/IGNITE
>
> More, in general, about how to get involved here:
> https://ignite.apache.org/our-community.html#faq
>
> On 13 Jun 2022, at 11:38, Surinder Mehra  wrote:
>
> Hello,
> Can someone please log a feature request. This would be very useful
> feature for everyone using SQL queries
>
> On Fri, Apr 8, 2022 at 8:03 PM Surinder Mehra  wrote:
>
>> Ok thanks for letting me know. Is there a guideline for raising feature
>> request.
>>
>> On Fri, 8 Apr 2022, 18:10 Stephen Darlington, <
>> stephen.darling...@gridgain.com> wrote:
>>
>>> That would be a nice feature but I don’t think it’s currently possible.
>>> It may be worth creating a feature request ticket.
>>>
>>> On 8 Apr 2022, at 13:30, Surinder Mehra  wrote:
>>>
>>> Hi,
>>> I was going through the below example and have a question if we can get
>>> fields using field names from query results.
>>> like "select name as person_name, p.age as person_age from Person p"
>>> It would return List. The inner list is each person row returned
>>> with results as per order in sql query. While this works fine but I was
>>> wondering if we can fetch row fields by name just like Oracle sql query
>>> rowMapper
>>>
>>>
>>> https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/sql/SqlDmlExample.java
>>>
>>>
>>>
>>>
>

Re: Get SqlFieldQueryResults by name

[Surinder Mehra]

Hello,
Can someone please log a feature request. This would be very useful feature
for everyone using SQL queries

On Fri, Apr 8, 2022 at 8:03 PM Surinder Mehra  wrote:

> Ok thanks for letting me know. Is there a guideline for raising feature
> request.
>
> On Fri, 8 Apr 2022, 18:10 Stephen Darlington, <
> stephen.darling...@gridgain.com> wrote:
>
>> That would be a nice feature but I don’t think it’s currently possible.
>> It may be worth creating a feature request ticket.
>>
>> On 8 Apr 2022, at 13:30, Surinder Mehra  wrote:
>>
>> Hi,
>> I was going through the below example and have a question if we can get
>> fields using field names from query results.
>> like "select name as person_name, p.age as person_age from Person p"
>> It would return List. The inner list is each person row returned
>> with results as per order in sql query. While this works fine but I was
>> wondering if we can fetch row fields by name just like Oracle sql query
>> rowMapper
>>
>>
>> https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/sql/SqlDmlExample.java
>>
>>
>>
>>

Re: Gridgain ultimate 2.8.18 : can't write to work directory

[Surinder Mehra]

Thanks Stephen. Will check with them

On Wed, 8 Jun 2022, 15:18 Stephen Darlington, <
stephen.darling...@gridgain.com> wrote:

> This is a mailing list for Apache ignite users. Please contact GridGain
> for support of their software.
>
> On 8 Jun 2022, at 09:53, Surinder Mehra  wrote:
>
> Hi,
> I deployed gridgain ultimate 2.8.18 to kubernetes with mount point for
> work directory to store ignite data as per steps mentioned  below
>
>
> https://www.gridgain.com/docs/latest/installation-guide/kubernetes/amazon-eks-deployment
>
> It works if I don't use an external  mount point for the work directory
> but  when I use the below configuration, it throws the below error
> saying it cannot write to the work directory.
> mounted volumes as mounted as root with below permission set
> *drwxr-xr-x -> root*
> But the container starts with user gridgain.
>
> Do those volumes need explicit permission or I am missing any other steps
> here ?
>
>
>
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> class org.apache.ignite.IgniteException: Cannot write to work directory:
> /opt/gridgain/work
> at
> org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1130)
> at org.apache.ignite.Ignition.start(Ignition.java:347)
> at
> org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:358)
> Caused by: class org.apache.ignite.IgniteCheckedException: Cannot write to
> work directory: /opt/gridgain/work
> at
> org.apache.ignite.internal.util.IgniteUtils.workDirectory(IgniteUtils.java:10126)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.initializeConfiguration(IgnitionEx.java:1874)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1700)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1143)
> at
> org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1058)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:944)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:843)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:713)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:682)
> at org.apache.ignite.Ignition.start(Ignition.java:344)
> ... 1 more
> Failed to start grid: Cannot write to work directory: /opt/gridgain/work
>
>
>
>
> config:
>
> 
>
>
> 
>
> 
>  class="org.apache.ignite.configuration.DataStorageConfiguration">
> 
>  value="/opt/gridgain/walarchive"/>
> 
> 
> 
>  class="org.gridgain.grid.configuration.GridGainConfiguration">
> 
>  class="org.gridgain.grid.configuration.SnapshotConfiguration">
> 
> 
> 
> 
> 
> 
>  class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
>
> 
> 
>
> 
>
> statefulset config:
> 
> image: gridgain/ultimate:8.8.18-openjdk11-slim
> volumeMounts:
> - mountPath: /opt/gridgain/config
>   mountPropagation: None
>   name: config-vol
> - mountPath: /opt/gridgain/work
>   mountPropagation: None
>   name: work-vol
>
>
>

Gridgain ultimate 2.8.18 : can't write to work directory

[Surinder Mehra]

Hi,
I deployed gridgain ultimate 2.8.18 to kubernetes with mount point for work
directory to store ignite data as per steps mentioned  below

https://www.gridgain.com/docs/latest/installation-guide/kubernetes/amazon-eks-deployment

It works if I don't use an external  mount point for the work directory
but  when I use the below configuration, it throws the below error
saying it cannot write to the work directory.
mounted volumes as mounted as root with below permission set
*drwxr-xr-x -> root*
But the container starts with user gridgain.

Do those volumes need explicit permission or I am missing any other steps
here ?



SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
class org.apache.ignite.IgniteException: Cannot write to work directory:
/opt/gridgain/work
at
org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1130)
at org.apache.ignite.Ignition.start(Ignition.java:347)
at
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:358)
Caused by: class org.apache.ignite.IgniteCheckedException: Cannot write to
work directory: /opt/gridgain/work
at
org.apache.ignite.internal.util.IgniteUtils.workDirectory(IgniteUtils.java:10126)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.initializeConfiguration(IgnitionEx.java:1874)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1700)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1143)
at
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1058)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:944)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:843)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:713)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:682)
at org.apache.ignite.Ignition.start(Ignition.java:344)
... 1 more
Failed to start grid: Cannot write to work directory: /opt/gridgain/work




config:



   



















   





statefulset config:

image: gridgain/ultimate:8.8.18-openjdk11-slim
volumeMounts:
- mountPath: /opt/gridgain/config
  mountPropagation: None
  name: config-vol
- mountPath: /opt/gridgain/work
  mountPropagation: None
  name: work-vol

Re: Re[2]: LEFT, RIGHT JOIN not working

[Surinder Mehra]

Hi,
Could you please provide an update on this.

On Mon, Jun 6, 2022 at 11:48 AM Zhenya Stanilovsky 
wrote:

>
> Hi ! thanks for example, i hope some updates will be here in a short time.
>
>
>
>
> Hi,
> Just wondering if you had an opportunity to look into this.
>
> On Thu, Jun 2, 2022 at 2:52 PM Surinder Mehra  > wrote:
>
> Hi,
> Please find the attached java file which reproduces the issue. As you can
> see, the cache key is used as a join condition but LEFT join is still
> giving only common values.
>
> output:
> [2, Keyboard, 2]
> Size of actual output 1
> Expected size 3 is not equal to Actual size 1
>
>
> On Thu, Jun 2, 2022 at 11:48 AM Zhenya Stanilovsky  > wrote:
>
> Hi, Surinder Mehra ! I check your sql and it work correct for me.
>
>1. You no need to define AffinityKeyMapped for Key, check additionally
>[1], you can simple modify [2] according to your case
>2. I problem still exist somehow, plz attach some code example.
>
> thanks !
>
> [1]
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/affinity/AffinityKeyMapped.html
> [2]
> https://github.com/apache/ignite/blob/master/modules/indexing/src/test/java/org/apache/ignite/internal/processors/cache/IgniteCacheJoinPartitionedAndReplicatedTest.java#L160
>
>
>
> Hi,
> I have the following sample code to demo issue in SQL joins. I have
> created an affinity key and value as shown below and added some sample data
> to it. When I try LEFT self join on this table it always gives me common
> rows   irrespective of LEFT or RIGHT JOIN
> Could you please help me find what am I doing wrong here.
>
> cache Key :
>
> public class OrderAffinityKey {
> Integer id;
> @AffinityKeyMapped
> Integer customerId;
> }
>
>
> cache value:
>
> public class Order implements Serializable {
> @QuerySqlField
> Integer id;
>
> @AffinityKeyMapped
> @QuerySqlField Integer customerId;
> @QuerySqlField String product;
> }
>
>
> Table C: (select customerID, product FROM "orderCache"."ORDER" WHERE
> CUSTOMERID IN ( 1, 2))
>
> 1 keyboard
> 2 Laptop
>
>
> Table O: (select customerID, product FROM "orderCache"."ORDER" WHERE
> CUSTOMERID IN ( 3, 2))
>
> 2 laptop
> 3 mouse
>
>
>
> JOIN:
>
> Query :
> select DISTINCT C.customerID, C.product, O.customerID
> FROM
>  (select customerID, product FROM "orderCache"."ORDER" WHERE CUSTOMERID IN
> ( 1, 2)) C
>  LEFT JOIN
> (select customerID, product FROM "orderCache"."ORDER" WHERE CUSTOMERID IN
> ( 3, 2)) O
> ON
> C.customerId = O.customerId
>
>
> Output:
>
> 2 laptop   2
> 3 mouse   3
>
> Expected output:
>
> 1 keyboard   null
> 2 laptop   2
> 3 mouse   3
>
>
>
>
>
>
>
>
>
>
>

Re: gridgain ultimate edition snapshot error

[Surinder Mehra]

Hi,
Thanks for your reply. Current limits are highlighted below. As suggested
in prev reply, I will change limits and try again.

Limit Soft Limit   Hard Limit   Units

Max cpu time  unlimitedunlimitedseconds

Max file size unlimitedunlimitedbytes

Max data size unlimitedunlimitedbytes

Max stack size8388608  unlimitedbytes

Max core file sizeunlimitedunlimitedbytes

Max resident set  unlimitedunlimitedbytes

Max processes 6330663306
 processes
*Max open files1024 4096 files
  *
Max locked memory 6553665536bytes

Max address space unlimitedunlimitedbytes

Max file locksunlimitedunlimitedlocks

Max pending signals   6330663306signals

Max msgqueue size 819200   819200   bytes

Max nice priority 00
Max realtime priority 00
Max realtime timeout  unlimitedunlimitedus

On Tue, Jun 7, 2022 at 1:38 PM Gianluca Bonetti 
wrote:

> Hello
>
> What is returned by this command?
>
> # cat /proc/PID/limits
>
> Cheers
> Gianluca
> Gianluca
>
> On Tue, 7 Jun 2022 at 07:35, Surinder Mehra  wrote:
>
>> Hi,
>> I was going through this post on stackoverflow which is about the same
>> issue. The fact that snapshot works for apache ignite bit not in ultimate
>> edition indicates there is some bug in later. Could you please confirm. We
>> have around 15 caches with 2 backups. I changed backups to zero but still
>> see this issue. Could you please advise further.
>>
>>
>> https://stackoverflow.com/questions/72041292/is-there-a-fix-for-too-many-open-files-error-in-gridgain-cluster
>>
>> On Mon, Jun 6, 2022 at 9:13 PM Surinder Mehra  wrote:
>>
>>> Hi,
>>> I was experimenting with the GG ultimate edition to take snapshots and
>>> encountered the below error and cluster stops. Please note that this works
>>> in the ignite free version and we don't see too many files open error. Is
>>> this a bug or we are missing some configuration?
>>>
>>> version:  gridgain-8.8.19
>>>
>>> /bin./snapshot-utility.sh snapshot -type=full
>>>
>>> [21:03:51,693][SEVERE][db-snapshot-executor-stripe-0-#35][] Critical
>>> system error detected. Will be handled accordingly to configured handler
>>> [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
>>> super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
>>> [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
>>> failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
>>> o.a.i.i.processors.cache.persistence.StorageException: Failed to initialize
>>> partition file:
>>> /home/usr/tools/gridgain-ultimate-8.8.19/work/db/node00-c221fe71-5d29-4cd7-ab0f-9fa8240711b2/cache-name/part-88.bin]
>>> class
>>> org.apache.ignite.internal.processors.cache.persistence.StorageException:
>>> Failed to initialize partition file:
>>> /home/usr/tools/gridgain-ultimate-8.8.19/work/db/node00-c221fe71-5d29-4cd7-ab0f-9fa8240711b2/cache-name/part-88.bin
>>> at
>>> org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.init(FilePageStore.java:519)
>>> at
>>> org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:405)
>>> at
>>> org.apache.ignite.internal.processors.cache.persistence.pagemem.PageReadWriteManagerImpl.read(PageReadWriteManagerImpl.java:68)
>>> at
>>> org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:577)
>>> at
>>> org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:911)
>>> at
>>> org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:730)
>>> at
>>> org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:711)
>>> at
>>> org.gridgain.grid.internal.processors.cache.database.snapshot.SnapshotCreateFuture.completeSavingAllocatedIndex(SnapshotCreateFuture.java:1304)
>>> at
>>> org.gridgain.grid.internal.processors.cache.database.s

Re: gridgain ultimate edition snapshot error

[Surinder Mehra]

Hi,
I was going through this post on stackoverflow which is about the same
issue. The fact that snapshot works for apache ignite bit not in ultimate
edition indicates there is some bug in later. Could you please confirm. We
have around 15 caches with 2 backups. I changed backups to zero but still
see this issue. Could you please advise further.

https://stackoverflow.com/questions/72041292/is-there-a-fix-for-too-many-open-files-error-in-gridgain-cluster

On Mon, Jun 6, 2022 at 9:13 PM Surinder Mehra  wrote:

> Hi,
> I was experimenting with the GG ultimate edition to take snapshots and
> encountered the below error and cluster stops. Please note that this works
> in the ignite free version and we don't see too many files open error. Is
> this a bug or we are missing some configuration?
>
> version:  gridgain-8.8.19
>
> /bin./snapshot-utility.sh snapshot -type=full
>
> [21:03:51,693][SEVERE][db-snapshot-executor-stripe-0-#35][] Critical
> system error detected. Will be handled accordingly to configured handler
> [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
> super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
> [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
> failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
> o.a.i.i.processors.cache.persistence.StorageException: Failed to initialize
> partition file:
> /home/usr/tools/gridgain-ultimate-8.8.19/work/db/node00-c221fe71-5d29-4cd7-ab0f-9fa8240711b2/cache-name/part-88.bin]
> class
> org.apache.ignite.internal.processors.cache.persistence.StorageException:
> Failed to initialize partition file:
> /home/usr/tools/gridgain-ultimate-8.8.19/work/db/node00-c221fe71-5d29-4cd7-ab0f-9fa8240711b2/cache-name/part-88.bin
> at
> org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.init(FilePageStore.java:519)
> at
> org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:405)
> at
> org.apache.ignite.internal.processors.cache.persistence.pagemem.PageReadWriteManagerImpl.read(PageReadWriteManagerImpl.java:68)
> at
> org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:577)
> at
> org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:911)
> at
> org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:730)
> at
> org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:711)
> at
> org.gridgain.grid.internal.processors.cache.database.snapshot.SnapshotCreateFuture.completeSavingAllocatedIndex(SnapshotCreateFuture.java:1304)
> at
> org.gridgain.grid.internal.processors.cache.database.snapshot.SnapshotCreateFuture.completeSnapshotCreation(SnapshotCreateFuture.java:1486)
> at
> org.gridgain.grid.internal.processors.cache.database.snapshot.SnapshotCreateFuture.doFinalStage(SnapshotCreateFuture.java:1171)
> at
> org.gridgain.grid.internal.processors.cache.database.snapshot.SnapshotOperationFuture.completeStagesLocally(SnapshotOperationFuture.java:2352)
> at
> org.gridgain.grid.internal.processors.cache.database.snapshot.SnapshotOperationFuture$10.run(SnapshotOperationFuture.java:2286)
> at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:567)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.nio.file.FileSystemException:
> /home/usr/tools/gridgain-ultimate-8.8.19/work/db/node00-c221fe71-5d29-4cd7-ab0f-9fa8240711b2/cache-name/part-88.bin:
> Too many open files
> at
> java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
> at
> java.base/sun.nio.fs.UnixFileSystemProvider.newAsynchronousFileChannel(UnixFileSystemProvider.java:201)
> at
> java.base/java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:253)
> at
> java.base/java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:311)
> at
> org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIO.(AsyncFileIO.java:65)
> at
> org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory.create(AsyncFileIOFactory.java:43)
> at
> org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.init(FilePageStore.java:491)
> ... 14 more
> [21:03:51,695][SEVERE][db-snapshot-executor-stripe-0-#35][FailureProcessor]
> No dea

gridgain ultimate edition snapshot error

[Surinder Mehra]

Hi,
I was experimenting with the GG ultimate edition to take snapshots and
encountered the below error and cluster stops. Please note that this works
in the ignite free version and we don't see too many files open error. Is
this a bug or we are missing some configuration?

version:  gridgain-8.8.19

/bin./snapshot-utility.sh snapshot -type=full

[21:03:51,693][SEVERE][db-snapshot-executor-stripe-0-#35][] Critical system
error detected. Will be handled accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
o.a.i.i.processors.cache.persistence.StorageException: Failed to initialize
partition file:
/home/usr/tools/gridgain-ultimate-8.8.19/work/db/node00-c221fe71-5d29-4cd7-ab0f-9fa8240711b2/cache-name/part-88.bin]
class
org.apache.ignite.internal.processors.cache.persistence.StorageException:
Failed to initialize partition file:
/home/usr/tools/gridgain-ultimate-8.8.19/work/db/node00-c221fe71-5d29-4cd7-ab0f-9fa8240711b2/cache-name/part-88.bin
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.init(FilePageStore.java:519)
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:405)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageReadWriteManagerImpl.read(PageReadWriteManagerImpl.java:68)
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:577)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:911)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:730)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:711)
at
org.gridgain.grid.internal.processors.cache.database.snapshot.SnapshotCreateFuture.completeSavingAllocatedIndex(SnapshotCreateFuture.java:1304)
at
org.gridgain.grid.internal.processors.cache.database.snapshot.SnapshotCreateFuture.completeSnapshotCreation(SnapshotCreateFuture.java:1486)
at
org.gridgain.grid.internal.processors.cache.database.snapshot.SnapshotCreateFuture.doFinalStage(SnapshotCreateFuture.java:1171)
at
org.gridgain.grid.internal.processors.cache.database.snapshot.SnapshotOperationFuture.completeStagesLocally(SnapshotOperationFuture.java:2352)
at
org.gridgain.grid.internal.processors.cache.database.snapshot.SnapshotOperationFuture$10.run(SnapshotOperationFuture.java:2286)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:567)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.nio.file.FileSystemException:
/home/usr/tools/gridgain-ultimate-8.8.19/work/db/node00-c221fe71-5d29-4cd7-ab0f-9fa8240711b2/cache-name/part-88.bin:
Too many open files
at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at
java.base/sun.nio.fs.UnixFileSystemProvider.newAsynchronousFileChannel(UnixFileSystemProvider.java:201)
at
java.base/java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:253)
at
java.base/java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:311)
at
org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIO.(AsyncFileIO.java:65)
at
org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory.create(AsyncFileIOFactory.java:43)
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.init(FilePageStore.java:491)
... 14 more
[21:03:51,695][SEVERE][db-snapshot-executor-stripe-0-#35][FailureProcessor]
No deadlocked threads detected.
[21:03:51,767][SEVERE][db-snapshot-executor-stripe-0-#35][FailureProcessor]
Thread dump at 2022/06/06 21:03:51 IST
Thread [name="main", id=1, state=WAITING, blockCnt=4, waitCnt=4169]
Lock [object=java.util.concurrent.CountDownLatch$Sync@5b60e356,
ownerName=null, ownerId=-1]
at java.base@11.0.14.1/jdk.internal.misc.Unsafe.park(Native Method)
at
java.base@11.0.14.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at
java.base@11.0.14.1/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
at
java.base@11.0.14.1/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
at
java.base@11.0.14.1/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)

Re: LEFT, RIGHT JOIN not working

[Surinder Mehra]

Hi,
Just wondering if you had an opportunity to look into this.

On Thu, Jun 2, 2022 at 2:52 PM Surinder Mehra  wrote:

> Hi,
> Please find the attached java file which reproduces the issue. As you can
> see, the cache key is used as a join condition but LEFT join is still
> giving only common values.
>
> output:
> [2, Keyboard, 2]
> Size of actual output 1
> Expected size 3 is not equal to Actual size 1
>
>
> On Thu, Jun 2, 2022 at 11:48 AM Zhenya Stanilovsky 
> wrote:
>
>> Hi, Surinder Mehra ! I check your sql and it work correct for me.
>>
>>1. You no need to define AffinityKeyMapped for Key, check
>>additionally [1], you can simple modify [2] according to your case
>>2. I problem still exist somehow, plz attach some code example.
>>
>> thanks !
>>
>> [1]
>> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/affinity/AffinityKeyMapped.html
>> [2]
>> https://github.com/apache/ignite/blob/master/modules/indexing/src/test/java/org/apache/ignite/internal/processors/cache/IgniteCacheJoinPartitionedAndReplicatedTest.java#L160
>>
>>
>> Hi,
>> I have the following sample code to demo issue in SQL joins. I have
>> created an affinity key and value as shown below and added some sample data
>> to it. When I try LEFT self join on this table it always gives me common
>> rows   irrespective of LEFT or RIGHT JOIN
>> Could you please help me find what am I doing wrong here.
>>
>> cache Key :
>>
>> public class OrderAffinityKey {
>> Integer id;
>> @AffinityKeyMapped
>> Integer customerId;
>> }
>>
>>
>> cache value:
>>
>> public class Order implements Serializable {
>> @QuerySqlField
>> Integer id;
>>
>> @AffinityKeyMapped
>> @QuerySqlField Integer customerId;
>> @QuerySqlField String product;
>> }
>>
>>
>> Table C: (select customerID, product FROM "orderCache"."ORDER" WHERE
>> CUSTOMERID IN ( 1, 2))
>>
>> 1 keyboard
>> 2 Laptop
>>
>>
>> Table O: (select customerID, product FROM "orderCache"."ORDER" WHERE
>> CUSTOMERID IN ( 3, 2))
>>
>> 2 laptop
>> 3 mouse
>>
>>
>>
>> JOIN:
>>
>> Query :
>> select DISTINCT C.customerID, C.product, O.customerID
>> FROM
>>  (select customerID, product FROM "orderCache"."ORDER" WHERE CUSTOMERID
>> IN ( 1, 2)) C
>>  LEFT JOIN
>> (select customerID, product FROM "orderCache"."ORDER" WHERE CUSTOMERID IN
>> ( 3, 2)) O
>> ON
>> C.customerId = O.customerId
>>
>>
>> Output:
>>
>> 2 laptop   2
>> 3 mouse   3
>>
>> Expected output:
>>
>> 1 keyboard   null
>> 2 laptop   2
>> 3 mouse   3
>>
>>
>>
>>
>>
>>
>

Re: Ignite node in maintenance mode

[Surinder Mehra]

Hey, thanks for reply. How would we recover deleted caches data. Wouldn't
that put cluster into inconsistent state if one of the related caches is
deleted.

Would this mean that we have to empty cluster and restore to prev
consistent state(say restore from snapshot)

On Thu, 2 Jun 2022, 20:16 Gary Ng Kwong Sang,  wrote:

> https://lists.apache.org/thread/76b271wdq3dog0ggxyo6rwjr9hrgc6ss
>
>
>
> On 2022/06/02 14:12:54 Surinder Mehra wrote:
>
> > Hi,
>
> > We are running 3 node cluster and suddenly it went into maintenance mode.
>
> > Now i am able to login into nodes but cant do any action like activating
>
> > the cluster or removing other nodes from it. All 3 nodes run in
>
> > isolated mode. I restarted the cluster, but it didn't help.
>
> > Is there a way to exit maintenance mode?  How do we recover from such a
>
> > state ?
>
> >
>
> > Error:
>
> >
> 
>
> > Failed to change cluster state to ACTIVE
>
> > Failed to activate cluster (node is in maintenance mode).
>
> > suppressed:
>
> >
>
> > Command [SET-STATE] finished with code: 4
>
> > Error stack trace:
>
> > class org.apache.ignite.internal.client.GridClientException: Failed to
>
> > activate cluster (node is in maintenance mode).
>
> > suppressed:
>
> >
>
> > at
>
> >
> org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.handleClientResponse(GridClientNioTcpConnection.java:632)
>
> > at
>
> >
> org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.handleResponse(GridClientNioTcpConnection.java:563)
>
> > at
>
> >
> org.apache.ignite.internal.client.impl.connection.GridClientConnectionManagerAdapter$NioListener.onMessage(GridClientConnectionManagerAdapter.java:691)
>
> > at
>
> >
> org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279)
>
> > at
>
> >
> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
>
> > at
>
> >
> org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:116)
>
> > at
>
> >
> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
>
> > at
>
> >
> org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3734)
>
> > at
>
> >
> org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175)
>
> > at
>
> >
> org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1211)
>
> > at
>
> >
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2508)
>
> > at
>
> >
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2273)
>
> > at
>
> >
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1910)
>
> > at
>
> >
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
>
> > at java.base/java.lang.Thread.run(Unknown Source)
>
> >
>
> > Control utility has completed execution at: 2022-06-02T14:06:03.205
>
> > Execution time: 209 ms
>
> > root@ignite-client-kaurv-tbabor1-0:/opt/ignite/apache-ignite/bin#
>
> >
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows
>
>
>

Ignite node in maintenance mode

[Surinder Mehra]

Hi,
We are running 3 node cluster and suddenly it went into maintenance mode.
Now i am able to login into nodes but cant do any action like activating
the cluster or removing other nodes from it. All 3 nodes run in
isolated mode. I restarted the cluster, but it didn't help.
Is there a way to exit maintenance mode?  How do we recover from such a
state ?

Error:

Failed to change cluster state to ACTIVE
Failed to activate cluster (node is in maintenance mode).
suppressed:

Command [SET-STATE] finished with code: 4
Error stack trace:
class org.apache.ignite.internal.client.GridClientException: Failed to
activate cluster (node is in maintenance mode).
suppressed:

at
org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.handleClientResponse(GridClientNioTcpConnection.java:632)
at
org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.handleResponse(GridClientNioTcpConnection.java:563)
at
org.apache.ignite.internal.client.impl.connection.GridClientConnectionManagerAdapter$NioListener.onMessage(GridClientConnectionManagerAdapter.java:691)
at
org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279)
at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
at
org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:116)
at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
at
org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3734)
at
org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175)
at
org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1211)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2508)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2273)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1910)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.base/java.lang.Thread.run(Unknown Source)

Control utility has completed execution at: 2022-06-02T14:06:03.205
Execution time: 209 ms
root@ignite-client-kaurv-tbabor1-0:/opt/ignite/apache-ignite/bin#

Re: LEFT, RIGHT JOIN not working

[Surinder Mehra]

Hi,
Please find the attached java file which reproduces the issue. As you can
see, the cache key is used as a join condition but LEFT join is still
giving only common values.

output:
[2, Keyboard, 2]
Size of actual output 1
Expected size 3 is not equal to Actual size 1


On Thu, Jun 2, 2022 at 11:48 AM Zhenya Stanilovsky 
wrote:

> Hi, Surinder Mehra ! I check your sql and it work correct for me.
>
>1. You no need to define AffinityKeyMapped for Key, check additionally
>[1], you can simple modify [2] according to your case
>2. I problem still exist somehow, plz attach some code example.
>
> thanks !
>
> [1]
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/affinity/AffinityKeyMapped.html
> [2]
> https://github.com/apache/ignite/blob/master/modules/indexing/src/test/java/org/apache/ignite/internal/processors/cache/IgniteCacheJoinPartitionedAndReplicatedTest.java#L160
>
>
> Hi,
> I have the following sample code to demo issue in SQL joins. I have
> created an affinity key and value as shown below and added some sample data
> to it. When I try LEFT self join on this table it always gives me common
> rows   irrespective of LEFT or RIGHT JOIN
> Could you please help me find what am I doing wrong here.
>
> cache Key :
>
> public class OrderAffinityKey {
> Integer id;
> @AffinityKeyMapped
> Integer customerId;
> }
>
>
> cache value:
>
> public class Order implements Serializable {
> @QuerySqlField
> Integer id;
>
> @AffinityKeyMapped
> @QuerySqlField Integer customerId;
> @QuerySqlField String product;
> }
>
>
> Table C: (select customerID, product FROM "orderCache"."ORDER" WHERE
> CUSTOMERID IN ( 1, 2))
>
> 1 keyboard
> 2 Laptop
>
>
> Table O: (select customerID, product FROM "orderCache"."ORDER" WHERE
> CUSTOMERID IN ( 3, 2))
>
> 2 laptop
> 3 mouse
>
>
>
> JOIN:
>
> Query :
> select DISTINCT C.customerID, C.product, O.customerID
> FROM
>  (select customerID, product FROM "orderCache"."ORDER" WHERE CUSTOMERID IN
> ( 1, 2)) C
>  LEFT JOIN
> (select customerID, product FROM "orderCache"."ORDER" WHERE CUSTOMERID IN
> ( 3, 2)) O
> ON
> C.customerId = O.customerId
>
>
> Output:
>
> 2 laptop   2
> 3 mouse   3
>
> Expected output:
>
> 1 keyboard   null
> 2 laptop   2
> 3 mouse   3
>
>
>
>
>
>


JoinQueryTest.java
Description: Binary data

Re: Docker image for Apache Ignite with Java 11

[Surinder Mehra]

Yeah hope so

On Thu, 2 Jun 2022, 14:06 Айсина Роза,  wrote:

> Hey!
>
>
> Thank you very much - I have already built the image with Java 11 based on
> your similar thread.
>
>
> But I want to address this problem to the community and Ignite developers.
>
> It is very painful to build custom image as it is required to have
> *run.sh* script that *is not included* in standard Ignite bundle.
>
> Currently I copy it directly from GitHub (with *ADD* command) but it is a
> workaround not a solution.
>
>
> Hope new image versions will be released!
>
>
> Best regards,
>
> Rose.
> --
> *From:* Surinder Mehra 
> *Sent:* Thursday, June 2, 2022 9:33:08 AM
> *To:* user@ignite.apache.org
> *Subject:* [POTENTIALLY_MALICIOUS] Re: Docker image for Apache Ignite
> with Java 11
>
> **ВНИМАНИЕ** Это письмо содержит подозрительное вложение или ссылку.
> Открывать только в случае 100% доверия к отправителю письма.
>
> Hey,
> I had a similar requirement some time back and created my own. Attached
> conversation might be useful for you until ignite  publishes their official
> docker image with java 11
>
>
> FROM adoptopenjdk/openjdk11
>
> # Settings
> ARG IGNITE_CFG_XML="node-configuration.xml"
> ARG IGNITE_VERSION="2.11.0"
> #ARG OPTION_LIBS="ignite-kubernetes,ignite-rest-http"
> ENV IGNITE_HOME /opt/ignite/apache-ignite
> #ENV CONFIG_URI config/$IGNITE_CFG_XML
> # Disabling quiet mode.
> ENV IGNITE_QUIET=false
> WORKDIR /opt/ignite
>
> # Add missing software
> RUN apt-get update &&\
> apt-get install bash && \
> apt-get install -y wget && \
> apt-get install unzip && \
> wget 
> https://dlcdn.apache.org//ignite/${IGNITE_VERSION}/apache-ignite-${IGNITE_VERSION}-bin.zip
>  && \
> unzip -o apache-ignite-${IGNITE_VERSION}-bin.zip && \
> mv apache-ignite-${IGNITE_VERSION}-bin apache-ignite && \
> rm apache-ignite-${IGNITE_VERSION}-bin.zip
>
> # Copy main binary archive
> #COPY apache-ignite* apache-ignite
>
> # Copy sh files and set permission
> COPY run.sh $IGNITE_HOME/
> COPY ./$IGNITE_CFG_XML $IGNITE_HOME/config
> # Grant permission to copy optional libs
> RUN chmod 777 ${IGNITE_HOME}/libs
>
> # Grant permission to create work directory
> RUN chmod 777 ${IGNITE_HOME}
>
> # Grant permission to execute entry point
> RUN chmod 555 $IGNITE_HOME/run.sh
>
> # Entry point
>
> RUN export JAVA_HOME="$(dirname $(dirname $(readlink -f $(which java"
>
> CMD $IGNITE_HOME/run.sh
>
> # Container port exposure
> EXPOSE 11211 47100 47500 49112 10800 8080
>
>
> On Thu, Jun 2, 2022 at 11:47 AM Айсина Роза  wrote:
>
>> Hello!
>>
>> I am looking for docker image with Apache Ignite using Java 11 instead of
>> Java 8.
>> I see that in provided Dockerfiles (
>> https://github.com/apache/ignite/tree/master/deliveries/docker/apache-ignite
>> )
>> image for x86_64 uses Java 8 but image for s390x uses Java 11.
>>
>> Can you please build the same x86-images but with Java 11?
>>
>> It's extremely painful to build them manually :(
>> And the reason is that all our Java applications use Java 11 or older and
>> it is not possible to join client node with different Java version.
>>
>> Thanks in advance!
>>
>> Best regards,
>>
>> Rose.
>> --
>>
>> Информация данного сообщения содержит коммерческую тайну Общества с
>> ограниченной ответственностью «ГПМ Дата», ОГРН 1207700499863 (далее – ООО
>> «ГПМ Дата») и предназначена только для лиц, которым адресовано данное
>> сообщение. Если Вы получили данное сообщение по ошибке, просим Вас удалить
>> его и не использовать полученную информацию, составляющую коммерческую
>> тайну ООО «ГПМ Дата».
>>
>> В соответствии с действующим законодательством Российской Федерации ООО
>> «ГПМ Дата» вправе требовать от лиц, получивших доступ к информации,
>> составляющей коммерческую тайну, в результате действий, совершенных
>> случайно или по ошибке, охраны конфиденциальности этой информации.
>>
>> Настоящее сообщение не является вступлением в переговоры о заключении
>> каких-либо договоров или соглашений, не свидетельствует о каком-либо
>> безусловном намерении ООО «ГПМ Дата» заключить какой-либо договор или
>> соглашение, не является заверением об обстоятельствах, которые должны быть
>> доведены до сведения другой стороны. Настоящее сообщение не является
>> офертой, акцептом оферты, равно как не является предварительным соглашением
>> и носит

Re: LEFT, RIGHT JOIN not working

[Surinder Mehra]

hello,
Just wondering if you got time to look into this one

On Wed, Jun 1, 2022 at 12:45 PM Surinder Mehra  wrote:

> Hi,
> I have the following sample code to demo issue in SQL joins. I have
> created an affinity key and value as shown below and added some sample data
> to it. When I try LEFT self join on this table it always gives me common
> rows   irrespective of LEFT or RIGHT JOIN
> Could you please help me find what am I doing wrong here.
>
> cache Key :
>
> public class OrderAffinityKey {
> Integer id;
> @AffinityKeyMapped
> Integer customerId;
> }
>
>
> cache value:
>
> public class Order implements Serializable {
> @QuerySqlField
> Integer id;
>
> @AffinityKeyMapped
> @QuerySqlField Integer customerId;
> @QuerySqlField String product;
> }
>
>
> Table C: (select customerID, product FROM "orderCache"."ORDER" WHERE
> CUSTOMERID IN ( 1, 2))
>
> 1 keyboard
> 2 Laptop
>
>
> Table O: (select customerID, product FROM "orderCache"."ORDER" WHERE
> CUSTOMERID IN ( 3, 2))
>
> 2 laptop
> 3 mouse
>
>
>
> JOIN:
>
> Query :
> select DISTINCT C.customerID, C.product, O.customerID
> FROM
>  (select customerID, product FROM "orderCache"."ORDER" WHERE CUSTOMERID IN
> ( 1, 2)) C
>  LEFT JOIN
> (select customerID, product FROM "orderCache"."ORDER" WHERE CUSTOMERID IN
> ( 3, 2)) O
> ON
> C.customerId = O.customerId
>
>
> Output:
>
> 2 laptop   2
> 3 mouse   3
>
> Expected output:
>
> 1 keyboard   null
> 2 laptop   2
> 3 mouse   3
>

LEFT, RIGHT JOIN not working

[Surinder Mehra]

Hi,
I have the following sample code to demo issue in SQL joins. I have created
an affinity key and value as shown below and added some sample data to it.
When I try LEFT self join on this table it always gives me common rows
 irrespective of LEFT or RIGHT JOIN
Could you please help me find what am I doing wrong here.

cache Key :

public class OrderAffinityKey {
Integer id;
@AffinityKeyMapped
Integer customerId;
}


cache value:

public class Order implements Serializable {
@QuerySqlField
Integer id;

@AffinityKeyMapped
@QuerySqlField Integer customerId;
@QuerySqlField String product;
}


Table C: (select customerID, product FROM "orderCache"."ORDER" WHERE
CUSTOMERID IN ( 1, 2))

1 keyboard
2 Laptop


Table O: (select customerID, product FROM "orderCache"."ORDER" WHERE
CUSTOMERID IN ( 3, 2))

2 laptop
3 mouse



JOIN:

Query :
select DISTINCT C.customerID, C.product, O.customerID
FROM
 (select customerID, product FROM "orderCache"."ORDER" WHERE CUSTOMERID IN
( 1, 2)) C
 LEFT JOIN
(select customerID, product FROM "orderCache"."ORDER" WHERE CUSTOMERID IN (
3, 2)) O
ON
C.customerId = O.customerId


Output:

2 laptop   2
3 mouse   3

Expected output:

1 keyboard   null
2 laptop   2
3 mouse   3

Re: Ignite Snapshots take minutes to complete

[Surinder Mehra]

Hey,
I have one more query on this one. Currently statefulset
"podManagementPolicy" is "OrderedReady". It means kubernetes will start one
pod at a time until it reaches the replica count given.
If we change  "podManagementPolicy" to "Parallel", all pods will start in
parallel and hence I can control init containers as intended to clean up
and copy from snapshot before any main container starts.

Question is, is it fine to start Ignite pods in parallel or should they
always start one after another. Documentation doesn't say anything about
this and I just want to make sure this has no side effects before I start
implementing my approach to restore data.

On Fri, May 27, 2022 at 11:33 PM Surinder Mehra  wrote:

> Hi Maxim,
> As I explained in original email, I do cleanup as part of init container.
> Since ignite nodes starts after one another, init container would also run
> in sequence.
> So when ignite node 1 completes startup(init container clean up work
> directory and copy data from snapshot) but node 2 is still running init
> container which might be still copying data. And node 3 since it is yet to
> start, it's wal, cp, and work directory hasn't been cleaned yet.
>
> So my question was, is there a way I can do cleanup of work directory  and
> copy from snapshot for all nodes before first ignite nodes starts.
>
> On Fri, 27 May 2022, 23:11 Maxim Muzafarov,  wrote:
>
>> Hello,
>>
>> If you're copying a snapshot part to the new node, then you have to be
>> sure that the /ignite/work/cp, /ignite/wal, /ignite/walarchive
>> directories are empty prior to the node start. Is it true for your
>> case?
>>
>> On Fri, 27 May 2022 at 10:29, Surinder Mehra  wrote:
>> >
>> > Hi,
>> > Please find ignite config and error log below
>> >
>> > config :
>> > 
>> > 
>> > > value="/opt/ignite/apache-ignite/config/ignite-log4j.xml"/>
>> > 
>> > 
>> > 
>> > 
>> > 
>> > 
>> > 
>> >
>> > 
>> > > class="org.apache.ignite.configuration.DataStorageConfiguration">
>> > 
>> > 
>> > 
>> > 
>> > 
>> > 
>> > > class="org.apache.ignite.configuration.DataRegionConfiguration">
>> > > value="true"/>
>> > 
>> > 
>> > 
>> > 
>> > 
>> > 
>> > > value="/ignite/walarchive"/>
>> > 
>> > 
>> >
>> >
>> > Error log:
>> >
>> > at
>> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.access$1000(FileWriteAheadLogManager.java:2763)
>> > at
>> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:870)
>> > at
>> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetaStorageSubscribersOnReadyForRead(GridCacheDatabaseSharedManager.java:3200)
>> > at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1116)
>> > at
>> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1799)
>> > at
>> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1721)
>> > at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1160)
>> > at
>> org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1054)
>> > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:940)
>> > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:839)
>> > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:709)
>> > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:678)
>> > at org.apache.ignite.Ignition.start(Ignition.java:353)
>> > ... 1 more
>> > Failed to start grid: WAL history is too short [descs=[FileDescriptor
>> [file=/ignite/walarchive/node00-44a0ade8-60c2-4190-aac3-7fb465129efe/0060.wal,
>> idx=60], FileDescriptor
>> [file=/ignite/walarchive/node00-44a0ade8-60c2-4190-aac3-7fb465129efe/0061.wal,

Re: Ignite Snapshots take minutes to complete

[Surinder Mehra]

Hi Maxim,
As I explained in original email, I do cleanup as part of init container.
Since ignite nodes starts after one another, init container would also run
in sequence.
So when ignite node 1 completes startup(init container clean up work
directory and copy data from snapshot) but node 2 is still running init
container which might be still copying data. And node 3 since it is yet to
start, it's wal, cp, and work directory hasn't been cleaned yet.

So my question was, is there a way I can do cleanup of work directory  and
copy from snapshot for all nodes before first ignite nodes starts.

On Fri, 27 May 2022, 23:11 Maxim Muzafarov,  wrote:

> Hello,
>
> If you're copying a snapshot part to the new node, then you have to be
> sure that the /ignite/work/cp, /ignite/wal, /ignite/walarchive
> directories are empty prior to the node start. Is it true for your
> case?
>
> On Fri, 27 May 2022 at 10:29, Surinder Mehra  wrote:
> >
> > Hi,
> > Please find ignite config and error log below
> >
> > config :
> > 
> > 
> >  value="/opt/ignite/apache-ignite/config/ignite-log4j.xml"/>
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> >
> > 
> >  class="org.apache.ignite.configuration.DataStorageConfiguration">
> > 
> > 
> > 
> > 
> > 
> > 
> >  class="org.apache.ignite.configuration.DataRegionConfiguration">
> >  value="true"/>
> > 
> > 
> > 
> > 
> > 
> > 
> >  value="/ignite/walarchive"/>
> > 
> > 
> >
> >
> > Error log:
> >
> > at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.access$1000(FileWriteAheadLogManager.java:2763)
> > at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:870)
> > at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetaStorageSubscribersOnReadyForRead(GridCacheDatabaseSharedManager.java:3200)
> > at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1116)
> > at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1799)
> > at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1721)
> > at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1160)
> > at
> org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1054)
> > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:940)
> > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:839)
> > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:709)
> > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:678)
> > at org.apache.ignite.Ignition.start(Ignition.java:353)
> > ... 1 more
> > Failed to start grid: WAL history is too short [descs=[FileDescriptor
> [file=/ignite/walarchive/node00-44a0ade8-60c2-4190-aac3-7fb465129efe/0060.wal,
> idx=60], FileDescriptor
> [file=/ignite/walarchive/node00-44a0ade8-60c2-4190-aac3-7fb465129efe/0061.wal,
> idx=61], FileDescriptor
> [file=/ignite/walarchive/node00-44a0ade8-60c2-4190-aac3-7fb465129efe/0062.wal,
> idx=62], FileDescriptor
> [file=/ignite/walarchive/node00-44a0ade8-60c2-4190-aac3-7fb465129efe/0063.wal,
> idx=63], FileDescriptor
> [file=/ignite/walarchive/node00-44a0ade8-60c2-4190-aac3-7fb465129efe/0064.wal,
> idx=64], FileDescriptor
> [file=/ignite/walarchive/node00-44a0ade8-60c2-4190-aac3-7fb465129efe/0065.wal,
> idx=65], FileDescriptor
> [file=/ignite/walarchive/node00-44a0ade8-60c2-4190-aac3-7fb465129efe/0066.wal,
> idx=66], FileDescriptor
> [file=/ignite/walarchive/node00-44a0ade8-60c2-4190-aac3-7fb465129efe/0067.wal,
> idx=67], FileDescriptor
> [file=/ignite/walarchive/node00-44a0ade8-60c2-4190-aac3-7fb465129efe/0068.wal,
> idx=68], FileDescriptor
> [file=/ignite/walarchive/node00-44a0ade8-60c2-4190-aac3-7fb465129efe/0069.wal,
> idx=69], FileDescriptor
> [file=/ignite/walarchive/node00-44a0ade8-60c2-4190-aac3-7fb465129efe/0070.wal,
> idx

Re: Ignite Snapshots take minutes to complete

[Surinder Mehra]

=87], FileDescriptor
[file=/ignite/walarchive/node00-44a0ade8-60c2-4190-aac3-7fb465129efe/0088.wal,
idx=88], FileDescriptor
[file=/ignite/walarchive/node00-44a0ade8-60c2-4190-aac3-7fb465129efe/0089.wal,
idx=89], FileDescriptor
[file=/ignite/walarchive/node00-44a0ade8-60c2-4190-aac3-7fb465129efe/0090.wal,
idx=90], FileDescriptor
[file=/ignite/walarchive/node00-44a0ade8-60c2-4190-aac3-7fb465129efe/0091.wal,
idx=91], FileDescriptor
[file=/ignite/walarchive/node00-44a0ade8-60c2-4190-aac3-7fb465129efe/0092.wal,
idx=92], FileDescriptor
[file=/ignite/walarchive/node00-44a0ade8-60c2-4190-aac3-7fb465129efe/0093.wal,
idx=93], FileDescriptor
[file=/ignite/walarchive/node00-44a0ade8-60c2-4190-aac3-7fb465129efe/0094.wal,
idx=94]], start=WALPointer [idx=0, fileOff=0, len=0]]


On Thu, May 26, 2022 at 8:56 PM Николай Ижиков  wrote:

> Can you, please, send your config and full log file that contains error
> message.
>
> 26 мая 2022 г., в 17:50, Surinder Mehra  написал(а):
>
> Hello,
> I upgraded to 2.13.0 and I am able to take sync snapshots now. However, I
> ran into another problem while restoring from snapshot using manual steps
> mentioned in documentation.
>
> We run ignite statefulset on kubernetes cluster so when we scale it to N
> nodes, it brings up one node at a time.
>
> Now I am trying to attach init container which will copy /db directory
> from snapshots to work directory after clearing db directory from work
> directory and then start main container which runs ignite.
>
> It works well on single node, it's able to start cluster with snapshot
> Data.
>
> When I start multiple nodes, init container will run each one of those as
> first step. Since nodes starts one at a time, it's runs into error saying
> "too small WAL segments data"
>
> I suppose that could be because 2nd node is still in init step while first
> one is in running mode. There are few which haven't started yet, waiting
> for 2nd node to be in running state.
>
> Any idea how can we make main containers wait until all init containers
> are completed
>
> Asking this here as its related to ignite setup in kubernetes.
>
> Any help wil be appreciated. Thanks
>
> On Wed, 25 May 2022, 00:04 Surinder Mehra,  wrote:
>
>> Thanks a lot. I will try this.
>>
>> On Tue, 24 May 2022, 23:50 Николай Ижиков,  wrote:
>>
>>> > Does it ensure consistency while copying data which is parallely
>>> getting updated by application writes
>>>
>>> Yes.
>>>
>>> From the documentation:
>>>
>>> «An Ignite snapshot includes a consistent cluster-wide copy of all data
>>> records persisted on disk and some other files needed for a restore
>>> procedure.»
>>>
>>> > will this be a stop the world process
>>>
>>> No.
>>>
>>>
>>> 24 мая 2022 г., в 21:17, Surinder Mehra  написал(а):
>>>
>>> Hi
>>> Thanks for reply.
>>>
>>> #1:  So it's not a stop the world task. Does it ensure consistency while
>>> copying data which is parallely getting updated by application writes. Or
>>> does it mark the data to copied and ignore further updates on it.
>>>
>>> #2:
>>> I will try sync snapshot. But just to confirm, will this be a stop the
>>> world process. Couldn't find anything on Documentation page about it
>>>
>>> On Tue, 24 May 2022, 23:12 Николай Ижиков,  wrote:
>>>
>>>> Hello, Mehra.
>>>>
>>>> > 1. Is it stop the world process.
>>>>
>>>> No, you can perform any actions.
>>>> Note, topology changes will cancel snapshot create process.
>>>>
>>>> > 2. If so, is it stop the world only during command execution
>>>> (500millis) or until snapshot Dara is fully copied(takes many minutes) to
>>>> complete.
>>>>
>>>> Please, take a look at `—sync` option of create snapshot command (you
>>>> can see help in `control.sh` output).
>>>> `EVT_CLUSTER_SNAPSHOT_FINISHED` raise on snapshot create finish.
>>>>
>>>> > 3. Is there a way around to speed up this other than increasing
>>>> snapshot threads
>>>>
>>>> Stop write operations.
>>>> The less you change the quicker snapshot will be created.
>>>>
>>>> 24 мая 2022 г., в 20:12, Surinder Mehra 
>>>> написал(а):
>>>>
>>>> Hi,
>>>> I have 3 node ignite cluster each node contains 60G work directory(ebs)
>>>> and I need to create snapshots.
>>>> I followed steps to create snapshots and run create snapshot command
>>>> using control utility. Command completed in 500millis but snapshot
>>>> directory only had 400Mb data. Later I realised directory size grew up 30G.
>>>> I suppose it would reach size of work directory.
>>>>
>>>>
>>>> I have few questions.
>>>> 1. Is it stop the world process.
>>>> 2. If so, is it stop the world only during command execution
>>>> (500millis) or until snapshot Dara is fully copied(takes many minutes) to
>>>> complete.
>>>> 3. Is there a way around to speed up this other than increasing
>>>> snapshot threads
>>>>
>>>>
>>>>
>>>
>

Re: Ignite Snapshots take minutes to complete

[Surinder Mehra]

Hello,
I upgraded to 2.13.0 and I am able to take sync snapshots now. However, I
ran into another problem while restoring from snapshot using manual steps
mentioned in documentation.

We run ignite statefulset on kubernetes cluster so when we scale it to N
nodes, it brings up one node at a time.

Now I am trying to attach init container which will copy /db directory from
snapshots to work directory after clearing db directory from work directory
and then start main container which runs ignite.

It works well on single node, it's able to start cluster with snapshot Data.

When I start multiple nodes, init container will run each one of those as
first step. Since nodes starts one at a time, it's runs into error saying
"too small WAL segments data"

I suppose that could be because 2nd node is still in init step while first
one is in running mode. There are few which haven't started yet, waiting
for 2nd node to be in running state.

Any idea how can we make main containers wait until all init containers are
completed

Asking this here as its related to ignite setup in kubernetes.

Any help wil be appreciated. Thanks

On Wed, 25 May 2022, 00:04 Surinder Mehra,  wrote:

> Thanks a lot. I will try this.
>
> On Tue, 24 May 2022, 23:50 Николай Ижиков,  wrote:
>
>> > Does it ensure consistency while copying data which is parallely
>> getting updated by application writes
>>
>> Yes.
>>
>> From the documentation:
>>
>> «An Ignite snapshot includes a consistent cluster-wide copy of all data
>> records persisted on disk and some other files needed for a restore
>> procedure.»
>>
>> > will this be a stop the world process
>>
>> No.
>>
>>
>> 24 мая 2022 г., в 21:17, Surinder Mehra  написал(а):
>>
>> Hi
>> Thanks for reply.
>>
>> #1:  So it's not a stop the world task. Does it ensure consistency while
>> copying data which is parallely getting updated by application writes. Or
>> does it mark the data to copied and ignore further updates on it.
>>
>> #2:
>> I will try sync snapshot. But just to confirm, will this be a stop the
>> world process. Couldn't find anything on Documentation page about it
>>
>> On Tue, 24 May 2022, 23:12 Николай Ижиков,  wrote:
>>
>>> Hello, Mehra.
>>>
>>> > 1. Is it stop the world process.
>>>
>>> No, you can perform any actions.
>>> Note, topology changes will cancel snapshot create process.
>>>
>>> > 2. If so, is it stop the world only during command execution
>>> (500millis) or until snapshot Dara is fully copied(takes many minutes) to
>>> complete.
>>>
>>> Please, take a look at `—sync` option of create snapshot command (you
>>> can see help in `control.sh` output).
>>> `EVT_CLUSTER_SNAPSHOT_FINISHED` raise on snapshot create finish.
>>>
>>> > 3. Is there a way around to speed up this other than increasing
>>> snapshot threads
>>>
>>> Stop write operations.
>>> The less you change the quicker snapshot will be created.
>>>
>>> 24 мая 2022 г., в 20:12, Surinder Mehra  написал(а):
>>>
>>> Hi,
>>> I have 3 node ignite cluster each node contains 60G work directory(ebs)
>>> and I need to create snapshots.
>>> I followed steps to create snapshots and run create snapshot command
>>> using control utility. Command completed in 500millis but snapshot
>>> directory only had 400Mb data. Later I realised directory size grew up 30G.
>>> I suppose it would reach size of work directory.
>>>
>>>
>>> I have few questions.
>>> 1. Is it stop the world process.
>>> 2. If so, is it stop the world only during command execution (500millis)
>>> or until snapshot Dara is fully copied(takes many minutes) to complete.
>>> 3. Is there a way around to speed up this other than increasing snapshot
>>> threads
>>>
>>>
>>>
>>

Re: Ignite Snapshots take minutes to complete

[Surinder Mehra]

Thanks a lot. I will try this.

On Tue, 24 May 2022, 23:50 Николай Ижиков,  wrote:

> > Does it ensure consistency while copying data which is parallely getting
> updated by application writes
>
> Yes.
>
> From the documentation:
>
> «An Ignite snapshot includes a consistent cluster-wide copy of all data
> records persisted on disk and some other files needed for a restore
> procedure.»
>
> > will this be a stop the world process
>
> No.
>
>
> 24 мая 2022 г., в 21:17, Surinder Mehra  написал(а):
>
> Hi
> Thanks for reply.
>
> #1:  So it's not a stop the world task. Does it ensure consistency while
> copying data which is parallely getting updated by application writes. Or
> does it mark the data to copied and ignore further updates on it.
>
> #2:
> I will try sync snapshot. But just to confirm, will this be a stop the
> world process. Couldn't find anything on Documentation page about it
>
> On Tue, 24 May 2022, 23:12 Николай Ижиков,  wrote:
>
>> Hello, Mehra.
>>
>> > 1. Is it stop the world process.
>>
>> No, you can perform any actions.
>> Note, topology changes will cancel snapshot create process.
>>
>> > 2. If so, is it stop the world only during command execution
>> (500millis) or until snapshot Dara is fully copied(takes many minutes) to
>> complete.
>>
>> Please, take a look at `—sync` option of create snapshot command (you can
>> see help in `control.sh` output).
>> `EVT_CLUSTER_SNAPSHOT_FINISHED` raise on snapshot create finish.
>>
>> > 3. Is there a way around to speed up this other than increasing
>> snapshot threads
>>
>> Stop write operations.
>> The less you change the quicker snapshot will be created.
>>
>> 24 мая 2022 г., в 20:12, Surinder Mehra  написал(а):
>>
>> Hi,
>> I have 3 node ignite cluster each node contains 60G work directory(ebs)
>> and I need to create snapshots.
>> I followed steps to create snapshots and run create snapshot command
>> using control utility. Command completed in 500millis but snapshot
>> directory only had 400Mb data. Later I realised directory size grew up 30G.
>> I suppose it would reach size of work directory.
>>
>>
>> I have few questions.
>> 1. Is it stop the world process.
>> 2. If so, is it stop the world only during command execution (500millis)
>> or until snapshot Dara is fully copied(takes many minutes) to complete.
>> 3. Is there a way around to speed up this other than increasing snapshot
>> threads
>>
>>
>>
>

Re: Ignite Snapshots take minutes to complete

[Surinder Mehra]

Hi
Thanks for reply.

#1:  So it's not a stop the world task. Does it ensure consistency while
copying data which is parallely getting updated by application writes. Or
does it mark the data to copied and ignore further updates on it.

#2:
I will try sync snapshot. But just to confirm, will this be a stop the
world process. Couldn't find anything on Documentation page about it

On Tue, 24 May 2022, 23:12 Николай Ижиков,  wrote:

> Hello, Mehra.
>
> > 1. Is it stop the world process.
>
> No, you can perform any actions.
> Note, topology changes will cancel snapshot create process.
>
> > 2. If so, is it stop the world only during command execution (500millis)
> or until snapshot Dara is fully copied(takes many minutes) to complete.
>
> Please, take a look at `—sync` option of create snapshot command (you can
> see help in `control.sh` output).
> `EVT_CLUSTER_SNAPSHOT_FINISHED` raise on snapshot create finish.
>
> > 3. Is there a way around to speed up this other than increasing
> snapshot threads
>
> Stop write operations.
> The less you change the quicker snapshot will be created.
>
> 24 мая 2022 г., в 20:12, Surinder Mehra  написал(а):
>
> Hi,
> I have 3 node ignite cluster each node contains 60G work directory(ebs)
> and I need to create snapshots.
> I followed steps to create snapshots and run create snapshot command using
> control utility. Command completed in 500millis but snapshot directory only
> had 400Mb data. Later I realised directory size grew up 30G. I suppose it
> would reach size of work directory.
>
>
> I have few questions.
> 1. Is it stop the world process.
> 2. If so, is it stop the world only during command execution (500millis)
> or until snapshot Dara is fully copied(takes many minutes) to complete.
> 3. Is there a way around to speed up this other than increasing snapshot
> threads
>
>
>

Ignite Snapshots take minutes to complete

[Surinder Mehra]

Hi,
I have 3 node ignite cluster each node contains 60G work directory(ebs) and
I need to create snapshots.
I followed steps to create snapshots and run create snapshot command using
control utility. Command completed in 500millis but snapshot directory only
had 400Mb data. Later I realised directory size grew up 30G. I suppose it
would reach size of work directory.


I have few questions.
1. Is it stop the world process.
2. If so, is it stop the world only during command execution (500millis) or
until snapshot Dara is fully copied(takes many minutes) to complete.
3. Is there a way around to speed up this other than increasing snapshot
threads

Re: log4j CVE fix in ignite 2.11.1

[Surinder Mehra]

Thanks for confirming. Just one thing. I just downloaded ignite 2.11.1 and
2.12.x and I can still see log4j.1.2.17 in it.

Is it removed in 2.13 onwards?

On Fri, 20 May 2022, 20:27 Stephen Darlington, <
stephen.darling...@gridgain.com> wrote:

> Ignite-log4j is the code that links Ignite to log4j. It does not contain a
> copy of log4j.
>
> Log4j is version 1.x of log4j, which wasn’t vulnerable. IIRC, log4j 1.x
> has subsequently been removed from Ignite.
>
> On 20 May 2022, at 15:03, Surinder Mehra  wrote:
>
> Hi, as per page below, log4j CVE is already fixed in ignite 2.11.1
> https://blogs.apache.org/ignite/entry/apache-ignite-2-11-1
>
> Affected log4j versions were 2.0-2.14. I can see ignite 2.11.1 contains
> two log4j jar files below. Can you please confirm these log4j versions are
> not affected by CVE anymore ? Or did I miss something?
>
> ignite-log4j-2.11.1.jar
> log4j-1.2.17.jar
>
>
>

log4j CVE fix in ignite 2.11.1

[Surinder Mehra]

Hi, as per page below, log4j CVE is already fixed in ignite 2.11.1
https://blogs.apache.org/ignite/entry/apache-ignite-2-11-1

Affected log4j versions were 2.0-2.14. I can see ignite 2.11.1 contains two
log4j jar files below. Can you please confirm these log4j versions are not
affected by CVE anymore ? Or did I miss something?

ignite-log4j-2.11.1.jar
log4j-1.2.17.jar

Page lock error on ignite restart

[Surinder Mehra]

Hi,
We are seeing this error on one of the nodes while restarting the ignite
cluster. We are using native persistence and version 2.11.1. It only
happens randomly.  Any idea what could be wrong?

class org.apache.ignite.IgniteException: GridWorker [name=sys-stripe-2,
igniteInstanceName=null, finished=false, heartbeatTs=1652936946081]
at
app//org.apache.ignite.internal.processors.cache.persistence.file.IgniteNativeIoLib.pwrite(Native
Method)
at
app//org.apache.ignite.internal.processors.cache.persistence.file.AlignedBuffersDirectFileIO.writeFromAlignedBuffer(AlignedBuffersDirectFileIO.java:444)
at
app//org.apache.ignite.internal.processors.cache.persistence.file.AlignedBuffersDirectFileIO.write(AlignedBuffersDirectFileIO.java:277)
at
app//org.apache.ignite.internal.processors.cache.persistence.file.AbstractFileIO$5.run(AbstractFileIO.java:117)
at
app//org.apache.ignite.internal.processors.cache.persistence.file.AbstractFileIO.fully(AbstractFileIO.java:53)
at
app//org.apache.ignite.internal.processors.cache.persistence.file.AbstractFileIO.writeFully(AbstractFileIO.java:115)
at
app//org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.write(FilePageStore.java:748)
at
app//org.apache.ignite.internal.processors.cache.persistence.pagemem.PageReadWriteManagerImpl.write(PageReadWriteManagerImpl.java:116)
at
app//org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.write(FilePageStoreManager.java:636)
at
app//org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointManager.lambda$new$0(CheckpointManager.java:175)
at
app//org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointManager$$Lambda$500/0x00084035c440.write(Unknown
Source)
at
app//org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointPagesWriterFactory.lambda$null$0(CheckpointPagesWriterFactory.java:170)
at
app//org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointPagesWriterFactory$$Lambda$697/0x0008404d9c40.writePage(Unknown
Source)
at
app//org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.copyPageForCheckpoint(PageMemoryImpl.java:1343)
at
app//org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.checkpointWritePage(PageMemoryImpl.java:1250)
at
app//org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointPagesWriterFactory.lambda$buildRecovery$2(CheckpointPagesWriterFactory.java:179)
at
app//org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointPagesWriterFactory$$Lambda$682/0x0008404dd040.run(Unknown
Source)
at
app//org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:569)
at
app//org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.base@11.0.14.1/java.lang.Thread.run(Unknown Source)
[05:18:02,603][INFO][grid-timeout-worker-#30][FailureProcessor] Thread dump
is hidden due to throttling settings. Set
IGNITE_DUMP_THREADS_ON_FAILURE_THROTTLING_TIMEOUT property to 0 to see all
thread dumps.
[05:18:02,603][WARNING][grid-timeout-worker-#30][CacheDiagnosticManager]
Page locks dump:

Thread=[name=main, id=1], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=main time=(1652937482603, 2022-05-19 05:18:02.603)

Re: Window functions support in ignite 2.13

[Surinder Mehra]

Hi,
Please reply

On Sat, 30 Apr 2022, 07:19 Surinder Mehra,  wrote:

> Hi
> In ignite documentation 2.12, we could see SQL window functions but
> removed in 2.13. could you please confirm if they are not supported. Apache
> calcite support window functions so we expected to see them in ignite.
>

Window functions support in ignite 2.13

[Surinder Mehra]

Hi
In ignite documentation 2.12, we could see SQL window functions but removed
in 2.13. could you please confirm if they are not supported. Apache calcite
support window functions so we expected to see them in ignite.

Re: Apache Ignite H2 Vulnerabilities

[Surinder Mehra]

Hey guys,
I wanted to know if window functions are supported in ignite 2.13. Calcite
supports it so wanted to check.

On Fri, 29 Apr 2022, 19:44 Nikita Amelchev,  wrote:

> Hello, guys.
>
> Thanks for pointing it out.
>
> The calcite module was properly published to the maven. [1] The sync
> with mirrors can take some time.
> The calcite documentation was updated on the site. [2]
>
> [1]
> https://repo.maven.apache.org/maven2/org/apache/ignite/ignite-calcite/2.13.0/
> [2] https://ignite.apache.org/docs/latest/SQL/sql-calcite
>
> пт, 29 апр. 2022 г. в 12:36, Stephen Darlington
> :
> >
> > It’ll be added to Maven soon — I’m not exactly sure what happened. It is
> included in the source and binary downloads (download.cgi) if you want to
> get a copy now.
> >
> > On 29 Apr 2022, at 02:19, Lokesh Bandaru 
> wrote:
> >
> > Hello Stephen, the document(ReadMe) you shared earlier, has mentioned
> that ignite-calcite must be declared as a dependency.
> > In this case, it would be, org.apache.ignite:ignite-calcite:2.13.0
> right!. But, which, at the moment, is not available.
> > Can you please advise?
> >
> > On Thu, Apr 28, 2022 at 5:21 PM Zhenya Stanilovsky 
> wrote:
> >>
> >> Seems it would be published with new documentation, Nikita Amelchev
> isn`t it ? check [1]
> >>
> >> [1] https://issues.apache.org/jira/browse/IGNITE-15189
> >>
> >>
> >> Thank you Stephen.
> >> Is there also a writeup summarizing what is/isn't supported with this
> 'experimental' feature?
> >>
> >> On Thu, Apr 28, 2022 at 4:30 PM Stephen Darlington <
> stephen.darling...@gridgain.com> wrote:
> >>
> >> https://github.com/apache/ignite/blob/2.13.0/modules/calcite/README.txt
> >>
> >>
> >> On 28 Apr 2022, at 11:46, Lokesh Bandaru 
> wrote:
> >>
> >> Thanks Ilya.
> >>
> >> Version 2.13 has come out but still seems to be shipping with the same
> vulnerability-ridden version of h2 database.
> >> The documentation doesn't mention if/how Calcite is turned on.
> >> Can you advise on how it can be enabled?
> >>
> >> On Wed, Apr 13, 2022 at 7:29 AM Ilya Korol 
> wrote:
> >>
> >> Hi Lokesh,
> >>
> >> Updates for running Ignite over Java 17 is already in master. Please
> >> take a look:
> >> https://github.com/apache/ignite/blob/master/bin/include/jvmdefaults.sh
> >>
> >> On 2022/04/12 10:11:57 Lokesh Bandaru wrote:
> >>  > You are fast. :) Was just typing a reply on top of the last one and
> yours
> >>  > is already here.
> >>  >
> >>  > Ignore the last question, found this,
> >>  >
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.13 .
> >>  > *Looking forward to this release. *
> >>  >
> >>  > *One slightly unrelated question, feel free to ignore. *
> >>  > *I know there is no support(or certified) for any version of Java
> greater
> >>  > than 11. *
> >>  > *What would it take for 2.13 to be able to run on Java17?*
> >>  >
> >>  > On Tue, Apr 12, 2022 at 3:36 PM Stephen Darlington <
> >>  > stephen.darling...@gridgain.com> wrote:
> >>  >
> >>  > > Code freeze was yesterday. The target release date is 22 April.
> >>  > >
> >>  > > More here: Apache+Ignite+2.13
> >>  > > <
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.13>
> >>  > >
> >>  > > On 12 Apr 2022, at 11:03, Lokesh Bandaru  wrote:
> >>  > >
> >>  > > Thanks for getting back, Stephen.
> >>  > > I am aware that Calcite is in the plans.
> >>  > > Any tentative timeline as to when 2.13(beta/ga) is going to be made
> >>  > > available?
> >>  > >
> >>  > > Regards.
> >>  > >
> >>  > > On Tue, Apr 12, 2022 at 2:15 PM Stephen Darlington <
> >>  > > stephen.darling...@gridgain.com> wrote:
> >>  > >
> >>  > >> The H2 project removed support for Ignite some time ago (
> >>  > >> https://github.com/h2database/h2database/pull/2227) which makes
> it
> >>  > >> difficult to move to newer versions.
> >>  > >>
> >>  > >> The next version of Ignite (2.13) has an alternative SQL engine
> >> (Apache
> >>  > >> Calcite) so over time there will be no need for H2.
> >>  > >>
> >>  > >> On 11 Apr 2022, at 20:34, Lokesh Bandaru  wrote:
> >>  > >>
> >>  > >> Resending.
> >>  > >>
> >>  > >> On Mon, Apr 11, 2022 at 6:42 PM Lokesh Bandaru 
> >>  > >> wrote:
> >>  > >>
> >>  > >>> Hello there, hi
> >>  > >>>
> >>  > >>> Writing to you with regards to the security
> >> vulnerabilities(particularly
> >>  > >>> the most recent ones, CVE-2022-xxx and CVE-2021-xxx) in the H2
> >> database and
> >>  > >>> the Apache Ignite's dependency on the flagged versions of H2.
> >>  > >>> There is an open issue tracking this,
> >>  > >>> https://issues.apache.org/jira/browse/IGNITE-16542, which
> doesn't
> >> seem
> >>  > >>> to have been fully addressed yet.
> >>  > >>> Have these problems been overcome already? Can you please advise?
> >>  > >>>
> >>  > >>> Thanks.
> >>  > >>>
> >>  > >>
> >>  > >>
> >>  > >
> >>  >
> >>
> >>
> >>
> >>
> >>
> >
> >
>
>
> --
> Best wishes,
> Amelchev Nikita
>

Re: Query performance

[Surinder Mehra]

Hey, it's enabled already. Please check the console log in my email

On Thu, 28 Apr 2022, 13:16 Zhenya Stanilovsky,  wrote:

>
> Hi, can you check the same with lazy [1] flag ?
>
> [1]
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/query/SqlFieldsQuery.html#setLazy-boolean-
>
>
> Hi,
> We are running a sql field query to fetch 4million records from ignite
> cache. We have created a group index for all fields used in where clause
> and can see group index used. But the query takes 20 minutes to fetch all
> records. If we provide more strict criteria to fetch only say 500 records,
> it completes in less than 200 millis. This makes me wonder if I am missing
> some configuration to make result fetching faster or I am not using ignite
> correctly for this use case. Below log is printed during query execution.
> Could you please advise me?
>
> [11:46:30,694][WARNING][query-#8568][GridMapQueryExecutor] Query produced
> big result set.  [fetched=10, duration=120331ms, type=MAP,
> distributedJoin=false, enforceJoinOrder=false, lazy=true,
>
>
>
>
>
>

Query performance

[Surinder Mehra]

Hi,
We are running a sql field query to fetch 4million records from ignite
cache. We have created a group index for all fields used in where clause
and can see group index used. But the query takes 20 minutes to fetch all
records. If we provide more strict criteria to fetch only say 500 records,
it completes in less than 200 millis. This makes me wonder if I am missing
some configuration to make result fetching faster or I am not using ignite
correctly for this use case. Below log is printed during query execution.
Could you please advise me?

[11:46:30,694][WARNING][query-#8568][GridMapQueryExecutor] Query produced
big result set.  [fetched=10, duration=120331ms, type=MAP,
distributedJoin=false, enforceJoinOrder=false, lazy=true,

Re: WAL cleanup

[Surinder Mehra]

Ok thanks, just more query.

"If all 10 segments are filled, then it will be expected (wait) for the
first of the segments to go to the archive."

Does it mean application writes will wait until one of the wal segment is
made free(that means atleast checkpoint  must complete so segment file can
move to wal archive) ?

On Fri, 22 Apr 2022, 18:06 ткаленко кирилл,  wrote:

> Set the maximum to 50% of the estimated value. If it is possible to use 60
> GB for the archive, set it to 30 GB.
>

Re: WAL cleanup

[Surinder Mehra]

Sorry for typo in last thread point 5, I meant to type "the ones which
moves segments from WAL to WAL Archive"

On Fri, 22 Apr 2022, 17:13 Surinder Mehra,  wrote:

> Ok thanks for clarifying. O have follow up queries
> 1. Wal has only 10 segments and default checkpoint frequency is 3 minutes
> What if application writes fill up all 10 segments before next checkpoint
> interval, would it trigger another checkpoint followed by moving of segments
>
> 2. Does it always move segments to archive after checkpoint? What if
> checkpoint is taking time and no space left in WAL. i.e. wal reached its
> max size(10*segmentsize)
>
> 3. If point 2 is true, why do we need walarchive. Since data has been
> flushed to disk with checkpoint. And any data not checkpointed is present
> in WAL, why do we need archive directory.
>
> 4. I have seen WAL size fixed(10G with segment size as 1G) to max size and
> WAL archive increases to 60GB. Does it mean cleaner thread wasn't frequent
> enough?
> 5. Are cleaner threads and the ones which love segments have any relation
> to checkpoint process or performance impact. If so, are they tunable by
> user ?
>
>
> On Fri, 22 Apr 2022, 16:52 ткаленко кирилл,  wrote:
>
>> Cleaning takes place in a separate thread.
>> Transfer of segments occurs in a separate thread after the checkpoint.
>>
>>

Re: WAL cleanup

[Surinder Mehra]

Ok thanks for clarifying. O have follow up queries
1. Wal has only 10 segments and default checkpoint frequency is 3 minutes
What if application writes fill up all 10 segments before next checkpoint
interval, would it trigger another checkpoint followed by moving of segments

2. Does it always move segments to archive after checkpoint? What if
checkpoint is taking time and no space left in WAL. i.e. wal reached its
max size(10*segmentsize)

3. If point 2 is true, why do we need walarchive. Since data has been
flushed to disk with checkpoint. And any data not checkpointed is present
in WAL, why do we need archive directory.

4. I have seen WAL size fixed(10G with segment size as 1G) to max size and
WAL archive increases to 60GB. Does it mean cleaner thread wasn't frequent
enough?
5. Are cleaner threads and the ones which love segments have any relation
to checkpoint process or performance impact. If so, are they tunable by
user ?


On Fri, 22 Apr 2022, 16:52 ткаленко кирилл,  wrote:

> Cleaning takes place in a separate thread.
> Transfer of segments occurs in a separate thread after the checkpoint.
>
>

WAL cleanup

[Surinder Mehra]

Hi,
I have a question about WAL file cleanup.
1. A cache update appends updates to WAl file and writes it to RAM. WAL can
rollover older than 10 files to WalArchive.
2. So checkpoint moves data from RAM to Disk

As per docs link below, after checkpoint, WAl files created before it can
be deleted. I also saw WAL cleanup logs in ignite nodes.
So I just want to know more about the cleanup process. Who does the
cleanup, checkpointing thread or separate cleaner process ?

Can someone please elaborate on it or share a link to docs which explain it.

https://ignite.apache.org/docs/latest/persistence/native-persistence#:~:text=After%20a%20checkpoint%20is%20passed,before%20that%20point%20in%20time
.

Re: Ignite query is taking long time even no data in that cache model

[Surinder Mehra]

this might help, scroll down to group indexes section:
https://ignite.apache.org/docs/latest/SQL/indexes

On Wed, Apr 20, 2022 at 10:59 AM Charlin S  wrote:

> Hi,
> why my query is having only one index field, it supposed to be two index
> column
>
> /* "TestModel".TESTMODEL_TESTFIELD3_ASC_IDX: TESTFIELD3 = 'EN' */
>
> Thanks & Regards,
> Charlin
>
>
> On Tue, 19 Apr 2022 at 19:34, Surinder Mehra  wrote:
>
>> I may be wrong but as an excercise, can you try group index on these
>> fields please and see if it makes any difference.
>> I would request Apache ignite Dev's to validate it.
>>
>> On Tue, 19 Apr 2022, 19:20 Charlin S,  wrote:
>>
>>> Hi,
>>> EXPLAIN SELECT
>>> TestField1,TestField2,TestField3,TestField4,TestField5
>>>  FROM
>>> TestModel
>>>  WHERE
>>> TestField2 = 'A02'
>>>  AND
>>> TestField3 = 'EN'
>>>
>>> resulview [0]: SELECT
>>> __Z0.TESTFIELD1 AS __C0_0,
>>> __Z0.TESTFIELD2 AS __C0_1,
>>> __Z0.TESTFIELD3 AS __C0_2,
>>> __Z0.TESTFIELD4 AS __C0_3,
>>> __Z0.TESTFIELD5 AS __C0_4
>>> FROM "TestModel".TESTMODEL __Z0
>>> /* "TestModel".TESTMODEL_TESTFIELD3_ASC_IDX: TESTFIELD3 = 'EN' */
>>> WHERE (__Z0.TESTFIELD2 = 'A02')
>>> AND (__Z0.TESTFIELD3 = 'EN')
>>>
>>> resultview[1]: SELECT
>>> __C0_0 AS TESTFIELD1,
>>> __C0_1 AS TESTFIELD2,
>>> __C0_2 AS TESTFIELD3,
>>> __C0_3 AS TESTFIELD4,
>>> __C0_4 AS TESTFIELD5
>>> FROM PUBLIC.__T0
>>> /* "TestModel"."merge_scan" */
>>>
>>> Thanks & Regards,
>>> Charlin
>>>
>>> On Tue, 19 Apr 2022 at 18:04, Surinder Mehra  wrote:
>>>
>>>> Looks correct to me. Can you run explain plain for this query and see
>>>> if it uses index.
>>>>
>>>> On Tue, 19 Apr 2022, 17:41 Charlin S,  wrote:
>>>>
>>>>> Hi,
>>>>> My query details are
>>>>> fieldsQuery="SELECT
>>>>> TestField1,TestField2,TestField3,TestField4,TestField5
>>>>>  FROM
>>>>> TestModel
>>>>>  WHERE
>>>>> TestField2 = 'A02'
>>>>>  AND
>>>>> TestField2 = 'EN'"
>>>>>
>>>>> //cache model
>>>>>  public class TestModel : IBinarizable
>>>>> {
>>>>>
>>>>> [QuerySqlField(IsIndexed = true)]
>>>>> public string TestField1 { get; set; }
>>>>> [QuerySqlField(IsIndexed = true)]
>>>>> public string TestField2 { get; set; }
>>>>> [QuerySqlField(IsIndexed = true)]
>>>>> public string TestField3 { get; set; }
>>>>> [QuerySqlField]
>>>>> public string TestField4 { get; set; }
>>>>> [QuerySqlField]
>>>>> public decimal? TestField5 { get; set; }
>>>>>
>>>>> public void ReadBinary(IBinaryReader reader){//implementation}
>>>>> public void WriteBinary(IBinaryWriter writer){//implementation}
>>>>> }
>>>>> implementation
>>>>>
>>>>> SqlFieldsQuery fieldsQuery = new SqlFieldsQuery(query) { Timeout =
>>>>> TimeSpan.FromMilliseconds(1) };
>>>>> List list = new List();
>>>>> // public ICache IgniteCache { get; set; }
>>>>> IFieldsQueryCursor queryCursor =
>>>>> IgniteCache.Query(fieldsQuery);
>>>>>
>>>>> //our implementation
>>>>>  queryCursor.Dispose();
>>>>>
>>>>> Thanks,
>>>>> Charlin
>>>>>
>>>>>
>>>>>
>>>>> On Mon, 18 Apr 2022 at 13:35, Surinder Mehra 
>>>>> wrote:
>>>>>
>>>>>> Can you please show slow query console log output if it's using index
>>>>>> scan or full cache scan.
>>>>>> I ran into one scenario where index wasn't used and it ended up
>>>>>> scaning whole cache.
>>>>>> You can try this locally by using control centre and run explain query
>>>>>>
>>>>>> On Mon, 18 Apr 2022, 13:08 Charlin S,  wrote:
>>>>>>
>>>>>>> Hi Ignite team,
>>>>>>> We are using Ignite 2.10.0 with 4.6.2 and .Net 5 WebAPI and we have
>>>>>>> a 16-nodes(including 2 server nodes) Ignite cluster.
>>>>>>> We are facing slowness issues with some particular cache model query
>>>>>>> and other models query are fine.
>>>>>>>
>>>>>>> query type: SqlFieldsQuery
>>>>>>> Index: index created for where clause columns.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Charlin
>>>>>>>
>>>>>>>

Re: Ignite query is taking long time even no data in that cache model

[Surinder Mehra]

I may be wrong but as an excercise, can you try group index on these fields
please and see if it makes any difference.
I would request Apache ignite Dev's to validate it.

On Tue, 19 Apr 2022, 19:20 Charlin S,  wrote:

> Hi,
> EXPLAIN SELECT
> TestField1,TestField2,TestField3,TestField4,TestField5
>  FROM
> TestModel
>  WHERE
> TestField2 = 'A02'
>  AND
> TestField3 = 'EN'
>
> resulview [0]: SELECT
> __Z0.TESTFIELD1 AS __C0_0,
> __Z0.TESTFIELD2 AS __C0_1,
> __Z0.TESTFIELD3 AS __C0_2,
> __Z0.TESTFIELD4 AS __C0_3,
> __Z0.TESTFIELD5 AS __C0_4
> FROM "TestModel".TESTMODEL __Z0
> /* "TestModel".TESTMODEL_TESTFIELD3_ASC_IDX: TESTFIELD3 = 'EN' */
> WHERE (__Z0.TESTFIELD2 = 'A02')
> AND (__Z0.TESTFIELD3 = 'EN')
>
> resultview[1]: SELECT
> __C0_0 AS TESTFIELD1,
> __C0_1 AS TESTFIELD2,
> __C0_2 AS TESTFIELD3,
> __C0_3 AS TESTFIELD4,
> __C0_4 AS TESTFIELD5
> FROM PUBLIC.__T0
> /* "TestModel"."merge_scan" */
>
> Thanks & Regards,
> Charlin
>
> On Tue, 19 Apr 2022 at 18:04, Surinder Mehra  wrote:
>
>> Looks correct to me. Can you run explain plain for this query and see if
>> it uses index.
>>
>> On Tue, 19 Apr 2022, 17:41 Charlin S,  wrote:
>>
>>> Hi,
>>> My query details are
>>> fieldsQuery="SELECT
>>> TestField1,TestField2,TestField3,TestField4,TestField5
>>>  FROM
>>> TestModel
>>>  WHERE
>>> TestField2 = 'A02'
>>>  AND
>>> TestField2 = 'EN'"
>>>
>>> //cache model
>>>  public class TestModel : IBinarizable
>>> {
>>>
>>> [QuerySqlField(IsIndexed = true)]
>>> public string TestField1 { get; set; }
>>> [QuerySqlField(IsIndexed = true)]
>>> public string TestField2 { get; set; }
>>> [QuerySqlField(IsIndexed = true)]
>>> public string TestField3 { get; set; }
>>> [QuerySqlField]
>>> public string TestField4 { get; set; }
>>> [QuerySqlField]
>>> public decimal? TestField5 { get; set; }
>>>
>>> public void ReadBinary(IBinaryReader reader){//implementation}
>>> public void WriteBinary(IBinaryWriter writer){//implementation}
>>> }
>>> implementation
>>>
>>> SqlFieldsQuery fieldsQuery = new SqlFieldsQuery(query) { Timeout =
>>> TimeSpan.FromMilliseconds(1) };
>>> List list = new List();
>>> // public ICache IgniteCache { get; set; }
>>> IFieldsQueryCursor queryCursor =
>>> IgniteCache.Query(fieldsQuery);
>>>
>>> //our implementation
>>>  queryCursor.Dispose();
>>>
>>> Thanks,
>>> Charlin
>>>
>>>
>>>
>>> On Mon, 18 Apr 2022 at 13:35, Surinder Mehra  wrote:
>>>
>>>> Can you please show slow query console log output if it's using index
>>>> scan or full cache scan.
>>>> I ran into one scenario where index wasn't used and it ended up scaning
>>>> whole cache.
>>>> You can try this locally by using control centre and run explain query
>>>>
>>>> On Mon, 18 Apr 2022, 13:08 Charlin S,  wrote:
>>>>
>>>>> Hi Ignite team,
>>>>> We are using Ignite 2.10.0 with 4.6.2 and .Net 5 WebAPI and we have a
>>>>> 16-nodes(including 2 server nodes) Ignite cluster.
>>>>> We are facing slowness issues with some particular cache model query
>>>>> and other models query are fine.
>>>>>
>>>>> query type: SqlFieldsQuery
>>>>> Index: index created for where clause columns.
>>>>>
>>>>> Regards,
>>>>> Charlin
>>>>>
>>>>>

Re: Ignite query is taking long time even no data in that cache model

[Surinder Mehra]

Looks correct to me. Can you run explain plain for this query and see if it
uses index.

On Tue, 19 Apr 2022, 17:41 Charlin S,  wrote:

> Hi,
> My query details are
> fieldsQuery="SELECT
> TestField1,TestField2,TestField3,TestField4,TestField5
>  FROM
> TestModel
>  WHERE
> TestField2 = 'A02'
>  AND
> TestField2 = 'EN'"
>
> //cache model
>  public class TestModel : IBinarizable
> {
>
> [QuerySqlField(IsIndexed = true)]
> public string TestField1 { get; set; }
> [QuerySqlField(IsIndexed = true)]
> public string TestField2 { get; set; }
> [QuerySqlField(IsIndexed = true)]
> public string TestField3 { get; set; }
> [QuerySqlField]
> public string TestField4 { get; set; }
> [QuerySqlField]
> public decimal? TestField5 { get; set; }
>
> public void ReadBinary(IBinaryReader reader){//implementation}
> public void WriteBinary(IBinaryWriter writer){//implementation}
> }
> implementation
>
> SqlFieldsQuery fieldsQuery = new SqlFieldsQuery(query) { Timeout =
> TimeSpan.FromMilliseconds(1) };
> List list = new List();
> // public ICache IgniteCache { get; set; }
> IFieldsQueryCursor queryCursor =
> IgniteCache.Query(fieldsQuery);
>
> //our implementation
>  queryCursor.Dispose();
>
> Thanks,
> Charlin
>
>
>
> On Mon, 18 Apr 2022 at 13:35, Surinder Mehra  wrote:
>
>> Can you please show slow query console log output if it's using index
>> scan or full cache scan.
>> I ran into one scenario where index wasn't used and it ended up scaning
>> whole cache.
>> You can try this locally by using control centre and run explain query
>>
>> On Mon, 18 Apr 2022, 13:08 Charlin S,  wrote:
>>
>>> Hi Ignite team,
>>> We are using Ignite 2.10.0 with 4.6.2 and .Net 5 WebAPI and we have a
>>> 16-nodes(including 2 server nodes) Ignite cluster.
>>> We are facing slowness issues with some particular cache model query and
>>> other models query are fine.
>>>
>>> query type: SqlFieldsQuery
>>> Index: index created for where clause columns.
>>>
>>> Regards,
>>> Charlin
>>>
>>>

Re: Ignite query is taking long time even no data in that cache model

[Surinder Mehra]

Can you please show slow query console log output if it's using index scan
or full cache scan.
I ran into one scenario where index wasn't used and it ended up scaning
whole cache.
You can try this locally by using control centre and run explain query

On Mon, 18 Apr 2022, 13:08 Charlin S,  wrote:

> Hi Ignite team,
> We are using Ignite 2.10.0 with 4.6.2 and .Net 5 WebAPI and we have a
> 16-nodes(including 2 server nodes) Ignite cluster.
> We are facing slowness issues with some particular cache model query and
> other models query are fine.
>
> query type: SqlFieldsQuery
> Index: index created for where clause columns.
>
> Regards,
> Charlin
>
>

Re: Checkpointing is taking long time

[Surinder Mehra]

Hey thanks for replying. we haven't configured the storage path so by
default it should be in the work directory. work, wal, walarchive all three
are SSDs. I have the following queries.

1. We have DirectIO disabled :  does enabling it impact the performance if
enabled?
2.  When we disabled throttling, we saw 5x better performance. Struggling
load test completed in 1/5th of time. What are its side effects if we keep
it disabled
3. Does MaxMemoryDirectSize have any relation to throughput rate.
4. Can the current configuration mentioned in the previous thread be scaled
further? like increasing WalSegment size beyond 1GB and related size of
walArchive, checkpointbufferSize and MaxMemoryDirectSize jvm parameter.
5. We see now due to throttling disabled, WalArchive size is going beyond
50G(WalSegment size 1G and checkpoint buffer size 6G). would decreasing
checkpoint frequency and/or increasing checkpoint threads count increase
throughput or impact application writes inversely. Currently checkpointing
frequency and threads are default


On Sun, Apr 17, 2022 at 6:33 AM Ilya Korol  wrote:

> Hi, From my perspective this looks like your physical storage is not
> fast enough to handle incoming writes. markDirty speed is 2x times
> faster that checkpointWrite even in the presence of throttling. You've
> mentioned that ignite work folder stored on SSD, but what about PDS
> folder (DataStorageConfiguration.setStoragePath())?
>
> Btw, have you tested your setup with DirectIO disabled?
>
> On 2022/04/14 10:55:23 Surinder Mehra wrote:
>  > Hi,
>  > We have an application with ignite thick clients which writes to ignite
>  > caches on ignite grid deployed separately. Below is the ignite
>  > configuration per node
>  >
>  > With this configuration, we see throttling happening and
> checkpointing time
>  > is between 20-30 seconds. Did we miss something in configuration or any
>  > other settings we can enable. Any suggestions will be of great help.
>  >
>  > * 100-200 concurrent writes to 25 node cluster
>  > * #partitions 512
>  > * cache backups = 2
>  > * cache mode partitioned
>  > * syncronizationMode : primary Sync
>  > * Off Heap caches
>  > * Server nodes : 25
>  > * RAM : 64G
>  > * maxmemoryDirectSize : 4G
>  > * Heap: 25G
>  >
>  > * persistenceEnabled: true
>  > * data region size : 24GB
>  > * checkPointingBufferSize: 6gb
>  > * walSegmentSize: 1G
>  > * walBufferSize : 256MB
>  > * walarchiveSize: 24G
>  > * writeThrotlingEnabled: true
>  > * checkPointingfreq : 60 sec
>  > * checkPointingThreads: 4
>  > * DirectIO enabled: true
>  >
>  > SSDs atatched:
>  > work volume : 20G
>  > wal volume : 15G
>  > Wal archive volume : 26G
>  >
>  >
>  > Checkpointing logs:
>  >
>  > [10:27:13,237][INFO][db-checkpoint-thread-#230][Checkpointer] Checkpoint
>  > started [checkpointId=11749dc0-fd0d-4b5f-8b9a-510e774fec38,
>  > startPtr=WALPointer [idx=26, fileOff=385214751, len=16683],
>  > checkpointBeforeLockTime=29ms, checkpointLockWait=0ms,
>  > checkpointListenersExecuteTime=2ms, checkpointLockHoldTime=3ms,
>  > walCpRecordFsyncDuration=11ms, writeCheckpointEntryDuration=3ms,
>  > splitAndSortCpPagesDuration=30ms, pages=40505, reason='timeout']
>  > [10:27:13,242][INFO][sys-stripe-7-#8][PageMemoryImpl] Throttling is
> applied
>  > to page modifications [percentOfPartTime=0.88, markDirty=2121 pages/sec,
>  > checkpointWrite=1219 pages/sec, estIdealMarkDirty=0 pages/sec,
>  > curDirty=0.00, maxDirty=0.02, avgParkTime=410172 ns, pages:
> (total=40505,
>  > evicted=0, written=10, synced=0, cpBufUsed=133, cpBufTotal=1554645)]
>  > [10:27:29,935][INFO][grid-timeout-worker-#30][IgniteKernal]
>  > Metrics for local node (to disable set 'metricsLogFrequency' to 0)
>  > ^-- Node [id=214f3c2b, uptime=00:45:00.227]
>  > ^-- Cluster [hosts=45, CPUs=540, servers=25, clients=20, topVer=75,
>  > minorTopVer=0]
>  > ^-- Network [addrs=[127.0.0.1, 192.168.98.141], discoPort=47500,
>  > commPort=47100]
>  > ^-- CPU [CPUs=12, curLoad=3.67%, avgLoad=0.82%, GC=0%]
>  > ^-- Heap [used=5330MB, free=79.18%, comm=20480MB]
>  > ^-- Off-heap memory [used=1019MB, free=95.92%, allocated=24775MB]
>  > ^-- Page memory [pages=257976]
>  > ^-- sysMemPlc region [type=internal, persistence=true,
>  > lazyAlloc=false,
>  > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%,
>  > allocRam=99MB, allocTotal=0MB]
>  > ^-- default region [type=default, persistence=true, lazyAlloc=true,
>  > ... initCfg=24576MB, maxCfg=24576MB, usedRam=1018MB, freeRam=95.86%,
>  > al

Checkpointing is taking long time

[Surinder Mehra]

Hi,
We have an application with ignite thick clients which writes to ignite
caches on ignite grid deployed separately. Below is the ignite
configuration per node

With this configuration, we see throttling happening and checkpointing time
is between 20-30 seconds. Did we miss something in configuration or any
other settings we can enable. Any suggestions will be of great help.

* 100-200 concurrent writes to 25 node cluster
* #partitions 512
* cache backups = 2
* cache mode partitioned
* syncronizationMode : primary Sync
* Off Heap caches
* Server nodes : 25
* RAM : 64G
* maxmemoryDirectSize :  4G
* Heap: 25G

* persistenceEnabled: true
* data region size : 24GB
* checkPointingBufferSize: 6gb
* walSegmentSize: 1G
* walBufferSize :  256MB
* walarchiveSize: 24G
* writeThrotlingEnabled: true
* checkPointingfreq :  60 sec
* checkPointingThreads: 4
* DirectIO enabled: true

SSDs atatched:
work volume : 20G
wal volume : 15G
Wal archive volume : 26G


Checkpointing logs:

[10:27:13,237][INFO][db-checkpoint-thread-#230][Checkpointer] Checkpoint
started [checkpointId=11749dc0-fd0d-4b5f-8b9a-510e774fec38,
startPtr=WALPointer [idx=26, fileOff=385214751, len=16683],
checkpointBeforeLockTime=29ms, checkpointLockWait=0ms,
checkpointListenersExecuteTime=2ms, checkpointLockHoldTime=3ms,
walCpRecordFsyncDuration=11ms, writeCheckpointEntryDuration=3ms,
splitAndSortCpPagesDuration=30ms, pages=40505, reason='timeout']
[10:27:13,242][INFO][sys-stripe-7-#8][PageMemoryImpl] Throttling is applied
to page modifications [percentOfPartTime=0.88, markDirty=2121 pages/sec,
checkpointWrite=1219 pages/sec, estIdealMarkDirty=0 pages/sec,
curDirty=0.00, maxDirty=0.02, avgParkTime=410172 ns, pages: (total=40505,
evicted=0, written=10, synced=0, cpBufUsed=133, cpBufTotal=1554645)]
[10:27:29,935][INFO][grid-timeout-worker-#30][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=214f3c2b, uptime=00:45:00.227]
^-- Cluster [hosts=45, CPUs=540, servers=25, clients=20, topVer=75,
minorTopVer=0]
^-- Network [addrs=[127.0.0.1, 192.168.98.141], discoPort=47500,
commPort=47100]
^-- CPU [CPUs=12, curLoad=3.67%, avgLoad=0.82%, GC=0%]
^-- Heap [used=5330MB, free=79.18%, comm=20480MB]
^-- Off-heap memory [used=1019MB, free=95.92%, allocated=24775MB]
^-- Page memory [pages=257976]
^--   sysMemPlc region [type=internal, persistence=true,
lazyAlloc=false,
  ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%,
allocRam=99MB, allocTotal=0MB]
^--   default region [type=default, persistence=true, lazyAlloc=true,
  ...  initCfg=24576MB, maxCfg=24576MB, usedRam=1018MB, freeRam=95.86%,
allocRam=24576MB, allocTotal=3820MB]
^--   metastoreMemPlc region [type=internal, persistence=true,
lazyAlloc=false,
  ...  initCfg=40MB, maxCfg=100MB, usedRam=1MB, freeRam=98.78%,
allocRam=0MB, allocTotal=1MB]
^--   TxLog region [type=internal, persistence=true, lazyAlloc=false,
  ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%,
allocRam=99MB, allocTotal=0MB]
^--   volatileDsMemPlc region [type=user, persistence=false,
lazyAlloc=true,
  ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%,
allocRam=0MB]
^-- Ignite persistence [used=3821MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=7, qSize=0]
^-- Striped thread pool [active=0, idle=12, qSize=0]
[10:27:38,261][INFO][db-checkpoint-thread-#230][Checkpointer] Checkpoint
finished [cpId=11749dc0-fd0d-4b5f-8b9a-510e774fec38, pages=40505,
markPos=WALPointer [idx=26, fileOff=385214751, len=16683],
walSegmentsCovered=[], markDuration=47ms, pagesWrite=25018ms, fsync=6ms,
total=25100ms]

Re: Get SqlFieldQueryResults by name

[Surinder Mehra]

Ok thanks for letting me know. Is there a guideline for raising feature
request.

On Fri, 8 Apr 2022, 18:10 Stephen Darlington, <
stephen.darling...@gridgain.com> wrote:

> That would be a nice feature but I don’t think it’s currently possible. It
> may be worth creating a feature request ticket.
>
> On 8 Apr 2022, at 13:30, Surinder Mehra  wrote:
>
> Hi,
> I was going through the below example and have a question if we can get
> fields using field names from query results.
> like "select name as person_name, p.age as person_age from Person p"
> It would return List. The inner list is each person row returned
> with results as per order in sql query. While this works fine but I was
> wondering if we can fetch row fields by name just like Oracle sql query
> rowMapper
>
>
> https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/sql/SqlDmlExample.java
>
>
>
>

Get SqlFieldQueryResults by name

[Surinder Mehra]

Hi,
I was going through the below example and have a question if we can get
fields using field names from query results.
like "select name as person_name, p.age as person_age from Person p"
It would return List. The inner list is each person row returned
with results as per order in sql query. While this works fine but I was
wondering if we can fetch row fields by name just like Oracle sql query
rowMapper

https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/sql/SqlDmlExample.java

Re: Return result stream from ignite compute

[Surinder Mehra]

Hey,
They are deployed already. What itbis complaining about is the lambda
inside stream.map(). If I remove it, it works fine.

Question is how can we work around stream operations which always take
lambda function.

On Sun, 3 Apr 2022, 00:15 Ilya Shishkov,  wrote:

> Hi Surinder,
>
> Your tasks and nested classes should be in the class path of the
> server node. You can deploy them manually or automatically by means of
> peer-class loading [1].
>
> 1.
> https://ignite.apache.org/docs/latest/code-deployment/peer-class-loading
>
> пт, 1 апр. 2022 г. в 15:12, Surinder Mehra :
>
>> Hi, I am trying to return the result stream from ignite compute task.
>> When compute task has a map() on stream it fails with below error . Can
>> someone please explain.
>>
>> "Exception in thread "main" class
>> org.apache.ignite.binary.BinaryObjectException: Failed to deserialize
>> object [typeName=java.util.stream.ReferencePipeline$3]"
>>
>> Setup: One default ignite node
>> Compute task
>> client node to submit compute task and collect the stream results.
>>
>> public class StreamTask implements IgniteCallable> {
>> @Override
>> public Stream call() throws Exception {
>>
>> Function integerDepartmentFunction = i -> new 
>> MyFunction().apply(i);
>> return IntStream.of(1, 2, 3).boxed().map(integerDepartmentFunction);
>> }
>> }
>>
>> class MyFunction implements Function {
>> @Override
>> public Department apply(Integer integer) {
>> return new Department(integer, "sdf"+ integer);
>> }
>> }
>>
>>
>> IgniteConfiguration cfg = new IgniteConfiguration();
>> cfg.setClientMode(true);
>> try (Ignite ignite = Ignition.start(cfg)) {
>>
>> ClusterGroup serversGrp = ignite.cluster().forServers();
>>
>> Stream stream = ignite.compute(serversGrp).call(new 
>> StreamTask());
>> System.out.println("Stream : "+ stream.collect(Collectors.toList()));
>>
>> }
>>
>> Error:
>>
>> Exception in thread "main" class 
>> org.apache.ignite.binary.BinaryObjectException: Failed to deserialize object 
>> [typeName=java.util.stream.ReferencePipeline$3]
>>  at 
>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:971)
>>  at 
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1769)
>>  at 
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1721)
>>  at 
>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:319)
>>  at 
>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:304)
>>  at 
>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:101)
>>  at 
>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:80)
>>  at 
>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10540)
>>
>> Caused by: java.lang.ClassNotFoundException: 
>> training.ignite.compute.StreamTask$$Lambda$894/0x000800670840
>>  at java.base/java.lang.Class.forName0(Native Method)
>>  at java.base/java.lang.Class.forName(Class.java:398)
>>  at 
>> org.apache.ignite.internal.util.IgniteUtils.forName(IgniteUtils.java:9064)
>>  at 
>> org.apache.ignite.internal.util.IgniteUtils.forName(IgniteUtils.java:9002)
>>  at 
>> org.apache.ignite.internal.MarshallerContextImpl.getClass(MarshallerContextImpl.java:376)
>>  at 
>> org.apache.ignite.internal.binary.BinaryContext.descriptorForTypeId(BinaryContext.java:693)
>>
>>

Return result stream from ignite compute

[Surinder Mehra]

Hi, I am trying to return the result stream from ignite compute task. When
compute task has a map() on stream it fails with below error . Can someone
please explain.

"Exception in thread "main" class
org.apache.ignite.binary.BinaryObjectException: Failed to deserialize
object [typeName=java.util.stream.ReferencePipeline$3]"

Setup: One default ignite node
Compute task
client node to submit compute task and collect the stream results.

public class StreamTask implements IgniteCallable> {
@Override
public Stream call() throws Exception {

Function integerDepartmentFunction = i ->
new MyFunction().apply(i);
return IntStream.of(1, 2, 3).boxed().map(integerDepartmentFunction);
}
}

class MyFunction implements Function {
@Override
public Department apply(Integer integer) {
return new Department(integer, "sdf"+ integer);
}
}


IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setClientMode(true);
try (Ignite ignite = Ignition.start(cfg)) {

ClusterGroup serversGrp = ignite.cluster().forServers();

Stream stream = ignite.compute(serversGrp).call(new
StreamTask());
System.out.println("Stream : "+ stream.collect(Collectors.toList()));

}

Error:

Exception in thread "main" class
org.apache.ignite.binary.BinaryObjectException: Failed to deserialize
object [typeName=java.util.stream.ReferencePipeline$3]
at 
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:971)
at 
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1769)
at 
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1721)
at 
org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:319)
at 
org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:304)
at 
org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:101)
at 
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:80)
at 
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10540)

Caused by: java.lang.ClassNotFoundException:
training.ignite.compute.StreamTask$$Lambda$894/0x000800670840
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:398)
at 
org.apache.ignite.internal.util.IgniteUtils.forName(IgniteUtils.java:9064)
at 
org.apache.ignite.internal.util.IgniteUtils.forName(IgniteUtils.java:9002)
at 
org.apache.ignite.internal.MarshallerContextImpl.getClass(MarshallerContextImpl.java:376)
at 
org.apache.ignite.internal.binary.BinaryContext.descriptorForTypeId(BinaryContext.java:693)

checkpointing timedout

[Surinder Mehra]

Hi,
We have a 4 node cluster with 64G RAM and 40G DISK per node attached for
/work and /walarchive each.  WAL dir is 10G per node

Below is our data region configuration and jvm_opts. We are getting
timeouts on checkpointing and WalArchive is getting filled up and no data
is moving to the /work directory. Checkpointing error is mentioned below

 could you suggest what's wrong with these configs(Note : with
walarchveSize as 16G and checkpointingBUffSize as 4G, it was writing but
very slow and throttling rate was 35% so I increased it, but it started
showing timeout errors)

JVM_OPTS: -XX:MaxDirectMemorySize=2g -Xms20g -Xmx25g
-XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC
-XX:+DisableExplicitGC

Ignite config:























Checkpointing logs:

[14:56:55,209][INFO][db-checkpoint-thread-#105][Checkpointer] Checkpoint
started [checkpointId=4b071279-8f25-4cd0-a5ef-e6f58f3a5653,
startPtr=WALPointer [idx=930, fileOff=527943956, len=51411],
checkpointBeforeLockTime=9ms, checkpointLockWait=0ms,
checkpointListenersExecuteTime=6ms, checkpointLockHoldTime=9ms,
walCpRecordFsyncDuration=7ms, writeCheckpointEntryDuration=3ms,
splitAndSortCpPagesDuration=45ms, pages=35542, reason='timeout']
[14:56:55,921][INFO][db-checkpoint-thread-#105][Checkpointer] Checkpoint
finished [cpId=4b071279-8f25-4cd0-a5ef-e6f58f3a5653, pages=35542,
markPos=WALPointer [idx=930, fileOff=527943956, len=51411],
walSegmentsCovered=[], markDuration=64ms, pagesWrite=108ms, fsync=604ms,
total=785ms]

Re: Unable to change maxWalArchiveSize in ignite 2.11.1

[Surinder Mehra]

thanks it worked

On Wed, Mar 23, 2022 at 8:59 AM Ilya Korol  wrote:

> Hi Surinder,
>
> I guess that there was Integer overflow in expression #{2 * 1024 * 1024
> * 1024} so it was evaluated as -2147483648. Try to add 'L' to one of
> multipliers like:
> #{2 * 1024 * 1024 * 1024L}
>
> 22.03.2022 23:14, Surinder Mehra пишет:
> > Hi,
> > We noticed that WalArchive size is going beyond the default max of
> > 1GB, so we tried to increase it in DataStorageConfiguration. But while
> > starting the ignite node, it always throws the below exception. Could
> > you please explain why. complete log in file attached.
> >
> > Reason for change:
> > Starting to clean WAL archive [highIdx=992, currSize=2.2 GB,
> > maxSize=1.0 GB]
> >
> > change done:
> > 
> >  > class="org.apache.ignite.configuration.DataStorageConfiguration">
> > 
> > 
> > 
> > 
> >  > class="org.apache.ignite.configuration.DataRegionConfiguration">
> >  value="true"/>
> > 
> > 
> >
> > 
> >  > value="/ignite/walarchive"/>
> > 
> > 
> >
> >
> > Error log while starting:
> > PropertyAccessException 1:
> > org.springframework.beans.MethodInvocationException: Property
> > 'maxWalArchiveSize' threw exception; nested exception is
> > java.lang.IllegalArgumentException: Ouch! Argument is invalid: Max WAL
> > archive size can be only greater than 0 or must be equal to -1 (to be
> > unlimited)]
> >
>

Unable to change maxWalArchiveSize in ignite 2.11.1

[Surinder Mehra]

Hi,
We noticed that WalArchive size is going beyond the default max of 1GB, so
we tried to increase it in DataStorageConfiguration. But while starting the
ignite node, it always throws the below exception. Could you please explain
why. complete log in file attached.

Reason for change:
Starting to clean WAL archive [highIdx=992, currSize=2.2 GB, maxSize=1.0 GB]

change done:

















Error log while starting:
PropertyAccessException 1:
org.springframework.beans.MethodInvocationException: Property
'maxWalArchiveSize' threw exception; nested exception is
java.lang.IllegalArgumentException: Ouch! Argument is invalid: Max WAL
archive size can be only greater than 0 or must be equal to -1 (to be
unlimited)]
[17:48:54,958][INFO][wal-file-cleaner%null-#59][FileWriteAheadLogManager] 
Starting to clean WAL archive [highIdx=992, currSize=2.2 GB, maxSize=1.0 GB]
[17:48:54,960][INFO][db-checkpoint-thread-#68][Checkpointer] Checkpoint 
finished [cpId=4dbfd782-8e8b-490b-ba1c-ecfa6eb83b7b, pages=217924, 
markPos=WALPointer [idx=993, fileOff=7340668, len=60987], 
walSegmentsCovered=[985 - 992], markDuration=203ms, pagesWrite=1397ms, 
fsync=5679ms, total=7287ms]
[17:48:55,003][INFO][db-checkpoint-thread-#68][Checkpointer] Checkpoint started 
[checkpointId=e18ab515-e3d7-4ec1-84fd-494adb430997, startPtr=WALPointer 
[idx=994, fileOff=10801377, len=60987], checkpointBeforeLockTime=5ms, 
checkpointLockWait=0ms, checkpointListenersExecuteTime=6ms, 
checkpointLockHoldTime=6ms, walCpRecordFsyncDuration=8ms, 
writeCheckpointEntryDuration=4ms, splitAndSortCpPagesDuration=20ms, 
pages=30961, reason='too big size of WAL without checkpoint']
[17:48:55,196][INFO][wal-file-cleaner%null-#59][FileWriteAheadLogManager] 
Finish clean WAL archive [cleanCnt=7, currSize=512.0 MB, maxSize=1.0 GB]
[17:48:56,562][INFO][db-checkpoint-thread-#68][Checkpointer] Checkpoint 
finished [cpId=e18ab515-e3d7-4ec1-84fd-494adb430997, pages=30961, 
markPos=WALPointer [idx=994, fileOff=10801377, len=60987], 
walSegmentsCovered=[993], markDuration=39ms, pagesWrite=195ms, fsync=1363ms, 
total=1602ms]




















class org.apache.ignite.IgniteException: Failed to instantiate Spring XML 
application context [springUrl=file:/ignite/config/node-configuration.xml, 
err=Error creating bean with name 
'org.apache.ignite.configuration.IgniteConfiguration#0' defined in URL 
[file:/ignite/config/node-configuration.xml]: Cannot create inner bean 
'org.apache.ignite.configuration.DataStorageConfiguration#6497b078' of type 
[org.apache.ignite.configuration.DataStorageConfiguration] while setting bean 
property 'dataStorageConfiguration'; nested exception is 
org.springframework.beans.factory.BeanCreationException: Error creating bean 
with name 'org.apache.ignite.configuration.DataStorageConfiguration#6497b078' 
defined in URL [file:/ignite/config/node-configuration.xml]: Error setting 
property values; nested exception is 
org.springframework.beans.PropertyBatchUpdateException; nested 
PropertyAccessExceptions (1) are:
PropertyAccessException 1: org.springframework.beans.MethodInvocationException: 
Property 'maxWalArchiveSize' threw exception; nested exception is 
java.lang.IllegalArgumentException: Ouch! Argument is invalid: Max WAL archive 
size can be only greater than 0 or must be equal to -1 (to be unlimited)]
at 
org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1098)
at org.apache.ignite.Ignition.start(Ignition.java:356)
at 
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:367)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to 
instantiate Spring XML application context 
[springUrl=file:/ignite/config/node-configuration.xml, err=Error creating bean 
with name 'org.apache.ignite.configuration.IgniteConfiguration#0' defined in 
URL [file:/ignite/config/node-configuration.xml]: Cannot create inner bean 
'org.apache.ignite.configuration.DataStorageConfiguration#6497b078' of type 
[org.apache.ignite.configuration.DataStorageConfiguration] while setting bean 
property 'dataStorageConfiguration'; nested exception is 
org.springframework.beans.factory.BeanCreationException: Error creating bean 
with name 'org.apache.ignite.configuration.DataStorageConfiguration#6497b078' 
defined in URL [file:/ignite/config/node-configuration.xml]: Error setting 
property values; nested exception is 
org.springframework.beans.PropertyBatchUpdateException; nested 
PropertyAccessExceptions (1) are:
PropertyAccessException 1: org.springframework.beans.MethodInvocationEx

Re: getAllkeys Api

[Surinder Mehra]

Thanks for pointing out heap side view on this. Will try it

On Tue, 8 Mar 2022, 21:09 Stephen Darlington, <
stephen.darling...@gridgain.com> wrote:

> Yes, the transformer runs on the server side, so it should be functionally
> the same. However, a scan is going to copy the key and value to the heap,
> so it’s likely to use more memory than a SQL query.
>
> On 8 Mar 2022, at 10:04, Surinder Mehra  wrote:
>
> Hi,
> Actually SQL is not enabled on these caches.
> We can try compute but I was thinking to use scan query with 'transformer'
> . I assume transformer runs on server node right so isn't it same as
> "select _key from cache"
>
> On Tue, 8 Mar 2022, 15:08 Stephen Darlington, <
> stephen.darling...@gridgain.com> wrote:
>
>> You could use SQL. “select _key from table”
>>
>> But really, copying all the data over the network is often not the best
>> strategy. Send a compute job to each node or partition and work on the data
>> “in place” on the data nodes.
>>
>> On 8 Mar 2022, at 03:56, Surinder Mehra  wrote:
>>
>> Hi,
>> I was looking for a way to fetch all cache keys in an efficient manner
>> from ignite cache. Looks like there is no such api yet.
>> Would it be correct to use a transformer in Scan query to just fetch the
>> key from entry being scanned. This will avoid fetching full cache entry
>> from server nodes
>>
>> Or is there a better way to do this.
>>
>>
>> https://www.gridgain.com/docs/latest/developers-guide/key-value-api/using-scan-queries
>>
>>
>>
>>
>

Activating persistence ignite cluster in EKS

[Surinder Mehra]

Hi,
As per ignite deployment on EKS docs, we need to activate the cluster once
all replicas are up.

Activating it manually is not a production ready setup. We were trying to
add a postStart hook to the pod but that causes issues since it activates
the cluster when first replicas start and the rest of replicas
don't become part of baseline as expected.

*Is there an automatic way to activate the cluster when statefulsets
complete creating replicas.*

P.S. We could do it from a thick client, but that is risky if the client
starts before all replicas are started. We will end up in same situation
where few nodes are out of baseline

[image: image.png]

https://ignite.apache.org/docs/latest/installation/kubernetes/amazon-eks-deployment

Re: getAllkeys Api

[Surinder Mehra]

Hi,
Actually SQL is not enabled on these caches.
We can try compute but I was thinking to use scan query with 'transformer'
. I assume transformer runs on server node right so isn't it same as
"select _key from cache"

On Tue, 8 Mar 2022, 15:08 Stephen Darlington, <
stephen.darling...@gridgain.com> wrote:

> You could use SQL. “select _key from table”
>
> But really, copying all the data over the network is often not the best
> strategy. Send a compute job to each node or partition and work on the data
> “in place” on the data nodes.
>
> On 8 Mar 2022, at 03:56, Surinder Mehra  wrote:
>
> Hi,
> I was looking for a way to fetch all cache keys in an efficient manner
> from ignite cache. Looks like there is no such api yet.
> Would it be correct to use a transformer in Scan query to just fetch the
> key from entry being scanned. This will avoid fetching full cache entry
> from server nodes
>
> Or is there a better way to do this.
>
>
> https://www.gridgain.com/docs/latest/developers-guide/key-value-api/using-scan-queries
>
>
>
>

getAllkeys Api

[Surinder Mehra]

Hi,
I was looking for a way to fetch all cache keys in an efficient manner from
ignite cache. Looks like there is no such api yet.
Would it be correct to use a transformer in Scan query to just fetch the
key from entry being scanned. This will avoid fetching full cache entry
from server nodes

Or is there a better way to do this.

https://www.gridgain.com/docs/latest/developers-guide/key-value-api/using-scan-queries

Why ignite chose Rendezvous hashing

[Surinder Mehra]

Hi,
While go through blogpost below, I couldn't completely understand why
ignite chose Rendezvous over consistent hashing for partition to node
mapping.

Does the later gives some clear performance boost.

https://www.gridgain.com/resources/blog/data-distribution-in-apache-ignite

Re: Ever increasing Disk space for a cache in ignite storage directory

[Surinder Mehra]

Hi,
As per ignite docs, " It is reused when new entries need to be added to the
storage on subsequent write operations. Therefore, the allocated size does
not decrease when you remove entries from the caches  ".
Ignite devs can comment more on this.

https://ignite.apache.org/docs/latest/monitoring-metrics/metrics

On Tue, Feb 22, 2022 at 12:48 PM Sumit Deshinge 
wrote:

> Hi Surinder,
>
> We have native persistence enabled, so I guess eviction policies should
> not come into the picture.
> Cache configuration is as follows:
> Atomicity: Transactional
> CacheMode: Replicated
> Rebalance mode: sync
>
> The cache entries are removed using cache APIs : remove and/or removeAll.
>
> This removes cache entries as in I can see cache get api returns nothing
> for the entries removed, but the disk space does not get reduced.
> Even after let's say all entries are removed from cache, disk space used
> for the cache remains as is.
>
> My observation is that the data is deleted from disk after
> cache.destroy(), but I can not do this operation as the data is
> continuously being read/written on the cache.
>
>
> On Tue, Feb 22, 2022 at 9:14 AM Surinder Mehra  wrote:
>
>> Do you manually remove that entry from cache or using eviction/expiration
>> policy.
>>
>> Can you share cache configuration please.
>>
>> This thread might help you(comments)
>>
>>
>> https://stackoverflow.com/questions/48951091/ignite-how-eviction-expiry-and-rebalancing-work-with-external-cachestore
>>
>> On Tue, Feb 22, 2022, 08:27 Sumit Deshinge 
>> wrote:
>>
>>> Hi,
>>>
>>> We are using ignite 2.12 with persistence enabled along with full sync -
>>> replicated mode.
>>> There is a cache whose data ranges from few Kilobytes to few Megabytes.
>>> Once a particular cache entry is worked upon, we remove that entry from
>>> cache. The cache goes through a large number of reads/writes parallely.
>>>
>>> But what we have observed is, the disk space utilized by the cache keeps
>>> on increasing. I had expected the disk space to be reused - (deleted cache
>>> entries disk partitions) for new cache entries being inserted , but this is
>>> not happening.
>>>
>>> Any reason for this behavior and any solution to avoid such problems?
>>>
>>> --
>>> Regards,
>>> Sumit Deshinge
>>>
>>>
>
> --
> Regards,
> Sumit Deshinge
>
>

Re: Ever increasing Disk space for a cache in ignite storage directory

[Surinder Mehra]

Do you manually remove that entry from cache or using eviction/expiration
policy.

Can you share cache configuration please.

This thread might help you(comments)

https://stackoverflow.com/questions/48951091/ignite-how-eviction-expiry-and-rebalancing-work-with-external-cachestore

On Tue, Feb 22, 2022, 08:27 Sumit Deshinge  wrote:

> Hi,
>
> We are using ignite 2.12 with persistence enabled along with full sync -
> replicated mode.
> There is a cache whose data ranges from few Kilobytes to few Megabytes.
> Once a particular cache entry is worked upon, we remove that entry from
> cache. The cache goes through a large number of reads/writes parallely.
>
> But what we have observed is, the disk space utilized by the cache keeps
> on increasing. I had expected the disk space to be reused - (deleted cache
> entries disk partitions) for new cache entries being inserted , but this is
> not happening.
>
> Any reason for this behavior and any solution to avoid such problems?
>
> --
> Regards,
> Sumit Deshinge
>
>

Re: Unable to execute sql in Control centre

[Surinder Mehra]

Yes thank you !

On Wed, Feb 16, 2022, 14:33 Pavel Tupitsyn  wrote:

> I think schema name is required:
> SELECT * FROM "deptCache".DEPARTMENT
>
> On Wed, Feb 16, 2022 at 11:55 AM Surinder Mehra 
> wrote:
>
>> Hi,
>> I tried to execute sql commands in control centre as per demo in ignite
>> training on 15h feb
>> I always get thai error " table not found"
>> As you can see table name and schema is present on control centre. Same
>> query works in java code.
>> Am I missing something?
>>
>> Cache Config:
>>
>> CacheConfiguration deptCacheConfig = new 
>> CacheConfiguration<>(DEPT_CACHE);
>> deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
>> deptCacheConfig.setBackups(1);
>> deptCacheConfig.setIndexedTypes(Integer.class, Department.class);
>>
>>
>> Department class:
>>
>> public class Department implements Serializable{
>> @QuerySqlField(index = true)
>> private final int deptId;
>> @QuerySqlField
>> private final String deptName;
>>
>>
>> [image: image.png]
>>
>>

Unable to execute sql in Control centre

[Surinder Mehra]

Hi,
I tried to execute sql commands in control centre as per demo in ignite
training on 15h feb
I always get thai error " table not found"
As you can see table name and schema is present on control centre. Same
query works in java code.
Am I missing something?

Cache Config:

CacheConfiguration deptCacheConfig = new
CacheConfiguration<>(DEPT_CACHE);
deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
deptCacheConfig.setBackups(1);
deptCacheConfig.setIndexedTypes(Integer.class, Department.class);


Department class:

public class Department implements Serializable{
@QuerySqlField(index = true)
private final int deptId;
@QuerySqlField
private final String deptName;


[image: image.png]

Re: Restore data from Ignite snapshots

[Surinder Mehra]

Hey guys,
Can someone explain pls why snaphot restore doesnt work woth control.sh

On Fri, Feb 4, 2022, 18:57 Surinder Mehra  wrote:

> hey,
> Did you get a chance to review my queries please.
>
> On Thu, Feb 3, 2022 at 4:40 PM Surinder Mehra  wrote:
>
>> Hi,
>> So the way I am thinking to use it is if we lose the EBS volume and we
>> need to restore the cluster state back. I would have a secondary EBS as my
>> snapshot directory so I can restore from it.
>> It means Application would need to be restarted after EBS data is copied
>> back to the work directory. I see two options here
>> 1. Manual as described in previous reply. manually copy data from
>> snapshot directory to work/db and restart cluster
>> 2. Use control script : I am not clear on how will this work because  If
>> I restart cluster, it is going to create directory structure again and then
>> when we run restore command, it does not copy data
>>
>> Could you please suggest how it would work. directory structure is
>> attached.
>> Also, can you suggest a better way to copy snapshot directory data to S3
>> . I am thinking of using a kubernetes CSI driver to do it. Any objections
>> to it
>>
>>
>>
>> On Thu, Feb 3, 2022 at 4:23 PM Maxim Muzafarov  wrote:
>>
>>> Hello,
>>>
>>> You don't need to stop the cluster or delete/move any snapshot files
>>> in case you are using the restore procedure from the control.sh, so
>>> the following should work:
>>> - create snapshot
>>> - stop the caches you are intended to restore
>>> - run ./control.sh --snapshot restore snapshot_1 --start
>>>
>>> Can you provide the directory structure of the Ignite working
>>> directory? (use `tree` command)
>>>
>>> On Wed, 2 Feb 2022 at 22:15, Surinder Mehra  wrote:
>>> >
>>> > Hi,
>>> > Could you please point out if i missed something?
>>> >
>>> > On Wed, Feb 2, 2022, 13:39 Surinder Mehra  wrote:
>>> >>
>>> >> Hey thanks for your suggestions.
>>> >>
>>> >> I tried restoring using control.sh but it doesn't seem to work. Below
>>> are steps
>>> >> 1. Started 3 nodes and added data using a thick client
>>> >> 2. created a snapshot using with  ./control.sh --snapshot create
>>> snapshot_1
>>> >> 3. I verified, the snapshot directory has data
>>> >> 4. Stopped the cluster and cleared binary_data, marshaler and nodes
>>> directory /db
>>> >> 5. Started the cluster again, all 3 nodes
>>> >> 6. Activate the cluster using ./control.sh --set-state ACTIVE
>>> >> 7. Run restore command : ./control.sh --snapshot restore snapshot_1
>>> --start
>>> >> 8. Command was successful but data is not copied to cluster nodes.
>>> >>
>>> >> Please note that when I restarted the cluster, it created
>>> binary_data, marshaler and nodes directories by default.
>>> >>
>>> >> Did I miss anything ?
>>> >>
>>> >>
>>> >> On Tue, Feb 1, 2022 at 8:21 PM Maxim Muzafarov 
>>> wrote:
>>> >>>
>>> >>> Hello,
>>> >>>
>>> >>> Your case looks correct to me, however, I'd like to mention some
>>> >>> important points that may help you:
>>> >>> - the directories structure of the snapshot has the same structure as
>>> >>> the Ignite native persistence, so you may backup the original cluster
>>> >>> node directory (for binary_data, marshaller and db) and move all the
>>> >>> files right from the snapshot.
>>> >>> - do not forget to backup and clear the original wal directory in
>>> case
>>> >>> of restoration.
>>> >>> - you may use control.sh --snapshot restore command to restore from a
>>> >>> snapshot (this was added in 2.11)
>>> >>>
>>> >>> [1] https://issues.apache.org/jira/browse/IGNITE-13805
>>> >>>
>>> >>> On Tue, 1 Feb 2022 at 16:28, Surinder Mehra 
>>> wrote:
>>> >>> >
>>> >>> > Hi,
>>> >>> > After a few hiccups, I managed to restore the cluster state from
>>> the snapshot. Please confirm if they look correct. If so documentation page
>>> needs to be updated
>>> >>> >
>>> >>> > Create N nodes
>>>

Re: Restore data from Ignite snapshots

[Surinder Mehra]

hey,
Did you get a chance to review my queries please.

On Thu, Feb 3, 2022 at 4:40 PM Surinder Mehra  wrote:

> Hi,
> So the way I am thinking to use it is if we lose the EBS volume and we
> need to restore the cluster state back. I would have a secondary EBS as my
> snapshot directory so I can restore from it.
> It means Application would need to be restarted after EBS data is copied
> back to the work directory. I see two options here
> 1. Manual as described in previous reply. manually copy data from snapshot
> directory to work/db and restart cluster
> 2. Use control script : I am not clear on how will this work because  If I
> restart cluster, it is going to create directory structure again and then
> when we run restore command, it does not copy data
>
> Could you please suggest how it would work. directory structure is
> attached.
> Also, can you suggest a better way to copy snapshot directory data to S3 .
> I am thinking of using a kubernetes CSI driver to do it. Any objections to
> it
>
>
>
> On Thu, Feb 3, 2022 at 4:23 PM Maxim Muzafarov  wrote:
>
>> Hello,
>>
>> You don't need to stop the cluster or delete/move any snapshot files
>> in case you are using the restore procedure from the control.sh, so
>> the following should work:
>> - create snapshot
>> - stop the caches you are intended to restore
>> - run ./control.sh --snapshot restore snapshot_1 --start
>>
>> Can you provide the directory structure of the Ignite working
>> directory? (use `tree` command)
>>
>> On Wed, 2 Feb 2022 at 22:15, Surinder Mehra  wrote:
>> >
>> > Hi,
>> > Could you please point out if i missed something?
>> >
>> > On Wed, Feb 2, 2022, 13:39 Surinder Mehra  wrote:
>> >>
>> >> Hey thanks for your suggestions.
>> >>
>> >> I tried restoring using control.sh but it doesn't seem to work. Below
>> are steps
>> >> 1. Started 3 nodes and added data using a thick client
>> >> 2. created a snapshot using with  ./control.sh --snapshot create
>> snapshot_1
>> >> 3. I verified, the snapshot directory has data
>> >> 4. Stopped the cluster and cleared binary_data, marshaler and nodes
>> directory /db
>> >> 5. Started the cluster again, all 3 nodes
>> >> 6. Activate the cluster using ./control.sh --set-state ACTIVE
>> >> 7. Run restore command : ./control.sh --snapshot restore snapshot_1
>> --start
>> >> 8. Command was successful but data is not copied to cluster nodes.
>> >>
>> >> Please note that when I restarted the cluster, it created binary_data,
>> marshaler and nodes directories by default.
>> >>
>> >> Did I miss anything ?
>> >>
>> >>
>> >> On Tue, Feb 1, 2022 at 8:21 PM Maxim Muzafarov 
>> wrote:
>> >>>
>> >>> Hello,
>> >>>
>> >>> Your case looks correct to me, however, I'd like to mention some
>> >>> important points that may help you:
>> >>> - the directories structure of the snapshot has the same structure as
>> >>> the Ignite native persistence, so you may backup the original cluster
>> >>> node directory (for binary_data, marshaller and db) and move all the
>> >>> files right from the snapshot.
>> >>> - do not forget to backup and clear the original wal directory in case
>> >>> of restoration.
>> >>> - you may use control.sh --snapshot restore command to restore from a
>> >>> snapshot (this was added in 2.11)
>> >>>
>> >>> [1] https://issues.apache.org/jira/browse/IGNITE-13805
>> >>>
>> >>> On Tue, 1 Feb 2022 at 16:28, Surinder Mehra 
>> wrote:
>> >>> >
>> >>> > Hi,
>> >>> > After a few hiccups, I managed to restore the cluster state from
>> the snapshot. Please confirm if they look correct. If so documentation page
>> needs to be updated
>> >>> >
>> >>> > Create N nodes
>> >>> > Add some data to them
>> >>> > Create snapshot
>> >>> > Stop all nodes(cluster)
>> >>> > Delete binary_data, marsheller and sub directories of /work/db
>> >>> > Copy snapshots/snapshotname/db/binary_data to /work/db/,
>> >>> > Copy snapshots/snapshotname/db/marshaller to /work/db/
>> >>> > Copy  snapshots/snapshotname/db/{nodeid} dir to /work/db/
>> >>> > Start cluster
>> >>

Re: Restore data from Ignite snapshots

[Surinder Mehra]

Hi,
So the way I am thinking to use it is if we lose the EBS volume and we need
to restore the cluster state back. I would have a secondary EBS as my
snapshot directory so I can restore from it.
It means Application would need to be restarted after EBS data is copied
back to the work directory. I see two options here
1. Manual as described in previous reply. manually copy data from snapshot
directory to work/db and restart cluster
2. Use control script : I am not clear on how will this work because  If I
restart cluster, it is going to create directory structure again and then
when we run restore command, it does not copy data

Could you please suggest how it would work. directory structure is
attached.
Also, can you suggest a better way to copy snapshot directory data to S3 .
I am thinking of using a kubernetes CSI driver to do it. Any objections to
it



On Thu, Feb 3, 2022 at 4:23 PM Maxim Muzafarov  wrote:

> Hello,
>
> You don't need to stop the cluster or delete/move any snapshot files
> in case you are using the restore procedure from the control.sh, so
> the following should work:
> - create snapshot
> - stop the caches you are intended to restore
> - run ./control.sh --snapshot restore snapshot_1 --start
>
> Can you provide the directory structure of the Ignite working
> directory? (use `tree` command)
>
> On Wed, 2 Feb 2022 at 22:15, Surinder Mehra  wrote:
> >
> > Hi,
> > Could you please point out if i missed something?
> >
> > On Wed, Feb 2, 2022, 13:39 Surinder Mehra  wrote:
> >>
> >> Hey thanks for your suggestions.
> >>
> >> I tried restoring using control.sh but it doesn't seem to work. Below
> are steps
> >> 1. Started 3 nodes and added data using a thick client
> >> 2. created a snapshot using with  ./control.sh --snapshot create
> snapshot_1
> >> 3. I verified, the snapshot directory has data
> >> 4. Stopped the cluster and cleared binary_data, marshaler and nodes
> directory /db
> >> 5. Started the cluster again, all 3 nodes
> >> 6. Activate the cluster using ./control.sh --set-state ACTIVE
> >> 7. Run restore command : ./control.sh --snapshot restore snapshot_1
> --start
> >> 8. Command was successful but data is not copied to cluster nodes.
> >>
> >> Please note that when I restarted the cluster, it created binary_data,
> marshaler and nodes directories by default.
> >>
> >> Did I miss anything ?
> >>
> >>
> >> On Tue, Feb 1, 2022 at 8:21 PM Maxim Muzafarov 
> wrote:
> >>>
> >>> Hello,
> >>>
> >>> Your case looks correct to me, however, I'd like to mention some
> >>> important points that may help you:
> >>> - the directories structure of the snapshot has the same structure as
> >>> the Ignite native persistence, so you may backup the original cluster
> >>> node directory (for binary_data, marshaller and db) and move all the
> >>> files right from the snapshot.
> >>> - do not forget to backup and clear the original wal directory in case
> >>> of restoration.
> >>> - you may use control.sh --snapshot restore command to restore from a
> >>> snapshot (this was added in 2.11)
> >>>
> >>> [1] https://issues.apache.org/jira/browse/IGNITE-13805
> >>>
> >>> On Tue, 1 Feb 2022 at 16:28, Surinder Mehra 
> wrote:
> >>> >
> >>> > Hi,
> >>> > After a few hiccups, I managed to restore the cluster state from the
> snapshot. Please confirm if they look correct. If so documentation page
> needs to be updated
> >>> >
> >>> > Create N nodes
> >>> > Add some data to them
> >>> > Create snapshot
> >>> > Stop all nodes(cluster)
> >>> > Delete binary_data, marsheller and sub directories of /work/db
> >>> > Copy snapshots/snapshotname/db/binary_data to /work/db/,
> >>> > Copy snapshots/snapshotname/db/marshaller to /work/db/
> >>> > Copy  snapshots/snapshotname/db/{nodeid} dir to /work/db/
> >>> > Start cluster
> >>> > Cluster should auto activate after all nodes join it
> >>> > Cluster is ready
> >>> >
> >>> >
> >>> > On Mon, Jan 31, 2022 at 7:14 PM Surinder Mehra 
> wrote:
> >>> >>
> >>> >> Hi,
> >>> >> We are using ignite 2.11.1 to experiment with ignite snapshots. We
> tried steps mentioned on below page to restore ignite data from snapshot
> >>> >> https://ignite.apache.org/docs/la

Re: Restore data from Ignite snapshots

[Surinder Mehra]

Hi,
Could you please point out if i missed something?

On Wed, Feb 2, 2022, 13:39 Surinder Mehra  wrote:

> Hey thanks for your suggestions.
>
> I tried restoring using control.sh but it doesn't seem to work. Below are
> steps
> 1. Started 3 nodes and added data using a thick client
> 2. created a snapshot using with  ./control.sh --snapshot create snapshot_1
> 3. I verified, the snapshot directory has data
> 4. Stopped the cluster and cleared binary_data, marshaler and nodes
> directory /db
> 5. Started the cluster again, all 3 nodes
> 6. Activate the cluster using ./control.sh --set-state ACTIVE
> 7. Run restore command : ./control.sh --snapshot restore snapshot_1 --start
> 8. Command was successful but data is not copied to cluster nodes.
>
> Please note that when I restarted the cluster, it created binary_data,
> marshaler and nodes directories by default.
>
> Did I miss anything ?
>
>
> On Tue, Feb 1, 2022 at 8:21 PM Maxim Muzafarov  wrote:
>
>> Hello,
>>
>> Your case looks correct to me, however, I'd like to mention some
>> important points that may help you:
>> - the directories structure of the snapshot has the same structure as
>> the Ignite native persistence, so you may backup the original cluster
>> node directory (for binary_data, marshaller and db) and move all the
>> files right from the snapshot.
>> - do not forget to backup and clear the original wal directory in case
>> of restoration.
>> - you may use control.sh --snapshot restore command to restore from a
>> snapshot (this was added in 2.11)
>>
>> [1] https://issues.apache.org/jira/browse/IGNITE-13805
>>
>> On Tue, 1 Feb 2022 at 16:28, Surinder Mehra  wrote:
>> >
>> > Hi,
>> > After a few hiccups, I managed to restore the cluster state from the
>> snapshot. Please confirm if they look correct. If so documentation page
>> needs to be updated
>> >
>> > Create N nodes
>> > Add some data to them
>> > Create snapshot
>> > Stop all nodes(cluster)
>> > Delete binary_data, marsheller and sub directories of /work/db
>> > Copy snapshots/snapshotname/db/binary_data to /work/db/,
>> > Copy snapshots/snapshotname/db/marshaller to /work/db/
>> > Copy  snapshots/snapshotname/db/{nodeid} dir to /work/db/
>> > Start cluster
>> > Cluster should auto activate after all nodes join it
>> > Cluster is ready
>> >
>> >
>> > On Mon, Jan 31, 2022 at 7:14 PM Surinder Mehra 
>> wrote:
>> >>
>> >> Hi,
>> >> We are using ignite 2.11.1 to experiment with ignite snapshots. We
>> tried steps mentioned on below page to restore ignite data from snapshot
>> >> https://ignite.apache.org/docs/latest/snapshots/snapshots
>> >>
>> >> But we get the below error when we start a cluster after copying data
>> manually as mentioned on the page.
>> >>
>> >> Steps:
>> >> 1.Created 3 nodes and added 3 records
>> >>
>> >> 2.Created snapshot.
>> >> 3. Stopped the cluster and removed files from binary_data and
>> marshellar, not the directories. they are present but empty
>> >> 4. removed nodeId directories and files under them from /work/db/
>> >>
>> >> 5. Copied node id directories from snapshot directory to /work/db/. I
>> guess the below step meant to say $IGNITE_HOME/work/db/ right ?
>> >>
>> >> Copy the files belonging to a node with the {node_id} from the
>> snapshot into the $IGNITE_HOME/work/ directory. If the db/{node_id}
>> directory is not located under the Ignite work dir then you need to copy
>> data files there.
>> >>
>> >> Error : do we need to copy binary_data and marshaler files as well or
>> something else missing ?
>> >>
>> >> Caused by: class org.apache.ignite.IgniteCheckedException: Cannot find
>> metadata for object with compact footer (Ignite work directory might have
>> been cleared after restart. Make sure that IGNITE_HOME does not point to a
>> temp folder or any other folder that is destroyed/cleared on restarts)
>> [typeId=-88020438, IGNITE_HOME='null']
>> >>
>> >> Please note that ignite HOEM/work/db directory has all nodes data
>> copied from snapshot, it is not cleared as indicated by error above
>> >>
>> >>
>>
>

Re: Restore data from Ignite snapshots

[Surinder Mehra]

Hey thanks for your suggestions.

I tried restoring using control.sh but it doesn't seem to work. Below are
steps
1. Started 3 nodes and added data using a thick client
2. created a snapshot using with  ./control.sh --snapshot create snapshot_1
3. I verified, the snapshot directory has data
4. Stopped the cluster and cleared binary_data, marshaler and nodes
directory /db
5. Started the cluster again, all 3 nodes
6. Activate the cluster using ./control.sh --set-state ACTIVE
7. Run restore command : ./control.sh --snapshot restore snapshot_1 --start
8. Command was successful but data is not copied to cluster nodes.

Please note that when I restarted the cluster, it created binary_data,
marshaler and nodes directories by default.

Did I miss anything ?


On Tue, Feb 1, 2022 at 8:21 PM Maxim Muzafarov  wrote:

> Hello,
>
> Your case looks correct to me, however, I'd like to mention some
> important points that may help you:
> - the directories structure of the snapshot has the same structure as
> the Ignite native persistence, so you may backup the original cluster
> node directory (for binary_data, marshaller and db) and move all the
> files right from the snapshot.
> - do not forget to backup and clear the original wal directory in case
> of restoration.
> - you may use control.sh --snapshot restore command to restore from a
> snapshot (this was added in 2.11)
>
> [1] https://issues.apache.org/jira/browse/IGNITE-13805
>
> On Tue, 1 Feb 2022 at 16:28, Surinder Mehra  wrote:
> >
> > Hi,
> > After a few hiccups, I managed to restore the cluster state from the
> snapshot. Please confirm if they look correct. If so documentation page
> needs to be updated
> >
> > Create N nodes
> > Add some data to them
> > Create snapshot
> > Stop all nodes(cluster)
> > Delete binary_data, marsheller and sub directories of /work/db
> > Copy snapshots/snapshotname/db/binary_data to /work/db/,
> > Copy snapshots/snapshotname/db/marshaller to /work/db/
> > Copy  snapshots/snapshotname/db/{nodeid} dir to /work/db/
> > Start cluster
> > Cluster should auto activate after all nodes join it
> > Cluster is ready
> >
> >
> > On Mon, Jan 31, 2022 at 7:14 PM Surinder Mehra 
> wrote:
> >>
> >> Hi,
> >> We are using ignite 2.11.1 to experiment with ignite snapshots. We
> tried steps mentioned on below page to restore ignite data from snapshot
> >> https://ignite.apache.org/docs/latest/snapshots/snapshots
> >>
> >> But we get the below error when we start a cluster after copying data
> manually as mentioned on the page.
> >>
> >> Steps:
> >> 1.Created 3 nodes and added 3 records
> >>
> >> 2.Created snapshot.
> >> 3. Stopped the cluster and removed files from binary_data and
> marshellar, not the directories. they are present but empty
> >> 4. removed nodeId directories and files under them from /work/db/
> >>
> >> 5. Copied node id directories from snapshot directory to /work/db/. I
> guess the below step meant to say $IGNITE_HOME/work/db/ right ?
> >>
> >> Copy the files belonging to a node with the {node_id} from the snapshot
> into the $IGNITE_HOME/work/ directory. If the db/{node_id} directory is not
> located under the Ignite work dir then you need to copy data files there.
> >>
> >> Error : do we need to copy binary_data and marshaler files as well or
> something else missing ?
> >>
> >> Caused by: class org.apache.ignite.IgniteCheckedException: Cannot find
> metadata for object with compact footer (Ignite work directory might have
> been cleared after restart. Make sure that IGNITE_HOME does not point to a
> temp folder or any other folder that is destroyed/cleared on restarts)
> [typeId=-88020438, IGNITE_HOME='null']
> >>
> >> Please note that ignite HOEM/work/db directory has all nodes data
> copied from snapshot, it is not cleared as indicated by error above
> >>
> >>
>

Re: Restore data from Ignite snapshots

[Surinder Mehra]

Hi,
After a few hiccups, I managed to restore the cluster state from the
snapshot. Please confirm if they look correct. If so documentation page
needs to be updated


   1. Create N nodes
   2. Add some data to them
   3. Create snapshot
   4. Stop all nodes(cluster)
   5. Delete binary_data, marsheller and sub directories of /work/db
   6. Copy snapshots/snapshotname/db/binary_data to /work/db/,
   7. Copy snapshots/snapshotname/db/marshaller to /work/db/
   8. Copy  snapshots/snapshotname/db/{nodeid} dir to /work/db/
   9. Start cluster
   10. Cluster should auto activate after all nodes join it
   11. Cluster is ready


On Mon, Jan 31, 2022 at 7:14 PM Surinder Mehra  wrote:

> Hi,
> We are using ignite 2.11.1 to experiment with ignite snapshots. We tried
> steps mentioned on below page to restore ignite data from snapshot
> https://ignite.apache.org/docs/latest/snapshots/snapshots
>
> But we get the below error when we start a cluster after copying data
> manually as mentioned on the page.
>
> Steps:
> 1.Created 3 nodes and added 3 records
>
> 2.Created snapshot.
> 3. Stopped the cluster and removed files from binary_data and marshellar,
> not the directories. they are present but empty
> 4. removed nodeId directories and files under them from /work/db/
>
> 5. Copied node id directories from snapshot directory to /work/db/. I
> guess the below step meant to say $IGNITE_HOME/work/*db/ *right *?*
>
>-
>
>Copy the files belonging to a node with the {node_id} from the
>snapshot into the $IGNITE_HOME/work/ directory. If the db/{node_id} 
> directory
>is not located under the Ignite work dir then you need to copy data
>files there.
>
>Error : do we need to copy binary_data and marshaler files as well or
>something else missing ?
>-
>
>Caused by:
> *class org.apache.ignite.IgniteCheckedException: Cannot find metadata for
>object with compact footer (Ignite work directory might have been cleared
>after restart. Make sure that IGNITE_HOME does not point to a temp folder
>or any other folder that is destroyed/cleared on restarts)
>[typeId=-88020438, IGNITE_HOME='null'] *
>Please note that ignite HOEM/work/db directory has all nodes data
>copied from snapshot, it is not cleared as indicated by error above
>
>
>

baseline command fails when node is down

[Surinder Mehra]

I was trying to add and remove nodes as mentioned in the blog below.
When I kill a node and try running ./control.sh --baseline. It consistently
throws errors
https://dzone.com/articles/apache-ignite-baseline-topology-by-examples

I tried it several times. When I restarted that node, it started working.
Again when I killed the same node, it stopped working.

This didn't happen when I killed some other node. That time it shows
correct online and offline nodes in baseline.
I saw a similar issue was reported before as well.
https://lists.apache.org/thread/sw163d42qtv0h9lcwdr1zjt1cv6cvlqq

I wonder if the control script was trying to connect to an offline node.
Could you please put some light on it.

[image: image.png]

Re: Failed to execute query because cache partition has been lostParts

[Surinder Mehra]

Hi Stephen,
I tried it again to identify issues. This time I commented out the line
which was activating the cluster everytime client started. It looks to work
properly now.

Thanks for your help.

On Mon, Jan 31, 2022 at 8:58 PM Surinder Mehra  wrote:

> Hi,
> Backups are configured, please recheck cacheconfig. Its set to 1, thats
> why node 3 has backups from node 1 and node two as shown in pic attached to
> my previous reply.
>
> On Mon, Jan 31, 2022, 20:46 Stephen Darlington <
> stephen.darling...@gridgain.com> wrote:
>
>> According to the configuration you shared, you don’t have backups.
>>
>> I would expect to see something like:
>>
>> CacheConfiguration deptCacheConfig = new
>> CacheConfiguration<>(DEPT_CACHE);
>> deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
>> *deptCacheConfig.setBackups(1);*
>> IgniteCache deptCache =
>> ignite.getOrCreateCache(deptCacheConfig);
>>
>> The default is zero backups, so loss of a single node results in lost
>> partitions.
>>
>> Note, that since you have persistence enabled, you’ll have to drop and
>> recreate the table for the change to take effect.
>>
>> On 31 Jan 2022, at 11:16, Surinder Mehra  wrote:
>>
>> Hi, thanks for pointing out the problem of why data is being stored on
>> the same machine.
>> The 2nd point:
>> As I mentioned in steps, we are getting the same exception even with
>> backups.
>> Could you please point out other problems with configuration as you
>> mentioned
>>
>>
>> On Mon, Jan 31, 2022 at 4:35 PM Stephen Darlington <
>> stephen.darling...@gridgain.com> wrote:
>>
>>> There’s a lot to unpack here, but there are (at least) two problems with
>>> your configuration.
>>>
>>> First, you should not be activating your cluster in your startup script.
>>> That’s an operation that needs to be completed only once, not every time
>>> the cluster (or node) starts. This is probably why all the data is being
>>> stored on a single machine.
>>>
>>> Second, you create a cache but without any backups. That means that when
>>> a node dies, you lose access to all the data.
>>>
>>> > On 31 Jan 2022, at 10:57, Surinder Mehra  wrote:
>>> >
>>> >
>>> > Hi Guys,
>>> > I observed below behavior with persistence enabled. Could you please
>>> check and confirm if it's a bug or I missed some configuration.
>>> >
>>> > Steps:
>>> > 1. Start ignite node1 and 2 with persistence enabled
>>> > 2. Step 1 will write few entries to ignite cache
>>> > 3. All entries go to one node
>>> > 4. Stop node containing the data
>>> > 5. Run a query to get data, below exception will be thrown. I guess
>>> this is expected because the node containing the data is down
>>> > 6. Restart killed node, exception shouldn’t be thrown because data is
>>> back in caches
>>> > 7. Run a query to get data, below exception will be thrown again. This
>>> looks weird.
>>> > 8. Kill node2 and try to query again, still the same exception
>>> > 9. Restart only the left node(data owning node). After restart, the
>>> query starts working again.
>>> > 10. I checked the cache directory under node1, it didn't have
>>> part0.bin ever. files start from part1 and so on.
>>> > 11. From the exception, it looks like it is trying to read non
>>> existent file
>>> > 12. Tried setting consistent Id for each node, still same exception.
>>> >
>>> >
>>> > Another observation : When backup is enabled, shouldn't the backup
>>> node serve the query. Why it throws same below exception when
>>> node1(primary) is down
>>> >
>>> > This exception is only thrown if data containing node goes down
>>> >
>>> > Caused by: class
>>> org.apache.ignite.internal.processors.cache.CacheInvalidStateException:
>>> Failed to execute query because cache partition has been lostParts
>>> [cacheName=deptCache, part=0]
>>> >
>>> > Node2{
>>> >
>>> > DataStorageConfiguration storageCfg = new DataStorageConfiguration();
>>> >
>>> storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
>>> > IgniteConfiguration cfg = new IgniteConfiguration();
>>> > cfg.setDataStorageConfiguration(storageCfg);
>>> >
>>> > try (Ignite ignite = Ignition.start(cfg)) {
>

Re: Failed to execute query because cache partition has been lostParts

[Surinder Mehra]

Hi,
Backups are configured, please recheck cacheconfig. Its set to 1, thats why
node 3 has backups from node 1 and node two as shown in pic attached to my
previous reply.

On Mon, Jan 31, 2022, 20:46 Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> According to the configuration you shared, you don’t have backups.
>
> I would expect to see something like:
>
> CacheConfiguration deptCacheConfig = new
> CacheConfiguration<>(DEPT_CACHE);
> deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
> *deptCacheConfig.setBackups(1);*
> IgniteCache deptCache =
> ignite.getOrCreateCache(deptCacheConfig);
>
> The default is zero backups, so loss of a single node results in lost
> partitions.
>
> Note, that since you have persistence enabled, you’ll have to drop and
> recreate the table for the change to take effect.
>
> On 31 Jan 2022, at 11:16, Surinder Mehra  wrote:
>
> Hi, thanks for pointing out the problem of why data is being stored on the
> same machine.
> The 2nd point:
> As I mentioned in steps, we are getting the same exception even with
> backups.
> Could you please point out other problems with configuration as you
> mentioned
>
>
> On Mon, Jan 31, 2022 at 4:35 PM Stephen Darlington <
> stephen.darling...@gridgain.com> wrote:
>
>> There’s a lot to unpack here, but there are (at least) two problems with
>> your configuration.
>>
>> First, you should not be activating your cluster in your startup script.
>> That’s an operation that needs to be completed only once, not every time
>> the cluster (or node) starts. This is probably why all the data is being
>> stored on a single machine.
>>
>> Second, you create a cache but without any backups. That means that when
>> a node dies, you lose access to all the data.
>>
>> > On 31 Jan 2022, at 10:57, Surinder Mehra  wrote:
>> >
>> >
>> > Hi Guys,
>> > I observed below behavior with persistence enabled. Could you please
>> check and confirm if it's a bug or I missed some configuration.
>> >
>> > Steps:
>> > 1. Start ignite node1 and 2 with persistence enabled
>> > 2. Step 1 will write few entries to ignite cache
>> > 3. All entries go to one node
>> > 4. Stop node containing the data
>> > 5. Run a query to get data, below exception will be thrown. I guess
>> this is expected because the node containing the data is down
>> > 6. Restart killed node, exception shouldn’t be thrown because data is
>> back in caches
>> > 7. Run a query to get data, below exception will be thrown again. This
>> looks weird.
>> > 8. Kill node2 and try to query again, still the same exception
>> > 9. Restart only the left node(data owning node). After restart, the
>> query starts working again.
>> > 10. I checked the cache directory under node1, it didn't have part0.bin
>> ever. files start from part1 and so on.
>> > 11. From the exception, it looks like it is trying to read non existent
>> file
>> > 12. Tried setting consistent Id for each node, still same exception.
>> >
>> >
>> > Another observation : When backup is enabled, shouldn't the backup node
>> serve the query. Why it throws same below exception when node1(primary) is
>> down
>> >
>> > This exception is only thrown if data containing node goes down
>> >
>> > Caused by: class
>> org.apache.ignite.internal.processors.cache.CacheInvalidStateException:
>> Failed to execute query because cache partition has been lostParts
>> [cacheName=deptCache, part=0]
>> >
>> > Node2{
>> >
>> > DataStorageConfiguration storageCfg = new DataStorageConfiguration();
>> >
>> storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
>> > IgniteConfiguration cfg = new IgniteConfiguration();
>> > cfg.setDataStorageConfiguration(storageCfg);
>> >
>> > try (Ignite ignite = Ignition.start(cfg)) {
>> > ignite.cluster().state(ClusterState.ACTIVE);
>> > CacheConfiguration deptCacheConfig =
>> new CacheConfiguration<>(DEPT_CACHE);
>> > deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
>> > IgniteCache deptCache =
>> ignite.getOrCreateCache(deptCacheConfig);
>> >
>> > Department d1 = new Department(1, "CS");
>> > Department d2 = new Department(2, "ECE");
>> > Department d3 = new Department(3, "CIVIL");
>> >
>> > if(deptCache.size(CachePeekMode.ALL) == 0){
>> > System.out.pr

Restore data from Ignite snapshots

[Surinder Mehra]

Hi,
We are using ignite 2.11.1 to experiment with ignite snapshots. We tried
steps mentioned on below page to restore ignite data from snapshot
https://ignite.apache.org/docs/latest/snapshots/snapshots

But we get the below error when we start a cluster after copying data
manually as mentioned on the page.

Steps:
1.Created 3 nodes and added 3 records

2.Created snapshot.
3. Stopped the cluster and removed files from binary_data and marshellar,
not the directories. they are present but empty
4. removed nodeId directories and files under them from /work/db/

5. Copied node id directories from snapshot directory to /work/db/. I guess
the below step meant to say $IGNITE_HOME/work/*db/ *right *?*

   -

   Copy the files belonging to a node with the {node_id} from the snapshot
   into the $IGNITE_HOME/work/ directory. If the db/{node_id} directory is
   not located under the Ignite work dir then you need to copy data files
   there.

   Error : do we need to copy binary_data and marshaler files as well or
   something else missing ?
   -

   Caused by:
*class org.apache.ignite.IgniteCheckedException: Cannot find metadata for
   object with compact footer (Ignite work directory might have been cleared
   after restart. Make sure that IGNITE_HOME does not point to a temp folder
   or any other folder that is destroyed/cleared on restarts)
   [typeId=-88020438, IGNITE_HOME='null'] *
   Please note that ignite HOEM/work/db directory has all nodes data copied
   from snapshot, it is not cleared as indicated by error above

Re: Failed to execute query because cache partition has been lostParts

[Surinder Mehra]

New setup :

Node configuration :

  




  

  

  

  


  



Started 3 different nodes with this config :
./ignite.sh ../config/custom-config.xml


Client : It adds data once and queries the cache


cfg.setClientMode(true);
try (Ignite ignite = Ignition.start(cfg)) {
ignite.cluster().state(ClusterState.ACTIVE);
CacheConfiguration deptCacheConfig = new
CacheConfiguration<>(DEPT_CACHE);
deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
deptCacheConfig.setBackups(1);
IgniteCache deptCache =
ignite.getOrCreateCache(deptCacheConfig);


Department d1 = new Department(1, "CS");
Department d2 = new Department(2, "ECE");
Department d3 = new Department(3, "CIVIL");


if(deptCache.size(CachePeekMode.ALL) == 0){
System.out.println("Adding dept data to cache");
deptCache.put(d1.getDeptId(), d1);
deptCache.put(d2.getDeptId(), d2);
deptCache.put(d3.getDeptId(), d3);
}

List> depts = deptCache.query(new
ScanQuery<>()).getAll();
depts.stream().forEach(entry -> System.out.println(entry.getValue()));
}


After inserting data, this is how cache data looks like on nodes :
[image: image.png]

Test :

1. first time it prints data
2. Kill node 3, query still works. I believe because node 1 and 2 are
primary nodes for 3 records we inserted.
3. Restart node3, kill node 2. query gets below error.

Caused by: class
org.apache.ignite.internal.processors.cache.CacheInvalidStateException:
Failed to execute query because cache partition has been lostParts
[cacheName=deptCache, part=513]

Base line log :
[17:40:44] Topology snapshot [ver=11, locNode=340bad33, servers=2,
clients=1, state=ACTIVE, CPUs=8, offheap=19.0GB, heap=24.0GB]
[17:40:44]   ^-- Baseline [id=0, size=3, online=2, offline=1]

Does it mean for query to run, baseline must have all nodes up ?

On Mon, Jan 31, 2022 at 4:46 PM Surinder Mehra  wrote:

> Hi, thanks for pointing out the problem of why data is being stored on the
> same machine.
> The 2nd point:
> As I mentioned in steps, we are getting the same exception even with
> backups.
> Could you please point out other problems with configuration as you
> mentioned
>
>
> On Mon, Jan 31, 2022 at 4:35 PM Stephen Darlington <
> stephen.darling...@gridgain.com> wrote:
>
>> There’s a lot to unpack here, but there are (at least) two problems with
>> your configuration.
>>
>> First, you should not be activating your cluster in your startup script.
>> That’s an operation that needs to be completed only once, not every time
>> the cluster (or node) starts. This is probably why all the data is being
>> stored on a single machine.
>>
>> Second, you create a cache but without any backups. That means that when
>> a node dies, you lose access to all the data.
>>
>> > On 31 Jan 2022, at 10:57, Surinder Mehra  wrote:
>> >
>> >
>> > Hi Guys,
>> > I observed below behavior with persistence enabled. Could you please
>> check and confirm if it's a bug or I missed some configuration.
>> >
>> > Steps:
>> > 1. Start ignite node1 and 2 with persistence enabled
>> > 2. Step 1 will write few entries to ignite cache
>> > 3. All entries go to one node
>> > 4. Stop node containing the data
>> > 5. Run a query to get data, below exception will be thrown. I guess
>> this is expected because the node containing the data is down
>> > 6. Restart killed node, exception shouldn’t be thrown because data is
>> back in caches
>> > 7. Run a query to get data, below exception will be thrown again. This
>> looks weird.
>> > 8. Kill node2 and try to query again, still the same exception
>> > 9. Restart only the left node(data owning node). After restart, the
>> query starts working again.
>> > 10. I checked the cache directory under node1, it didn't have part0.bin
>> ever. files start from part1 and so on.
>> > 11. From the exception, it looks like it is trying to read non existent
>> file
>> > 12. Tried setting consistent Id for each node, still same exception.
>> >
>> >
>> > Another observation : When backup is enabled, shouldn't the backup node
>> serve the query. Why it throws same below exception when node1(primary) is
>> down
>> >
>> > This exception is only thrown if data containing node goes down
>> >
>> > Caused by: class
>> org.apache.ignite.internal.processors.cache.CacheInvalidStateException:
>> Failed to execute query because cache partition has been lostParts
>> [cacheName=deptCache, part=0]
>> &

Re: Failed to execute query because cache partition has been lostParts

[Surinder Mehra]

Hi, thanks for pointing out the problem of why data is being stored on the
same machine.
The 2nd point:
As I mentioned in steps, we are getting the same exception even with
backups.
Could you please point out other problems with configuration as you
mentioned


On Mon, Jan 31, 2022 at 4:35 PM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> There’s a lot to unpack here, but there are (at least) two problems with
> your configuration.
>
> First, you should not be activating your cluster in your startup script.
> That’s an operation that needs to be completed only once, not every time
> the cluster (or node) starts. This is probably why all the data is being
> stored on a single machine.
>
> Second, you create a cache but without any backups. That means that when a
> node dies, you lose access to all the data.
>
> > On 31 Jan 2022, at 10:57, Surinder Mehra  wrote:
> >
> >
> > Hi Guys,
> > I observed below behavior with persistence enabled. Could you please
> check and confirm if it's a bug or I missed some configuration.
> >
> > Steps:
> > 1. Start ignite node1 and 2 with persistence enabled
> > 2. Step 1 will write few entries to ignite cache
> > 3. All entries go to one node
> > 4. Stop node containing the data
> > 5. Run a query to get data, below exception will be thrown. I guess this
> is expected because the node containing the data is down
> > 6. Restart killed node, exception shouldn’t be thrown because data is
> back in caches
> > 7. Run a query to get data, below exception will be thrown again. This
> looks weird.
> > 8. Kill node2 and try to query again, still the same exception
> > 9. Restart only the left node(data owning node). After restart, the
> query starts working again.
> > 10. I checked the cache directory under node1, it didn't have part0.bin
> ever. files start from part1 and so on.
> > 11. From the exception, it looks like it is trying to read non existent
> file
> > 12. Tried setting consistent Id for each node, still same exception.
> >
> >
> > Another observation : When backup is enabled, shouldn't the backup node
> serve the query. Why it throws same below exception when node1(primary) is
> down
> >
> > This exception is only thrown if data containing node goes down
> >
> > Caused by: class
> org.apache.ignite.internal.processors.cache.CacheInvalidStateException:
> Failed to execute query because cache partition has been lostParts
> [cacheName=deptCache, part=0]
> >
> > Node2{
> >
> > DataStorageConfiguration storageCfg = new DataStorageConfiguration();
> >
> storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
> > IgniteConfiguration cfg = new IgniteConfiguration();
> > cfg.setDataStorageConfiguration(storageCfg);
> >
> > try (Ignite ignite = Ignition.start(cfg)) {
> > ignite.cluster().state(ClusterState.ACTIVE);
> > CacheConfiguration deptCacheConfig =
> new CacheConfiguration<>(DEPT_CACHE);
> > deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
> > IgniteCache deptCache =
> ignite.getOrCreateCache(deptCacheConfig);
> >
> > Department d1 = new Department(1, "CS");
> > Department d2 = new Department(2, "ECE");
> > Department d3 = new Department(3, "CIVIL");
> >
> > if(deptCache.size(CachePeekMode.ALL) == 0){
> > System.out.println("Adding dept data to cache");
> > deptCache.put(d1.getDeptId(), d1);
> > deptCache.put(d2.getDeptId(), d2);
> > deptCache.put(d3.getDeptId(), d3);
> > }
> >
> > }
> >
> > }
> >
> > Node1{
> >
> > DataStorageConfiguration storageCfg = new DataStorageConfiguration();
> >
> storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
> > IgniteConfiguration cfg = new IgniteConfiguration();
> > cfg.setDataStorageConfiguration(storageCfg);
> >
> > try (Ignite ignite = Ignition.start(cfg)) {
> > ignite.cluster().state(ClusterState.ACTIVE);
> > CacheConfiguration deptCacheConfig =
> new CacheConfiguration<>(DEPT_CACHE);
> > deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
> > IgniteCache deptCache =
> ignite.getOrCreateCache(deptCacheConfig);
> >
> > Department d1 = new Department(1, "CS");
> > Department d2 = new Department(2, "ECE");
> > Department d3 = new Department(3, "CIVIL");
> >
> > if(deptCache.size(CachePeekMode.ALL) == 0){
> > System.out.println("Adding dept data to cache");
> > deptCache.put(d1.getDeptId(), d1);
> > deptCache.put(d2.getDeptId(), d2);
> > deptCache.put(d3.getDeptId(), d3);
> > }
> >
> > }
> > }
> >
> > Client{
> >
> > IgniteConfiguration cfg = new IgniteConfiguration();
> > cfg.setDataStorageConfiguration(storageCfg);
> >
> > try (Ignite ignite = Ignition.start(cfg)) {
> > IgniteCache deptCache =
> ignite.getOrCreateCache(DEPT_CACHE);
> > List> depts = deptCache.query(new
> ScanQuery<>()).getAll();
> > depts.stream().forEach(entry ->
> System.out.println(entry.getValue()));
> > }
> > }
>
>

Failed to execute query because cache partition has been lostParts

[Surinder Mehra]

Hi Guys,
I observed below behavior with persistence enabled. Could you please check
and confirm if it's a bug or I missed some configuration.

Steps:
1. Start ignite node1 and 2 with persistence enabled
2. Step 1 will write few entries to ignite cache
3. All entries go to one node
4. Stop node containing the data
5. Run a query to get data, below exception will be thrown. I guess this is
expected because the node containing the data is down
6. Restart killed node, exception shouldn’t be thrown because data is back
in caches
7. Run a query to get data, below exception will be thrown again. This
looks weird.
8. Kill node2 and try to query again, still the same exception
9. Restart only the left node(data owning node). After restart, the query
starts working again.
10. I checked the cache directory under node1, it didn't have part0.bin
ever. files start from part1 and so on.
11. From the exception, it looks like it is trying to read non existent file
12. Tried setting consistent Id for each node, still same exception.


Another observation : When backup is enabled, shouldn't the backup node
serve the query. Why it throws same below exception when node1(primary) is
down

This exception is only thrown if data containing node goes down

Caused by: class
org.apache.ignite.internal.processors.cache.CacheInvalidStateException:
Failed to execute query because cache partition has been lostParts
[cacheName=deptCache, part=0]

Node2{

DataStorageConfiguration storageCfg = new DataStorageConfiguration();
storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setDataStorageConfiguration(storageCfg);

try (Ignite ignite = Ignition.start(cfg)) {
ignite.cluster().state(ClusterState.ACTIVE);
CacheConfiguration deptCacheConfig = new
CacheConfiguration<>(DEPT_CACHE);
deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
IgniteCache deptCache =
ignite.getOrCreateCache(deptCacheConfig);

Department d1 = new Department(1, "CS");
Department d2 = new Department(2, "ECE");
Department d3 = new Department(3, "CIVIL");

if(deptCache.size(CachePeekMode.ALL) == 0){
System.out.println("Adding dept data to cache");
deptCache.put(d1.getDeptId(), d1);
deptCache.put(d2.getDeptId(), d2);
deptCache.put(d3.getDeptId(), d3);
}

}

}

Node1{

DataStorageConfiguration storageCfg = new DataStorageConfiguration();
storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setDataStorageConfiguration(storageCfg);

try (Ignite ignite = Ignition.start(cfg)) {
ignite.cluster().state(ClusterState.ACTIVE);
CacheConfiguration deptCacheConfig = new
CacheConfiguration<>(DEPT_CACHE);
deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
IgniteCache deptCache =
ignite.getOrCreateCache(deptCacheConfig);

Department d1 = new Department(1, "CS");
Department d2 = new Department(2, "ECE");
Department d3 = new Department(3, "CIVIL");

if(deptCache.size(CachePeekMode.ALL) == 0){
System.out.println("Adding dept data to cache");
deptCache.put(d1.getDeptId(), d1);
deptCache.put(d2.getDeptId(), d2);
deptCache.put(d3.getDeptId(), d3);
}

}
}

Client{

IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setDataStorageConfiguration(storageCfg);

try (Ignite ignite = Ignition.start(cfg)) {
IgniteCache deptCache =
ignite.getOrCreateCache(DEPT_CACHE);
List> depts = deptCache.query(new
ScanQuery<>()).getAll();
depts.stream().forEach(entry ->
System.out.println(entry.getValue()));
}
}

Re: Server cache reads less number of entries than number of entries put by client cache

[Surinder Mehra]

Just curious, why can't we use continuous query here with "appropriate"
event type to write to another cache. So your listener will do below things
1. Write entry to another cache
2. remove entry from source cache

Just an idea, please correct if I am wrong

On Fri, Jan 28, 2022 at 3:49 PM Sumit Deshinge 
wrote:

> Hi,
>
> We are running apache ignite 2.11 with cache configuration FULL_SYNC and
> REPLICATED mode.
>
> Our use case is :
>
> 1. *Multiple thin clients are adding data* into a cache using putAll
> operation.
> 2. *Simultaneously the server is reading the data* using server cache
> iterator.
> 3. *While iterating over the cache, data is removed from the cache and
> added into new cache using a transaction*, i.e. transaction with remove
> and put operations. We have transaction *concurrency - pessimistic and
> isolation levels - repeatable_read*.
>
> But we are seeing few missing entries at server side, i.e. server is not
> able to read all the data put by client. E.g. in one of the run, all thin
> clients put 5000 entries, server is able to read only 4999 entries. Here we
> saw 1 entry was not read by server.
>
> *Another observation is that, if we remove the transaction in the second
> step above, or use optimistic transaction with serializable isolation
> level, then this issue is not observed*.
>
> What could be the possible problem in this use case
> with pessimistic concurrency and repeatable_read isolation level? This is
> particularly important as this configuration is resulting in data loss.
>
> --
> Regards,
> Sumit Deshinge
>
>

Use S3 as persistent Store for ignite caches

[Surinder Mehra]

Hi,
Asking further queries on this post here. Is it possible to use S3 as
backend persistent store for ignite caches.
We are exploring options to backup and restore data into ignite.

2.11.x snapshots don't support incremental backups and PITR.

https://stackoverflow.com/questions/70875613/back-and-restore-apache-ignite-volumes-in-kubernetes/70876463#70876463

Re: SqlQuery deprecated

[Surinder Mehra]

I found the link how to read full object. But would still like to
understand why SqlQuery is deprecated.

https://stackoverflow.com/questions/66248464/how-to-get-ignite-cache-sql-query-to-return-value-object


On Mon, Jan 17, 2022 at 4:54 PM Surinder Mehra  wrote:

> Hi,
> With Ignite 2.9 or before Ignite supported SqlQuery but in 2.11 I see it
> is deprecated. Could you please tell us why and what is the alternative to
> it. SqlFieldQuery does not allow to read full object like below but
> individual fields which is cumbersome
>
> String sql1 = "select * from Person";
> cache.query(new SqlQuery(Person.class, sql1))
> .getAll()
> .forEach(e ->System.out.println("Value: "+ e.getValue()));
>
>
>