Re: nodes are restarting when i try to drop a table created with persistence enabled
Ivan, Igor, Andrey, as SQL experts, Does this sound like a known limitation or issue? If not, what do we need to reproduce the scenario - heapdums? - Denis On Thu, Sep 26, 2019 at 2:12 AM Shiva Kumar wrote: > Hi dmagda, > > When I insert many records (~ 10 or 20 million) to the same table and try > to drop table or delete records from the table, nodes are restarting, the > restarts happens In the middle of drop or delete operation. > According to the logs the cause for restart looks like OOM in the data > region. > > regards, > shiva > > On Wed, Sep 25, 2019 at 1:12 PM Denis Mekhanikov > wrote: > >> I think, the issue is that Ignite can't recover from >> IgniteOutOfMemory, even by removing data. >> Shiva, did IgniteOutOfMemory occur for the first time when you did the >> DROP TABLE, or before that? >> >> Denis >> >> ср, 25 сент. 2019 г. в 02:30, Denis Magda : >> > >> > Shiva, >> > >> > Does this issue still exist? Ignite Dev how do we debug this sort of >> thing? >> > >> > - >> > Denis >> > >> > >> > On Tue, Sep 17, 2019 at 7:22 AM Shiva Kumar >> wrote: >> >> >> >> Hi dmagda, >> >> >> >> I am trying to drop the table which has around 10 million records and >> I am seeing "Out of memory in data region" error messages in Ignite logs >> and ignite node [Ignite pod on kubernetes] is restarting. >> >> I have configured 3GB for default data region, 7GB for JVM and total >> 15GB for Ignite container and enabled native persistence. >> >> Earlier I was in an impression that restart was caused by >> "SYSTEM_WORKER_BLOCKED" errors but now I am realized that >> "SYSTEM_WORKER_BLOCKED" is added to ignore failure list and the actual >> cause is " CRITICAL_ERROR " due to "Out of memory in data region" >> >> >> >> This is the error messages in logs: >> >> >> >> ""[2019-09-17T08:25:35,054][ERROR][sys-#773][] JVM will be halted >> immediately due to the failure: [failureCtx=FailureContext >> [type=CRITICAL_ERROR, err=class o.a.i.i.mem.IgniteOutOfMemoryException: >> Failed to find a page for eviction [segmentCapacity=971652, loaded=381157, >> maxDirtyPages=285868, dirtyPages=381157, cpPages=0, pinnedInSegment=3, >> failedToPrepare=381155] >> >> Out of memory in data region [name=Default_Region, initSize=500.0 MiB, >> maxSize=3.0 GiB, persistenceEnabled=true] Try the following: >> >> ^-- Increase maximum off-heap memory size >> (DataRegionConfiguration.maxSize) >> >> ^-- Enable Ignite persistence >> (DataRegionConfiguration.persistenceEnabled) >> >> ^-- Enable eviction or expiration policies]] >> >> >> >> Could you please help me on why drop table operation causing "Out of >> memory in data region"? and how I can avoid it? >> >> >> >> We have a use case where application inserts records to many tables in >> Ignite simultaneously for some time period and other applications run a >> query on that time period data and update the dashboard. we need to delete >> the records inserted in the previous time period before inserting new >> records. >> >> >> >> even during delete from table operation, I have seen: >> >> >> >> "Critical system error detected. Will be handled accordingly to >> configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, >> timeout=0, super=AbstractFailureHandler >> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext >> [type=CRITICAL_ERROR, err=class o.a.i.IgniteException: Checkpoint read lock >> acquisition has been timed out.]] class org.apache.ignite.IgniteException: >> Checkpoint read lock acquisition has been timed out.| >> >> >> >> >> >> >> >> On Mon, Apr 29, 2019 at 12:17 PM Denis Magda >> wrote: >> >>> >> >>> Hi Shiva, >> >>> >> >>> That was designed to prevent global cluster performance degradation >> or other outages. Have you tried to apply my recommendation of turning of >> the failure handler for this system threads? >> >>> >> >>> - >> >>> Denis >> >>> >> >>> >> >>> On Sun, Apr 28, 2019 at 10:28 AM shivakumar >> wrote: >> >> HI Denis, >> >> is there any specific reason for the blocking of critical thread, >> like CPU >> is full or Heap is full ? >> We are again and again hitting this issue. >> is there any other way to drop tables/cache ? >> This looks like a critical issue. >> >> regards, >> shiva >> >> >> >> -- >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >> >
Throttling/ Queue length for DataStreamers
Hello, I'm using datastreamers to ingest large amounts of data in batches. So the load on the grid is pretty spiky Some time I'm seeing pretty heavy GCing and that causes the ingestion to slow down on the grid, but the client continues to pump data which makes the GC pauses worse because I suspect that the queues on the grid keep bloating with requests and it really gets into a death spiral sometimes. It seems like having some throttling will help with these scenarios. Two questions - 1. Is there a way to see the length of the MSG queue building for datastreamers 2. Is there a way to throttle this? I.e. Set a max queue size or some way to slow down the data streaming clients? Thanks, Abhishek
Does any one have working Ignite cluster on AWS
Hi All, can someone help me run a working ignite cuter on AWS , did anyone able to figure out steps required for setting up working ignite cluster on AWS. Ignite documentation doesnt make sense it's just launching a docker instance on Ec2. -- Thanks & Regards Sri Tummala
Re: Apache Ignite Cloud Formation Template
I found one example of cloud formation after launching the template and
started to execute ignite.sh it is failing with class not found exception
AWSCredentialsProviderWrapper.
I sued:-
Ignitefabric.1.5.0
aws java sdk 1.3.21.1
https://github.com/aws-samples/aws-big-data-blog/blob/master/aws-blog-real-time-in-memory-oltp-and-analytics-with-apache-ignite/cloudformation/configignite.json
ConfireIgnite.sh:-
#!/bin/bash
#
# This is a modified version of the file stored at
s3://publicbucketbabupe/ignitelibrary/configureIgnite.sh
# which changes the config to use the instance provided credentials
rather than requiring access/secret to be passed in
#
# Parameters are
# 1 - Cache Name
# 2 - Number of replicas
# 3 - S3 Bucket Name
#
echo "
http://www.springframework.org/schema/beans\;
xmlns:util=\"http://www.springframework.org/schema/util\;
xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\;
xsi:schemaLocation=\"
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd\;>
" > /tmp/igniteconfig.xml
echo "
" >> /tmp/igniteconfig.xml
availfreeMemory=$(cat /proc/meminfo|grep MemTotal|awk '{print $2}')
memoryOverhead=$((availfreeMemory/1024/1024/10))
availfreeMemoryinGB=$((availfreeMemory/1024/1024 - memoryOverhead))
if [[ $availfreeMemoryinGB -gt 8 ]]; then
offheapmemoryinGB=$((availfreeMemoryinGB-8))
echo "
" >> /tmp/igniteconfig.xml
echo "8g" > /tmp/heapsize.log
else
echo "${availfreeMemoryinGB}g" > /tmp/heapsize.log
fi
echo "
" >> /tmp/igniteconfig.xml
echo "" >>
/tmp/igniteconfig.xml
echo "" >>
/tmp/igniteconfig.xml
echo "" >> /tmp/igniteconfig.xml
echo " " >> /tmp/igniteconfig.xml
echo " " >> /tmp/igniteconfig.xml
echo "" >> /tmp/igniteconfig.xml
echo "
" >> /tmp/igniteconfig.xml
On Thu, Sep 26, 2019 at 5:34 PM sri hari kali charan Tummala <
kali.tumm...@gmail.com> wrote:
> looks complex and hard to get it right, it would be nice if Ignite
> community builds one working cloud formation template to launch ignite
> cluster and in the cloud formation template output a JDBC connection for
> connecting ignite cluster which would be awesome.
>
> Many would leverage this stack to build Ignite cluster on AWS or how about
> ignite on EMR , while launching emr add ingite bootstrap script which
> launches ignite in emr which would be lot easier also JDBC connection for
> connecting ignite, this would change things for Ignite world.
>
> Thanks
> Sri
>
>
>
> On Thu, Sep 26, 2019 at 3:51 PM Denis Magda wrote:
>
>> Hello Sri,
>>
>> You’re right, you need to start several aws instances to create a
>> distributed cluster if you follow the guide below. Ignite configuration for
>> all the nodes/instances has to have a properly configured IPFinder. You can
>> use a static IP finder if IPs are know in advance or AWS IPfinder if IPs
>> are assigned dynamically.
>> https://apacheignite.readme.io/docs/tcpip-discovery
>>
>> Let us know if it helps and we’ll update the docs.
>>
>> Denis
>>
>> On Thursday, September 26, 2019, sri hari kali charan Tummala <
>> kali.tumm...@gmail.com> wrote:
>>
>>> Hi Community,
>>>
>>> this doesn't launch a multinode ignite cluster am I right? it just
>>> launches a single node ignite cluster on one ec2 instance.
>>>
>>> https://apacheignite.readme.io/docs/aws-deployment
>>>
>>> Thanks
>>> Sri
>>>
>>> On Thu, Sep 26, 2019 at 8:49 AM sri hari kali charan Tummala <
>>> kali.tumm...@gmail.com> wrote:
>>>
Hi All,
did anyone have an cloud formation example for deploying Apache Ignite
on AWS or steps to deploy or some successful deployment of Ignite onto AWS.
this below blog has missing artifacts so not working.
https://aws.amazon.com/blogs/big-data/real-time-in-memory-oltp-and-analytics-with-apache-ignite-on-aws/
--
Thanks & Regards
Sri Tummala
>>>
>>> --
>>> Thanks & Regards
>>> Sri Tummala
>>>
>>>
>>
>> --
>> -
>> Denis
>>
>>
>
> --
> Thanks & Regards
> Sri Tummala
>
>
--
Thanks & Regards
Sri Tummala
Map Reduce over cache items, where values are sequences
Hello everyone,
I would like to use MapReduce over cache items representing events happened
in a process to calculate certain statistics. Could you be so kind to help
me how can I do that with apache ignite?
I have tens of millions of processes that happened in the past. The
processes look like a sequence of events [event1, event2, event3, ...
eventN], where number of events per process could vary (50-100). Every
event has certain sets of attributes like timestamp, event type, set of
metrics. I put these data to a cache as process_id => [e1, e2, e3, e4,
...]. What I would like to get is to get a histogram how often event of a
certain type happens in all the processes or processes that have certain
condition. What I managed to do is to broadcast a callable that lands on
ignite nodes and can access local cache items and counts what I want and
returns it back to the caller in K chunks which I have to aggregate on the
client.
Ignite localIgnite = Ignition.localIgnite();
IgniteCache localCache = localIgnite.cache("processes");
MyHistogram hist = new MyHistogram()
for (Cache.Entry e : localCache.localEntries()) {
hist.process(e.getValue());
}
return hist;
The problem with the approach is it utilizes only a single core on the
ignite node, while I have 64. How could I do something similar in more
efficient manner?
thank you in advance.
Re: Grid suddenly went in bad state
Hello! I'm not really sure, maybe it is because nodes tried to enter PME when they were already unable to communicate and therefore not all cache operations were completed. Anyway, since your logs don't start from beginning, it's impossible to know if there were any other clues. Currently there are none. Please also consider https://issues.apache.org/jira/browse/IGNITE-11365 Regards, -- Ilya Kasnacheev чт, 26 сент. 2019 г. в 19:11, Abhishek Gupta (BLOOMBERG/ 731 LEX) < agupta...@bloomberg.net>: > Thanks for the response Ilya. > So from a sequence of events perspective, first the logs show "Partition > states validation has failed for group" for many minutes. And only after > that we see the "Failed to read data from remote connection" caused by > "java.nio.channels.ClosedChannelException". So the question remains - what > could cause "Partition states validation has failed for group" in the first > place? > > Will also appreciate insights into my question 2. Below about 'client' > being nominator the coordinator. Is that by design? > > Thanks, > Abhishek > > > > From: ilya.kasnach...@gmail.com At: 09/26/19 11:33:36 > To: Abhishek Gupta (BLOOMBERG/ 731 LEX ) > Cc: user@ignite.apache.org > Subject: Re: Grid suddenly went in bad state > > Hello! > > "Failed to read data from remote connection" in absence of other errors > points to potential network problems. Maybe you have short idle timeout for > TCP connections? Maybe they get blockaded? > > Regards, > -- > Ilya Kasnacheev > > > вт, 24 сент. 2019 г. в 20:46, Abhishek Gupta (BLOOMBERG/ 731 LEX) < > agupta...@bloomberg.net>: > >> Hello Folks, >> Would really appreciate any suggestions you could provide about the below. >> >> >> Thanks, >> Abhishek >> >> From: user@ignite.apache.org At: 09/20/19 15:11:33 >> To: user@ignite.apache.org >> Subject: Re: Grid suddenly went in bad state >> >> >> Find attached the logs from 3 of the nodes and their GC graphs. The logs >> from the other nodes look pretty much the same. >> >> Some questions - >> 1. What could be the trigger for the "Partition states validation has >> failed for group" in node 1 ? Seems like it came on suddenly >> 2. If you look at the logs, there seems to be a change in coordinator >> 3698 2019-09-19 15:07:04.487 [INFO ] [disco-event-worker-#175] >> GridDiscoveryManager - Coordinator changed [prev=ZookeeperClusterNode >> [id=d667641c-3213-42ce-aea7-2fa232e972d6, addrs=[10.115.226.147, 127.0.0.1, >> 10.126.191.211], order=91, loc=false, client=true], cur=ZookeeperCluste >> rNode [id=0c643dd0-a884-4fd0-acb3-a6d7e2c5e71d, addrs=[10.115.248.110, >> 10.126.230.37, 127.0.0.1], order=109, loc=false, client=false]] >> 3713 2019-09-19 15:09:19.813 [INFO ] [disco-event-worker-#175] >> GridDiscoveryManager - Coordinator changed [prev=ZookeeperClusterNode >> [id=2c4a25d1-7701-407f-b728-4d9bcef3cb5b, addrs=[10.115.226.148, >> 10.126.191.212, 127.0.0.1], order=94, loc=false, client=true], >> cur=ZookeeperCluste rNode [id=0c643dd0-a884-4fd0-acb3-a6d7e2c5e71d, >> addrs=[10.115.248.110, 10.126.230.37, 127.0.0.1], order=109, loc=false, >> client=false]] >> >> What is curious is that it seems to suggest, a client was a coordinator. >> Is that by design? Clients are allowed to be coordinators? >> >> >> 3. It just seems like the grid went into a tailspin as show in the logs >> for node 1. Any help in understanding what triggered these series of event >> will be very helpful. >> >> >> Thanks, >> Abhishek >> >> >> >> >> From: user@ignite.apache.org At: 09/20/19 05:24:59 >> To: user@ignite.apache.org >> Subject: Re: Grid suddenly went in bad state >> >> Hi, >> >> Could please also attach logs for other nodes? And what version of Ignite >> you're currently using? >> >> Also you've mentioned high GC activity, is it possible to provide GC logs? >> >> Regards, >> Igor >> >> On Fri, Sep 20, 2019 at 1:17 AM Abhishek Gupta (BLOOMBERG/ 731 LEX) < >> agupta...@bloomberg.net> wrote: >> >>> Hello, >>> I've got a 6 node grid with maxSize (dataregionconfig) set to 300G each. >>> The grid seemed to be performing normally until at one point it started >>> logging "Partition states validation has failed for group" warning - see >>> attached log file. This kept happening for about 3 minutes and then stopped >>> (see line 85 in the attached log file). Just then a client seems to have >>> connected (see line 135 where connection was accepted). But soon after, it >>> kept logging the below exception. After a while (~1 hour), it started >>> showing logging "Partition states validation has failed for group" again >>> (line 284). >>> >>> >>> 2019-09-19 13:28:28.601 [INFO ] [exchange-worker-#176] >>> GridDhtPartitionsExchangeFuture - Completed partition exchange >>> [localNode=0c643dd0-a884-4fd0-acb3-a6d7e2c5e71d, >>> exchange=GridDhtPartitionsExchangeFuture [topVer >>> =AffinityTopologyVersion [topVer=126, minorTopVer=0], evt=NODE_JOINED, >>> evtNode=ZookeeperClusterNode [id=af5f33f4-842a-4691-8e84-da4fb19eafb2, >>> addrs=[10.126.90.78,
Re: Getting javax.cache.CacheException after upgrading to Ignite 2.7 (IGNITE-10884 )
Hi,
this issue still exists in ignite 2.7.6 version but the ticket is marked as
Fixed. I am still able to reproduce the issue using following reproducer.
Because this issue I am not able to upgrade to latest version from 2.6.0
version.
GitHub project: https://github.com/prasadbhalerao1983/IgniteTestPrj.git
Please run IgniteTransactionTester class to check the issue.
Thanks,
Prasad
On Thu, Jan 10, 2019 at 7:22 PM Ilya Kasnacheev
wrote:
> Hello!
>
> I have filed a blocker ticket about it:
> https://issues.apache.org/jira/browse/IGNITE-10884
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 9 янв. 2019 г. в 17:43, Prasad Bhalerao >:
>
>>
>> Hi Ilya,
>>
>> I have created a reproducer for this issue and uploaded it to GitHub.
>>
>> GitHub project: https://github.com/prasadbhalerao1983/IgniteTestPrj.git
>>
>> Please run IgniteTransactionTester class to check the issue.
>>
>>
>> Exception:
>>
>> Exception in thread "main" javax.cache.CacheException: Only pessimistic
>> repeatable read transactions are supported at the moment.
>> at
>> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:697)
>> at
>> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:636)
>> at
>> org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.query(GatewayProtectedCacheProxy.java:388)
>> at
>> IgniteTransactionTester.testTransactionException(IgniteTransactionTester.java:53)
>> at IgniteTransactionTester.main(IgniteTransactionTester.java:38)
>> Caused by: class
>> org.apache.ignite.internal.processors.query.IgniteSQLException: Only
>> pessimistic repeatable read transactions are supported at the moment.
>> at
>> org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.tx(MvccUtils.java:690)
>> at
>> org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.tx(MvccUtils.java:671)
>> at
>> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.runQueryTwoStep(IgniteH2Indexing.java:1793)
>> at
>> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.doRunDistributedQuery(IgniteH2Indexing.java:2610)
>> at
>> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.doRunPrepared(IgniteH2Indexing.java:2315)
>> at
>> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.querySqlFields(IgniteH2Indexing.java:2209)
>> at
>> org.apache.ignite.internal.processors.query.GridQueryProcessor$4.applyx(GridQueryProcessor.java:2135)
>> at
>> org.apache.ignite.internal.processors.query.GridQueryProcessor$4.applyx(GridQueryProcessor.java:2130)
>> at
>> org.apache.ignite.internal.util.lang.IgniteOutClosureX.apply(IgniteOutClosureX.java:36)
>> at
>> org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuery(GridQueryProcessor.java:2707)
>> at
>> org.apache.ignite.internal.processors.query.GridQueryProcessor.querySqlFields(GridQueryProcessor.java:2144)
>> at
>> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:685)
>>
>> Thanks,
>>
>> Prasad
>>
>>
>>
>> On Wed, Jan 9, 2019 at 6:22 PM Ilya Kasnacheev
>> wrote:
>>
>>> Hello!
>>>
>>> It was discussed recently:
>>> http://apache-ignite-users.70518.x6.nabble.com/Migrate-from-2-6-to-2-7-td25738.html
>>>
>>> I don't think you will be able to use SQL from transactions in Ignite
>>> 2.7. While this looks like a regression, you will have to work around it
>>> for now.
>>>
>>> Do you have a small reproducer for this issue? I could file a ticket if
>>> you had. You can try to do it yourself, too.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> ср, 9 янв. 2019 г. в 15:33, Prasad Bhalerao <
>>> prasadbhalerao1...@gmail.com>:
>>>
Hi,
My cache configuration is as follows. I am using TRANSACTIONAL and not
TRANSACTIONAL_SNAPSHOT.
private CacheConfiguration ipContainerIPV4CacheCfg() {
CacheConfiguration ipContainerIpV4CacheCfg = new
CacheConfiguration<>(CacheName.IP_CONTAINER_IPV4_CACHE.name());
ipContainerIpV4CacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
ipContainerIpV4CacheCfg.setWriteThrough(ENABLE_WRITE_THROUGH);
ipContainerIpV4CacheCfg.setReadThrough(false);
ipContainerIpV4CacheCfg.setRebalanceMode(CacheRebalanceMode.ASYNC);
ipContainerIpV4CacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
ipContainerIpV4CacheCfg.setBackups(1);
Factory storeFactory =
FactoryBuilder.factoryOf(IpContainerIpV4CacheStore.class);
ipContainerIpV4CacheCfg.setCacheStoreFactory(storeFactory);
ipContainerIpV4CacheCfg.setIndexedTypes(DefaultDataAffinityKey.class,
IpContainerIpV4Data.class);
ipContainerIpV4CacheCfg.setCacheStoreSessionListenerFactories(cacheStoreSessionListenerFactory());
ipContainerIpV4CacheCfg.setSqlIndexMaxInlineSize(84);
RendezvousAffinityFunction affinityFunction = new
Re: Application for Ignite Contributor
Hi, Deepak, this is an Umbrella ticket for newbies in ML https://issues.apache.org/jira/browse/IGNITE-9281 I'll freshen up it and add more tasks next week to have more option to start with ML Of course, you could free contact with me here, on dev-list, or on github by mention @zaleslaw preparing PR for the Ignite Good luck! чт, 26 сент. 2019 г. в 12:38, Dmitriy Pavlov : > Hi Deepak, > > Thank you for your interest in Apache Ignite. > > Have you ever considered the applicability of usage of Apache Ignite in > Apache OFBiz. Would it be useful as scalable storage? As storage option? > > Maybe we can think about some integration of these two projects. WDYT? > > One more thing, do you know about static code inspections Apache OFBiz was > set up recently. Was it successful? > > Sincerely, > Dmitriy Pavlov > > ср, 25 сент. 2019 г. в 22:49, Denis Magda : > > > Hi Deepak and welcome! > > > > Sorry for the late response. You sent this message to the user list, > > forwarded it to the dev list. > > > > I added you to the Ignite contributors list in JIRA. Feel free to take > > over any ticket you like. Do you have any specific interest (SQL, ML, > > caching, etc.? > > > > - > > Denis > > > > > > On Sat, Sep 21, 2019 at 2:35 AM Deepak Nigam > > > wrote: > > > >> Hello all, > >> > >> Please consider my application to become an Ignite Contributor. Here are > >> the details: > >> > >> Full Name: Deepak Nigam > >> ASF User Name: deepaknigam > >> Email Address: deepaknigam.1...@gmail.com > >> > >> I am already a committer in the Apache OFBiz project, hence already > >> signed ICLA. > >> > >> Regards, > >> -- > >> Deepak Nigam > >> > > >
