Re: Streamer and data loss
Hi, To minimise data loss during streamer node failure I think we can use the following steps: 1. Use autoFlushFrequency param to set the desired flush frequency, depending on desired consistency level and performance you can choose how frequently you would like the data to be flush to Ignite nodes. 2. Develop a automated checkpointing process to capture and store the source data offset, it can be something like kafka message offset or cache keys if keys are sequential or timestamp for last flush and depending on that the Ignite client can restart the data streaming process from last checkpoint if there are node failure. HTH Regards, Saikat On Fri, Jan 10, 2020 at 4:34 AM narges saleh wrote: > Thanks Saikat for the feedback. > > But if I use the overwrite option set to true to avoid duplicates in case > I have to resend the entire payload in case of a streamer node failure, > then I won't > get optimal performance, right? > What's the best practice for dealing with data streamer node failures? Are > there examples? > > On Thu, Jan 9, 2020 at 9:12 PM Saikat Maitra > wrote: > >> Hi, >> >> AFAIK, the DataStreamer check for presence of key and if it is present in >> the cache then it does not allow overwrite of value if allowOverwrite is >> set to false. >> >> Regards, >> Saikat >> >> On Thu, Jan 9, 2020 at 6:04 AM narges saleh wrote: >> >>> Thanks Andrei. >>> >>> If the external data source client sending batches of 2-3 MB say via TCP >>> socket connection to a bunch of socket streamers (deployed as ignite >>> services deployed to each ignite node) and say of the streamer nodes die, >>> the data source client catching the exception, has to check the cache to >>> see how much of the 2-4MB batch has been flushed to cache and resend the >>> rest? Would setting streamer with overwrite set to true work, if the data >>> source client resend the entire batch? >>> A question regarding streamer with overwrite option set to true. How >>> does the streamer compare the content the data in hand with the data in >>> cache, if each record is being assigned UUID when being inserted to cache? >>> >>> >>> On Tue, Jan 7, 2020 at 4:40 AM Andrei Aleksandrov < >>> aealexsand...@gmail.com> wrote: >>> Hi, Not flushed data in a data streamer will be lost. Data streamer works thought some Ignite node and in case if this the node failed it can't somehow start working with another one. So your application should think about how to track that all data was loaded (wait for completion of loading, catch the exceptions, check the cache sizes, etc) and use another client for data loading in case if previous one was failed. BR, Andrei 1/6/2020 2:37 AM, narges saleh пишет: > Hi All, > > Another question regarding ignite's streamer. > What happens to the data if the streamer node crashes before the > buffer's content is flushed to the cache? Is the client responsible > for making sure the data is persisted or ignite redirects the data to > another node's streamer? > > thanks. >>>
Re: Cache entries expiring in pageEvictionMode=DISABLED
Hi, Are you not seeing cache entries and therefore making the assumption that eviction has kicked in? You could inadvertently be using a different config w/persistence turned on? search in ignite logs for the words "persistenceEnabled=false" If you've not enabled onHeapCaching or persistence, and page eviction is disabled and no memory is left, Ignite will throw IgniteOutOfMemoryException. docs for your version: https://apacheignite.readme.io/v2.3/docs/evictions Can you try with only one cache(NonEvictable_256MB_Region) and see whether the problem repeats itself? Also, you have an old version, and if possible, can you do a small test using a newer version(2.7.6) https://ignite.apache.org/download.cgi and see whether the behavior repeats itself. If you are able to reproduce this on 2.7.6, then send us your entire config, a reproducer project (or the code that you use to populate the cache) and we will take a look. Thanks, Alex -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Cache entries expiring in pageEvictionMode=DISABLED
I'm running 3 container instances of ignite fabric 2.3.0. Below is my
datastorageconfiguration:
Here's the snippet of the cacheConfig for the cache using my non-evictable
data region:
After a while, when entries in the main region cache probably get filled
(0.9 threshhold), I'm starting seeing non-evictable cache entries also
getting evicted. What's worse - they are evicted on one node, and eviction
is not propagated on the other two nodes (so now every node has different
count of entries for that cache, even though it's replicated). So eviction
is not even working properly.
Please help!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Failed to reinitialize local partitions (rebalancing will be stopped)
Hello! Actually, it turns out you needn't file a ticket, rather you should fix your data region configuration, as seen when your reproducer is ran against 2.8 builds: Caused by: org.apache.ignite.spi.IgniteSpiException: Failed to join node (Incompatible data region configuration [region=Data_Region, locNodeId=fd66182a-89a6-4443-8f5e-2fba7d4ad36b, isPersistenceEnabled=true, rmtNodeId=16825cee-57cf-454e-94ce-46daad4f6b93, isPersistenceEnabled=false]) You should not have data regions with same name but different persistence flag settings in your cluster, it turns out. This is due to https://issues.apache.org/jira/browse/IGNITE-10983 Regards, -- Ilya Kasnacheev ср, 15 янв. 2020 г. в 09:51, ashishb888 : > Hello Ilya, > > Okay, will file a ticket. > > BR > Ashish > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: JDBC Connectivity
Hello! 4) I actually think that if you specify caches in thick client's config file, and they are absent on server, they will be created. (However, they will not be changed if configuration differs) Regards, -- Ilya Kasnacheev ср, 15 янв. 2020 г. в 15:59, narges saleh : > Hi All, > > I am trying to use ignite's cache grid with native persistence and prefer > to use JDBC for cache/db connectivity. > > 1) Is this possible, in either client or server mode? > 2) If yes, I assume, I'd need one JDBC connection per cache, as I see it > is possible to specify only one cache per JDBC connection. Is this right? > 3) Is this also true if I need to join multiple tables/caches? > 4) Can I specify my caches in XML config file and just pass the config > file to the JDBC connection? > 5) Will I get the same load performance if I JDBC with streaming set to > true as I'd using the streamer module directly (I see that I can specify > most of the streamer config options on JDBC connection configuration)? > > thanks. >
Re: JDBC Connectivity
> 1) Is this possible, in either client or server mode? Yes. > 2) If yes, I assume, I'd need one JDBC connection per cache, as I see it is > possible to specify only one cache per JDBC connection. Is this right? No. You can access any cache that has SQL enabled as long as you fully qualify it. > 3) Is this also true if I need to join multiple tables/caches? See above. > 4) Can I specify my caches in XML config file and just pass the config file > to the JDBC connection? If you use the thin JDBC driver, you just need a host name and port. For a thick client you wouldn’t need to specify the caches in the XML file (though if they’re in there it won’t be a problem). > 5) Will I get the same load performance if I JDBC with streaming set to true > as I'd using the streamer module directly (I see that I can specify most of > the streamer config options on JDBC connection configuration)? You mean the IgniteDataStreamer? If so, no. At the very least, it has to parse the SQL which the data streamer doesn’t have to do. Regards, Stephen
JDBC Connectivity
Hi All, I am trying to use ignite's cache grid with native persistence and prefer to use JDBC for cache/db connectivity. 1) Is this possible, in either client or server mode? 2) If yes, I assume, I'd need one JDBC connection per cache, as I see it is possible to specify only one cache per JDBC connection. Is this right? 3) Is this also true if I need to join multiple tables/caches? 4) Can I specify my caches in XML config file and just pass the config file to the JDBC connection? 5) Will I get the same load performance if I JDBC with streaming set to true as I'd using the streamer module directly (I see that I can specify most of the streamer config options on JDBC connection configuration)? thanks.
