Re: Defrag?

[Ryan Trollip]

Thanks, Ilya, we may have to consider moving back to non-native storage and
caching more selectively as the performance degrades when there is a lot of
write/delete activity or tables with large amounts of rows. This is with
SQL with indexes and the use of query plans etc.

Is there any easy way to rebuild the entire native database after hours?
e.g. with a batch run on the weeknds?

On Wed, Jun 23, 2021 at 7:39 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> I don't think there's anything ready to use, but "killing performance"
> from fragmentation is also not something reported too often.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 16 июн. 2021 г. в 04:39, Ryan Trollip :
>
>> We see continual very large growth to data with ignite native. We have a
>> very chatty use case that's creating and deleting stuff often. The data on
>> disk just keeps growing at an explosive rate. So much so we ported this to
>> a DB to see the difference and the DB is much smaller. I was searching to
>> see if someone has the same issue. This is also killing performance.
>>
>> Founds this:
>>
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation
>>
>> Apparently, there is no auto-rebalancing of pages? or cleanup of pages?
>>
>> Has anyone implemented a workaround to rebuild the cache and indexes say
>> on a weekly basis to get it to behave reasonably?
>>
>> Thanks
>>
>

persistent caches not rebalancing when new node is added

[Alan Ward]

I have a 16 node Ignite (v2.10.0) cluster with persistence enabled, and
about 20 caches, all of which are configured as cacheMode = partitioned,
backups = 1, with a rebalanceMode of ASYNC and rebalanceDelay of -1 (such
that rebalancing will only happen manually). The auto baseline adjustment
feature is disabled. The cluster uses TcpDiscoveryVmIpFinder and each of
the 16 nodes has a list of all 16 ip addresses.

I want to expand the cluster and add a 17th node and rebalance the data
accordingly. In the new node, I update the config to include all 16 nodes
plus itself, then start it up. Using ./control.sh --baseline on one of the
original 16 nodes, I see all 16 nodes in the baseline, plus the new one in
a different section at the bottom (e.g. not yet part of the baseline). I
run ./control.sh --baseline add , and it seems to work, as I now
have 17 nodes in the baseline topology, and the metrics that are logged out
every minute from each node indicate that there are now 17 servers. I see
these same logs/info on the new node as well as the 16 original ones.

On the newly added node, I see logs like these after updating the baseline
topology:

Local state for group durability has changed [name=MyCache1Name,
enabled=false]
Local state for group durability has been logged to WAL [name=MyCache1Name,
enabled=false]
...
Prepared rebalancing [grp=ignite-sys-cache, mode=SYNC, supplier=...]
...
Starting rebalance routine [grp=ignite-sys-cache, mode=SYNC, supplier=...]
...
Completed rebalancing [rebalanceId=42, grp=ignite-sys-cache, supplier=...]
Local state for group durability has changed [name=ignite-sys-cache,
enabled=true]

I don't know what ignite-sys-cache is, but this all seems fine and good,
but my actual caches are not rebalanced and I have no data for them on this
new node. I tried using ignite.cache(cacheName).rebalance() on all of my
caches, but that also appeared to have no effect, even after sitting
overnight.

Is there something I'm missing with regards to how cluster expansion,
rebalancing, and baseline topology works? I've tried for a couple weeks to
get this working with no success. The official docs don't say much on the
subject other than 'update the baseline topology and data rebalancing
should occur based on your rebalanceMode and rebalanceDelay settings'.

Re: Defrag?

[Ilya Kasnacheev]

Hello!

I don't think there's anything ready to use, but "killing performance" from
fragmentation is also not something reported too often.

Regards,
-- 
Ilya Kasnacheev


ср, 16 июн. 2021 г. в 04:39, Ryan Trollip :

> We see continual very large growth to data with ignite native. We have a
> very chatty use case that's creating and deleting stuff often. The data on
> disk just keeps growing at an explosive rate. So much so we ported this to
> a DB to see the difference and the DB is much smaller. I was searching to
> see if someone has the same issue. This is also killing performance.
>
> Founds this:
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation
>
> Apparently, there is no auto-rebalancing of pages? or cleanup of pages?
>
> Has anyone implemented a workaround to rebuild the cache and indexes say
> on a weekly basis to get it to behave reasonably?
>
> Thanks
>

Re: Ignite client hangs forever when performing cache operation with 6 servers, works with 3 servers

[Ilya Kasnacheev]

Hello!

Unfortunately, these links are dead (404).

If it's still relevant, please consider re-uploading.

Regards,
-- 
Ilya Kasnacheev


пн, 14 июн. 2021 г. в 21:59, mapeters :

> Problem: Ignite client hangs forever when performing a cache operation. We
> have 6 ignite servers running, the problem goes away when reducing this to
> 3. What effect does expanding/reducing the server cluster have that could
> cause this?
>
> See attached for sample stack trace of hanging client thread, server config
> snippet, client config snippet, and cache key snippet. From looking through
> the logs, there essentially seem to be various TCP communication errors
> such
> as the attached client and server errors. We tried increasing the (client)
> failure detection timeout values as suggested by the server error message,
> but that just made system startup hang for a long time (close to an hour).
>
> Usage:
>
> We have large number data objects (64k-400M) stored within HDF5 files and
> process hundreds of millions of records a day, with total data throughput
> ranging from 500GB - 10TB of data a day. We utilize ignite as an in memory
> distributed cache in front of the process that interacts with the HDF5
> files.
>
> Configuration:
>
> 1. Ignite version is 2.9.
> 2. The configuration is a 6 node ignite cluster using a partitioned cache.
> 3. Ignite’s persistence is disabled and we wrote a cache store
> implementation to persist the cache entries to the backing hdf5 files.
> 4. Ignite is configured in a write behind / read through manner.
> 5. There are four primary caches split up by data type to reduce amount of
> traffic on any one cache. The caches are all configured the same except for
> write behind properties and the data types within each cache to help manage
> how much data is in a specific cache.
> 6. The cache key is a compound object of path to the file and then a group
> /
> locator string within the file.
>
> Hardware:
>
> 1. In our failure site, there are 6 physical systems running Red Hat
> Hyperconverged Infrastructure.
> 2. Each physical node had a pinned VM running apache ignite. The VM has
> 128GB of memory. Ignite is configured with 16GB of heap memory, and 64GB of
> off heap cache.
> 3. There are 6 other VMs, each running 3 processes that all store to
> ignite.
> 4. There is a single VM that fronts the HDF5 files that Ignite talks to for
> persistent storage.
>
> hangingStackTrace.txt
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t3178/hangingStackTrace.txt>
>
> serverConfig.xml
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t3178/serverConfig.xml>
>
> clientConfig.xml
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t3178/clientConfig.xml>
>
> DataStoreKey.java
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t3178/DataStoreKey.java>
>
>
> serverErrors.txt
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t3178/serverErrors.txt>
>
> clientErrors.txt
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t3178/clientErrors.txt>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Restoring Ignite cluster from the persistent store

[Ilya Kasnacheev]

Hello!

Why do you need to change the consistent id of the restored node at all?

Regards,
-- 
Ilya Kasnacheev


чт, 17 июн. 2021 г. в 13:52, Naveen :

> Hi
>
> AM using Ignite 2.8.1, am trying to bring up new cluster by restoring the
> persistent store, wal, work directory of an existing cluster, which I was
> able to do it.
> As part of this, I had to rename the folders name to match it with the
> consistent id of the destination node etc.
> I had to delete this metastorage folder under persistent store to bring up
> the node and once all the nodes are up, I do activate the cluster and works
> fine. But this way, I am not able to retain all the users which I have
> created on the source cluster.
> And if I dont delete metastorage folder, it would try to connect to the
> source cluster not the new cluster, so I had to delete this folder to make
> it work.
> Is there any way I can retain all the users that were created on the source
> cluster while brining up the new cluster by restoring the file system data
> of source cluster
>
> Thanks
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: SQL EXPLAIN ANALYZE

[Ivan Fedorenkov]

https://issues.apache.org/jira/browse/IGNITE-14981

чт, 17 июн. 2021 г. в 14:51, Ilya Kasnacheev :

> Hello!
>
> This is unfortunate. Do we have a ticket for it?
>
> As a workaround, one may set query warning timeout to 1 ms and then see
> these detailed messages in the log.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 17 июн. 2021 г. в 14:31, :
>
>> Actually this is true that neither plain explain nor explain analyze
>> shows the number of scanned rows. This information is only available when
>> ignite shows the warning about query exceeding time limits.
>>
>> Regards,
>> Ivan
>>
>> From: Ilya Kasnacheev 
>> Reply-To: "user@ignite.apache.org" 
>> Date: Thursday, June 17, 2021 at 1:04 PM
>> To: "user@ignite.apache.org" 
>> Subject: Re: SQL EXPLAIN ANALYZE
>>
>> Hello!
>>
>> What happens if you just do EXPLAIN ?
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пт, 11 июн. 2021 г. в 14:34, Devakumar J > >:
>> Hi,
>>
>> In general. I can see in server logs if any query executes longer, log
>> shows
>> Long running query with information about scan count and index usage.
>>
>> But if i execute from sql client EXPLAIN ANALYZE   which returns
>> two
>> rows as per below link:
>>
>>
>> https://apacheignite-sql.readme.io/docs/performance-and-debugging#using-explain-statement
>> <
>> https://urldefense.com/v3/__https:/apacheignite-sql.readme.io/docs/performance-and-debugging*using-explain-statement__;Iw!!PNmPxbqrAcYR!2T0TckYHNwL5PQSafBc2_K02iBzP2a8yH5qgI6KEZvtFggKtptsQ5a5pci0jvOJ2h8sl0gFE6g$
>> >
>>
>> But i dont scan count returned in this. Is there any to enable it from
>> ignite config?
>>
>> Thanks & Regards,
>> Devakumar J
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/<
>> https://urldefense.com/v3/__http:/apache-ignite-users.70518.x6.nabble.com/__;!!PNmPxbqrAcYR!2T0TckYHNwL5PQSafBc2_K02iBzP2a8yH5qgI6KEZvtFggKtptsQ5a5pci0jvOJ2h8tAXO9IlQ$
>> >
>>
>

Re: SQL query for nested collections and map in POJO

[Taner Ilyazov]

Hello, I've tried that but still hit a wall because the key-value object
that we have can include values with multiple types, like a date. I need to
be able to filter or order based on the value of that date.
f.e. Isn't there a simpler mechanism to store and work with the key-value
data using SQL?
I've thought of having a table like this:
`
Map -> table with columns(id:INT, pseudo-foreign
key:INT, key:STRING, value:STRING, valueType:STRING)
`
Where valuetype is the concrete type of GenericType. But that way the
ordering still won't work, because I can't use custom sql functions on the
order by expression of the sql query.

On Fri, 4 Jun 2021 at 11:49, Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> As I said in your other thread, you need to think in relational terms.
> That is, you need to normalise the data. Then your query would just be a
> join/filter against the CacheNameType table.
>
> You could do that as you feed the data in. You could do it using something
> like a ContinuousQuery to update the child tables dynamically.
>
> On 4 Jun 2021, at 08:05, Taner Ilyazov  wrote:
>
> We need to be able to filter based on CacheName.nestedClass.types ->
> example CacheName.nestedClass.types.contains("firstType)
> and filter on CacheName.nestedClass.dynamicFields.thirdKey.greaterThan(0L).
> DynamicFields values consist of custom implementations of Long, String,
> Date wrappers.
>
> Filtering as needed works with ScanQueries, but we need to be able to
> order the results by the same fields.
>
> With specifying QueryEntities using XML I've managed to get the results
> mapped into a POJO with a SqlFieldsQuery like: "select * from CacheName",
> but I can't specify the nested fields using SQL like "select
> nestedClass.type from QueryEntities" or  "select type from QueryEntities".
> Without specifying as so we can't filter or order based on those
> properties.
>
> 
> 
> 
> 
> 
> 
> 
> 
>
> 
>
> 
>  value="com.tanerilyazov.test.CacheName"/>
>
> 
> 
> 
> 
> 
>  value="java.util.Map"/>
> 
> 
>
>
> On Fri, 4 Jun 2021 at 09:54, Taner Ilyazov 
> wrote:
>
>> Hello everyone, I'm trying to find a solution for the current case:
>>
>> I have generated classes, which can't be changed.
>>
>> public static class CacheName {
>> private Long id;
>> private NestedClass nestedClass;
>> }
>>
>> public static class NestedClass {
>> private List types;
>> private Map dynamicFields;
>> }
>>
>>
>>
>
>