Re: very fast loading of very big table

[vtchernyi]

Hi Denis,Data space is 3.7Gb according to MSSQL table properriesVladimir9:47, 19 февраля 2021 г., Denis Magda :Hello Vladimir, Good to hear from you! How much is that in gigabytes?-DenisOn Thu, Feb 18, 2021 at 10:06 PM  wrote:Sep 2020 I've published the paper about Loading Large Datasets into Apache Ignite by Using a Key-Value API (English [1] and Russian [2] version). The approach described works in production, but shows inacceptable perfomance for very large tables.The story continues, and yesterday I've finished the proof of concept for very fast loading of very big table. The partitioned MSSQL table about 295 million rows was loaded by the 4-node Ignite cluster in 3 min 35 sec. Each node had executed its own SQL queries in parallel and then distributed the loaded values across the other cluster nodes.Probably that result will be of interest for the community.Regards,Vladimir Chernyi[1] https://www.gridgain.com/resources/blog/how-fast-load-large-datasets-apache-ignite-using-key-value-api[2] https://m.habr.com/ru/post/526708/
-- Отправлено из мобильного приложения Яндекс.Почты

Re: very fast loading of very big table

[Denis Magda]

Hello Vladimir,

Good to hear from you! How much is that in gigabytes?

-
Denis


On Thu, Feb 18, 2021 at 10:06 PM  wrote:

> Sep 2020 I've published the paper about Loading Large Datasets into Apache
> Ignite by Using a Key-Value API (English [1] and Russian [2] version). The
> approach described works in production, but shows inacceptable perfomance
> for very large tables.
>
> The story continues, and yesterday I've finished the proof of concept for
> very fast loading of very big table. The partitioned MSSQL table about 295
> million rows was loaded by the 4-node Ignite cluster in 3 min 35 sec. Each
> node had executed its own SQL queries in parallel and then distributed the
> loaded values across the other cluster nodes.
>
> Probably that result will be of interest for the community.
>
> Regards,
> Vladimir Chernyi
>
> [1]
> https://www.gridgain.com/resources/blog/how-fast-load-large-datasets-apache-ignite-using-key-value-api
> [2] https://m.habr.com/ru/post/526708/
>
>

very fast loading of very big table

[vtchernyi]

Sep 2020 I've published the paper about Loading Large Datasets into Apache Ignite by Using a Key-Value API (English [1] and Russian [2] version). The approach described works in production, but shows inacceptable perfomance for very large tables.The story continues, and yesterday I've finished the proof of concept for very fast loading of very big table. The partitioned MSSQL table about 295 million rows was loaded by the 4-node Ignite cluster in 3 min 35 sec. Each node had executed its own SQL queries in parallel and then distributed the loaded values across the other cluster nodes.Probably that result will be of interest for the community.Regards,Vladimir Chernyi[1] https://www.gridgain.com/resources/blog/how-fast-load-large-datasets-apache-ignite-using-key-value-api[2] https://m.habr.com/ru/post/526708/

Re: Ignite 2.81. - NULL pointer exception

[narges saleh]

Was there a resolution for this issue?

On Mon, Sep 7, 2020 at 9:40 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> I wonder why there's no stack traces for all the threads.
>
> I wonder if somebody from the development side will step in (2.8.1):
>
> [05:57:16,193][SEVERE][sys-stripe-15-#16][] Critical system error
> detected. Will be handled accordingly to configured handler
> [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
> super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
> [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
> failureCtx=FailureContext [type=CRITICAL_ERROR,
> err=java.lang.NullPointerException]]
> java.lang.NullPointerException
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:1064)
> // tx is null?
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:953)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:909)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$200(IgniteTxHandler.java:123)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:217)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:215)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
> at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:745)
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 2 сент. 2020 г. в 11:05, Mahesh Renduchintala <
> mahesh.renduchint...@aline-consulting.com>:
>
>> I sent the logs again. There is no specific activity.
>> We have a cluster - 2 servers and about 15 thick clients
>>
>> Just happened without much info. I can say it is likely a new node joined
>> in and it may have triggered this crash.
>>
>>
>>

Unsubscribe

[Ryan Trollip]

I sent an unsubscribe request but am still getting emails. Please remove
from the list

Re: Graceful shutdown and request draining of Ignite servers

[Raymond Wilson]

I agree, but there is a core element here that might be worth considering
for IA, which is the ability to flag a node as [temporarily] unhealthy or
unavailable so application logic can use that as a part of the IA toolset.
Just a thought... :)

Thanks,.
Raymond.

On Fri, Feb 19, 2021 at 2:05 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> This sounds like a too detailed and peculiar scenario that should be taken
> care of on the application level, as you already do.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 17 февр. 2021 г. в 23:50, Raymond Wilson :
>
>> I Ilya,
>>
>> Sorry, that was a response to another problem!
>>
>> In this case, we have a more asynchronous mode of query-response where
>> the processing node can asynchronously send back a response to a query. The
>> reasons for this are: (1) Some responses are effectively streams of data
>> and we can't structure them as a single response, and (2) we can have
>> thousands of concurrent requests per node, which causes thread pool
>> exhaustion and response starvation due to the synchronous nature of the
>> IComputeFunc.Invoke() method.
>>
>> eg: We may have a request sequence like this where A, B and C are nodes
>> in the grid
>>
>> Request: A -> B -> C
>> Response: C -> B -> A
>>
>> If node B goes away unexpectedly, requests executing on 'C' can't send
>> their response and the request fails.
>>
>> From the perspective of A, it may attempt a retry after failing to
>> receive the response from B, but that's unsatisfactory for other reasons.
>>
>> I have built a POC that permits nodes to emit an application level
>> availability state which requestors can use to exclude certain nodes from
>> their request topology projections. This means a node being removed due to
>> auto-scale down or container scheduling can gracefully exit the grid after
>> ensuring the active requests it is involved in can complete normally. In
>> the case above, node B would be a client node providing services through a
>> web api gateway (A) and requesting results from co-located processing on
>> node C.
>>
>> Thanks,
>> Raymond.
>>
>>
>> On Thu, Feb 18, 2021 at 9:15 AM Raymond Wilson <
>> raymond_wil...@trimble.com> wrote:
>>
>>> Hi Ilya,
>>>
>>> That is the current method we use to stop the grid.
>>>
>>> However, this can leave uncheckpointed changes in the in-memory stores
>>> (only in the WAL), so when we restart the grid it goes into the cache
>>> recovery mode which is very slow.
>>>
>>> Raymond.
>>>
>>> On Thu, Feb 18, 2021 at 3:34 AM Ilya Kasnacheev <
>>> ilya.kasnach...@gmail.com> wrote:
>>>
 Hello!

 Why can't you just use Ignite.stop(instanceName, false)?

 Just make sure your projections are not singleton and the tasks will be
 rolled over.

 Regards,
 --
 Ilya Kasnacheev


 вт, 9 февр. 2021 г. в 06:41, Raymond Wilson >>> >:

> All,
>
> We have a very similar requirement as described in this item:
> https://issues.apache.org/jira/browse/IGNITE-10872
>
> Namely, when removing a node from a Ignite grid, we want to do two
> things:
>
> 1. Prevent new requests from reaching it
> 2. Allow all running requests the node is involved in to complete
> before it terminates.
>
> The solution outlined in 10872 partially solves these elements within
> our architecture in that it allows Ignite to pause shutdown of the node
> until all requests are completed (and, I assume, prevent new requests from
> reaching the node being shut down).
>
> In our architecture the phrase 'requests the node is involved in' made
> be opaque from the context on Ignite due to an asynchronous calling model
> we are using to permit very large numbers of concurrent requests to 
> execute
> without saturating the Ignite thread pools. What this means is that a node
> that may be a candidate to be shut down may be waiting for a response from
> another node on the grid in a way that Ignite can't see, so would 
> determine
> the node was safe to shut down when it is not.
>
> A good example of this in our system is an Apply style Ignite call
> where the request is sent to one of a set of nodes. That set of nodes may
> scale in/out due to request demand. On a scale in operation, the node to 
> be
> removed needs to be excluded from the topology projection constructed to
> perform the Apply() against. Once we are satisfied the node has no further
> request involved (eg: by a simple timeout) then we would proceed with
> actual shut down of that node.
>
> I have not seen any capability in Ignite today where a node can be
> 'un-blessed'; does one exist? Or should we construct this facility within
> our application logic layer?
>
> Thanks,
> Raymond.
>
>
> --
> 
> Raymond Wilson
> Solution Architect, Civil Construction Software Systems (CCSS)
> 11 

Re: Graceful shutdown and request draining of Ignite servers

[Ilya Kasnacheev]

Hello!

This sounds like a too detailed and peculiar scenario that should be taken
care of on the application level, as you already do.

Regards,
-- 
Ilya Kasnacheev


ср, 17 февр. 2021 г. в 23:50, Raymond Wilson :

> I Ilya,
>
> Sorry, that was a response to another problem!
>
> In this case, we have a more asynchronous mode of query-response where the
> processing node can asynchronously send back a response to a query. The
> reasons for this are: (1) Some responses are effectively streams of data
> and we can't structure them as a single response, and (2) we can have
> thousands of concurrent requests per node, which causes thread pool
> exhaustion and response starvation due to the synchronous nature of the
> IComputeFunc.Invoke() method.
>
> eg: We may have a request sequence like this where A, B and C are nodes in
> the grid
>
> Request: A -> B -> C
> Response: C -> B -> A
>
> If node B goes away unexpectedly, requests executing on 'C' can't send
> their response and the request fails.
>
> From the perspective of A, it may attempt a retry after failing to receive
> the response from B, but that's unsatisfactory for other reasons.
>
> I have built a POC that permits nodes to emit an application level
> availability state which requestors can use to exclude certain nodes from
> their request topology projections. This means a node being removed due to
> auto-scale down or container scheduling can gracefully exit the grid after
> ensuring the active requests it is involved in can complete normally. In
> the case above, node B would be a client node providing services through a
> web api gateway (A) and requesting results from co-located processing on
> node C.
>
> Thanks,
> Raymond.
>
>
> On Thu, Feb 18, 2021 at 9:15 AM Raymond Wilson 
> wrote:
>
>> Hi Ilya,
>>
>> That is the current method we use to stop the grid.
>>
>> However, this can leave uncheckpointed changes in the in-memory stores
>> (only in the WAL), so when we restart the grid it goes into the cache
>> recovery mode which is very slow.
>>
>> Raymond.
>>
>> On Thu, Feb 18, 2021 at 3:34 AM Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com> wrote:
>>
>>> Hello!
>>>
>>> Why can't you just use Ignite.stop(instanceName, false)?
>>>
>>> Just make sure your projections are not singleton and the tasks will be
>>> rolled over.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> вт, 9 февр. 2021 г. в 06:41, Raymond Wilson >> >:
>>>
 All,

 We have a very similar requirement as described in this item:
 https://issues.apache.org/jira/browse/IGNITE-10872

 Namely, when removing a node from a Ignite grid, we want to do two
 things:

 1. Prevent new requests from reaching it
 2. Allow all running requests the node is involved in to complete
 before it terminates.

 The solution outlined in 10872 partially solves these elements within
 our architecture in that it allows Ignite to pause shutdown of the node
 until all requests are completed (and, I assume, prevent new requests from
 reaching the node being shut down).

 In our architecture the phrase 'requests the node is involved in' made
 be opaque from the context on Ignite due to an asynchronous calling model
 we are using to permit very large numbers of concurrent requests to execute
 without saturating the Ignite thread pools. What this means is that a node
 that may be a candidate to be shut down may be waiting for a response from
 another node on the grid in a way that Ignite can't see, so would determine
 the node was safe to shut down when it is not.

 A good example of this in our system is an Apply style Ignite call
 where the request is sent to one of a set of nodes. That set of nodes may
 scale in/out due to request demand. On a scale in operation, the node to be
 removed needs to be excluded from the topology projection constructed to
 perform the Apply() against. Once we are satisfied the node has no further
 request involved (eg: by a simple timeout) then we would proceed with
 actual shut down of that node.

 I have not seen any capability in Ignite today where a node can be
 'un-blessed'; does one exist? Or should we construct this facility within
 our application logic layer?

 Thanks,
 Raymond.


 --
 
 Raymond Wilson
 Solution Architect, Civil Construction Software Systems (CCSS)
 11 Birmingham Drive | Christchurch, New Zealand
 raymond_wil...@trimble.com


 

>>>
>>
>> --
>> 
>> Raymond Wilson
>> Solution Architect, Civil Construction Software Systems (CCSS)
>> 11 Birmingham Drive | Christchurch, New Zealand
>> raymond_wil...@trimble.com
>>
>>
>> 
>>
>
>
> --
> 

Re: Reliably duplicate SQL cache

[Ilya Kasnacheev]

Hello!

It will be available in cache's Query Entities so you can try:

((CacheConfiguration)ignite.cache("SQL_PUBLIC_")

.getConfiguration(CacheConfiguration.class)).getQueryEntities().iterator().next().getValueType()


Regards,
-- 
Ilya Kasnacheev


чт, 18 февр. 2021 г. в 11:40, Courtney Robinson :

> Hi Illya,
> Thanks for responding.
> That makes sense - I figured something like that but didn't know exactly
> what.
> Is it possible to get the existing key_type and value_type for tables?
> The reason is because we have tables in production and they were not
> created with key_type and value_type. We actually thought this only applied
> when you use Java classes with annotations.
>
> In the SYS table somewhere perhaps?
>
>
> On Mon, Feb 15, 2021 at 11:19 AM Ilya Kasnacheev <
> ilya.kasnach...@gmail.com> wrote:
>
>> Hello!
>>
>> Two tables have different name of an indexed binary type by default.
>>
>> Try
>> repo.query("create table page1(a varchar, b varchar, c varchar, PRIMARY
>> KEY (a, b)) WITH \"cache_name=page1, key_type=PageKey, value_type=Page\"")
>> repo.query("create table page2(a varchar, b varchar, c varchar, PRIMARY
>> KEY (a, b)) WITH \"cache_name=page2, key_type=PageKey, value_type=Page\"")
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> сб, 13 февр. 2021 г. в 19:32, Courtney Robinson <
>> courtney.robin...@hypi.io>:
>>
>>> Due to an issue I posted about in a previous thread
>>> http://apache-ignite-users.70518.x6.nabble.com/Basic-SQL-pagination-returning-incorrect-results-td35443.html
>>>
>>> I've written a work around to use the streamer interface with a
>>> ScanQuery to duplicate a cache.
>>> Both are created from SQL using something like this:
>>>
>>> repo.query("create table page1(a varchar, b varchar, c varchar, PRIMARY KEY 
>>> (a, b)) WITH \"cache_name=page1\"")
>>> repo.query("create table page2(a varchar, b varchar, c varchar, PRIMARY KEY 
>>> (a, b)) WITH \"cache_name=page2\"")
>>>
>>> The data is copied, printing the size shows 100 as expected in the test
>>> but a SQL query on page2 table returns 0 rows.
>>>
>>> def copied = repo.query("SELECT * FROM page2 LIMIT 101")
>>>
>>> Gets nothing. The copy function used is below. I'm presuming I've missed
>>> a step and the SQL index or something else is not being done. How should
>>> this be written to duplicate all data from page1 into page2 table/cache.
>>>
>>> public void copy(String fromTableName, String toTableName) {
>>>   var ignite = ctx.ignite;
>>>   try (
>>> IgniteCache from = ignite.cache(fromTableName);
>>> IgniteCache to = ignite.cache(toTableName)
>>>   ) {
>>> if (from == null || to == null) {
>>>   throw new IllegalArgumentException(format("Both from and to tables 
>>> must exist. from: %s, to: %s", fromTableName, toTableName));
>>> }
>>> try (
>>>   IgniteDataStreamer strmr = 
>>> ignite.dataStreamer(toTableName/*from.getName()*/);
>>>   var cursor = from.withKeepBinary().query(new ScanQuery<>())
>>> ) {
>>>   strmr.allowOverwrite(true);
>>>   strmr.keepBinary(true);
>>>   //strmr.receiver(StreamVisitor.from((cache, e) -> to.put(e.getKey(), 
>>> e.getValue(;
>>>   for (Cache.Entry e : cursor) {
>>> strmr.addData(e.getKey(), e.getValue());
>>>   }
>>>   //strmr.flush();
>>> }
>>> log.info("Total in target cache {}", to.sizeLong(CachePeekMode.ALL));
>>>   }
>>> }
>>>
>>>
>>>
>>> Regards,
>>> Courtney Robinson
>>> Founder and CEO, Hypi
>>> Tel: ++44 208 123 2413 (GMT+0) 
>>>
>>> 
>>> https://hypi.io
>>>
>>

Re: Reliably duplicate SQL cache

[Courtney Robinson]

Hi Illya,
Thanks for responding.
That makes sense - I figured something like that but didn't know exactly
what.
Is it possible to get the existing key_type and value_type for tables?
The reason is because we have tables in production and they were not
created with key_type and value_type. We actually thought this only applied
when you use Java classes with annotations.

In the SYS table somewhere perhaps?


On Mon, Feb 15, 2021 at 11:19 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> Two tables have different name of an indexed binary type by default.
>
> Try
> repo.query("create table page1(a varchar, b varchar, c varchar, PRIMARY
> KEY (a, b)) WITH \"cache_name=page1, key_type=PageKey, value_type=Page\"")
> repo.query("create table page2(a varchar, b varchar, c varchar, PRIMARY
> KEY (a, b)) WITH \"cache_name=page2, key_type=PageKey, value_type=Page\"")
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> сб, 13 февр. 2021 г. в 19:32, Courtney Robinson  >:
>
>> Due to an issue I posted about in a previous thread
>> http://apache-ignite-users.70518.x6.nabble.com/Basic-SQL-pagination-returning-incorrect-results-td35443.html
>>
>> I've written a work around to use the streamer interface with a ScanQuery
>> to duplicate a cache.
>> Both are created from SQL using something like this:
>>
>> repo.query("create table page1(a varchar, b varchar, c varchar, PRIMARY KEY 
>> (a, b)) WITH \"cache_name=page1\"")
>> repo.query("create table page2(a varchar, b varchar, c varchar, PRIMARY KEY 
>> (a, b)) WITH \"cache_name=page2\"")
>>
>> The data is copied, printing the size shows 100 as expected in the test
>> but a SQL query on page2 table returns 0 rows.
>>
>> def copied = repo.query("SELECT * FROM page2 LIMIT 101")
>>
>> Gets nothing. The copy function used is below. I'm presuming I've missed
>> a step and the SQL index or something else is not being done. How should
>> this be written to duplicate all data from page1 into page2 table/cache.
>>
>> public void copy(String fromTableName, String toTableName) {
>>   var ignite = ctx.ignite;
>>   try (
>> IgniteCache from = ignite.cache(fromTableName);
>> IgniteCache to = ignite.cache(toTableName)
>>   ) {
>> if (from == null || to == null) {
>>   throw new IllegalArgumentException(format("Both from and to tables 
>> must exist. from: %s, to: %s", fromTableName, toTableName));
>> }
>> try (
>>   IgniteDataStreamer strmr = 
>> ignite.dataStreamer(toTableName/*from.getName()*/);
>>   var cursor = from.withKeepBinary().query(new ScanQuery<>())
>> ) {
>>   strmr.allowOverwrite(true);
>>   strmr.keepBinary(true);
>>   //strmr.receiver(StreamVisitor.from((cache, e) -> to.put(e.getKey(), 
>> e.getValue(;
>>   for (Cache.Entry e : cursor) {
>> strmr.addData(e.getKey(), e.getValue());
>>   }
>>   //strmr.flush();
>> }
>> log.info("Total in target cache {}", to.sizeLong(CachePeekMode.ALL));
>>   }
>> }
>>
>>
>>
>> Regards,
>> Courtney Robinson
>> Founder and CEO, Hypi
>> Tel: ++44 208 123 2413 (GMT+0) 
>>
>> 
>> https://hypi.io
>>
>