Re: very fast loading of very big table

[vtchernyi]

Hi Denis,

Data space is 3.7Gb according to MSSQL table properries

Vladimir

9:47, 19 февраля 2021 г., Denis Magda <dma...@apache.org>:

Hello Vladimir, 

Good to hear from you! How much is that in gigabytes?

-

Denis

On Thu, Feb 18, 2021 at 10:06 PM <vtcher...@gmail.com> wrote:

Sep 2020 I've published the paper about Loading Large Datasets into Apache Ignite by Using a Key-Value API (English [1] and Russian [2] version). The approach described works in production, but shows inacceptable perfomance for very large tables.

The story continues, and yesterday I've finished the proof of concept for very fast loading of very big table. The partitioned MSSQL table about 295 million rows was loaded by the 4-node Ignite cluster in 3 min 35 sec. Each node had executed its own SQL queries in parallel and then distributed the loaded values across the other cluster nodes.

Probably that result will be of interest for the community.

Regards,

Vladimir Chernyi

[1] https://www.gridgain.com/resources/blog/how-fast-load-large-datasets-apache-ignite-using-key-value-api

[2] https://m.habr.com/ru/post/526708/

--
Отправлено из мобильного приложения Яндекс.Почты